SolarWinds hack: Did DOJ know 6 months earlier?

What did the U.S. Justice Department know about the SolarWinds fiasco? How early did it find out? And who did it tell?

The answers to these questions are in flux, as an investigative reporter digs into the story, some 26 months after it went public. She says the DOJ knew much earlier than we first thought, but the news didn’t reach the FBI nor the NSA for half a year.

It’s complicated. But Hanlon’s razor probably applies. In this week’s Secure Software Blogwatch, we look at the story from all sides.

Your humble blogwatcher curated these bloggy bits for your entertainment. Not to mention: AI TV spot.

See Webinar: Lessons Learned from the SolarWinds AttackGet report: The State of Software Supply Chain Security 2022-23

DOJ on down-low for 6 months

What’s the craic? Steve Prentice summarizes in his podcast — “DOJ detected the SolarWinds hack 6 months earlier than first disclosed”:

“Unable to find a vulnerability”
The US Department of Justice, Mandiant, and Microsoft stumbled upon the SolarWinds breach six months earlier than previously reported. … Suspicions were triggered when the [DOJ] detected unusual traffic emanating from one of its servers that was running a trial version of the Orion software suite made by SolarWinds.
…
Investigators reached out to SolarWinds to assist with the inquiry, but the company’s engineers were unable to find a vulnerability in their code. In August 2020, the DOJ purchased the Orion system, suggesting that the department was satisfied that there was no further threat posed.

Sauce? Kim Zetter broke the story — “Noticed Russian hackers in its network but did not realize the significance”:

“Other compromised customers”
[It] involved Russian hackers … inserting a backdoor into software served to about 18,000 [SolarWinds] customers. … The hackers had been in [the] networks [of] at least nine US federal agencies [and] top tech and security firms for between four and nine months before the campaign was exposed.
…
Six months earlier, in late May 2020 [the DOJ] detected unusual traffic. … Investigators suspected the hackers had breached the DOJ server directly, possibly by exploiting a vulnerability in the Orion software. … It notified the US Cybersecurity and Infrastructure Agency (CISA) about the breach at the time … though a US National Security Agency spokesperson expressed frustration that the agency was not also notified.
…
[They] weren’t the only ones to stumble upon early evidence of the breach. Around the same time … security firm Volexity … was also investigating a breach … and traced it to the organization’s Orion server. … In September, the security firm Palo Alto Networks also discovered anomalous activity in connection with its Orion server.
…
In December 2020, when the public learned that a number of federal agencies were compromised … neither the DOJ nor CISA revealed to the public that the operation had unknowingly been found months earlier. The DOJ initially said its chief information officer had discovered the breach on December 24. … When asked why [Mandiant] didn’t publicly disclose that it had been tracking … the SolarWinds campaign … months earlier, a spokesperson noted only that, “When we went public, we had identified other compromised customers.”

Ouch. Bruce Schneier offers a sympathetic reading — “SolarWinds Detected Six Months Earlier”:

The Department of Justice detected the SolarWinds attack six months before Mandient [did]. But didn’t realize what they detected. And so ignored it.

As opposed to rst, who alleges an allegation:

“Leaking a report that was just false”

If there's any justification for this kind of delay between detecting an intrusion and acting on it, it would have to be giving the government time to investigate. … To correctly identify the source and means of intrusion, so the right parties were charged and innocents weren't dragged in. Which, unfortunately, was not the case.

…

Almost immediately after the intrusion was publicly disclosed, there were a bunch of stories, in both industry outlets [and] the New York Times, suggesting that software from JetBrains might have somehow been implicated in the hack — citing no evidence other than that Solarwinds had bought JetBrains products, and that they were, y'know … Russian.

…

And yet, when a full technical writeup of the way the build servers got breached was available, it turned out that JetBrains software was not at fault. … So, extra time to investigate didn't keep the investigators from leaking a report that was just false.

Never ascribe to malice that which is adequately explained by incompetence. gweihir shaves with a modded Hanlon: [You’re fired—Ed.]

Well, not a surprise. For things to go this badly, the defenders have to be simply incompetent. These three seeing it and not realizing this was a major supply chain attack fits the picture perfectly.
…
Detection is, at best, a half-measure. The only thing that will cut it is secure systems.

Did we learn nothing from 9/11? u/Hrmbee is deeply concerned at the silos:

The lack of communication between departments, between organizations, and between vendors, security consultants, and clients is deeply concerning. … There should be policies and protocols in place to not just report to a single entity but to ensure that all involved are notified of both the breaches and the potentials for damage, and options for mitigation.

But what about SolarWinds’ own failures? iAPX whispers, conspiratorially:

There is something really weird on this story. … “The company’s engineers were unable to find a vulnerability in their code.” Naturally they won’t! You don’t ask people who created code with a security hole to find it.

…

You don’t do QA by code developers, but by QA people. You don’t search a flaw by code developers, you use a hacker.

Have we been given the whole story yet? boomboomsubban notes this entertaining angle:

They noticed the breach in the trial version and still bought the product. Makes me wonder what would have led to them not buying the product.

Still, that time around the holidays must have been “fun” for feds. u/Pyro1934 describes it from the inside:

The federal agency I work for used the impacted SolarWinds, but luckily was not compromised. … But I find it rather shocking to know we didn’t at least get a heads up.

…

We got informed about 3–4 days before it went public. That’s when I was rushed in an hour early by my director and tossed into a meeting with the CISO/CIO on it with all the other technical folks.

Meanwhile, myowntrueself “knows” the real reason for the delay:

They had to keep it quiet so the NSA could keep using it.

Previously in And finally

You have been reading Secure Software Blogwatch by Richi Jennings. Richi curates the best bloggy bits, finest forums, and weirdest websites … so you don’t have to. Hate mail may be directed to @RiCHi or ssbw@richi.uk. Ask your doctor before reading. Your mileage may vary. Past performance is no guarantee of future results. Do not stare into laser with remaining eye. E&OE. 30.

Image sauce: Jelleke Vanooteghem (Unsplash license) (edited)