Microsoft blamed for “a cascade of security failures” in Exchange breach report

A federal Cyber Safety Review Board has issued its report on what led to last summer's capture of hundreds of thousands of emails by Chinese hackers from cloud customers, including federal agencies. It cites "a cascade of security failures at Microsoft" and finds that "Microsoft's security culture was inadequate" and needs to adjust to a "new normal" of cloud provider targeting.

The report, mandated by President Biden in the wake of the far-reaching intrusion, details the steps that Microsoft took before, during, and after the breach and in each case finds critical failure. The breach was "preventable," even though it cites Microsoft as not knowing precisely how Storm-0558, a "hacking group assessed to be affiliated with the People's Republic of China," got in.

"Throughout this review, the board identified a series of Microsoft operational and strategic decisions that collectively points to a corporate culture that deprioritized both enterprise security investments and rigorous risk management," the report reads.

The report notes that Microsoft "fully cooperated with the Board's review." A Microsoft spokesperson issued a statement regarding the report. "We appreciate the work of the CSRB to investigate the impact of well-resourced nation state threat actors who operate continuously and without meaningful deterrence," the statement reads. "As we announced in our Secure Future Initiative, recent events have demonstrated a need to adopt a new culture of engineering security in our own networks." Along with hardening its systems and implementing more sensors and logs to "detect and repel the cyber-armies of our adversaries," Microsoft said it would "review the final report for additional recommendations."

“Inaccurate public statements” and unsolved mysteries

The Cyber Safety Review Board (CSRB), formed two years ago, is composed of government and industry officials, from entities including the Departments of Homeland Security, Justice, and Defense, the NSA, FBI, and others. Microsoft provides cloud-based services, including Exchange and Azure, to numerous government agencies, including consulates.

Microsoft has previously offered a version of the intrusion story, one that notably avoids the words "vulnerability," "exploit," or "zero-day." A Microsoft post in July 2023 cited an inactive signing key acquired by Storm-0558, which was then used to forge tokens for the Azure AD cloud service that stores keys for logins. This was "made possible by a validation error in Microsoft code," Microsoft wrote.

Congress and government agencies called on Microsoft to offer far more disclosure, and others, including Tenable's CEO, offered even harsher assessments. In September, the company met them partway. It was an engineer's account that was hacked, Microsoft claimed, giving attackers access to a supposedly locked-down workstation, the consumer signing key, and, crucially, access to crash dumps moved into a debugging environment. A "race condition" prevented a mechanism that strips out signing keys and other sensitive data from crash dumps from functioning. Furthermore, "human errors" allowed for an expired signing key to be used in forging tokens for modern enterprise offerings.

Those kinds of unrevealing, withholding public statements were cited by the CSRB in its finding of Microsoft's failures. The report cites "Microsoft’s decision not to correct, in a timely manner, its inaccurate public statements about this incident, including a corporate statement that Microsoft believed it had determined the likely root cause of the intrusion when in fact, it still has not." It also notes that Microsoft did not update its September 2023 blog post about the invasion cause until March 2024, "as the Board was concluding its review and only after the Board’s repeated questioning about Microsoft’s plans to issue a correction." (The updated blog post notes that Microsoft has "not found a crash dump containing the impacted key material.")

Audit logging and “pay-to-play” security

The CSRB report also raises the issue of what some have called Microsoft's "pay-to-play" security model. The report notes that State Department officials detected the Chinese breach in June and notified Microsoft. That only happened because the department paid for a "G5" tier of Microsoft's cloud services that provided "Microsoft Purview Audit (Premium)." With that, the State Department had an alert set up for notable mail item access, which was triggered by the invaders' download of roughly 60,000 emails off-site. The report calls for cloud service providers to "adopt a minimum standard for default audit logging in cloud services" for better intrusion detection.

Once Microsoft realized that the intruders had used a theoretically expired 2016 consumer signing key to forge tokens for an enterprise customer, it launched an "all-hands-on-deck" investigation that went through the night, June 26–27. The company arrived at 46 hypotheses for the intrusion, including "a theoretical quantum computing capability to break public-key cryptography." Despite assigning teams to investigate every hypothesis, "nine months after the discovery of the intrusion, Microsoft says that its investigation into these hypotheses remains ongoing."

While Microsoft was ultimately able to remove attackers' access to 22 enterprise organizations and 503 individual accounts, by the end of the board's review, the company could not "demonstrate to the Board that it knew how Storm-0558 had obtained the 2016 MSA key."

Throughout 24 pages (plus appendices), the CSRB report also cites numerous flaws in the way Microsoft handled the messaging of a breach involving some of its most high-profile customers. It cites Rep. Don Bacon (R-Nebraska), who serves on the House Armed Services Committee and is a member of the House Taiwan Caucus.

Microsoft sent Bacon and other compromised users an "Unusual Sign-In Activity" email, with a password change prompt. That prompt, which claimed that "this notification does not mean that Microsoft's own systems have in any way been compromised," looked to Bacon and some other victims like "possible spam," so they disregarded it. Bacon changed his password directly rather than click on the email's reset link. He later learned from the FBI that his email had been compromised by that point. He and others later received more detailed notice from Microsoft about the intrusion later.

“Cascade of Microsoft’s avoidable errors”

The CSRB's conclusion is that Microsoft's security culture is "inadequate" and that a "cascade of Microsoft's avoidable errors allowed this intrusion to succeed." It cites in particular:

• Lacking security practices of other cloud providers
• Failure to detect a compromise on a laptop from an employee at an acquired company before connecting it to its network
• Letting inaccurate public statements stand for months
• A "separate incident" from January 2024 that, while not in the CSRB's purview, allowed another nation-state actor access to emails, code, and internal systems
• A need to "demonstrate the highest standards of security, accountability, and transparency."

"Unfortunately, throughout this review, the Board identified a series of operational and strategic decisions that collectively point to a corporate culture in Microsoft that deprioritized both enterprise security investments and rigorous risk management," the report states. "These decisions resulted in significant costs and harm for Microsoft customers around the world. The Board is convinced that Microsoft should address its security culture." The report cites a 2002 email from then-CEO Bill Gates, noting that "when we face a choice between adding features and resolving security issues, we need to choose security."

The full report includes far more detail, including a timeline of the Storm-0558 attack from the 2016 key to Microsoft's last 2023 blog post regarding the breach.

Storm-0558 and a very rich target

While it receives far less attention in the report, the entity involved in the other side of the summer 2023 incident, Storm-0558, is among the most successful, skilled, and relentless nation-state threat actors working today. Microsoft and other global-scale providers are frequent targets of such groups. The report notes Storm-0558's history of compromising clouds and, in particular, stealing authentication keys. It notes "industry links" of Storm-0558 to Operation Aurora, which led Google to largely leave China in 2010, along with the 2011 RSA SecurID failure.

"Indeed, security researchers have tracked Storm-0558’s activities for over 20 years," the report notes.