How Chinese hackers got their hands on Microsoft’s token signing key

The mystery of how Chinese hackers managed to steal a crucial signing key that allowed them to breach Microsoft 365’s email service and access accounts of employees of 25 government agencies has been explained: they found it somewhere where it shouldn’t have been – Microsoft’s corporate environment.

The theft of a Microsoft signing key

In short:

• The key was included in the crash dump of a consumer signing system located in Microsoft’s “highly isolated and restricted production environment
• Microsoft didn’t notice it
• The crash dump was moved to the company’s debugging environment on the internet-connected corporate network
• Some time later the hackers managed to compromise a Microsoft engineer’s corporate account, access the debugging environment, get the crash dump, and extract the key

Or, at least, Microsoft believes that it all went down like that. “Due to log retention policies, we don’t have logs with specific evidence of this exfiltration by this actor, but this was the most probable mechanism by which the actor acquired the key,” the company said on Wednesday.

The signing key was included in the snapshot of the crashed process of a consumer signing system because of an unexpected race condition, and its presence in the crash dump wasn’t detected by Microsoft’s credential scanning methods. (The race condition has been resolved and credential scanning enhanced, Microsoft says.)

But how come a consumer key was able to grant access to enterprise mail?

The company has previously said that MSA (consumer) keys and Azure AD (enterprise) keys are issued and managed from separate systems and should only be valid for their respective systems, but that attackers exploited a token validation issue.

“To meet growing customer demand to support applications which work with both consumer and enterprise applications, Microsoft introduced a common key metadata publishing endpoint in September 2018. As part of this converged offering, Microsoft updated documentation to clarify the requirements for key scope validation – which key to use for enterprise accounts, and which to use for consumer accounts,” the company now explained.

“As part of a pre-existing library of documentation and helper APIs, Microsoft provided an API to help validate the signatures cryptographically but did not update these libraries to perform this scope validation automatically (this issue has been corrected). The mail systems were updated to use the common metadata endpoint in 2022. Developers in the mail system incorrectly assumed libraries performed complete validation and did not add the required issuer/scope validation. Thus, the mail system would accept a request for enterprise email using a security token signed with the consumer key (this issue has been corrected using the updated libraries).”

Some questions are still unanswered

Wiz researchers previously discovered that the key in question was replaced sometime between June 27th and July 5th, 2023, but that it expired on April 4th, 2021.

So why hasn’t this stopped it from being considered valid by Microsoft’s cloud services two years later? Microsoft didn’t say.

The breach did lead to one positive outcome, though: starting this month, more federal government and commercial Microsoft customers will get expanded cloud logging capabilities for free, to make it easier to investigate intrusions.

UPDATE (April 4, 2024, 04:10 a.m. ET):

Microsoft subsequently found no evidence of a crash dump containing the 2016 MSA key material, and still doesn’t know how the attackers got their hands on the signing key.