Microsoft releases tool to speed up recovery of systems borked by CrowdStrike update
By now, most people are aware of – or have been personally affected by – the largest IT outage the world have ever witnessed, courtesy of a defective update for Crowdstrike Falcon Sensors that threw Windows hosts into a blue-screen-of-death (BSOD) loop.
“We currently estimate that CrowdStrike’s update affected 8.5 million Windows devices, or less than one percent of all Windows machines. While the percentage was small, the broad economic and societal impacts reflect the use of CrowdStrike by enterprises that run many critical services,” David Weston, Microsoft’s VP of Enterprise and OS Security, stated on Saturday.
CrowdStrike claimed earlier today that “a significant number” of affected systems are back online and operational.
“Together with customers, we tested a new technique to accelerate impacted system remediation. We’re in the process of operationalizing an opt-in to this technique,” they noted on their remediation and guidance hub. “Customers are encouraged to follow the Tech Alerts for latest updates as they happen and they will be notified when action is needed.”
Microsoft collaborates with Crowdstrike, provides recovery tool
Microsoft is, understandably, doing everything it can to speed up worldwide recovery from the issue, has deployed hundreds of Microsoft engineers and experts to work with customers to restore services, and is collaborating with CrowdStrike.
“CrowdStrike has helped us develop a scalable solution that will help Microsoft’s Azure infrastructure accelerate a fix for CrowdStrike’s faulty update. We have also worked with both AWS and GCP to collaborate on the most effective approaches,” Weston explained.
Microsoft has also released a recovery tool that can be downloaded and used by IT admins to make the repair process less time-consuming.
The tool provides two repair options.
The first one – Recover from WinPE (Preinstallation Environment) – does not require local admin privileges, but requires the person to manually enter the BitLocker recovery key (if BitLocker is used on the device).
The second one – Recover from safe mode – may allow recovery without entering the BitLocker recovery keys.
“For this option, you must have access to an account with local administrator rights on the device. Use this approach for devices using TPM-only protectors, devices that are not encrypted, or situations where the BitLocker recovery key is unknown,” the Intune Support Team noted.
They also included detailed recovery steps for Windows clients, servers, and OSes hosted on Hyper-V.
Microsoft has previously confirmed that the buggy CrowdStrike update affected Windows 365 Cloud PCs and that users “may restore their Windows 365 Cloud PC to a known good state prior to the release of the update (July 19, 2024)”. The company has also provided guidance for restoring affected Azure virtual machines.
Cloud security company Orca has released a script that automates the remediation of Windows virtual machines hosted on AWS.
Threat actor exploiting the situation
As expected, scammers and threat actors have immediately started taking advantage of the chaos that resulted from the faulty update.
Trend Micro researchers provided examples of tech support scams doing the rounds, and even legal scams.
A tech support scam exploiting the situation (Source: Trend Micro)
CrowdStrike warned about:
- Attackers offering a fake utility for automating recovery that loads the Remcos remote access tool
- Phishers and vishers impersonating CrowdStrike support and contacting customers
- Scammers posing as independent researchers, claiming to have evidence the technical issue is linked to a cyberattack and offering remediation insights
“CrowdStrike Intelligence recommends that organizations ensure they are communicating with CrowdStrike representatives through official channels,” the company said.
UPDATE (July 23, 2024, 05:15 a.m. ET):
CrowdStrike has provided a way for remediating affected systems more quickly. Customers must opt in to use the technique via the support portal. (A Reddit user has explained the process involved.)
The company has also released a video explaining how users can self-remediate affected remote Windows laptops.