Faulty CrowdStrike update takes out Windows machines worldwide
Houndreds of housands and possibly millions of Windows computers and servers worldwide have been made inoperable by a faulty update of Crowdstrike Falcon Sensors, and the outage affected transport, broadcast, financial, retail and other organizations in Europe, Australia, the US and elsewhere.
Sky News is off the air in the UK; Delta, United and American Airlines have paused flights around the world.
What happened?
What initially seemed like it might be a Microsoft problem is now confirmed to have been created by Crowdstrike, i.e., its endpoint security agent.
The malfunctioning update throws Windows hosts into a blue-screen-of-death (BSOD) loop that – as advised by Crowdstrike – can be interrupted by:
1. Booting Windows into Safe Mode or the Windows Recovery Environment
2. Navigating to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locating the file matching “C-00000291*.sys” and deleting it, then
4. Booting the host normally.
Unfortunately, in many cases this will have to be a manual intervention that has to be performed via a local admin account, and it will take a while at companies with huge fleets of Windows PC workstations to restore them – and on a Friday, too. IT/support teams will have to work through the weekend, it seems.
Crowdstrike is surely analyzing the “bad” update to see what happened, and security researchers are trying to do the same.
While the cause of the outage is likely to end up being a simple coding error (and not deliberate sabotage due to unnoticed supply-chain compromise), it affected one component of the CIA triad (availability), making this effectively an information security issue.
UPDATE (July 19, 2024, 05:45 a.m. ET):
“The .sys files causing the issue are channel update files, they cause the top level CS driver to crash as they’re invalidly formatted. It’s unclear how/why Crowdstrike delivered the files and I’d pause all Crowdstrikes updates temporarily until they can explain,” security researcher Kevin Beaumont noted.
“This is going to turn out to be the biggest ‘cyber’ incident ever in terms of impact, just a spoiler, as recovery is so difficult.”
UPDATE (July 19, 2024, 06:25 a.m. ET):
CrowdStrike President & CEO George Kurtz says that the company is actively working with customers impacted by this issue, caused by a single content update for Windows hosts.
“Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed,” he explained.
“We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels.”
The fix may have been deployed, but many machines stuck in the BSOD loop won’t be able to get it without a manual intervention first. In organizations that have a great number of Windows machines (whether physical or virtual) and a small IT support team, the process might last days, or even weeks.
UPDATE (July 19, 2024, 07:20 a.m. ET):
The fact that most Microsoft 365 apps and services have been intermittently inaccessible in the US in the last 12 hours is not connected to the Crowdstrike-caused IT outage.
“The underlying cause [a configuration change in a portion of our Azure backend workloads] has been fixed, however, residual impact is continuing to affect some Microsoft 365 apps and services. We’re conducting additional mitigations to provide relief,” Microsoft says.
The company has also confirmed that the faulty Crowdstrike update affected Windows 365 Cloud PCs and that “users may restore their Windows 365 Cloud PC to a known good state prior to the release of the update (July 19, 2024)”.