Microsoft lost some customers’ cloud security logs
Microsoft has lost several weeks of cloud security logs that its customers rely on to spot cyber intrusions.
What happened
As reported by Business Insider earlier this month, Microsoft privately notified affected customers of this incident and told them the failure was “not related to any security compromise.”
The preliminary post incident review has since been made public, and says that the cause was a bug in the internal monitoring agent that was triggered when a fix for a bug in the log collection service was rolled out.
“Starting around 23:00 UTC on 2 September 2024, a bug in one of Microsoft’s internal monitoring agents resulted in a malfunction in some of the agents when uploading log data to our internal logging platform. This resulted in partially incomplete log data for the affected Microsoft services,” the company said.
Two weeks after the issue was detected on 5 September, the company’s engineering teams introduced a temporary and partially effective workaround for the problem, which consisted of periodically restarting the agent or server to restart the log collection process.
Still, some of the log data has been lost and cannot be recovered.
Which services were affected?
The incident resuted in potentially incomplete logs for the following services:
- Azure Logic Apps (platform logs)
- Azure Healthcare APIs (platform logs)
- Microsoft Sentinel (security alerts)”
- Azure Monitor (diagnostic settings routed to Azure Monitor)
- Azure Trusted Signing (incomplete SignTransaction and SignHistory logs)
- Azure Virtual Desktop (logs in Application Insights)
- Power Platform (data discrepancies across reports), and
- Microsoft Entra (sign-in logs, activity logs).
“Entra logs flowing via Azure Monitor into Microsoft Security products, including Microsoft Sentinel, Microsoft Purview, and Microsoft Defender for Cloud, were also impacted,” the company said. This potentially affected tenants’ ability to analyze data, detect threats, or generate security alerts.
The importance of logs
Logging – and having complete logs – is crucial for security products to work as they should and enterprise defenders and incident responders to be able to do their jobs.
After Chinese hackers managed to access email accounts belonging to US organizations and government agencies last year, Microsoft was lambasted for not providing specific cloud logging capabilities to customers that don’t have premium Microsoft Purview Audit accounts.
Access to those logs would have potentially resulted in the intrusion being spotted earlier than it has. The incident pushed Microsoft to make logs available to all agencies using Microsoft Purview Audit (regardless of license tier) and to increase the default log retention period from 90 days to 180 days.