Diffy: A triage tool for cloud-centric incident response

The Netflix Security Intelligence and Response Team (SIRT) has released Diffy, an open source triage tool that allows digital forensics and incident response teams to quickly pinpoint compromised hosts during a security incident on cloud architectures.

The name of the tool comes from its function: it identifies differences between instances that might point to a compromise (an unexpected listening port, a running process with an unusual name, a strange crontab entry, a surprising kernel module, etc.).

“Diffy finds outliers among a group of very similar hosts (e.g. AWS Auto Scaling Groups) and highlights those for a human investigator, who can then examine those hosts more closely. More importantly, Diffy helps an investigator avoid wasting time in forensics against hosts that don’t need close examination,” Forest Monsen and Kevin Glisson, of the Netflix SIRT, explained.

Spotting the difference

For the moment, Diffy uses the “functional baseline” method to do its work, but a “clustering” method will be implemented soon, they say. When that happens, IR teams will be able to use one or both of the methods at the same time.

The functional baseline method works thusly: an osquery table output representing system state is collected from a single newly-deployed instance and stored for later comparison. When an incident happens, another osquery table output is collected from all instances in an application group. The two outputs are compared, and interesting, security-relevant differences are highlighted for the investigator.

There will be no need to collect a baseline for the clustering method to work. Instead, when an incident happens, the osquery table output will be collected from all instances in an application group, and an algorithm will identify the differenced between them. But, this method can only work if the instances in an application group are very similar.

“Diffy is currently focused on Linux instances running within Amazon Web Services (AWS), but owing to our plugin structure, could support multiple platforms and cloud providers,” the team explained. They have, so far, included plugins for collection using osquery via AWS EC2 Systems Manager, but are actively adding more plugins and tests.

Diffy can also be integrated into the continuous integration or continuous delivery (CI/CD) pipeline.

“In today’s cloud architectures, automation wins. Digital forensics and incident response teams need straightforward help to help them respond to compromises with swift action, quickly identifying the work ahead. Diffy can help those teams,” Monsen and Glisson concluded.