Kubernetes security matures: Inside the project’s first audit
Auditing 1.5 million lines of code is a heroic undertaking. With resources provided by the Cloud Native Computing Foundation (CNCF), the Kubernetes Project leadership created the Security Audit Working Group to perform an audit in an open, transparent, and repeatable manner, while also paving the way for future Kubernetes security reviews and research. It included members from Google, Red Hat, Salesforce, InGuardians, and input from the broader security community.
We felt that the two most critical components of this project were developing a process for vendor selection and determining the shape and deliverables the audit would produce.
From the start, we knew that no matter how many person-months we allocated to this initiative, or how many dollars were provided, we would be unable to perform a completely comprehensive review of Kubernetes in one go. That’s why we focused on work that would improve the quality and speed of the next audit or the next independent researcher. That doesn’t mean we didn’t want to find a bunch of novel bugs, though!
Our goals included deliverables that would help information security researchers, Kubernetes project developers, and cluster administrations. They included two White Papers (1 | 2), a Threat Model, a Security Review report, and our published RFP. We documented our selection process both for transparency and to help future initiatives; the Security Audit Working Group ultimately selected Atredis Partners and Trail of Bits to perform the audit.
The results were deeply illuminating, and I’m very proud to have been a part of the group that helped produce and release this to the community. While the Security Audit Working Group put in a lot of time and energy to lead and coordinate the process, participate in threat modeling work as well as report review, the individual researchers (credited on each document) should be applauded for their world-class work.
The research yielded 37 findings, five of which were considered “high severity.” All of these findings have been officially reported to the Kubernetes Product Security Committee and will be, and have already begun to be addressed in the open.
The findings spanned the breadth of security control families from authentication and authorization to cryptography and timing attacks. They also spanned severity from bypassing critical security controls like Pod Security Policy, to “I’d-prefer-it-didn’t” issues like supporting older and less secure TLS ciphers. Beyond the technical findings, the audit produced 241 pages of supporting data, recommendations, threat models, reference architectures, and prime areas for future research.
This body of research represents a deep foundation for the community to build upon. Kubernetes has been maturing for some time, and now its security posture has begun to mature as well. Audits like what we did for Kubernetes are a common part of a product lifecycle and help us stay a step ahead of potential security issues. Despite many important findings, we did not see fundamental architectural design flaws, or critical vulnerabilities that should cause pause when adopting Kubernetes for high-security workloads or critical business functions.
This was only the first of what we hope are many security research and review efforts done for Kubernetes. The members of the Security Audit Working Group will continue to help the Kubernetes project, distributions and users improve their security stance based on the findings and knowledge we gained here. For Kubernetes users, you can read through the full report or the two white papers for more details on the audit findings.