Managing and maturing Kubernetes security in the enterprise

The TL;DR version of the Infoworld article went something like this: “Companies are shying away from managing their own Kubernetes clusters and more and more, turning to managed Kubernetes solutions” – and I was not surprised.

Kubernetes: Complex and ever-evolving ecosystem

Even though Kubernetes adoption seems to be at an all-time high (48% according to the Container Journal, from 27% in 2018), security awareness for teams working on Kubernetes projects at their workplaces and running mission-critical workloads on Kubernetes is surprisingly low. I should know: my team and I trained over a thousand folks on managing container and Kubernetes security in 2020 alone. In addition, we work on security testing, assessment and consulting engagements where we run into and often successfully break into Kubernetes deployments (with permission, of course).

Kubernetes has emerged as a viable platform for companies to deploy their traditional workloads, mysterious microservices and miscellaneous to Kubernetes. Moreover, it has emerged as a platform of platforms, where you can build out viable and extremely powerful Platform-as-a-Service infrastructure for your organization. However, platforms of platforms come with security rabbit holes.

Kubernetes is a complex and ever-evolving ecosystem. The security perspective is naturally an extension of that complexity. From RBAC to encryption providers, monitoring services and service meshes, it’s easy to get lost in Kubernetes security.

Here are some things that I think would be good to focus on, as a way of maturing your way into robust Kubernetes security, and more importantly, without tearing your hair out.

Start with purpose

The more I see insecure Kubernetes clusters, the more I realize no attention was paid to purpose during the genesis of the project. The project might have started off as a “pilot” for an application or a small set of applications but has now ballooned into a mess of different workloads, running on haphazardly configured namespaces, leading to poor RBAC policies and permissions, and a bad mix of improperly implemented security controls across the board. What’s worse, add CI/CD and GitOps to the mix, these clusters start to look like run-down, derelict buildings that were built a couple of years ago by a shoddy contractor.

Set out the purpose and define the boundaries of a new cluster. For instance, running complex microservices with service meshes and advanced tracing functionality alongside several of “lift and shift” legacy projects on the same cluster tends to produce porous RBAC policies and security policies that cannot be implemented correctly – even if the cluster is namespaced to the hilt.

Kubernetes is a distributed system, so the answer may be to leverage distributed, decentralized systems and create purpose-driven clusters that can be designed appropriately, and for which security can be enforced consistently.

Get awareness into the room

With Kubernetes projects, people go from 0-100 very quickly and before they know it, they are running a massive tranche of mission-critical workload on the Kube.

The important thing for any team or organization undertaking Kubernetes implementations is to get a baseline amount of security awareness of Kubernetes and its broader ecosystem. Just like you train users to not respond to phishing emails with security awareness training, you need to have your DevOps staff, architects and engineering teams understanding Kubernetes security implementations: how it can be compromised and how it can be fixed.

In fact, my team and I always find eye-opening reactions when we train technology teams on Kubernetes security and they’re always shell-shocked to find privilege-escalation possibilities and attack patterns during the lab sessions, which are powerful against Kubernetes and workloads hosted on Kubernetes.

With Kubernetes, it’s important that technology teams are exposed to both offensive and defensive techniques. In fact, I would go so far as to say that understanding defense with Kubernetes is even more important than understanding offense.

The complex nature of the system calls for understanding the different configuration parameters, runtime security implementations, security monitoring techniques, encryption possibilities, integrations and tools. This helps immensely when these teams are building and maintaining their Kubernetes clusters. Security awareness helps, especially when implementing systems like Kubernetes.

Keep it simple

Not a day goes by where I don’t see a splashy ad by vendors proclaiming that their Kubernetes security solution protects clusters against all evils, deliberate or unintentional.

I have nothing against tools, and I am sure the right tools can really help you secure your Kubernetes deployments extremely well. However, I always urge folks to go back to the basics.

After establishing consistency with cluster design and namespaces, it’s important to home in on RBAC and work your way from there into other areas like secrets management, security monitoring, runtime security and policy management.

Designing and implementing a scalable and relevant access management system for your cluster goes a long way in securing it against even the most unexpected and “sophisticated” threat actors.

On the basis of strong RBAC, other controls including monitoring, supply-chain security, workload security and other aspects become much easier to implement.

Automate! Automate! Automate!

One of the biggest benefits of the massive Kubernetes ecosystem is the number of tools available to do $THING. You literally have a tool for every possible utility and use-case, ranging from deployment to encryption to security monitoring. The good thing is that a lot of these tools, yes, even the good ones, cost you a total of $0. The open-source ecosystem for Kubernetes is burgeoning and finding high-quality security tools to automate discovering of flaws to auto fixing YAML manifests is not a problem.

Leveraging these tools is something that you and your team should be doing. Right from including it as part of your DevOps pipelines to using them to identify vulnerabilities.

Managing Kubernetes security

The Admission Controller object in Kubernetes is probably the most important security paradigm that one can leverage to add additional security controls like resource constraints, namespace lifecycle controls.

Previously, the Pod Security Policy was the object that would validate and enforce granular security controls. However, its deprecation created the need for alternatives solutions.

Solutions like the Open Policy Agent and Kyverno have resulted in the rise of Policy-as-Code frameworks, where you can frame desired policy controls and implement them across your cluster or specific namespaces. These policy frameworks allow you to set granular security policies that with Domain Specific Languages (DSLs) or YAML and leveraging them can render your cluster a LOT more secure and more scalable.