Keys, tokens and too much trust found in container images

We are all aware of the risks introduced by good old third party code. Where would we be without it? Apparently not very far. It is estimated that between 30 to 70 percent of code comes from 3rd party applications. This is why we patch up old libraries and update open source packages.

While the risks of 3rd party code are well known, the risks of using 3rd party containers are more obscure. In this article I will discuss one such risk: the introduction of 3rd party secrets; and look at examples from public registries.

To get a taste of the prevalence of such secrets, we scanned the top 1,000 most popular container images found on public registries. We were not only looking for default passwords, but mostly for less obvious examples of secrets. We selected only the latest images, from the top starred public repositories. What we found convinced us that the risk is very real, as 67% of images had at least one form of a secret.

Tokens anyone?

We didn’t find many, but we did find a handful of images that had access tokens stored inside configuration files or source code. Luckily, it looks as though these were either test files, comments or examples – not real tokens forgotten in the image.

Should you respect certificate authorities?

Well yes, but only knowingly! We were not surprised when we found that 67% of images had certificates installed on them. Many applications rely on certificates as part of their function. One notable example is OpenSSL, which was responsible for almost 80% of certificates. However, this raises the question: does your container really need to trust hundreds of CAs?

The usage of CAs is to establish a chain of trust. A domain in the internet that communicates using SSL/TLS needs a valid certificate signed by a trusted CA. By trusting hundreds of CAs, you actually trust hundreds of thousands – if not millions – of domains. While this may be a tolerable practice for a web browser, it is not so for a container that communicates with only a handful of services. If you use public containers, we recommend reviewing which certificates are installed by default, and remove all unnecessary ones.

Proprietary certificates

We found several images for specific applications that come with their own “handy” certificate, which should be added to the certificate store of their clients. The private key in these cases is found in GitHub. Usually, the authors of these images explain that these certificates should be changed after installation.

If you just realized that you already have such an application running, using a default certificate, no big deal – just update your image and deploy another container. But by that time, you may already have multiple clients that already trust an insecure certificate. This is exactly the type of risk that can be easily overlooked, but easily mitigated by scanning for certificates in your images. If you just realized that you have this exact problem, you may already have a very large number of clients trusting an insecure certificate, which could be a real headache to fix now.

Untrustworthy hosts

It is always good manners to respect your hosts, but to trust them? That is what host keys are used for. They are a method for SSH clients to trust known hosts. An SSH client keeps a list of public keys of trusted hosts, and can be configured to alert or even to refuse to connect to unknown hosts. We found that 5% of images already contained default private and public host keys. These keys can be used for MITM (“man in the middle”) attacks on containers’ clients that trust them. To remove the risk, as in the previous example, you need to revoke these keys from all of their clients.

Pay root forward

About 3% of images had some default insecure key configured, or an authorized_keys file present for a user. At first, this number seemed somewhat high to us. Could it be that so many images enable SSH connection by default using publicly available keys? When we dug a little deeper, we noticed that about two thirds of these images shared exactly the same insecure key. They all used the same base image. While the authors of the base image disabled the ssh server by default, other authors that built on the basis of this image enabled it by default; as can be clearly seen from their Dockerfiles:

To test that this is indeed insecure, we pulled one image that enabled this key by default, and ran two containers: one detached and another with bash. Then we used the insecure key from the second container to SSH to the first one. The following figure shows this successful experiment. Note how the container ID changes after we SSH from the first container to the second.

Another interesting finding was that some authors of images were really trying to improve their images’ security. Instead of using “insecure passwords”, they chose to generate SSH keys in the Dockerfile itself:

The keys indeed changed for each new image tag, but remained identical throughout all of the containers based on that image – enabling each container to SSH to its siblings. Not to mention that if you pulled the image directly (instead of building it using the Dockerfile), than all of your container use the same publicly known SSH key.

These examples show that leaving SSH keys on an image, may be great in term of convenience, but detrimental to security. Even if a key is not meant to be used by default, we saw that other images conveniently make it enabled by default. Additionally, “randomization” of a key should not happen on the image, as it is carried to all of its containers.

Summary

Whether enterprises use a selected few images, or don’t rely on public repositories for their images at all, our data shows that secrets are left unattended on images. As we saw in the case of host keys and proprietary certificates, it is best to know about these secrets as soon as possible, because removing them from clients, reissuing them, and then redeploying them can be a tedious task.

To protect your containers, make sure that your images only have the necessary packages and configurations. Validate that these configurations don’t include default passwords and keys. Also, continuously scan your images for such secrets on a regular basis. We recommend using tools to distribute relevant secrets to containers in run time (making the need for hardcoding default secrets superfluous) such as Amazon KMS, HashiCorp Vault and Aqua’s Secrets.