Build-time security: Block risk and security issues from production rings

Build-time security has become a standard part of any security program and continues to grow in popularity with the shift left movement. In its most popular form, it’s a series of checks that take place as code makes its way from a developer’s laptop into production to ensure that the code is free from known vulnerabilities.

While they share some similarities with production environments, it’s important to realize that build servers have a unique threat model and require additional security measures that map to the unique set of risks.

Build-time security is not just about securing code that is pushed through the pipeline; it is about analyzing and monitoring the process and tools that enable that code to be pushed out and implementing sufficient monitoring and controls to ensure that it is done safely.

Build servers for example, not just the code residing on them, can represent a significant risk to organizations because in order to be effective they must be granted a lot of power. This risk is often forgotten by teams, leading them to inadequately account for it in their build time security program.

Why build servers are a popular target

While build servers are not typically a high priority for security teams, they have a number of characteristics that make them a common target for attackers.

• Build servers are a critical transit hub – Code and value flows through a build server from corporate endpoints into the other parts of the organization. So, an attacker exploiting a build server could ultimately gain access to other parts of the business.
• Build servers link environments with unequal security – By their very nature, build servers are natural bridges between corporate environments (IT owned), pre-production cloud environments (engineering and operations owned), and production cloud environments (operations owned.) By breaching the least secure of these environments, an attacker gains a path to the other environments.
• Build servers provide access to source code managers – Because they are closely related to source code management (SCM) systems (they may even be the same system) SCM and build systems are likely passing code back and forth. This could allow an attacker to compromise the SCM itself by infecting the repository or moving laterally to it.
• Build servers are publicly accessible – Because they are often left exposed to the LAN or even the open internet – teams regularly put build system on the corporate network next to WiFi and VoIP hardware – they can function as a bridge between those end user systems and production cloud infrastructure.

The value for an attacker and impact to an organization

Because build servers are largely misunderstood and thus not prioritized in security as noted above, they often represent hidden risk for organizations. The impact of a build system being compromised is meaningful to the organization even if it never becomes public because of the sensitive information that lives on the build systems themselves and how it can be used. For example:

• Artifacts left behind – Beyond the build software itself, build servers can include data, machine learning models, documentation, and other archived information. While most companies have policies against engineers putting customer data in build environments, it is not uncommon for engineers to make exceptions as a result of pressure from the business side.
• Source code – Other than intellectual property theft, source code access allows for easier development of production attacks by abusing business logic, identifying vulnerabilities, and more easily discovering hidden functionality. It additionally allows for malicious code, such as web shells and cryptomining software, to be embedded in the mobile and web application code that is shipped to production and ultimately run on a production server, or the end user’s client.
• Code signing keys – Code signing keys allows malware and other malicious software to be installed and run in environments, circumventing security software and OS controls.
• Infrastructure keys – Infrastructure keys are often left on the disk or can be accessed by the build servers, allowing the attacker to pivot their attack into other parts of the infrastructure. This is often how build systems are able to push out releases and update your environment, access which the attackers now also have.

Conclusion

Every team within an organization is playing catch up as environments increasingly overlap with one another. And since there is less-and-less differentiation between corporate and production environments, not all of the overlapping sections and bridges – some of them major transit hubs – are obvious.

The massive hubs that are turned into build systems are among the most impactful and under appreciated links that both attackers and defenders can leverage. Any effective build-time security strategy must account for, and monitor, not just the code to be pushed out, but the tools and processes involved in the development cycle.