What the food and building industry can teach us about securing embedded systems

As one of the leading experts in product security with over 15 years of experience in security engineering and 120 cybersecurity patents under his belt, Adam Boulton is one of the most experienced software security professionals in the industry.

Currently the SVP of Security Technology and Innovation at Cybellum, the Left to Our Own Devices podcast invited Adam Boulton to share his experience and his tips on building a product security strategy.

Adam didn’t expect his career to lead to embedded systems. For years, he was involved in typical security-critical systems: web applications, mobile applications, source code reviews, without any real exposure to embedded devices.

As he became more involved in embedded systems, he discovered that as the industry matured, people have become too comfortable with source code analysis. “It’s considered witchcraft, where you compile software, and it’s this source code that goes into this black box and it spits out executable code.”

Although there are a lot of reasons why compiled software is advantageous, Adam finds it problematic “How much time is spent on the source code and the review of that? Because, that’s not what you are actually executing or distributing.”

3 limitations of compiled software and analysis

Adam shared with the podcast what he sees at the three main limitations of reliance on compiled software and analysis:

1. Testing the source code doesn’t consider the full system

While companies may choose to test their source code within their processes it can’t be the final test as it isn’t a true representation of what will be delivered to the end user. “You have the representation of what has happened. You’re drawing multiple translation phases, and then you actually distribute that out to the market.”

“So why isn’t more time spent on what you distribute, what the end users, what the threat actors will have access to? What is going to be executed on the platform? Did anyone check the build environments and all the scripts and what was actually going to get built? Often you miss that and it’s a huge blind spot.”

2. Many checks can’t be conducted at a source level

Source code is only one level of a complex skyscraper. Testing the foundation is critical, but engineers must also check each floor as it’s developed to ensure the strength and integrity needed to execute its function while being able to support whatever will be built upon it.

In addition, Adam warns, “There are tons of non-functional requirements, such as hardening requirements that you cannot check because they don’t exist at a source code level. There are things the compiler will do and the way you compile the software and give those instructions to the compiler to help harden the software and make it more secure. It’s impossible to do that in a source code analysis.”

3. People place too much faith in their compiler

Adam reveals that many vulnerabilities emerge from the way the compiler was configured, or the way that build environments have just included all sorts of hidden secrets or documents. Without proper review, these vulnerabilities remain in the device, becoming embedded into devices and sent out to become active on networks across the globe. Remaining secure, and being viewed by customers as secure, requires checks throughout the development process along with SBOM management to protect devices down to the component level, should a vulnerability be discovered in the future.

Delivering a level of QA similar to the food or building industry

Adam expressed that embedded systems should take lessons from other industries in terms of the quality assurance of many products.

Take the food industry, for example: “Whether the chefs check it or the service staff check it, you’d like to check what is going out the door. You know, the chef isn’t just going to read about the ingredients,” he adds. “The same is true for the building industry. A building inspector does not just review the blueprints. They have to check what is actually built. Is it built to specification? The size of that property?”

“The lesson we can take from other industries is to check what has actually gone out,” he concludes.

Developing a quality software security strategy – with metrics and KPIs

Calling upon his experience, Adam shared strategies and KPIs that can be used by C-level executives to track and measure the ROI of product security.

“I looked back at previous software security strategies over the years, and ones that I had to align to as an individual contributor, a security researcher. And they just weren’t measured. There was no clear sort of KPIs.”

“You don’t do this in most other teams but yet for software security, there seems to be this lack of maturity in a lot of product software security strategies to ask: Where’s your metrics? How do we know our ROI? How are we going to improve?”

Adam explains that even with the complexity of software supply chains, it’s all about visibility and goals. Product teams should develop that visibility and let that direct the quality. He stresses the need to use technology to organize software assets and inventory and draw conclusions from that.

“For large products, for example, like an infotainment system, a modern one has more than 140,000 files on there, right? These are large, complex systems. Where do we start? I say we start with a quantified or measurable software security strategy where you ask and answer the following questions and track these answers over time.”

Some of these questions are:

• Where are we today?
• What do we want to be?
• Where do we want to be in a couple of years?
• What are we trying to achieve?”

How product teams can secure a budget in 2023

Adam shared a few practical tips on how product teams can secure a budget in a difficult economy:

• Understand the business – CEOs aren’t interested in CVEs and CVSS scores, no matter how passionate you are. Ask yourself, what do they want to build? Not what you want to build.
• Know your regulatory practices – They are quantified when it comes to business and risk. Every C-level executive will be held accountable and receive the penalty.
• Know your standards – Even if you don’t follow regulatory practices, it’s great to help align to standards to improve quality. They’re a very good reference material.
• Take a hard look at the strategy – Before building out an actual product security team, take a hard look at the strategy. If you are faced with large and complex problems, look at the inventory of your software. “I’ve done many assessments with huge organizations and if you look at the skillsets with the product security team, they aren’t aligned with the technology stack with what’s being developed. And that’s where the inventory comes back into play.”
• Have measurable data points – Data points build credibility, especially when you are working with a CEO. “This is why I’m always so focused on the inventory and assets of software. We do this with the ingredients of food. We do this with building materials, to manage anything we’re gonna build and develop.”

Sharing data points with executives as a foundation for how you plan on using given resources. This requires numbers and the identification of internal trends over time, not just based on a gut feeling. “Because we do this all over the place. We do this with the ingredients of food. We do this with building materials. To manage anything we’re going to build and develop. It doesn’t have to be any different than software,” he says.

Yet just like the food and construction industries, regulations have helped put security standards in place so we can trust any building we enter. Companies must ask themselves if their industry can retain high security standards on their own or if regulatory bodies will need to do the heavy lifting in the future.