Secure workloads without slowing down your DevOps flows

In this Help Net Security podcast recorded at RSA Conference 2019, David Meltzer, CTO at Tripwire, and Lamar Bailey, Senior Director of Security Research at Tripwire, discuss the challenges of securing DevOps.

Here’s a transcript of the podcast for your convenience.

David: Welcome to the Help Net Security podcast. This is David Meltzer, the CTO at Tripwire. Today I’m joined by Lamar Bailey, Senior Director of Security Research at Tripwire. Today we’re going to be talking about DevOps and DevOps security.

Lamar, DevOps is certainly gaining a lot of traction in the security world and we’ve seen the proliferation of DevSecOps, SecDevOps, SecOps. What are you seeing happening this year around DevOps at RSA?

Lamar: It’s definitely a hot topic here at RSA and lots of companies are looking into it, trying to figure out how to merge between the security teams and the development teams. The development teams are definitely running ahead, trying to get things out to market and security teams are trying to make sure that everything they are pushing is secure and it’s going to the delivering processes. We’re seeing a lot of companies and services trying to bridge that gap, and make sure that those two teams can work together and be able to get their process and products out.

David: From an IT security perspective, starting to understand what are these DevOps parts of the organization doing, what’s a good way to get started?

Lamar: I’d say the best way to get started is to go talk to the developers and see what they’re planning. There’s a lot of open source DevOps tools that are out there, and we see a lot of developers going, grabbing those, working with those, playing with those, and I’m bringing those into the workplace and using them. A lot of times it becomes things that security teams don’t know about, IT doesn’t know about yet. But talk to them, see what they’re using and see how to bridge and make sure we can add security to those tools.

David: One of the things that I’ve seen myself talking to application developers is, how often security just wasn’t even in their considerations, as they started to adopt more of these DevOps tools, especially as they were moving to the cloud. As you see customers adopting cloud infrastructure, what are some of the top security challenges they’d need to be thinking about?

Lamar: One of the interesting security challenges of the cloud we see, is kind of a throwback, it’s access. It tends to be where everybody has root access. They don’t go through and take the time to set up all the least privilege. Everybody that has access to the cloud, has the same root access, and it causes problems.

David: What kind of solutions are there out there for figuring out where these access systems are misconfigured or where these settings exist?

Lamar: There’s not as many as you would think. It is pretty hard to configure the access in some of these cloud services. Some of the cloud services providers themselves have tools, but then they have to be set up correctly. You’re looking a lot of third parties to look to come in and manage that and look at that.

David: Containerization is another area. I’ve seen a whole lot of growth of over the last couple of years, people using Docker and now Kubernetes. Are there security challenges you’ve seen around those areas?

Lamar: Definitely. On the containers, it’s kind of the next segment from virtual machines and for a period of time people thought “oh, it’s just a container, there’s no security issues with it”, which we find that’s not the case. Scanning those containers need to happen pre-production and during production. There are tools out there that you need to be looking at and looking at your containers before they move into production, make sure they are secure and check compliance.

David: One of the Tripwire researchers I saw had been doing some assessments of open source operating systems available in containerized form, and actually found some of the default operating systems being shipped as containers were still vulnerable systems, as they existed.

Lamar: We’ve seen a lot of a lot of that actually, in some of the research. There’s a lot of free containers you can just go grab and start your work from there, but they’re not necessarily secure or they’re not configured correctly. So it’s like grabbing any open source, you’ve got to vet it. You’ve got to make sure it’s what you need for your environment, make sure is secure and then build on top of it.

David: That makes a lot of sense. In terms of looking into that pipeline, what are some of the key areas of integration that people should be thinking about as they start to figure out: “How do I work in this DevOps life cycle and insert security into it?”

Lamar: Definitely, preproduction. We’re seeing a lot of tools now where it’s easier for the developers to scan their own containers and scan their own even serverless, to see what their security stance is before they start. As long as that’s easy, they’ll continue to do that, and they don’t mind doing that. There’s an easy way to fix them before they move into production, because they want to get their code into production, and they don’t want to have IT or SecDevOps say: “All right, well we didn’t do this. Let’s hold off a week or a month.” They’re more interested in getting their code out, so they’ll follow the processes. Having it done beforehand is huge. Looking at it after you push into production, that’s also always going to be required.

David: I remember the old way of doing security assessment which is someone makes a build, they put it into test, and they let security have at it for a couple of days, or they’ll spend a week doing their security assessment. Clearly, those approaches aren’t going to work anymore when people are trying to push code into production on an hourly basis. I think that’s the part that people are often missing around the DevOps cycle which is really all about agility and speed and the velocity that the developers are moving at. Any security solutions that we look at, absolutely have to be moving at the pace of DevOps. That means we have to rethink some of the traditional security controls, maybe even the idea of taking a few hours to do these assessments, really doesn’t work in this new world again, does it?

Lamar: Definitely not. We used to do Waterfall, and Agile killed that. Now DevOps is killing Agile. It’s just not fast enough and IT has got to be at that same pace. I think it’s imperative that you scan and your security is throughout your whole process, and like you say it’s minutes, it’s not hours, it’s not days. Then you’ve got to have a way to revert, if something is wrong, and revert has to be almost instantaneous also.

David: When you are seeing those changes that are happening, and you’re trying to address the security issue right now, what’s the most effective way to deal with that? What are the new ways that people are thinking about securing the CI/CD pipelines that didn’t exist a couple of years ago?

Lamar: It’s integrations and automation mainly. Integration with different tools to check where you’re at in the process, check your pipeline. Once you get into production basically, it’s kind of the most effective way. Any change happens to production, then you just rollback. You don’t have to wait and see what the change was, you can do the forensics on the back end. But if this container changes, for instance, roll it back, put in the known good go container and go forward. That should be completely automated and be in a matter of seconds, not even minutes, for that to happen.

David: Let’s talk about one other topic which is something I hear DevOps talk a lot about which is immutable. That design pattern that everything that we move to production won’t change at all anymore. All the changes will happen from the developer perspective. Have you heard people talk about or use immutability? How does that affect the security of these applications that are built with this immutable paradigm?

Lamar: It is. It comes up a lot, and I think the theory is pretty good. But there’s also the gap of, once this is moved into production, even though the developer did the changes, IT made, and what changes happened. Should that particular device be doing something different now than it did before? You go back and ask the developer what was the change control, and the developer tells you: “Oh, that’s in GitHub or in my code control system to tell you what all my changes were.” There’s a gap there that’s happening, so they’re not as immutable as they think they are.

David: The other thing people also need to keep in mind is the idea that, even if the container or the immutable object itself doesn’t change, the threat environment is always changing. So, a system that might not have been vulnerable when we first scanned it a couple days ago, actually maybe a new vulnerability came out and is vulnerable today.

Lamar: Absolutely. Continuous scanning of these devices is required, these assets. You can’t scan like we used to. Companies will scan their assets once a week, once a quarter, once a month. That’s not valid anymore. Our landscape changes so fast. We’re looking at DevOps switched the code and the products are changing so fast, we almost need to be in a complete continuous security assessment.

David: When you see DevOps being applied in organizations, is it just a cloud thing that you’re seeing or is it happening for on-premise systems and virtualization environments as well?

Lamar: It’s happening all across the environments. It’s interesting because it’s not the same teams. What you’ll see in large enterprises is, maybe there’s multiple teams that are doing this, and some are doing it in cloud, some are doing it on-prem, and they don’t even use the same tool sets. There’s not as much standardization across here either, which is also an issue from an IT’s standpoint of trying to make sure everything’s secure.

David: Moving at the pace of the internet and trying to secure all those systems, certainly a difficult challenge. Thanks for sharing your thoughts on this. This has been Dave Meltzer, the CTO of Tripwire, along with Lamar Bailey, Senior Director of security research at Tripwire.

If you want to find out more information about how DevOps is affecting security or the kind of solutions that Tripwire can provide to integrate into your CI/CD pipeline and help you with your DevOps security, go to www.tripwire.com.