George Gerchow, CSO at Sumo Logic: Our DevSecOps strategy

Sumo Logic was founded in 2010 by experts in log management, scalable systems, big data, and security. Today, their purpose-built, cloud-native service analyzes more than 100 petabytes of data, more than 16 million searches, and delivers 10s of millions of insights daily – positioning Sumo among the most powerful machine data analytics services in the world.

In this podcast, George Gerchow, CSO with Sumo Logic, talks about their DevSecOps strategy.

Here’s a transcript of the podcast for your convenience.

Hi, my name is George Gerchow, Chief Security Officer with Sumo Logic, and we’re here today to talk about DevSecOps and our approach. Sumo Logic is a company that was created in AWS eight years ago. The security analytics multitenant solution. And one of the reasons why that is important is because, from day one, from minute one, we’ve always had security baked into the DNA of our company. The best way to approach that is to start embedding security with development to create a more agile and secure delivery mechanism and pipeline for software innovation and software development.

And so, as most organizations are showing today with the continuous delivery, continuous innovation pipeline, CI/CD pipeline, if security is not out in front of that, or part of it, you’re going to find yourself in a situation where you’re constantly in this mode of reacting or trying to bolt security on afterwards. So, from the very beginning as I mentioned, we’ve put a certain set of steps in place to make sure that we work well with development, we provide guardrails and not blockers to the innovation that they’re doing.

The first piece that we focus on is code analysis, and so we work quite closely with our developers upfront as they release code in small chunks, to make sure that we’re reviewing that code in an automated fashion to identify new vulnerabilities upfront that can then be remediated quickly, and put into our overall library so then that way the situation doesn’t happen again. Code analysis, and again this could be you know whether you’re working with containers or outside of containers, there is a lot of open source software out there that’s low friction, turnkey, and not disruptive to the development process, to be able to check that code quickly and then make it more secure and more efficient as it goes out into the wild.

The second piece is around change management. So, having a very progressive, agile change management process is key. You know, when we come from mostly waterfall type methodologies, where you’re releasing code once or twice a year, change management is a long stringent process that is not really going to work in a DevOps or DevSecOps shop. It just takes too long. You know if you look a company like ours who are doing 8 to 10 code releases a day, or Etsy that’s doing anywhere from 60 to 80 code releases a day. That’s just not going to work when you have to sit around and wait a week, you know, two weeks, three weeks for a change. The idea is to modernize your change management program to be able to absorb a higher rate of change with less impact to the business while keeping everything secure using modern day tools which are also part of this CI/CD pipeline like Slack and Jira.

Again, I would look at your change management process quite heavily. Ours works out really well to be able to make sure that any developer can submit a change at any given time, that change gets executed within 24 hours.

The third step is around compliance monitoring – why you’re making these changes. You can now gather evidence for compliance, things like GDPR, PCI, HIPAA, ISO 27001 while they take in place instead of disrupting the organization, and going back to the developers and security team and saying: “What actually changed over the course of a year or six months?” You can actually gather that information right then and there once that code has been analyzed, it’s been gone through the change management process. It’s a great time to start doing evidence gathering around compliance.

The fourth one is to start doing threat investigation the minute that code hits a production environment. So, what kind of new threats have been exposed because now this code is out in the wild? So, even though you do upfront code analysis it doesn’t mean that once it hits a production permit and now it has relationships with other services that are running, and other micro services, that no new threats have been added to the environment. And so it’s one of the things that we constantly look for right when code hits production is – Okay, based on the relationship it has with other micro services, or other services running within the organization, did we expose any new threats? And what do we do about those threats?

And so having an automated way to identify those, and remediate those pretty quickly, keeps that lifecycle moving, and again, you tightly couple yourself with a development team, SREs or site recovery engineers to make sure that this is an ongoing continuous effort.

The next piece is around vulnerability management and constantly improving that, so this is the fifth step. You got to always step up your game. So, whether it’s penetration testing on an app, or external penetration testing we believe that now the modern way to do things is via bug bounties. So, you set a bug bounty and you hire essentially, you know, some of the top hackers in the world to try to crack your service. And what this does is it sets up your level of security to an optimum level. And so, in that way, by exposing it and by saying “Look here we are, we’re trying to make ourselves more secure, come and break it”, and you end up finding things that that would typically not happen through a straight methodological pen test.

And so, vulnerability management has been a very key piece for us. And then, finally the last one, the sixth step is continually training developers out of the security budget on security. You know, so what most ops typically do is they do OWASP training, which you know is fine and it’s a good checkbox for compliance, but you’ll see things like SQL injections still being within the top three of the last top 10. And it’s never going to change. It has been that way forever, it’s not going to change. It’s again something that you’re probably already familiar with, and once a year is not enough for touchpoint. What we have to do is based off of the code analysis which was step 1, and then threat investigation, you know be in step four, and then the work they do on step 5 on vulnerable management and collaborating in those bug bounties, you now want to have a grading system for developers.

And so what a lot of modern organizations like ours do is, you actually have a budget set aside for serious security training, like DEF CON for example in Las Vegas. It’s the top hacker conference in the world with hands-on engineers, and there’s live hacking taking place, and then you can now do a scoring system where you can send your top three developers who have had the best scores, the least amount of vulnerabilities that are weighted, and the most amount of collaboration in the bug bounty portal with security team and the hackers, to make sure that they’re on top as security. Send them off to something like DEF CON, and then they’re sitting next to people who have the same exact, if not better coding and development skills, but with mal intention. So, they’re actually getting to see real hackers work in a very malicious way, and then they then can in turn bring that back to the organization, do lunch and learns, cross pollinate. And so it keeps everyone very aware of security, and of course these get competitive. You know, all developers want the opportunity to go to DEF CON, and when it comes out of the CSOs budget, they are going to want to take advantage of that.

So, again, you’re collaborating with the VP of development throughout this entire lifecycle, and it’s a great way to start moving forward. So, whether those six steps work for most organizations or not you kind of determine that, for us it’s ever changing, but those are the six steps that are working right now as we start approaching 2019.