Making informed decisions: The importance of data driven security

When deciding what product to buy, the information the vendor offers about the product is helpful, but not nearly enough: you need to analyze individual product results and peruse comparative reports to make the right choice for your organization.

In this podcast recorded at Black Hat USA 2018, Vikram Phatak, CEO of NSS Labs, talks about data driven security.

Here’s a transcript of the podcast for your convenience.

Hi, this is Vikram Phatak, CEO of NSS Labs. We’ll be talking today about the importance of data driven security. We do independent testing for security products around the globe. One of the things that’s important for us is our business model, enterprises and governments are our customer. So, a vendor can’t pay to be in a test, they can’t pay to not be in a test. We’re working for you.

In my prior life I had an intrusion prevention company, and we had good customers, but folks who could have the sophistication to test products for themselves. This is the mid-2000s. It was clear to me that there was a lot of marketing in security, and that the claims vendors were making were not always matching with the capabilities of their products.

But for somebody purchasing those products, there was very difficult for them to tell. You could tell if a product slowed you down, we could tell if a product was alerting you to things that were not real, false positives, but you couldn’t tell what it was missing. And of course security vendors would go and talk about how no product’s perfect. My view was – well yeah, no product’s perfect, but that doesn’t mean you have to not even try.

Not being perfect doesn’t mean that you only block 30 percent when your customers would reasonably expect that you would be approaching 100 percent, but just don’t get it right sometimes. That’s sort of the journey we’ve been on, trying to help customers, help folks around the world really understand how different security products perform, what’s important, how to really assess, and even you know from an architectural perspective what’s the right approach to solving your needs. Because it’s not a one size fits all, and there tends to be sort of “throw money at it, and just buy another product” when oftentimes it’s a matter of really understanding and utilizing the products and designing your network so that you get the most out of each dollar.

One of the things that we’ve actually been doing over the years is evolving our capabilities, as threat actors have become more sophisticated and specifically as they’ve gotten faster. We realized that if we started, we’re testing products with attacks that were a week old, a month old, that all the vendors would get 100 percent. Not because they were great, but because they could get there fairly quickly, but the bad guys were rotating Web sites, and rotating attacks hourly, or maybe even you know within a day or two that they would be changing attacks. So, if we weren’t fast at capturing the attacks and then running them through the security products, than we would be giving a false sense of security to purchasers. We developed a technology that is able to continuously, in an automated way, assess the capabilities of various security products.

We’ve been using that for testing ourselves, internally, for a number of years, and it’s been a very enlightening to see the ups and downs of different products, and the cycles they run through, and the cat and mouse game that happens between the attackers and the security vendors.

One of the things that, now that it’s all automated, that we’ve been able to package up and offer to customers is that they can actually take advantage of the harness that we’ve developed for themselves. So, instead of it being just a product that say we purchase from someone, and we drop into the test rig with the recommended settings from the vendor. Chances are that if you’re a larger company, you’re tuning, you’re changing policies, etc., you’ve got your own specific needs. Now, again it’s all automated, you can set up a product and have us do continuous test of that product on your behalf. We call that continuous security validation.

And that would be with your policy and your configuration. So, you get a much better sense of how do you really stand. Because, as you can imagine, when we test products you’re either using recommended settings, or the vendors come in and they try and put their best foot forward to know that you’re going to probably have something different, if there’s a false positive. In fact, we’ve got a customer that had an endpoint product (should say has, they are about to replace it), where they turned off over 100 different protections because they were raising alerts (it’s a big oil and gas company) in their production environment, it was raising a lot of alerts, and obviously you can’t have that.

But what they didn’t know, still don’t know, and now they are getting visibility is what was the impact of turning those off. And the short version is – it was pretty bad. I mean, the product was maybe blocking 10 percent of the attacks after they disabled all those different features. Knowing that now, you know it allows them to either select a different product, or go back to the vendor and say “hey, hold on a second, I’ve got the specific need. You can’t be writing your protections in such a way that they block my legitimate applications.” The vendors themselves, not that they’re bad people, they never had that feedback before because it wasn’t something that they were being told.

So, it really helps the entire industry, but the vendors and the enterprises, governments to get a better handle on the decisions they’re making and what the impact is from an operational perspective. Set up for that’s pretty easy if anybody is interested, just treat it like a remote office or branch office, ship us your product and provide us some licenses for your endpoint. That’s what it is. And then install it like you would any other branch office, with your central management system, your policies, your configs, and we’ll take it from there. If you need help with that, we’re going to help with that as well.

What we’ve been finding is that, for lack of a better description, if you were here at Black Hat right now and if you walked around the show floor, industry analysts are estimating that the security market is worth about 96 billion dollars. And let’s just say that two billion of it is vulnerability management, and that a billion of it is threat intelligence, but there’s 93 billion of that 96 that’s designed or supposed to be keeping those threats, those bad guys away from those vulnerabilities. If you ask folks, you know where they stand, nobody can really tell you. This is one of those things that can help. So, instead of just simply “select a vendor hopefully they’re good”, you know most in my experience have good intentions even though their marketing can be a bit grandiose at times. But you pick a vendor, and you trust in them, and you hope, but without the data to really understand what’s happening, that’s really not an effective strategy.

Taking a data driven approach where you’re able to uncover, have the visibility, understand what’s going on, work with your vendors or your SOC team can modify, make protections available, is really important or could be as the example with a different customer. When they looked at the data they found that the single greatest problem they had, was the root cause of it was that old JD Edwards system they were running, which forced them to run IE6, which meant they couldn’t go to Windows 10. You know, by default, they were having an attack surface that was antiquated, and was easily exploitable by the bad guys. When the CISO found out, you know being able to go to the CIO and to the business folks, and explain that the most important thing they could do from a security perspective was actually to update their business application, and spend the money there as opposed to buying a bunch of security products that would imperfectly try and protect around that problem, that that was really the best course of action.

Taking a data driven approach once you have the visibility really lets you be thoughtful and make decisions, and hopefully get away from the constant firefighting that ends up happening right now in security teams. And I know it’s hard, but the folks that we’ve been working with that have taken that approach really, their lives have gotten better. Right. And that’s one of the things I’m proud of in talking to them about it, is we’ve helped them make their lives better.

As you guys know, we run group tests all the time, endpoint products like the advanced endpoint for next gen firewall. We just released today the Software Defined Wide Area Network (SD-WAN) group test and results were really interesting, the products overall. One of our takeaways is that the SD-WAN technologies in general are ready for prime time, we saw some pretty solid products. There were three that we had recommended. That was Fortinet, Talari, and to the third was VMware. And then we have several verified products which are also really good products. It was just mostly on a cost basis that they maybe a little bit more expensive compared to some of the others, such as Versa, Forcepoint and some others.

One of the things that we didn’t, which we tested for the products that were, that had it availble, was security. So, primary use case with SD-WAN, there was all kinds of things we tested, everything from a zero touch, you know remote configurations to application steering and resiliency failover capabilities, and so on to maintain quality of user experience. Most of the products did pretty well there. The big differentiator we found was in the quality of experience for Voice over IP, Video over IP, and then overall application performance. We also did for those products that had it available – we tested security.

We didn’t put that in our official comparison, because not everybody had security yet. We do see that is something that, as SD-WAN moves into a wind edge kind of a technology, that it will incorporate security. There are three products that did have it in there, and they did for all did very well. The Fortinet as I mentioned was recommended, and then Forcepoint and Versa were verified, and they all had very strong security capabilities as well.

So, depending on which we’re looking for, if you want to have an architecture that is security separate from SD-WAN, lots of different choices! We encourage you to read the reports on our website at www.nsslabs.com, or if security is important, then and as I mentioned there’s three products that you should be putting it through the top of your list, because you know the security is embedded in those, and so you don’t have to have a separate security product from an SD-WAN product, which again long term is I think where it’s going.

With that said, thank you for your time, and hopefully this was an interesting podcast.