You must build a security team. Where do you start?

Security veteran Chris Deibler, the new VP of Security at DataGrail, has been brought in to build the company’s security team to support its growth.

A former Director of Security Engineering at Shopify and Director of Security at Twitch, he knows a thing or two (or 52) about successfully instituting a security organization within an enterprise, so we decided to pick his brain on the subject.

[The answers have been lightly edited for clarity.]

Scaling a security team as the company grows is a delicate endeavor. What are the notes those in charge of the effort must strike correctly from the get go? And what things never, ever work?

Great question! I think the first thing you have to communicate is that you can’t do it all at once, and that will necessarily mean lots of engineering-visible posture improvements over time. Small things will change, large things will change, you’ll try an approach you saw work before that fails here and now you have to pivot rapidly.

To your customers, this can seem like chaos or lack of focus. You need to provide transparency into what you’re addressing, when, and why; you need to ask the grace to miss a few shots. You also need to communicate your (hopefully growing) capacity effectively, and educate that as you scale, some smaller fires necessarily get left burning in the pursuit of better wins.

The thing that never works, or at least not in my experience, is attempting to bridge too many rungs on the maturity model at once. You don’t go from “We should do some observability some time” to “Here is our omni-source self-tuning event ingestion and filtration flow”. You can’t overmatch your engineering org’s tolerance for process, and especially bad process. Walking before you run is cliche, but it’s a less painful way to discover a nail in your shoes.

Can you share some advice on how new team members should be brought on board (either from inside or outside of the organization) and some tips for how to look for and what to look for in new hires?

A lot of orgs approach this differently. I am personally a fan of the full-service comprehensive approach where a new hire’s first two weeks are actually spent learning the company, its product, and its customers vs. teams, tools, and tech. Sans check-in with the hiring manager at useful intervals for questions, the foundation of the education should be the value proposition for the people you’re serving. Working a warehouse shift, taking on a set of easy customer support tickets, shadowing a remote sale; whatever lets you experience the customer’s needs and wants.

Then you move to “And this is how we fit in”. Security can become a very siloed and inward looking function and while you can succeed like that, your odds are better deeply aligned with the business. In that vein, getting everyone an onboarding buddy outside their function is a useful way to build that relationship.

I don’t have a lot of tips around what to look for in new hires because I don’t presuppose I know all the ways someone can be successful in the space. I think we’re seeing a general trend towards “You can approach this job from a lot of backgrounds” in job descriptions and hiring practices and that’s spot on. Certainly there are easy wins like “curiosity” and “humility” but you want those assessed further up the funnel. You as the security context interviewer should be providing practical problems to solve and getting out of the way to see the approach. One thing I do like to see in this model is the thoughtfulness around the “why” of the approach and the ability to pivot with new factors added.

In your experience, what are the best ways to build cohesion in the security team?

Share a culture, make it as inclusive as possible, and boot rotten apples. That’s it.

Sharing culture means involving your people in the generation and maintenance of that culture. Your tenets should be sourced and voiced by your people. Your rituals should be at a cadence and structure that supports their neurodiversity.

The fun things you do together should be their suggestions. Content for show-and-tell should be their content. None of this excuses you from editorial control, but if you treat growing your culture like growing your organization – identifying leaders and investing in them – you’re going to do well.

There are guard rails. Games seem easy, but not everyone likes games. Drinking is an old security standard, but not everyone drinks, nor wants to be around it. You need to be paying attention to who shows up to the fun stuff and who doesn’t – lack of participation could be nothing, but it could also mean that your chosen rituals are not inclusive and you need to find something better.

Booting rotten apples is self-explanatory, but there’s never been an exit I’ve done where I said afterwards “Man, I jumped on that just in time.” More frequently it’s “I should have managed this person up or out a year ago.” It’s an easy trap, especially for natively congenial people. Don’t fall into it. Toxicity does astoundingly broad damage to your culture in a very short span of time. People take that memory to their next org. You get a rep. Hiring is harder. Velocity suffers. Do not abide rotten apples.

What are the best ways to make the security team an integral and welcome cog in the organization’s machine?

Learn and invest in their rituals. The trust you build playing the game their way will buy you leeway when you come back to the table with improvements.

Sometimes there’s nothing, and you have to create a process from whole cloth. When you do that, you have to treat the thing you created as sacred, and hold yourself accountable to it, or no one is going to take you seriously.

I like severity systems as an example, because they’re easy to understand and (by a quirk of fate) I get put on the hook for building one almost everywhere I go. You go to all the trouble of building out this severity system, go to Eng and say “Everything with this trait is a sev-2, burn it down in two weeks.” What’s the first thing they’re going to do? Pull up the security repos and see how long the sev-2s have been rotting on the vine. Eng teams love “Do as I say, not as I do”. This is a complicated way of saying that you should “eat your own dogfood”, but the reality is your credibility rests on it.

It is an often repeated axiom that a diverse security team is a must for tackling security problems in a comprehensive way. Do you find that to be true? If not, why not? If yes, can you share past experiences that confirmed that for you?

I do find this to be true, and I’ve been on both ends of it. You need people in, or adjacent to, your organization that think differently. That can be background, work experience, cultural origin, race, gender, personality, neurotypicality, language – all those things contribute to what comes out of your mouth when someone asks a hard question, or challenges an assumption.

Challenging assumptions is the bread and butter of security, whether they’re yours or a partner’s. I think the easiest example of this I can remember is way back in my career as an architect, where I was brought into an existing team mulling over remote access changes for a risky class of users. They’d been grinding on it for weeks. I joked “Throw ‘em in Citrix and let’s roll!”, because that’s the last solution I implemented and I couldn’t possibly be serious. This turned out to be the best move by far.

Similarly, I am reminded every day that my grasp of a solution space is incomplete and I have blinders on viable options. On implementing command-level logging for a bastion environment, our new intern dropped “Well, in the Army, we did this.” No one in old guard tech space was even close. Eureka, buddy. You don’t get diversity of thought in groups of people following each other around from gig to gig. Get new thought. Open the top of that funnel wide.

Growing a security team is a welcome challenge, but what if the organization must scale it back? Is there a correct way to do it, and what errors should be avoided – and how?

Ideally, any circumstance in which a security org needs to be scaled back is accompanied by an equivalent scaling of the business. This is one of the reasons why understanding your scaling function(s) is so important as a security leader.

You should know how many application security engineers are needed to support a given product engineering team. You should know what incident load a certain number of employees produces, and how big your on-call rotation should be to process it. You should know how many infrastructure engineers are necessary to maintain adequate coverage over your portfolio of platforms and clouds. There’s no gold standard for these ratios, but the point is that for any given level of risk you are staffed somewhat elastically with the business. Change those ratios, change the risk. That’s a choice the business can make, but your job, beyond defending your people and your risk posture, is to scale. Scaling works both ways. Be transparent about the trades with your leadership. Document the decisions. Decide how much risk you can personally stomach (and still sleep). Act accordingly.

No matter how you got to this point, make sure you are a resource for the humans you are losing. The security industry is small and memories are long.

Small businesses need cybersecurity as much as the big companies, but they are often short on funds. With that in mind, do you have any advice for them on how to start building cybersecurity competency within their organization on a budget?

Someone influential in my career once told me that security is applied pain management, i.e., that you need to apply the correct pain to the correct people. Someone in your organization is already building cybersecurity competency whether they want to or not by virtue of managing the consequences – pain – of not having a program. Find out who that is.

I take the “designated resource” approach over the “security is everyone’s job” approach because I believe the axiom that things that belong to everyone belong to no one. You (and they!) need to be realistic about what that first resource can do, but matrixing security early only makes sense only if you have a designated owner with resourcing authority to direct the matrix.

Conferences are cheap; some of them are high quality. Reddit is free. CTFs and their kin are fun. Collegiate cybersecurity teams need volunteers. Wiresharking your home network is easy and frequently informative. The economical resources are there, all you need is the human with the spark.

On a more personal note: What challenges await you at DataGrail?

Hiring and the dread of being a threat aggregator.

It’s no secret that security hiring can be slow and you will be pressured to take shortcuts, whether it’s contractors or staff augmentation or prioritizing pain reduction over building the right team for the long term. Some of those things will be necessary to maintain your sanity and motivation along the way. It’s a fun challenge to think about because you’re pre-negotiating with your future self that’s sitting in the middle of a flaming incident, running their own queries, scraping their own logs, looking up law enforcement contacts, and idling wondering whether security was the correct career choice. Maybe get that guy an IR contractor, yeah? But recovering from a bad hire is worse. Put in the time, say “no” more often, and demand that your interview panels send clear signals. If that muscle doesn’t exist yet, teach it. Never: “Eh, I guess they’re a three”.

On the threat aggregator front, it’s scary to have such a deep impact on things that downstream customers value. You never want to be the vector that gets someone else popped. I get that the world is a massive ugly web of transitive trust now and that customer security and privacy expectations feel like they’re at a global low, but that’s the problem, right? That trust is gone. We – the company, the industry, the ecosystem – have to rebuild that trust. That means we have to be a trust anchor, not a threat aggregator.