Preparing for your first security breach

So you’ve finally accepted it’s just a matter of time before you experience your first major breach. Despite all the work you’ve put in to your monitoring and response program, the long hours chasing down those last unidentified systems on the network, the endless meetings with department stakeholders and the uncountable hours optimizing your SIEM’s correlation rules, you’ve come to terms with the reality of “when, not if”.

You have probably realized this in your first week on the job – security professionals are not well-renowned for the quality of their sleep, or the health of their livers. This is a guide for everyone who is dreading the day when the excrement impacts the oscillator. I’m not going to tell you about dealing with a breach in a technical or legal sense; I’m going to talk about maintaining your mental health and career prospects during one.

Before anything else, no matter what field you work in during times of crisis you will see everyone’s true colors brought forth – not least of which – your own. You will know more about yourself and your co-workers after the event than you ever did before.

It comes as no surprise that Murphy’s Law will likely bring itself to the fore (true to its nature) at the very worst time. That one web proxy that isn’t yet configured to log to the SIEM – that will be the one your attacker filtrates your data through. That virtual machine cluster that was provisioned a month ago, but is not yet in active use, is where they will stage the attack from. That four-hour quarterly maintenance window that went badly and you lost all the logs for – that is when the breach will happen.

You are going to have your deductive skills tested to the limit; breaches happen through the places you were not looking. If you are lucky, you will be able to infer what happened through the remaining audit artifacts on your network. You are going to have to make large leaps of deduction, and justify them to people desperate to hear “at least they didn’t take everything!” and “It’s not as bad as we thought!”

During a breach, you will find a whole chain of people that previously were merely names on an Org Chart become imminently real. If your experience at the job has been constrained to sitting quietly at your desk doing ‘your thing’, you are going to have more exposure to the executive leadership of your enterprise than you ever imagined. They are going to require fast and decisive answers from you – welcome to their world – you will be asked to make quick assessments of the information you have available and be held accountable for them afterwards.

You will perceive everything you’ve spent your time on amount to nothing, you will rant and rave to yourself (and all listeners) that this is just the proof you should quit and find another job. Take solace in the realization that things may have been much worse without the work you rendered. It is not that a breach occurred; it is the scale of the breach that matters.

Remember that business endures, for better or worse. Realize that the truth of what you saw will never see the light of day – it will be spun into an acceptable story and you will be bound by law to keep the secrets of someone else’s failure. The real trick is to survive the process with your sanity intact.

Your first responsibility will be to create a complete and detailed timeline. Your job now is to discover and document how this happened – but not your interpretation of why this happened – as much as you want to invoke all your “I told you so!” instincts, this is not the time. A complete blow-by-blow timeline of how everything happened within your network is the primary information your command chain needs of you.

This information is what is required for legal, PR, and the board members – it should be the primary deliverable that all other workflow is derived around. Most importantly, this is what will most effectively keep management off your back. Expect to receive constant requests for updated status, but don’t let updating too often get in the way of work. Do not be afraid to push back and give yourself time to report more accurate findings. Make it clear that you can either deliver inaccurate information now, or accurate information in another hour. Your job is to enable informed executive decisions at this point, so set expectations that this is your goal clearly.

Things are going to get a little crazy, requests become orders and niceties fall to the wayside. In times of crisis, sanity becomes more important than pleasantries. Studies have shown that people would rather work with unfriendly, competent people, than unfriendly, incompetent people. This effect becomes more pronounced during times of crisis; do not worry about offending people by not being nice to them, worry about not adding to the insanity.

Inevitably, you are going to end up making some judgment calls that may be above your station and tasking people that you normally would have no authority over, on the understanding you’ll answer for it later on; so long as you make this clear at the time, any reasonable person should support you on this.

As the long hours and sleepless nights count up, remember that there is an end and life will return to normal once more. If public disclosure of your breach is required, know that it is a double-edged sword. You may well experience great catharsis in knowing that the truth is finally out there, but you must come to terms beforehand that the PR spin engine will be operating at full pace and you will be under a mountain of non-disclosure.

People working in information security generally tend to be self-reliant types and the idea of using support resources outside of your own network of friends and family may seem alien, even repellant to you, but if you work for a suitably sized organization they will have employee councilors, who can be bound to the same NDA’s you are. Don’t underestimate the value of having someone you can legally discuss things with.

Handling a corporate breach is likely to be one of the most intense moments of your security career; you wouldn’t be faulted for wondering if it’s time for a career change because of it. Remember however, that in the world of incident response, there are two types of people – those that have been through a major breach, and those that haven’t. Your employer will, in all likelihood, continue to remain in business and you will continue to remain employed. In this day and age, it is an accepted truth that all organizations will be breached at some point – what is important is how you handle it. Manage the stress, try not to say anything you can’t take back and realize that you are going to come out of this with experience that you can’t learn in any lab, or simulated exercise.