What 17 years as an infosec trainer have taught me
July 2016 shall see me complete 17 years in the infosec training circuit. It has been an amazing journey, with humble beginnings.
How it all started
I had a strong academic background in Computer Science – Operating Systems, TCP/IP and Cryptography. I was fortunate to work on my master’s degree under Eugene Spafford in the COAST lab (now CERIAS) at Purdue.
The late 90s witnessed a meteoric rise of what became known as Silicon Valley Bubble 1.0 – job offers everywhere. I ended up picking the most oddball job description (and the lowest paying of them all): “Member of the Attack and Penetration team.”
My first introduction to the larger world of information security outside academia was Black Hat and DEF CON 1999. Those were my early years as a professional penetration tester, pulling off exploits from Technotronic and Packetstorm, reading Phrack and Textfiles and popping rootshells on Solaris and Irix boxes.
But the fun was not destined to last. Firewalls killed all opportunities to own Solaris boxes over RPC buffer overflows, and I needed a new way of getting into my target networks. Rather than bypass what is blocked, focus on what is available – this was my approach when I started finding and exploiting weaknesses in web applications. I had to walk up to the front door called “HTTP” and jiggle the doorknob until it opened.
Infosec conference talks those days were full of buffer overflows and DLL injection and memory corruption attacks. There was no research on “web hacking” – even the term was yet to be coined.
In 2000, I was working on techniques to achieve total compromise of a target network simply by packaging attack vectors in HTTP. I wrote a research paper called One Way Web Hacking which formed the basis of web exploitation as we know it today – webshells, SQL shells over HTTP, web uploaders, and even tunneling arbitrary protocols such as RDP over HTTP proxies. I presented many talks on web hacking, starting with Black Hat 2000 and continuing on several other conferences around the world.
How I began security trainings
The company I was working for wanted to offer private trainings on web hacking. I wrote up the course syllabus and taught the first training in our offices in California in 2000.
I continued my independent research on web application security, developing the first HTTP fingerprinting tools, the first webshells, filter evasion and also came up with the first software WAF prototype. It was then that I decided to continue offering web hacking training at Black Hat, followed by Hack In The Box, and several other conferences around the world.
Training kept me challenged, as it brought a lot of curious minds together in a room for two full days. As I taught my students, I learned, too. The best ideas come to me when I am staring at the whiteboard trying to explain a concept to my students for the eleventy-first time. This is where new inspiration strikes, new opportunities unfold, new avenues open as I rethink age old infosec problems again and again.
In 2010, when I was teaching browser exploits, a student asked me: “How can you make browser attacks bypass malware inspection engines?” This question got me thinking very hard, and five years later, Stegosploit was born out of my passion for browser exploits and photography. Steganographically encoding a browser exploit in an image polyglot, i.e. a file that is a representation of two different data types, makes for some incredibly stealthy exploit delivery, and can be a visual treat as well, depending upon the chosen photograph.
In 2001, I was invited to keynote the Malaysian government’s IT security conference in Kuala Lumpur. I was to speak on my findings from the Honeynet Project (a very different topic than web hacking). It was then that I met up with SK Chong.
SK was a hacker specialising in Windows shellcode and binary level attacks. He had followed my research on one way web hacking, and we met up to discuss how one way techniques can be applied directly to shellcode. SK eventually went on to publish his technique in Phrack and we kept in touch regularly.
Binary exploitation, working directly with memory layouts, pointers, registers and assembly code, had always been my first love. I used to reverse engineer DOS viruses back in the 90s. I had come a long way teaching web hacking and it was time to go back to my binary hacking roots.
In 2006, SK and I decided to team up and conceptualised The Exploit Laboratory over drinks at the Telawi Street Bistro in Kuala Lumpur. To me and SK, this was a historic moment that we look back upon every year. TSB has long shuttered its doors, but The Exploit Laboratory continues into its 10th year in 2016!
The Exploit Laboratory has been a fantastic journey. Teaching along with SK helped us keep a fantastic pace and overhaul topics and introduce new examples rapidly. We had a very simple philosophy: we wanted to teach the latest and greatest, in a very simple manner. Our challenge was to bring rocket science down to earth, and so we did.
It was through The Exploit Lab that I learned one of the most fascinating concepts in offensive techniques – Return Oriented Programming. Over the years, we taught several advanced concepts in exploit development. We created three more classes as a continuation to the basic Exploit Lab class – a Red Team class, a Master class and a class on fuzzing and vulnerability discovery. And to keep up with the times, the 10th year of the Exploit Laboratory will see a brand new class on ARM exploit development.
With the weight of the Internet shifting from desktops to mobile and IoT platforms, ARM exploit development is going to be an essential offensive skill to be acquired. I already taught two iterations of the ARM Exploit Laboratory at CanSecWest and SyScan this year, and am looking forward to advancing ARM exploit development even more.
The ARM Exploit Lab reminds me of the early days when we just started the Exploit Lab classes. There were little or no tools for assisting with exploit development. Today the x86 exploit development world is full of mature tools and processes. ARM exploit development is still a new area with lots of opportunities to build tools and discover new techniques.
Saumil Shah during training at HITBSecConf
The challenges of infosec training
Infosec training demands a lot of background work: soaking up new research, improvising existing techniques, identifying new topics to be added to the course.
I pride myself on providing cutting edge topics with every class. The rate at which I add topics and rework the content ends up overhauling my entire course once every 6 months (on an average). I have been teaching for 17 years with more than 200 classes in my track record, and I have enough data points to back up my statistics.
My classes have followed a learn-by-doing pedagogy from the start. Today, hands-on training is the norm at infosec conferences. Students are expected to bring their laptops and work with a portable lab environment. In my early days, we used to rent laptops for our students to provide a consistent training environment, and I used to spend an entire day ghosting disk images onto laptop drives.
In 2003, I switched over to using virtual machines as hypervisor technology matured and became mainstream. But even with virtual machines, I spend more than half of my preparation time fine-tuning the images and ironing out the hands-on exercises.
One of the constant challenges of training is time. Two days started becoming an increasingly short time duration to start from the basics and progress up to the cutting edge of offensive techniques. New topics needed to be added very rapidly, yet the basics cannot be compromised.
After every class, I make it a point to revisit my notes and identify topics that could have been explained more efficiently. I have been extremely fortunate to have had a fantastic training coach – Mr. Udayan Shah – who also happens to be my father.
My father went back to college in 1982, taught himself programming, and eventually started teaching programming professionally in 1986. I used to observe how he prepared diligently for each class. His flowcharts, hand written notes, talking points, everything. It stayed with me.
My father and I were also members of a computer hobby club during 1990-93. It was there that I conducted several meetings and public workshops on various emerging topics in computing such as Windows 3.1, Slackware Linux 1.0 and how to recover from DOS viruses such as Dark Avenger.
I got to learn the finer points of delivering a high energy workshop from my father. Most importantly, he taught me how to “sing to the audience”. Everything mattered: the size of fonts used on the projection screen, high contrast text and background, legibility of on-screen demos from the very last row of students, the art of handling questions and answers and fostering discussions, the importance of demo rehearsal. And even after 17 years, if I fail to “pray to the demo gods”, I still fall flat on my face.
I have the good fortune to still be able to pick my father’s brain on teaching style every now and then, and he never fails to teach me a new trick or two!
My day job, and how it helps me to teach
Many people have asked if I teach for a living. I don’t. My day job involves running my company Net-Square, doing what we do best for the past 15 years – penetration testing and reverse engineering. Starting up and running a pen-test shop has enriched me with several real world scenarios which end up being modeled in hands-on exercises in my classes. I never use textbook or artificial examples.
Teaching for a living is a very different profession. It wouldn’t have allowed me to make frequent changes to my classes and keep them up to date at the pace at which I do. My day job provides the inputs, innovation and fresh new perspectives needed for my classes. For me, training is an intense workout. It is very taxing, yet very gratifying.
Infosec training and certification
Every discussion on training eventually brings up the unavoidable topic of certification. The entire IT industry is obsessed with certification. Here I shall quote the Saumil Shah theorem on IT certifications – “The value of a certification program is inversely proportional to the number of students certified annually,” and its corollary – “Mass manufactured certification is not even worth the paper it is printed on.”
We need to step back and understand the purpose of certification. Most certificates are given for participation in the training programme – they provide no insight into the capabilities of the student at the end of the training. A few certifications do conduct tests at the end of the training. These provide a statement of capabilities, but keep in mind that the statement is like a baseline – a lowest common denominator.
The problem is exacerbated when certification becomes the criteria for recruitment, business development and compliance. It then becomes a means to an end, and not a vehicle for gaining knowledge.
I personally fell for the CISSP certification hype back when it was really new. I passed my CISSP in 1999. The only thing I got out of it was a rectangle with my name printed on it along with the letters CISSP and a few signatures.
That having been said, I am increasingly leaning towards the concept of limited numbered certificates. This would provide a means of recognizing exceptional efforts and identify students who bring sincerity and a high level of proficiency to the table.
Saumil Shah during training at HITBSecConf in Malaysia
Infosec training DOs and DONTs
Although 2016 will be my seventeenth year teaching at Black Hat USA, in the past five years I have preferred teaching at smaller conferences. I like a focused conference crowd, and a sharp and active mix of students in my class.
Hack In The Box, SyScan, REcon, Cansecwest, 44CON – these have been some of my favourite conferences to teach at. These conferences are places I call “home” – familiar turf, warm and friendly crew members, compact class size and extended 3 and 4 day training sessions make for high energy training.
My ideal class size is 24 students. Again, empirical data from my past trainings have shown that a class size of 24 contains the right critical mass for meaningful classroom discussions and Q&A sessions, while still maintaining a very good student-to-instructor ratio.
Black Hat is on its way to become a training factory, with many classes now having over 100 students each. Our Black Hat training features a larger crew, with two teaching assistants to ensure that even a larger class runs smoothly. A class size beyond 50 just doesn’t work. The diversity in capabilities becomes too wide and I risk the class being held up for a few insistent stragglers. I’d rather stick with quality and depth over quantity at this point in my journey.
We have seen student groups undergo a transformation over the past decade and a half. These days, students seem more shy and reserved, but the greatest value of instructor-led training is derived from discussions and Q&A sessions in class. Sometimes we instructors have to work on uncorking the questions bottle.
Every now and then, we get a fantastic group with a critical mass of proactive students and the pace and energy picks up instantly! We love teaching a vocal crowd, and there are times when I will risk breaking out unrehearsed material and go way above and beyond the planned syllabus. At the end of the class, I have only my students to thank for bringing out an extended performance.
The other challenge we face is in managing expectations. It took a couple of years for us to figure out the gaps. We took great pains to ensure that our syllabus and learning objectives are very clearly communicated in the course description. For private infosec training, I like to have a conference call with the stakeholders to discuss the topics they want, and then work out the final syllabus after a couple of iterations.
Matching expectations is very critical, as it can make or break the class. We also started writing tutorials and exercises to help students prepare in advance for the classes. I have seen several proactive students take advantage of my free tutorials and exercises and come to the class loaded with questions and ready for action. As an instructor, I am delighted to see students armed and ready to go.
There are exceptions though. I’ll never forget when a student at the Black Hat Abu Dhabi infosec training rocked up with an iPad when I had clearly asked for a laptop running VMware as a prerequisite. He was pretty insistent that the iPad would suffice. At that point, I told him to install VMware for the iPad and when he was ready, I’d be glad to transfer the VMs over. He needed about 20GB free space for it. He vanished after the first coffee break.
My plans for the future
I intend to continue teaching. With a firm base in x86 exploit development, I am excited to dive deeper into the world of ARM Exploitation and continue maturing The ARM Exploit Laboratory over the next few years.
I have been writing tutorials to help students prepare core concepts for my classes. I continue to seek feedback from students for areas to improve upon. Last year, I published two hands-on challenges – Tinysploit and Tinysploit2. These act as a litmus test of preparation for students wishing to take the ultra-advanced Exploit Laboratory classes.
Many people have encouraged me to make my training available online. I still feel that there is no substitute for an in-class instructor led training. After all, I am the son of a teacher-man!