How to become an incident responder
Keatron Evans, Infosec instructor and managing consultant at KM Cyber Security, LLC, discusses the path you can take to become an incident responder.
Chris Sienko: Hello and welcome to the third installment of the InfoSec Institute Video of the Week series. Once again this week we are talking career paths and this week we were talking the career path of incident responder.
Today’s guest is Keatron Evans. He’s been one of our longest running and most revered teachers here at Infosec, and I wanted to tell you a little bit about him here. Keatron Evans is a senior instructor for both the Infosec Institute and the Intense School. He has 12 years’ experience and training and as an information security consultant, trainer, and world-renowned subject matter expert and he works to help businesses and government entities understand and prevent compromises to data, infrastructure, and information systems.
He has led several of the nation’s elite red teams. In addition to working with complex technology for more than 14 years, he regularly consults with or trains many federal intelligence and defense agencies on threats to digital defense systems and computer forensics. Please welcome Keatron Evans. Thanks for being here today.
Keatron Evans: All right, thanks Chris. Appreciate the opportunity.
Chris: Okay, so let me start by getting a bit of your background and your professional journey. When did you start getting interested in computer security?
Keatron: Well, you know, it’s kind of interesting. A lot of people that you see in security today, they kind of came out of college looking to get into security, but my journey was slightly different. You know, I started off as just a break fix PC technician way, way back in the day in rural Mississippi. And kind of graduated from that into networking because at the time there was, this certification called the Novell CNE that was going around. And that was really when certifications started to gain some traction there. And Novell used to be the networking operating system. So I got deep into that and, became a master Novell, CNEs and CNI, and you know, got into some Banyan VINES and stuff like that.
And around that same time Microsoft was coming out with NT4 and shortly after that, Active Directory. So I got heavy into that. So I was actually just a network engineer, moreso on the application and infrastructure side than anything else. And I remember going to a conference and seeing some guys talking about some MIT lab that was compromised and I was like, “Man, that sounds cool. I would like to see more about how that happened.”
So I reached out to those guys and they gave me kind of like a makeshift white paper on how the attack happened. I was like, this is so cool. I want to do this, I don’t want to do network engineering and architecture anymore. I want to do this. So I kind of started to take other network jobs that put me in a position to have the opportunity to delve into, sort of, security a little bit.
So my focus on jobs and career changed as to where I was looking for networking jobs because I knew that’s what I had experienced in, but I was looking for ones that kind of allowed me to be able to touch some security stuff. And that was kind of how I got into it.
And shortly after that, got my first job that allowed me to do that in Illinois in a town called Wheaton right outside of Chicago and met, actually, the founder of Infosec Institute, Jack Koziol. Because we kind of came up in the hacking underground a little bit together. We were kind of in some of the same circles doing some of the same things. So in 2000, I took my first Certified Ethical Hacker class from him, under him.
And I guess about six years later, I was in an article in the Wall Street Journal about how hackers compromise the enterprise and all that kind of thing. And it was funny because I was featured in the top of the article and then right under me they interviewed Jack. So we kind of like reconnect like, “Hey, what are you doing in my article? ” He’s like, “What are you doing in my article?”, you know?
So we reconnected and it’s kind of been history ever since then. You know, because the training with Infosec and the training with other organizations as well has it’s given me a lot of opportunities to meet a lot of people and do a lot of things and it just kind of opened the door. So, you know, for me, I would say there’s a lot of different steps that led me to getting into security. But training was kind of like the glue or the hub, that allowed me to meet the right people, interact with the right people, and be able to get into those roles.
Chris: Hmm. Now, I mean we’re kind of going a little ways back, especially for younger folks. Do you think these sort of milestones are still applicable? When you say you sort of started in networking and you kind of jumped sideways into security and stuff like that. Is that, do you think that’s still sort of a viable game plan?
Keatron: I do think it is. And you know, it’s interesting, you don’t hear a lot of people recommend that if you look at the average, “Hey, here’s your recommended path”, the first thing you usually see is like, Security+ or something like that. But you know, I still think it is viable because what I’m finding more and more in the industry is I find people that have a ton of security certs and you know, got all this experience quote unquote pentesting and things like that. But you know, when you look at their understanding of basic networking and how packets actually get from one place to the other, it really limits them in their ability to do things like incident response, network forensics, and even malware analysis. I mean, you have to have some good understanding of network protocols and stuff like that to reverse engineer malware.
So I’m finding that while they have those top-level skills to be able to do that kind of work, that it’s the foundational stuff that I’m seeing more and more lacking with people newer coming into the industry. So I still think it’s viable and the difference in today and back then is now, you can kind of do it in parallel, right?
So you can take a security job, but as a personal goal, you should still be kind of on the side, making sure your foundational skills, your networking skills, your pack analysis skills, you should be building that stuff up on your own, even if it’s not job related. So the difference now and then is now, it’s so easy to do it in parallel because there’s online training, you know, everywhere for everything. So there’s really no excuse not to kind of still get that foundation.
Chris: So there’s still a lot of benefit to be had in having a much more broad, sort of, rather than being hyper specialized that you should really know sort of, like, the ins and outs of the entire,
Keatron: Well, absolutely. Because when somebody calls you to do security work, it’s weird. They kind of expect you to know everything, right? Like they don’t look at you and say, “Well, we’re going to bring somebody else in that understands networking because we don’t expect you to understand that.” They want you to know everything.
It’s just like if your mom, has you to come and fix her computer, she doesn’t understand that you don’t know everything there is to know about Excel formulas. You’re a computer guy, so you know you, you need to fix everything. And we find in security, if you go into a place and you’re doing incident response and you say, “I don’t know how networks work, you need to get your network guy here.”, it’s not going to be a good look for you. So you know, if someone’s paying you a 300, $400 an hour a rate to come in and manage an incident, they expect you to, you know, know how a network works and know how to be able to do basic packet captures.
And I’ve done that. You know, I’ve went into places where I was like, “Hey I need packets from here and here.” And they’re like, “Oh, this got to take us a couple of hours to get the Sim team on it. And for them to set up this and set up that.” I’m like, “Cool, keep doing that.”
But in the meantime I’m going to put tcpdump up here, put tcpdump here, or Wireshark, or whatever I’m using at the time and I’m just going to do some basic analysis from this point to this point while we’re waiting for them to do what they need to do with logarithm or, or Splunk or whatever they’re using to get me to packets that I was asking for.
So, I mean, that kind of just shows experience and it also shows that you can still slowly move forward even when you’re waiting on the rest of the organization to do something.
And I think that’s probably really encouraging for people who are wanting to make what they see as maybe like kind of an unsurpassable jump. Maybe they’re in one profession that it seems, sort of too far away to get into security. So that’s great to hear.
Chris: So because I know many of our viewers today have a minimal IT or cybersecurity experience and might be looking at this profession for the first time, could you walk me a little bit through the day-to-day activities of an incident response expert? Like what kind of jobs do you do on a day to day basis?
Keatron: Yes. So first of all, it ranges from managing the incident. Like a lot of times, I’ve literally had customers tell me, “Look, we’ve got the team, they can handle it. We just want you to manage the team so that we can say that you’re on site. No, we just want to be able to say on paper that we got you in here and you’re helping us manage this incident.”
And then at other times we’ll have teams that are very mature policy and procedure wise. So they have a good manager, they have good procedures in place, they just don’t have the technical depth necessarily. So I’ll literally come in and being hands on the keyboard, you know, grabbing packet captures, doing memory dumps, trying to analyze malware and stuff like that. And then bringing in other people from my team to do the other parts that either I’m not as skilled in, or I would rather hand off to somebody else.
So day to day works like that. Today, for example, I’m working on a few incidents but I’m doing it, you know, here at home because a lot of it is just directing traffic and things like that. So that’s kind of what my day to day is. But at the same time I might get called out to Seattle next week and I’ll be out there for two weeks straight helping them manage a 24/7 incident responsive shift.
So it just ranges, and that’s kind of exciting, actually, if you think about it, because I get to travel to a lot of different places. Every incident is slightly different. I feel like I learn something every time, and I get to do it while making a ridiculous amount of money. So I think it’s, how could you ask for more, you know, it’s very exciting career to be in.
And I just want to touch back on, you know, you talk about people having a different career and trying to get into this. You know, what I can tell them, when I started off in security, I was managing the network for a small engineering firm. An engineering firm of like 15 engineers, one server and 15 workstations. That’s what I was doing not that long ago. Like, you know, 16 years ago or so. And my transition was literally me pulling up an Excel spreadsheet, writing out on that Excel spreadsheet, all the different security certifications that I wanted to pursue, and I would go back every time I’d get one, I check it off as done. And because I kind of had, in my mind, if I get all these skills, I should be able to get a security job somewhere.
So I wasn’t even really worried about the opportunity so much. I was just looking to build the skillset that, you know, basically when that opportunity came I would be ready to execute because I like that quote, “Luck is the meeting of preparation and opportunity.”
And what I like to tell even my students sometimes is, “Look, you don’t control the opportunity part, but the part that you control 100% is the preparation part. So overdo the preparation so that when the opportunity comes, you can actually take advantage of it.” Because what we ended up doing is we’ll trip ourselves up and think it’s too hard or it’s too much work. So when the opportunity finally presents itself, we haven’t even prepared enough to recognize that the opportunity’s there.
So I always stress over prepare and prepare yourself for moving into this field. And you know, you run into somebody like me, like half of my staff are people that I just met randomly, either in classes or at conferences or some other places and they just happen to have the skillset or they have the problem-solving skills that attracted me to that person.
And I’m telling you, my top pentester, her job or her degree is in liberal arts and she was doing a liberal arts-type profession when I met her and hired her as a pentester and it’s just because she was interested in computers and she had the natural curiosity, you know, that was her hobby is messing around with computers and being on, you know, fence servers and things like that. So when I interviewed her, she blew me away with, she didn’t know anything about pentesting, but when I do my technical interviews, part of what I do is I give you a couple of packet captures. I give you a laptop with no OS on it. A Kali DVD, you got to install Kali, you got to pentest past a firewall and get to these servers. Well she didn’t know how to do any of that, but within that two hour timeframe, she was very methodically figuring out how to do every single piece of it.
So while she didn’t get, as far as some of the guys that were seasoned, she got almost as far with no experience. So what that showed me is like, hey, if she’s got zero experience and she figured this out in two hours and this dude right here has got 10 years experience and he only got a little bit further than her in two hours, I’m going to go with her because her problem solving skills are off the charts. So you know, that’s kind of like one of my little secrets. I look at that even more than I do certifications and training is you know, what the problem solving approach is.
Chris: That’s a really good transition to my next question here because it sounds like problem solving sort of plays a big part in incident response. So I was going to ask what sort of activities or projects should be you be interested in or enjoy doing on a day-to-day basis if you’re thinking of incident response as a career or an area of study?
Keatron: Okay, so definitely interacting with people because one of the key things that I find that I have to kind of remediate when we come into organizations is the communications, right? Because they don’t necessarily know who needs to be communicated with when.
You have HR and PR and legal and all of these other people that need to be involved in an incident that you on the surface wouldn’t think about because we think about incident response as more of a technical thing and it, you know, a lot of it is, but there’s a whole soft and management side to it to where you have to communicate details of that incident to the right people at the right time and you have to make sure that you don’t communicate certain information, you know, kind of prematurely or ahead of time.
We can look back at some cases where, for example, when LinkedIn had a breach a few years ago, they immediately reported and said, “Hey, we had a breach, we lost 7 million records.” And then they had to come back a month or so later and say, “oh, we were wrong at 117 million records.” So it kind of looks like maybe you didn’t know what you were talking about the first time or maybe you released that information a little too soon. So you know, helping organizations understand how to navigate that is sometimes the biggest challenge.
You know the technical stuff. Either it is or it isn’t, you know, that’s what I love about the tech stuff, but it’s binary. Either this technique will work or it doesn’t and if it doesn’t, you move on to something else and try to solve the problem another way. But the people side can be more challenging sometimes. So I think you have to, if you want to be kind of a manager or do all things incident response, you have to be good at communicating with people and calming people down.
Because I, I tell people all the time, that’s one of my biggest roles is to come in and be the calm. Because a lot of these organizations, regardless of how big they are, they haven’t had a lot of experience going through major breaches. So when they have one and the media is involved, a lot of times the first day of my job is just calming everyone down and saying, “Look guys, okay, they’re in.” Like rushing and running around. And you know, acting like the world’s going to end, it’s not going to speed up this process. So they’re in, you know, we’re not going to get them out instantly. So let’s go to our methodical approach. What’s your policy and procedure and your playbook say? Let’s look at that and let’s see if we can operate within that, and if I find that it’s got too many flaws or it’s too limiting than I will, with approval, go outside that and maybe you can go back and adjust that procedural document once this incident’s handled.
So, you know, I just kind of slow everybody down and calm them down. You know, I’ll sit down and have coffee with the CSO and just to kind of let him see like, “Hey look, if this guy’s not freaking out then maybe we shouldn’t be freaking out either. You know, cause he’s obviously done this a bunch of times and he seems to be a-okay with all of this.”
So that’s a big part of it is to have that calming factor, that calming demeanor to not come in and act like the sky is falling. Because I’ve run into other incident responders that have the opposite approach where they come in and make it seem like, “Oh my God, everything’s going to be so bad if you don’t do this”
Chris: Come with me if you want to live!
Keatron: And that’s really not my what you want. Yeah, exactly. Yeah, that’s really not the approach you want to take. If you want to get called back then, another incident.
Chris: That’s, that’s really interesting that the two main things you’ve discussed with regards to being a good incident responder are sort of human interaction and problem solving. Like when you think of sort of computer tech jobs, they always say, well you have to, you know, be able to stare at a screen for a long time or you have to really like analyze data in a certain way. But it’s this, this sounds like sort of a very human centered job and also a very cause and effect procedural sort of role.
Keatron: Yep. And you know, not to take away from staring at the screen and digging into packet captures. Because my assumption is look, if you’re getting ready to come an incident response, you should already know you’re going to be doing that. You know, I’m just trying to kind of make sure the person understands the things that maybe is not talked about as much. But yep, absolutely. You know, sitting and staring at a screen for eight hours a day, we, each tend to do rotating eight hour shifts.
So if I’m managing an incident, I will immediately get that customer to approve of me bringing in two more people because we’re gonna you know, you can overwork yourself, especially if you’re looking at packet captures and looking at memory dumps and stuff like that. So we’ll do three, eight hour shifts. I’ll do eight hours and I’ll bring in another responder to manage it for, and I’m still working obviously, you know, outside of my eight hours, but it’s more of a slow roll and I’m just looking at packet analysis and I’m not having to coordinate everything. I’m doing more of a team role versus a managing the team role. So-
Chris: Yeah, you were saying before that you’re kind of on call 24 hours, but you’re not really like, it’s not like 24 where you’re up in the middle of the night, you know, doing-
Keatron: Oh, absolutely not. Absolutely not. I’ve worked too hard and too long to be still doing that. But you know, there’s, that happens sometimes, right? Like if I have to go to San Diego and I can’t get somebody there until Monday, like today’s Friday afternoon. So if I get called out to an incident tonight in San Diego, the chances of me getting someone else out there to assist me over the weekend is pretty slim. So I might have to do that 24 hour thing from now until Monday.
But you know, basically a part of my process is getting it into a manageable, digestible thing. And a large part of that is having access to the right people and resources to bring in to help you manage it. Heroics is not the thing that you want to integrate into an incident response process. That’s the opposite. It’s cool to look at. It’s fun to watch on TV, but in the real world, you want it to be something that can be managed and everything’s interchangeable. Even my role. If I do it right, it should be interchangeable. We should be able to put somebody else in there and do it.
Chris: Okay. Now you said before that obviously, your prime associate’s problem solving ability was more desirable and stuff, but can we also, since we’re talking here on behalf of Infosec Institute, let’s talk about certs a little bit. Are there any particular certs that you think are good to study to sort of put you down this path? Are there ones that are sort of more tied to incident response than others? And where would you start?
Keatron: Well sure! So there’s some specific incident response certifications. There’s the, the EC Council has a, you know, Certified Incident Handler certification. Sans obviously has a GCIH, they have an incident handler cert, and as well Carnegie Mellon, you know they have a cert specific for incident response.
And I recommend all three of those because even though a cert just a piece of paper, studying for it and preparing for it exposes you to the concepts of the things that you need to be able to master. Now, one of the things about it is having the cert doesn’t mean that you’re going to come out on the other side of that certification and be an excellent or an expert incident handler. But what it means is now you understand what it takes to do that.
So you should be able to, from that point, go forward and constantly and exponentially improve your skillset to where you can become an expert incident handler in a much shorter amount of time than if you didn’t go through that certification track process. It kind of organizes and it says, “Hey, here’s what you need to know. You know, some of this stuff you might already be an expert at. Some of it you’re not, but you need to be an expert in all these things.” And it kind of gives you a nice footprint or blueprint of where you need to go from there.
Chris: Okay. Now for people who have the incident response job or are considering it, is this something that you, it sounds like you sort of go to where the incidents are. Do people to get hired by a company to be the incident response person, or do you kind of work sort of contract or freelance going from company to company?
Keatron: So I do mostly contract work. That’s one of the services that my company provides, but there are absolutely incident handlers for larger organizations. They’ll have a designated incident response team and then leading that team will be an incident manager or an incident handler that’s responsible for running the day to day or just managing the strategy, you know, depending on how it’s structured.
So absolutely, there’s roles where that’s your job is to do that for one individual organization. And I do work with a lot of those people that have that job. You know, again, they have an incident and one of the key indicators that it’s a skilled incident handler is they realize right away which skills they have on their team and which skills they don’t, and they know when to get us involved. You know, if it’s a skill set that they’re looking for it if they don’t have.
So there’s absolutely roles where you’re, you’re stagnant in one job for one company, but the beauty of that role is it gives you the opportunity to learn from outside people like myself. When you bring me in to handle an incident, theoretically, the next time you have an incident, if it’s exactly the same, you should have to use me less than you did the first time because you should have learned some things from that time that I came in and handled that first one.
So that’s a good, I think being an incident handler for it, an organization is a good starting point because it gives you the opportunity to kind of slowly learn what it is you’re supposed to be doing.
Now going back to the certs just for a minute there, becoming a good incident handler to me, I believe and there’s other people out there that do this, that will tell you the same. I mastered ethical hacking and penetration testing and offensive stuff first and then shortly after that I got into forensics.
So for me, having mastery of those two topics definitely improves your ability to be an incident handler because if you have a reach, what’s one of the first things you’re looking at? You’re trying to figure out did a malicious threat actor get in and if they got in, what they do? And having mastered that skill set, you have a lot of insight into what a person would do once they break in, how they would go about doing it, what types of things to look for.
For example, recently I was working on an incident where the hackers get in using, a very old exploit call Mimikatz where they got into a box and then once they got into that box, they started to move horizontally inside the organization.
Now they got onto that box via the Mimikatz exploit, eventually getting something called the Meterpreter shell which is just a type of payload built into the Metasploit that allows you to remote back-door access. Well, once they got in they didn’t use Meterpreter other than just for that one machine. And there was another incident response firm that was there before. So they were like, “Oh yeah, they got on this machine, they took stuff off of it and that was it.”
And I was adamant that it, nope, you know, this was machine really had nothing on it, so they probably move horizontally. And it turned out that they were moving horizontally using remote desktop the same way an admin managing network internally. And the reason I knew to look for that is because if I broke in, that’s what I would do. I would try to blend in, you know, with normal network admin traffic as much as possible. And how do we manage networks? Our ATP remote desktop.
So just instinctively I knew that that’s probably what happened. Whereas the other firm’s technician that was there was strictly looking at it from a, “I just took my first Metasploit class. I know Meterpreter, I’m excited because I found Meterpreter use here.” And he was just caught up in that completely forgetting the primary focus of a threat actor is to blend in. You know, once they get inside. So that’s, that’s what I mean by if you mastered that skill first, then when you’re responding to an incident and trying to figure out what an attacker did, you have a little more visibility and instinct as to what probably happened. And you can probably get from point A to point B a little bit faster.
Chris: And that again speaks to the importance of kind of having as much information and as much experience from as many different aspects of the security and networking spectrum as you can. It seems like you come to better conclusions that way, or-
Keatron: Yeah, and you have more reference points. I like to look at it from a machine learning standpoint. You have more data to process and make decisions on. If you’ve never seen that attack or if you’ve never executed that attack, you don’t have as many data points to try to piece it together from is someone that’s seen that attack and executed at attack 50 times.
Chris: Sort of speaking of this sort of hypothetical, incident response person who didn’t have that background, what are some of the common mistakes that instant response aspirants make along the way and how can they avoid them?
Keatron: One of the biggest ones is when they come into an organization, they don’t examine the in-place incident response playbook and policies. They come in doing heroics like, “Hey, I’m the guy. You know, I’m world renown for this.” And they go on and just start doing things and saying, “This should be done this way, and this should be done that way.” And they find themselves very quickly, making an organization violate their own approved incident response procedure and policy documents.
So that’s one mistake I see new people make as they come in and want to project their knowledge versus figuring out what’s there already and then trying to work within that, you know, put your knowledge inside that framework and try to work it that way. That keeps everybody safe.
The other thing is not doing a good enough job of what I like to call discovery first. You know, when you come into an incident, even if it’s your environment, your first order of business should be just to do a basic discovery. What’s our egress and ingress points, how many devices do we have? Have we accounted for all the devices on the enterprise? Where’s all the critical data? Where’s the data that someone would be after? You have to have answers to all these questions.
And amazingly, even today, a lot of organizations just don’t have those answers. When you’re going to start asking those questions, it sometimes takes days to get answers back as to even which servers the critical data lie on. It’s a disturbing, scary thing that happens when you see that. But it’s still the common mistake that I see happen.
Chris: Okay. So I think we’ve made instant response sounded like a pretty exciting and interesting job, which it obviously is. So again, going back to people who are thinking, “boy that sounds like something I would rather be doing than what I’m doing now, but I don’t really know where to start”. Like what is one action that you could take in your current job today that would put you a little bit closer to incident response as, as a profession, Whether it’s the study, or,
Keatron: Yeah. If you have no money. You know, cause I like to give the free options as well, if you have zero money, you know immediately go out and start looking at the BackTrack forums and you know all the forums that are related to hacking. You know, start with that.
That’s to me, I look at packing and pentesting is like a foundation for incident response, forensics and all these other things. Because again that’s, that’s really, when you do incident response, it’s primarily what you’re investigating. When you do forensics, that’s primarily what you’re investigating nowadays when you go into an organization to handle an incident. So it would make sense to me that having a firm understanding of what the problem is that you’re trying to investigate, will definitely make you better at investigating that problem.
So I would say if nothing else, there’s all kinds of hacking tutorials and things like that. Go download Kali, get it running in a virtual environment and just start learning how to do things like run Nmap and find vulnerabilities and learn how to exploit a vulnerability. Don’t even bother with trying to find all the latest vulnerabilities.
Find one machine that you can pull down and build a VM from. Find one vulnerability and become an expert at exploiting that one vulnerability. Do it over and over and over again. Because the thing is, where people get confused is there’s always going to be new vulnerabilities and new exploits. So trying to say “I’m up to speed with that”, that doesn’t matter, when you’re coming into the field. What you want to do is get the process of how discovery, enumeration, vulnerability mapping and exploitation works, like what that looks like. You want to get that down to where it’s second nature to you because again, every attacker that comes into an organization, they generally go through that process. Even if it’s via social engineering, phishing or whatever the case may be, what does that look like?
And if I could just pick two things, I would say learn how to do a basic exploit against a server that’s like sequel injection or something like that. And then also learn how to do a client-side phishing type exploit. Learn what it looks like from a technical side to send someone an email, have them open that email, and as a result of them opening that email, you get control of their machine.
Make yourself learn how to do that process from beginning to end. And then when you learn it, do it over and over again until it’s second nature. Because the level of understanding you’ll get from that, you can’t parallel at any other way. So now when you go and investigate that thing happening in an organization, it becomes very, very transparent for you to see how the pieces got laid out, what happened, and what they were after.
Chris: Wow. That’s fantastic. So let’s look a little bit at the future of incident response as a profession, do you see your role and the way it’s performed changing in coming years based on up-and-coming technologies, new hacker strategies, increased threats?
Keatron: Well, absolutely. So a couple of really big changes that’s happened, I’d say the last two years. Okay. So first of all, everybody’s moving everything to the cloud. So the whole concept of physically going to site to examine a server, a physical server, those days are numbered. It’s becoming more and more, “Oh, we’ve got all of that stuff inside AWS and Microsoft Azure or Google Cloud.” So those servers are there, so coming to our premise doesn’t really do much for you because you know, this is how we access them. So there was a lot of that.
And then even from an incident response standpoint, I’ve probably had four or five incidents already just within the last 60 days where we’re using the service. There’s actually several companies offer it, and I don’t want to get into trying to brand anything here, but basically the way the service works is you call me up, Chris calls me up and says, “Hey, we just had an incident. We think the server was compromised. You know, we don’t want to shut it down because you told us that that might be a bad idea if it’s a hack because you’ll get rid of critical evidence. So what do we do?”
Well, I can send you an email and say, “Hey, take this link, go to that server, click on this link and install this agent in memory.” That agent gets installed. And then from that point I can send a command and it will actually start to take a forensics image of your memory and of your logical hard drive and pull that up to a cloud server that’s close to your physical location because an auto detects where you are and it’ll pull up the image to that server and I can start analyzing that image in minutes versus having to wait 24 hours for me to get on site and all this good stuff.
So as that image is being acquired, you know the memory dump happens first and we talk about malware and breaches, a lot of times, and more and more frequently, memory is becoming the primary source of evidence versus stuff on the hard drive because everybody knows how to cover their tracks and get stuff off the hard drive. So the memory dump happens first, and I can usually within an hour or so start analyzing that memory dump to look for sources of malware, sources of exfiltration and that type of thing and memory. And just five years ago that wasn’t really that common of a thing to do and now I’ve worked five cases in the last two or three months where that’s exactly what happened.
We’re able to say this happened, this is in memory, this is the data they were after, here’s encryption key that their exfiltration tool is using the encrypt the outbound data, which is why your IDS and your FireEye and all your appliances is that an alert to it is because it’s encrypted. Here’s a key. So if we can get that data from your Splunk or your logarithm or QRadar or whatever, we can encrypt it or decrypt it using this key that was used to encrypt that and we can tell you what happened.
So I mean all of that without ever leaving the seat and I’m sitting in right now just because of those services. And also hackers have for a while been wise enough to use AWS and cloud services to do their attacks. So there proxying through that and it’s interesting, there’s a conference that I did about, it was in 2012 or 13 for the Secret Service down at the Nashville or, excuse me, the East Tennessee Security Summit and it was based on Oak Ridge Nuclear Labs down there. And it’s a public article, a public presentation where if, you know, back in 2012, I’ll say, “Hey look, here’s what’s going to happen in the cloud. Attackers are going to use it their pivot point.”
Because everybody else’s presentation was about, “How do you protect your data in the cloud?” My presentation was, you know, let’s get past that. “How are you going to protect yourself when you’re being attacked from the cloud?” And sure enough, today that’s exactly what’s happening is the cloud has become a primary pivot point for threat actors to proxy between you and the attack point that they’re actually coming from. So those things have changed the way that we do forensics and incident response.
Chris: All right. Any final tips or words of encouragement for the incident response experts of tomorrow?
Keatron: Just don’t stop trying to figure it out. Like whatever it is that you’re trying to learn and master, just don’t stop trying to figure it out. If you go to a website and you ask a question and you get flamed, just keep going. If you spend an entire weekend trying to get this one command to work in that command doesn’t work, keep trying. Don’t give up is the main thing. Be tenacious and aggressive at trying to increase your knowledge base because those of us that are hiring, we’ll see that. We’ll pick up on that and you know, those are the people that we try to get out of the crowd.
Chris: Fantastic. Keatron Evans, thank you very much for being with us today. And for those of you watching, if you’d like to know more information about security and certifications, please visit www.Infosecinstitute.com and again, going back to the free option, if you’d like to read lots and lots and lots of security-related information, we have a daily-updated resources website. It’s resources.infosecinstitute.com. That also includes several interactive labs, capture the flag exercises, and other sorts of things that will bring you up to code. So again, thank you for watching and thank you, Keatron, for being here.
Keatron: Thank you.
Chris: Take care.
Weekly career advice
Learn how to break into cybersecurity, build new skills and move up the career ladder. Each week on the Cyber Work Podcast, host Chris Sienko sits down with thought leaders from Carbon Black, IBM, CompTIA and others to discuss the latest cybersecurity workforce trends.
Get the hands-on training you need to learn new cybersecurity skills and keep them relevant. Every other week on Cyber Work Applied, expert Infosec instructors and industry practitioners teach a new skill — and show you how that skill applies to real-world scenarios.
Q&As with industry pros
Have a question about your cybersecurity career? Join our special Cyber Work Live episodes for a Q&A with industry leaders. Get your career questions answered, connect with other industry professionals and take your career to the next level.