Cyber threat hunting: Identify and hunt down intruders

Chris Sienko: Hello, and welcome to today's episode of the CyberSpeak with Infosec Institute podcast. This is an audio rebroadcast of a recent webinar we hosted entitled Cyber Threat Hunting, Identify and Hunt Down Intruders. Our guest speaker today, once again, is InfoSec instructor, Jeremy Martin. Over the course of this 40 minute webinar, Jeremy will discuss what it takes to be a modern day threat hunter. At the end of this podcast, you'll have a better understanding of how the defensive blue team operations can work in tandem with the offensive red team strategies to create a stronger security posture than ever before.

Among the topics discussed in this webinar are the job duties of a threat hunting professional, frameworks and strategies for cyber threat hunting, how to get started, and progress, your defensive security career, and we also have time to answer questions from live viewers. Just as a reminder, if you'd also like to see this webinar as it unfolds, including presentation slides, you can find this podcast on our YouTube page by searching InfoSec Institute at YouTube and visiting our channel. As a special opportunity for podcast listeners, you can receive up to $500 worth of ethical hacking toys with class signups. Just visit infosecinstitute.com/hacking-toys to learn more. Without further ado, here along with moderator Camille DuPuis is Infosec's own Jeremy Martin.

Camille DuPuis: Jeremy Martin is a senior security researcher, and he has focused his profession around red team penetration testing, computer forensics, open source intelligence, and cyber warfare. So he started his career in 1995, and he has worked with fortune 200 companies and federal government agencies, receiving a number of awards for his service. Jeremy currently provides training, and works with several governmental incident response and computer forensics departments.

Outside of consulting, he is an instructor here for InfoSec Institute. He is a security researcher. He's a published author, and he speaks at security conferences around the world. Mr. Martin's current research projects include vulnerability analysis, threat profiling, exploitation automation, anti-forensics, open source intelligence gathering, and reverse engineering malware.

He's also a member of several organizations, including the Information Security and Assurance Community organizations. He also volunteers for local ISSA and ACFEI chapters. With that, I'll go ahead and turn it over to you to get us started on what blue team is, and really what we're going to be talking about today.

Jeremy Martin: Good morning. She mentioned a little bit about red team, the attackers. Blue team primarily is trying to find threats. So with the blue team, it does require quite a few different subject matter expert types. So I know there's a term out there, Jack of many trades. It is a good idea to try to be that yourself as well, definitely know some of the basics on networking. And I know that there are things like deep learning, machine learning is starting to become a big hit here two which does help with the analyzation of data. Yeah, absolutely. Primary issue here is trying to find the bad guys, and that does get a little bit more challenging, especially if they're more advanced. So again, that's where, especially if you can even bring things like a Torch and TensorFlow, those are AI items into the mix that can help find things that people may be missing.

With that being said, I guess biggest thing that, especially with the threat hang community, is risk management. So I know here they're talking about a shift in mindset. There's always going to be residual risk. It all comes down to trying to find that special point to where risk is acceptable level. So there's a lot of issues out there over the years that people would think that they're 100% secure, where they're air gapped. And a lot of times they may be, majority of times they're not. I've seen so many scenarios, for example, there're sites out there like Showdown, Showdown is a great demonstration of people that think they're air gapped and they're not. So going back to here, there's a lot of vulnerabilities that get found, and the vendors don't get notified right away. And the people that are compromised, they don't know it. And so even though it's almost a year to identify a breach, it's just that one key indicator, or indicator of compromise, that one of the analysts might see. What's this? They investigate a little bit further and then identify what the potential issue is.

There's a one project that I was working on, this is a while ago, and unfortunately the organization worked with, did not either have the funding or they didn't understand the potential benefit of it, but we did what we called the scanner darkly. And a colleague of mine basically took a bunch of NetFlow data and did historical analysis over a year period of time. And he was great. Basically brilliant a programmer as well, network engineer, and he wrote this application to look at all the traffic, and what we’re looking for was basically a heartbeat. And we saw consistent traffic, and this was a cool instant because every five minutes there was a port connection to what looked like a random port from what looked like a random IP address. And there were five of those every five minutes. And then every three days, one specific computer would then verify a connection, One of those ports that were found open. It would connect only the ports that came back with a SYN, SYN-ACK, ACK handshake.

So in that instance that attacker, or the person scanning, took nine months to scan a class C. so none of the security tools caught this information, because it was way beneath the radar. And what we were able to surmise based off that was there were probably using a botnet to do the initial scanning, and then one specific system did the verification scan. So taking that into mind, the bigger the company you are, the bigger the organization, the bigger the attack fund for you're probably going to have. The more you're dealing with government, money, politics, more people are going to be attacking you. So it's definitely a good idea to assume that you've probably already been compromised.

So what you're looking for is again, indicators of possibly top talkers. If it's an advanced crew, they're not going to be top talkers, but you'd be looking for strange data. So you always need defense in depth, constant monitoring and the more tools you have, unfortunately sometimes it becomes a little bit more convoluted. But if the tools are set up very well, let's say if you have SEMs, a street event management tools, or event correlation tools, so you can start correlated contents, sometimes that helps people to. But the only downside of that is some people start to lose the ability to see basically what's happening in the weeds.

Camille: Now, Jeremy, question. I know a lot of times we see organizations taking a long time to identify that there's been a breach, or what seems like a long time to people maybe not in the industry or not aware of what these breaches look like. But why do you think that a lot of companies take 197 days? Do you think that's because of the team members that don't know what they're looking for, or do you have any thought on that?

Jeremy: Oh, absolutely. There's a lot of variables there. I do know that a lot of the organizations I've had opportunity to work with may not have had the budget to hire enough people to do the proper monitoring. They didn't have the training to know what to look for. So we're organizations just... Trying to be nice here, but they may not care as much until it happens to them. For example, I'm not sure if anybody here remembers Nortel, but after they had some financial issues, a third party came in and then basically asked the question, why is there so much traffic going to Beijing? And they didn't have any business need for that traffic, and come to find out that they've been compromised for over 10 years and they never knew it until they went out of business. And then the third party came in to look at the data.

And so sometimes you're looking at content which may look normal, and this is where behavioral based and machine learning fails, because if you already quote unquote trust the traffic, and trust that environment, that's going to benchmark what's already there. So that's sometimes how they get a missed. But then other times it could be, it's just so slow it looks like normal traffic, that even a seasoned professional that looks for that on daily basis misses it because it looks normal.

Camille: Sure. Definitely true with the thought that hackers and bad people online are getting quite a bit smarter, and realizing what these people are looking for and then doing the opposite of that.

Jeremy: Oh, absolutely. And that ties in with here, the quality of the data's huge. There's a term that a lot of people use garbage in, garbage out, and if you don't have a truly good idea of what the network is supposed to be, then that drastically minimizes the ability to catch it. I know we did talk about tools a little bit. The primary thing comes down to, again, knowing the data. For example, I know the term ICS, IOT and SCADA get thrown around quite a bit, and they should. ICS, industrial control systems, and SCADA, those are interesting environments because they're very controlled. And so an environment like that, it should be a lot easier to catch something that is outside of ordinary. Rather than an enterprise network, when unfortunately you have so many people, their traffic is just going to be... almost seems random in some cases.

For example, let's say football game, or some sports event or something actually happens on the news, chances are there's going to be more traffic. People looking it up through Google, watching video, which may or may not be corporate policy issue, but just the randomization of regular normal traffic does add some challenge. And if it's a closed environment, that makes things a lot easier. Assuming that you have the resources and tools to monitor for changes.

 

So as far as the hunter, you're looking at... Think of this as more of an analyst role, rather than any sort of engineer role. So you're looking for a pattern, and trying to prove this, prove a pattern. So think of it like call homes. I know that a lot of the APTs over the years have been using DNS traffic for exfiltrating information or infiltrating their malware, things in that area. For example, there's a commercial tool out there, if anybody's ever heard of it or used it, called Cobalt Strike. Cobalt Strike, it's basically the commercial version of Armitage, which used to be an interface into some other tools including Metasploit. But with a Cobalt Strike, one of the things it does is it has a mimicking capability for APTs. One is actual DNS traffic, so not just using port 53, but using actual DNS queries to get information in and out, or using HTTPS. So basically the hunter, you're trying to identify what shouldn't be there. One of these things is not like the other. So, that's a huge skill that needs to be basically built in, so that's the analyst skill.

Camille: So before we move on to this simple threat hunting process, so Jeremy will go over this with us. Just want to remind everyone that, feel free to ask us questions in the panel here. You don't always get the chance to ask your questions to an expert like Jeremy. So feel free to pass those on to us.

Jeremy: One way to look at this too, especially if anybody here has any forensic background. There's instant response, computer forensics, but it all comes down to collecting the data. The other thing though is you need some sort of scope, what to look for. So we were talking before about trying to identify anything that doesn't fit in. At that point, again, you're looking at it from a forensic standpoint to prove or disprove a theory. Why doesn't it fit in? Where did it come from? So that's where you establish hypothesis and then you, during the hunt processes, you're trying to prove it. So prove or disprove that theory. Once you're able to identify that it is or is not an actual risk or threat, that's when you go to the actual response phase.

But going back to the identification, that's where it all comes down to, you might have to put stuff in a lab to try to reproduce certain types of traffic. And this is also where, if you're able to identify a specific, let's say process, calling back to a site that you shouldn't be calling back to, you might be able to pull that process out in the response side, put it into a virtual environment and then see if it's acting the same way. So just think of the threat hunting process as a forensics mindset.

When creating the hypothesis, experience helps a lot. So if you've seen certain types of activity before, you can absolutely use that as part of what you believe is actually, or could be, happening. So I know here we're seeing many results in alerts and log entries can be prioritized. I know with SIMs, usually just a dashboard of a bunch of different events. A lot of people will call those top talkers, so they'll basically prioritize certain events that are either the most of those events, or the most dangerous critical, those type of events, which could be something like a port scan where it scans usually pretty low on the totem pole. But if that port scan or the IP address that scanning is seen anywhere further on the network, then that might jump up quite a bit.

Jeremy: But the thing is you actually... If you're monitoring, you have to have somebody looking at the logs. This is also where, unfortunately, a lot of organizations do not have enough resources. They fail because they would have, let's say, a team dedicated to intrusion detection monitoring, but then the team members get tasked for other things too. So they may not be able to monitor 24/7, or even at that aspect, they don't have the team to actually work 24 hours a day. So it might only be an eight to five job.

Camille: So, Jeremy, I'll interrupt you real quick. Talking about that team. So this is obviously one of the most important teams in an organization that has, that collects a lot of data. But besides these cybersecurity and IT skills that we're talking about, what else have you seen that really makes a successful threat hunting team, or blue team, in an organization?

Jeremy: Oh, that's a great question. And to be quite honest, communication. One of the biggest things that a lot of very good teams I've seen in the past have had a difficulty with would be able to talk about what they've found into layman's terms for management, so management can actually make a good decision based off of the findings. So report writing, being able to communicate in a meeting, or however. And also, an educator. So if management doesn't see that as a threat, educate them to the threat. But then also know when to... I don't know other ways to say it, but back off. So if management decides to accept the risk after you've let them know about the risk, then they've accepted it, so then go to the next risk.

Camille: Sure. Thank you. That's helpful I think for people looking to join this team, and thinking about not just the job skills, but also the personality and behavior skills that you want to think about as well.

Jeremy: That can definitely make things a little bit easier for you.

Camille: So next, let's move on to, what are we hunting for on the blue team?

Jeremy: Some of the biggest types of threats you're dealing with, of course the bad guys. Worst case there, an advanced persistent threat. I guess even worst-case scenario after that would be internal employees. So the easy way to think about it is a disgruntled employee can be worse than any other threat, based off of they could have the same skill sets as the worst outsider. They also have physical and internal access, and they have a trust level.

So things which you would be looking for, like your indicator compromise. What kind of things would have red flag? So something's already happened, and you're trying to identify certain things. For example, if you have white listing capability for all your applications or certain data, and if a certain program runs on a system that shouldn't be run on, that in itself be a red flag, or indicator of compromise. Things like indicator of attacks. Again, here understanding the attacks and progress, such as, how they're communicating. Is it through a file share, or is it through a HTTPS, encrypted HTTP? Would somebody be potentially a breaking an air gap?

Worst case scenario, you might have it to where you might be in an organization that is actually air gapped, but you're still exfiltrating data. And I've seen some very interesting research and actual tools in the past were a CCTV would be vulnerable to an IR camera, so where you can basically infiltrate or upsell trait data based off their CCTV system. Or one was a DNA sequencer, somebody actually put malware into the DNA sequencer, and then had it read and then exploit the system, which then dropped a RAT onto it. So going back to things that... Trying to think about how the attacker could possibly do it. If it's air gaps are not a hundred percent safe or secure, but those are the types of things you're looking for, is something that's outside the pattern, and things that are trusted.

You were mentioning network-based artifacts going into bad domains. That's a very common one that a lot of people will use. They'll look for blacklisted domain names. But if you're, for example, you've banned FTP, if FTP traffic's being seen on the network, that should also definitely red flag. The primary thing you're looking for is anything that would... is going outside the bounds of what you trust me environment. And with host based, registry keys are huge. A lot of malware will try to bind themselves, because that's an easy way to get it... When the system reboots it can resurrect to the application. If it's a memory resident, that's another potential issue.

But what's interesting about most malware, yes there's a lot of memory resident malware out there, but a lot of malware, or most malware, will try to run, especially if it's a targeting a windows box, because they know these things are going to reboot. And unless they have something in place to re exploit that system as soon as reboots, there's probably going to be some sort of artifact. Or even if it is memory resident only, the page file dot sys might have remnants of the malware. So that's basically what you're looking for is anything that, again, goes out outside the bounds of the normal pattern.

Camille: So, interesting question that came through the chat. Do you see hunters specializing in different areas? So maybe on a large team, would there be people on the team looking just for network-based artifacts, or looking just for host based artifacts? Do people specialize in certain things, or do you see more of a general practice on that?

Jeremy: Absolutely. Specialization is very common, so it's good to know a little bit about what your colleagues are doing, and especially in case they get stuck on another case or if for whatever reason they leave the organization. So there's still some cross, I guess, training cross skillset, but absolutely specialization is actually a good thing to have to an extent. Because network, there might be somebody that's extremely good at network, but they may not know some of the tools for doing data carving on a hard drive. So yes, specialization is good. It's always good to know everything, but you can't know everything.

Camille: Sure.

Jeremy: I'm assuming a lot of people here have probably heard of the CMM, capability maturity model. This is basically taken from that, which is basically a threat hunting maturity model, and where you are as an organization based off of how... or the capability that you have to identify threats.

So I know here they have zero initial, relies on automated alerting, so you have little to no routine data collection. And if you're not collecting data, it makes things more difficult. So with one, minimal, you have some indicator searches, but you also have moderate or high level of routine data collection. So things even like NetFlow sFlow, that's just basically the metadata of the network traffic. So you're looking at ports times, basically the communications between two systems without the payload, But at least you can correlate some of the content. Procedural [inaudible 00:22:40] analysis procedures created by others, high and very high level of routine data collection. So this is where, again, server logs... It becomes a lot more consistent. And then three, very high level, but creates new analysis procedures.

So this is where it gets really interesting, going back to things like the traditional policies, standards and procedures. This is where you would have not only timed changes, but you'd also have event driven changes. So when you're creating your own, creating your analysis, and you see something strange, your team may create a new IDS signature to look for certain types of traffic. If the system's online, that's, let's say beaconing back home, or it looks... or tripped a buffer overflow, then you would put that into a quarantine. So you're constantly changing things to try to identify threats.

And then four, leading, automates the majority of the data analysis procedures. So same basic principle, where might also have things like more intelligent IPS is in line, but it's constantly managed. That's the trick, is not only you're creating new procedures, you're testing and validating them, but this is where SIMs also come in. But primary thing is the testing and validation, and it's constant in level four.

Camille: So with that, right before we get to some more of the questions that we've got, just wanted to talk briefly about InfoSec Institute's Cyber Threat Hunting course. So in our course, what you'll do is you'll learn to identify, hunt down and analyze these cyber threats that we went over today. And it will focus a lot on that maturity model that Jeremy just talked us through. So you'll be learning how to measure your organization's threat hunting capabilities and finding solutions on how to better that. You'll also build an effective threat hunting solution based on open source tools, and you'll have the opportunity to take the exam to become a certified cyber threat hunting professional. Of course with that certification you can prove to employers and recruiters that you have the skills needed to do this job.

One cool thing we have going on, and I know, Jeremy, you have some of these ethical hacking toys, and we will be giving away some of these with course enrollments until the end of the year. Jeremy, could you tell us briefly what some of these are?

Jeremy: So a couple of ones that I do like quite a bit. So you have your Tetra, which is that wireless access point to mid icon, that is a very good utility to basically put in a line to test the security of your wireless and also the security of your clients, because it'll broadcast any access points, the clients that are connected to and have trusted, and then pretend to be them automatically. So you can do all this stuff on your own, but this automates the process and saves a lot of time. And then of course the USB thumb drives, those actually are not thumb drives. Those are going to be rubber duckies. So they look like a thumb drive, but they act as a generic keyboard. So this bypasses most of the USB security out there. There's a lot of good utilities on there.

Camille: Sure. Some cool stuff to play with to get some hands on experience with the job. So moving on from those, let's take some of the questions that we got. So starting out, I think I have an interesting question here. So since this is a newer role, blue team threat hunting hasn't been around for all that long, if you'd agree to that, Jeremy. But who do you see transition into this role? What kind of job titles have you seen move into this role, and with what certifications, or what background, have you seen successful transitions with?

Jeremy: I've seen people with, for example, an Intel background, which are good analysts, pen testers, people that have, for example, a basic cert team capability or forensics capability move into this. And it is relatively new, but realistically think of it as more of an active incident response team, rather than your traditional reactive. But as far as size of companies, definitely larger companies are trying to get into this because they're trying to minimize their risk. Medium sized companies, I have seen not as much based off of budget. And they're usually reactive if they have any security capability at all.

Camille: So when do you think would be the right time for a company to transition into a little bit more of a mature cyber threat hunting? Is there a size of the company or does it depend on what data your company has? What do you think? What type of companies should be really concerned with this?

Jeremy: If they care about their security, any of those companies, primarily because you're going from reactive to proactive and that makes 100% of the difference. If you actually care about getting compromised, I would, as a company, I would push this route. Saves money in the long run.

Camille: Right, to be proactive instead of reactive is definitely a safer bet I'd say. We've got a question from AJ, if someone wants to move to security from maybe a support type role, where would they start?

Jeremy: So some of the basic areas I would definitely start, knowing networking is huge. So I know some of the certifications do help depending on where you're at. A degree is another way to go. But yeah, certifications like Net+, Security+, and then target what's that security group needs? That's the biggest thing. I know a lot of people are saying automatically go to CEH or CHFI or a specific route, but if you have a group in mind, see what they're the weakest in or what their biggest need is, focus on that, because you're trying to make the team better too. And then you're far more likely to get what you want out of it as well.

Camille: Sure. Are there any particular skills that you'd say would be a good place to focus on if the goal is to become a threat hunter in that security team?

Jeremy: Yeah, identify anomalies. If you can look at something and identify what's not like the rest, that definitely helps quite bit. And that's more of a, I guess skill you have to play with or practice on. But outside that, personally I've always liked the vulnerability identification, exploitation, being able to do a forensic analysis. So I guess depending on what you want to focus on later on, go down that route. But primarily keeping your eye on new threats. So if that means attending conferences like DEF CON or Black Hat, those are also very good. You're always trying to keep ahead of the game. So identify changes, and then also keep your eye out for what's on the horizon.

Camille: Question through the chat here. So this person has an organization with less than 50 staff. What would be your recommended chain of command in terms of security? So they have CISO, ISM, et cetera listed, but who would you think would be the main person that should be in charge of this in a small organization?

Jeremy: So in a small company, usually you're looking at the CISO, and the CSO, or CISO, whatever their title is going to be, should talk directly to the CEO and not the CIO, because there's a conflict of interest if you're dealing directly with the CIO or chief technology officer. Because their goal is to keep everything up and running online as cheap as possible within budget. And your goal is to fix things, make sure that confidentiality, integrity and availability is intact. So absolutely, CISO would be the best one in the small company. If it's an extremely large company, then the CISO would have different groups. And then you'd have your ISM. But yeah, CISO.

Camille: Sure. Thank you. I think that'll help answer your question, Bernard. So looking at the overall career, because we talked about a lot of ways to get into that career, do you think threat hunting is a growing career compared to other cybersecurity roles? So this person is looking to transition into a role with more long-term job security. So do you think threat hunting would be a safe bet to go?

Jeremy: Yes. And then learning a little bit more of what you like better. For example, reverse engineering is technically part of the threat hunting, that's just a lot more specialized. But yes, easiest way to think about threat hunting I guess would be an incident response team that tries to catch things as they're happening, and before they happen, rather than just react. So yeah, it's definitely going to be huge, especially with all the litigation happening now, and leaks.

Camille: Another question here is, what dev or scripting language or tools are useful for a security role?

Jeremy: Anything that's common in the environment. For example, if you're on Windows, PowerShell is actually extremely powerful. It was probably not a good idea for Microsoft to release that to the public. So I know a lot of attackers are using that, if it's on the system. There's Python. I know with, especially with Linux and Windows, you might have to add the dependencies on the system. Windows, you have PowerShell, Python, Ruby. Basically whatever is already on the environment, and or whatever works best with specifically what you're trying to do. It's going to be variable there.

Camille: Just depending on what the organization primarily uses as well, I'm sure. So another question, we have time for just a couple more here. So how do you catch traffic on the OT networks for detecting anomalies or a specific... Or excuse me, suspicious traffic?

Jeremy: So when doing the traffic analysis, basically what you're dealing with is you'd have to have some sort of aggregators. For example, in a lot of networks, they are capturing NetFlow, sFlow data, so basically the med information of all the traffic. You need at least that. Most organizations cannot do full packet capture, either due to legal issues or just storage-based, because that's extremely large traffic and expensive. You'd have to put aggregators throughout the network, and then look for anomalies.

Camille: So, we'll do just one or two more here. Thank you to everyone who's submitting questions for us. So somebody asking, in an educational organization without a security team that is attempting to run vulnerability tests in order to stop common malware, what would your advice be as a proper course of action on tools? They currently have used only Kali Linux.

Jeremy: So, tools. I guess there are a lot of good open source tools out there. I would definitely suggest looking into things like either Snort or Security Onion with that capability to monitor. You also have Open Boss, which you can absolutely put on Kali, that's a decent phone or ability scanner, but I would definitely suggest on putting out some either Snort or Security Onion sensors throughout the network.

Camille: Let's just get to... Let's try and get to two more here, and then we will let Jeremy go and wrap up the presentation here. Jeremy, do you have any experience working with anomaly detection algorithms when analyzing logs or network traffic? That pcaps that we were talking about, I believe. And how would you recommend getting a start on machine learning for threat hunting? Any experience there that you could share?

Jeremy: Yeah. So I'm working with a colleague right now, he actually is a data scientist, so that's when he went to college when he loves it. We were using a TensorFlow as basically the algorithm that we're messing around with to try to identify not only potential risks and threats from attackers, but also potential vulnerabilities, so just based off of traffic going back and forth. But at this point a lot of it is going to be, I don't want to say necessarily homegrown, because a lot of the vendors are pushing this right now, but it's not as good as it probably should be, or it could be, sorry, with as much effort has been put into it by the security community. TensorFlow and Torch are some pretty good-

Camille: Perfect. And last question here before we wrap up. I know you said that scripting is a little bit different for each organization, but do you have to know scripting to get into this profession?

Jeremy: No, but it makes life a lot easier, especially if you're trying to automate things. Full-blown programming helps you better, but you definitely don't need to know scripting, especially depending on what you're going to want to get out of it. But if you did want to go into the deep learning side, some things like that, you might actually have to pick up a language.

Camille: Sure, to be a little bit more successful. Sounds good. Jeremy, just wanted to thank you for joining us today. And I want to thank all the audience. We had some great questions and some great participation. I hope this was useful and informative to everyone.

Chris: This concludes today's episode of Cyber Speak with InfoSec Institute. Thank you all for listening. Remember, if you enjoyed today's episode, you can find many more, including webinars, tutorials, and interviews with security thought leaders by visiting infosecinstitute.com/cyberspeak for the full list of episodes. Also, if you'd like to try our free security IQ package, which includes phishing simulators you can use to fake phish and then educate your colleagues and friends in the ways of security awareness, visit infosecinstitute.com/securityiq. Thanks once again to our guests, Jeremy Martin, and thank you all again for listening. We'll speak to you next week.