How to become an APT hunter with Carbon Black

Chris Sienko: Hello and welcome to another episode of Cyber Speak with InfoSec, the weekly podcast where industry thought leaders share their knowledge and experiences in order to help us all stay one step ahead of the bad guys. David Balcar has every security newcomer's dream job. He's a security strategist with Carbon Black with over 18 years in conducting security research, network penetration testing, incident response and computer forensics, hunting down advanced persistent threats and other high level attackers. He's a member of the High Technology Crime Investigation Association, a featured speaker at security conferences worldwide and is responsible for the pre-sales activity of the Carbon Black portfolio, which includes CB Response, CB Protection, CB Defense, CB Threat Hunter, CB Live Ops, and CB Threat site. And today we're going to talk about his security journey and a little bit about some of the most exciting or unusual APTs out there. David, thank you very much for your time today.

David Balcar: Oh, thanks for having me.

Chris: So let's... Since we're talking about your career journey in cybersecurity today, let's start at the very beginning. How did you first get started in computers and later into security specifically?

David: Yeah. I actually started when I was about 10, 11 years old back on the Apple II. And then Apple II, IIe timeframe. I actually still have my first Apple IIe. It still runs. Cool. I got into programming. I was doing Pascal and assembly language. Then after high school and stuff, I did all that stuff, and I then I went into the Navy for a few years. Six years actually. And then once I got out, I actually went to work for a good friend of mine who owned a computer repair shop. So I was actually fixing Macs, PCs, building them. That gave me that great foundation. Then I got really got into the networking piece of it, did a lot of Novell stuff, did some of the Microsoft for work groups, did all this stuff in the very beginning.

It was really, really cool stuff. And then I just started working for an engineering firm doing that for them. Building large scale Novell networks and whatnot. And then I just got to a point where headhunters were calling that kind of stuff. People want you to move from place to place and I just didn't want to be a number. So I actually started my own company for 16 years called NDI and I had two sides. One was a security business, we did nothing but firewalls, configurations, investigations. Then, the other side was your traditional integration partner, building server farms and switches and all that kind of stuff.

Chris: What was the interest in jumping over to sort of security over networking or other things?

David: It was just seeing the viruses and stuff, like the Michelangelo and stuff way back in the day. It was pretty funny getting a call saying, "Hey, what's going on with my computer?" And really customers want it... The internet was coming online at that point and people really wanted to communicate and I was like, God, I know there's bad guys out there. I just found it too easy that I could, after networks were built, it was so easy just to jump right in. Everyone was using public IP addresses. There was no natting, no firewalls and even people that had firewalls, they were still not doing natting. So it was easy to get around and do stuff. And then I started seeing, people stealing stuff, mainly from employees, ex-employees going back into their former employee taking documents and what so. So that really got me interested in the whole security aspect, and really for the last 18 plus years we've really gone down that path.

Chris: Well this sort of ties into that, but in your opinion, how has the cybersecurity landscape changed since you first got involved? Whether procedurally or directionally or [inaudible 00:03:46]...

David: Yeah, so it's changed a lot. Like I said, throw up Michelangelo, I know there's going to be a few people out there laughing at that one, but it really went from people just doing boot sector stuff to locking your screen, showing you a funny picture or whatever to all the way to cyber espionage and cyber theft and then to stealing money. I mean, that's really where the money is, right? You want to go steal it for the banks and the companies, ATM fraud, Swift transfers, all that kind of stuff. So that aspect of it, it's really, really come a long way.

And it's the same old moniker, right? Where people go for the money. If that's what they're going to go, they just do it in different ways. So they're not robbing a bank. There's still a few out there. But you know... I love doing pin testing on banks because I always ask them where's the money? And they go, Oh, we got this awesome vault in the back and everything. And you're like, no. It's that computer that's sitting there right there on the teller line and I just walked out of your building with. Guess what? I'm putting malware on it. I'm going to bring it back the next day so I could do all my transfers. So yeah, I mean the landscape has definitely changed a lot.

Chris: Let's start with how you got to where you are. Now like I said at the top of the show, security strategist for Carbon Black is a lot of security folks' dream job. So what are some of the job titles and responsibilities along the way that gave you the tool belt you needed to excel in this position? You talked a little bit about some of the other jobs you did, but sort of let's sort of concretize what each one sort of gave you that got you to where you are now?

David: Yeah, I mean starting from the beginning, really the programming piece. Knowing how computers work, knowing what's going on, the process of what's going on, the Ram, what's going up and down the network. I live in the ones and zeros, so I want to live... One of my best tools is like stuff like sniffer Wireshark type stuff. And I want to know everything that's happening, going on. Why is that happening? Why is it doing this? That plays a lot into my job today to where I can dissect something and go, "Hey, this doesn't look right." Well, why doesn't it look right? Well, I know because of everything I've done in the past. So you know, I've done stuff like a network architect, network security engineer, those kinds of things. So you know how these networks are put together. So you can visualize, pretty quickly say, Hey. There's a hole here. There's ingress egress points. Whether it's modems or where it's, T3 lines or fiber coming into the building. It doesn't matter. You know where all this stuff plays.

And that really goes a long way to saying, okay [gollee 00:06:22], I can put my thinking cap on for a second and go okay. I know you've got two ingress and egress points. What are we doing there? Are we doing VPN or how are we terminating these connections? And now with SAS and cloud and hybrid clouds and private clouds and... The data is just everywhere. And that's a huge piece of it. Knowing where that data and who's got access to it.

Chris: One of the regular series we do here on Cyber Speak is sort of helping people who might feel like they're never going to sort of get into their dream job. Sort of see the steps along the way. So for someone who is say, feeling stuck in a lower level security job, like at a help desk or some other place where they don't feel like they can break out, what is one action that they can take today? Whether read up on a new skill, sign up for a course, that can get them one step closer to high level APD on it.

David: I would say the biggest thing is training. You got to get the training. Some people say, Oh, you've got to have this cert, this cert, this cert. Not necessarily. There's plenty of companies out there that will actually hire you without a certification. My thing is learn from the ground up. Know how this stuff works. If you don't know how it works, everything else is a moot point because I can't teach you how to reverse malware if you don't know some programming or some assembly and how a [PE 00:07:45] execution is happening on a windows box or whatnot. That's very, very important.

So I always tell people, okay, get a network certification. Then get your basic security stuff so you can get this. So you know what crypto is and you know what asymmetric versus symmetric... You know all these different things. Just add to your tool belt cause you're going to use every single piece of it right now. And I'd say the biggest one beside all that is knowing the OS's. Don't go into a company going, "Oh well, I know windows." Well guess what? They've got Linux, they've got OSX, they've got CP. Whatever they've got, AS400. If you don't know the OS's then you're not doing yourself justice. You really got to know the OS's.

Chris: So for those just coming to the topic for the first time, we're talking about advanced persistent threats today. What exactly are APTs and how does hunting APTs differ from standard threat hunting?

David: I wouldn't say it really differs a lot from your standard threat hunting. The biggest thing with APTs is generally their nation states or highly sophisticated cyber gangs. That's where really where we put the term APT into it, right? These are going to be wanting to be on your network for a long time, not be discovered and change their tactics constantly. But they've always got to have a way in and a way out. Cause that's the thing, right? Once they get in, they want to get data out or they want to be able to send commands back in. That's a big thing.

The biggest thing as far as the threat hunting piece really is going to be around knowing what's on the endpoint, what's your standard, what's normal, and being able to look at unfiltered data. That means every process that's run, every registry, every cross proc that's running every net con. You've got to know all this stuff and be able to watch it. Right? And then... So if you're hunting, you're looking for those things. Well, what exactly could you possibly be looking for? Well you look for stuff that's out of the ordinary? You know, notepad.exe should not be communicating with the network. This shouldn't happen. Your accounting application should not be talking to China or it should not be talking to Iceland or wherever they're VPN'd in from so that, that's really big.

What else can I say? Also on the threat hunting side, I would say, if I go into an organization and they said, "Hey Dave, we think we've been attacked." I start at some real basic stuff. I start at DNS and work my way down. I'm looking for stuff that's out of the ordinary, stuff that I know is going to known command and control, that kind of stuff and just work my way back up until I can get to that either patient zero or not. Because once they get to patient zero, we start doing [LotL's 00:10:37], which is living off the land. So they're going to look like a normal user or they're going to look like the admin user. And that makes it extremely difficult. I know this one particular case I was working on, this attacker got in and basically only used malware on one machine.

From there they stole credentials, and then used the built in tools to windows to go laterally across the network. And I actually caught them looking, doing traces of RDP, and I knew that no one in the company RDPs except for the administrators. So I knew what three machines they use and I just could search the network looking for, okay is RDP ever touched any of these other machines? And sure enough I found it and you could just see them, I could just follow them with a timeline of all the machines they compromised using RDP, just stolen credentials. But the admin said, "Oh no, everything looks great." Man.

Chris: So, I mean it sounds to me like if you want to get to the top here, you sort of learn the skills and you learn... you get the toolbox by doing lots of different types of sort of searching and analyzing on sort of lower stakes threats and stuff like that. So there's... So what would you recommend for security professionals who want to move into the realm of APT hunting and analysis? Is there a particular combination of skill sets? Training [crosstalk 00:12:01]...

David: Yeah, I would classify it into three pieces, right? Number one, like I said before, know your OS, windows, Linux, OSX. That's your base. Then look at... because they all have their different challenges, right? Different way of executing code, different memory usage, file system access, all that kind of stuff. Look at something like GIAC right? Because they offer classes around pin testing, ICS, cyber defense, all these different things where you can get certifications on this stuff.

And my third and probably one of my most important is set up your own lab. Set up your own lab at home to play and be able to get samples and detonate malware. So you can see, "Oh now I see what it's doing." Now I don't just read it., I've actually understand why it's doing this to memory. I know why this is doing a buffer overflow or under run and I can see it physically. I'm a pretty visual person so I'm not great at... I can read some texts, but boy, let me put my hands on it and that will make a whole lot more sense to me.

Chris: Yeah, you sort of wreck it to build it. So what, if any, are some downsides to the kind of work you do since your job is sort of a dream job for a lot of folks? What are some of the sort of... It's 2:00 AM and I can't believe I'm still dealing with this nonsense aspects of the job that people should know about as well?

David: Yeah, I mean, I guess my biggest thing is when I was doing a lot of pin testing and IR stuff, yeah we would have the late nights working on a case or something. But what keeps me up at night is just how fast the malware's changing, how fast these tools are coming out. And me personally, every day I'm thinking of a new way of breaking into a system because if I can think of a way to break into it, someone else can too. But then I think it from a defense. So how do I stop myself from doing that and put that all into context. So I literally will probably wake up at three or four in the morning and go back down to my lab and says, okay. I just got this great idea. Does this work? Oh no, it doesn't work. Crap. Start over again. That's one of the downsides too. Cause you lose sleep over that and...

Chris: Yeah, sort of an obsession about it.

David: Yeah. So I mean it's definitely OCD I guess around that, but it's...

Chris: Plus you kind of feel like you have the weight of the world on your shoulders here because you're feel like you're sort of watching bombs go off all around you and yeah...

David: Yeah. I mean I don't like any customer to go down. So if I hear of something, if I get an email during the day or a text or something, say, Hey Dave, we're having this kind of problem, what can we do? Then I'm just I'm on it 24 seven cause I want to figure out why it's happening and be able to eradicate that, get it out of their network so they can get back to business.

Chris: Give us some examples of some of the biggest and scariest APS out there right now. Like who are their targets and what types of security measures are being put in place to try and combat them?

David: Sure. This is a great answer I'm going to give you here. So some of the biggest ones are the ones we don't know about yet. Right? That's the key with APTs` for sure. But you know, if you look at some of the big ones right now, if you look at them, a story that was published on motherboard just recently and all over the place was the supply chain attack against [Asus 00:00:15:11], their live updates software. So it affected, I don't know, a ton of million machines. How ever many people downloaded that new firmware... not firmware, but updates for their Asus live software. Right? That's crazy. The supply chain is the Holy Grail because if I can get that, you're going to trust everything walking through your door. If you look at, and you can definitely Google this stuff, look up Lenovo hard-coded password for their fingerprint reader.

Are you kidding me? So I don't even need a fingerprint. I can just type in a hard-coded password. That's crazy. Or like the keyboard scraper that was on HP laptops last year, from the factory. You know, I was asked when I give speeches and stuff about security, I said okay. How many people have HPs or whatever. Not to pick on them. I mean it could apply to anybody, but how many are actually re-imaging their machine clean when they get it? You get maybe 20%, maybe 15% of the crowd that says, yeah, I'm doing that. The rest of them go, no, I just take whatever's coming at me. Those are big.

Some of the other big ones, I guess mainly the financial. My specialty is financial and insurance, I would say around like Lazarus and Finn, seven slash carbon hoc. However, whichever name you want to give them this week. And it's really about, they're going after the money. They're going after Swift transfers. They're going after ATM fraud, creating accounts, depleting those accounts. That's really big. You know, if you look at what the Carbonak gang did, they stole close to a billion dollars and they were very persistent. They stayed in these networks for a long time and stealing money multiple ways, that's for sure.

Chris: You noted in our conversation before this, that there are nation state attack tools out there in the wild and because of that attributions are getting harder and harder to identify. Can you speak more about this? What are some of the targets of these attack tools and why are they so hard to identify?

David: Yeah, I mean if you, if you go back to like 2016, 2017 we've got like the shadow brokers, we've got a vault seven, vault eight leaks, that kind of stuff. Nation state tools. Now if you look at Wanna Cry, let's take the famous one, like Wanna Cry. That actually used three pieces of the nation state tools that were released. It affected like 200,000 machines in like a 24 hour period or something. It was crazy, all over the world. That stuff's getting out there. So if these nation state tools, and if someone could just go download them, which people are, they're using them against us, right? Those tools were used for a nation state to do something, whether it was cyber espionage or whatnot. The problem with those sophisticated tools is now the cyber gangs have them. Your script kitties have them, mom and pop has them.

Oh, what's this do? Let's click on this. That makes it very difficult. So far as attributions, I'm not a big proponent of attribution's cause unless I can get the guy behind the keyboard, which actually in March of last year in Spain, they actually arrested the leader of the Carbonak gang, at the time. So that was pretty cool. But you know, how do you prove that it's who? I mean we look for TTPs, the techniques, tactics and procedures that an attack looks like. If I can look at all the... they did this then this. I'm looking at fingerprints, but there's also a lot of false flags.

If you look at what happened in Seoul last year in the Olympic destroyer attack, there was a bunch of false flags in that. There was pointing to their neighbors. It was pointing to China. It was pointing to everybody to throw off analysts like myself when we're doing the research on it and reverse engineering it. They want to throw us... Go down a rabbit hole and you've got to be very, very vigilant on raising up the... some companies will just go, Whoa, yep, it's Iran or Iraq or it's whoever. It's North Korea, but how do you know? How do you really know?

When you look at the country itself with North Korea, it's very, very small, very small internet footprint. But that's not where their cyber gangs are hanging out. They're living in Singapore, they're living in Hong Kong, they're living someplace else. They're using VPN, they're doing all kinds of stuff. So that's why it's getting... It's very hard to identify these cyber gangs for sure.

Chris: Yeah, no-

David: Go ahead.

Chris: Excuse me. You know, we're talking about these sort of nation state level attacks and so forth and you said that they need to get into what they do and get out. On your end, are you aiming just to sort of like close the breach to get them out of there to minimize damage or do you go on the offensive and really try to sort of like shut these gangs down or is that sort of out of your purview?

David: Yeah, that's definitely out of my purview. I mean there's laws against a lot of the reverse hacking, so to speak. I know that's been brought up before here in the U S and a few other in the EU and stuff is like giving companies permission to hack back basically. But the problem with that is, if you're good and a lot of these cyber gangs are really good, they're going to attack one company. Then they're going to bounce off of that company too. So when you're actually trying to hack back, you're not hacking the hackers or the cyber criminals... Sorry, used the wrong word. They're attacking another company, so that makes it very difficult. So I'm not a big proponent of that. So my main thing is block everything you can and detect everything else and remediate as fast as possible. You've got to stop the bleeding and the biggest thing is know your IP.

I can't tell you how many companies I've gone into and you ask them, "Oh, what's your IP? What are you trying to protect?" Oh, nothing. We're just a manufacturer or such. I was like, okay. Do you have CAD drawings of this stuff? Do you have any patents on any of this stuff you make? And they're like, "Oh, well yeah." Okay. You still need to protect this stuff. You have certain manufacturing processes that you do. There is IP and some people just don't see it. And where is the data located? Is it encrypted? I love to encrypt everything, everything, everything.

If I get a virus or a piece of malware that's doing memory scraping, yeah, it's going to be un-encrypted in memory. But how can we stop it before they can get to it? And don't allow access for everybody. Not everyone needs the crown jewels. Right? They don't need to access. The people in HR don't need access to your manufacturing methods, so they get breached. It's very contained. That's a big point.

Chris: Back at the beginning of March, Carbon Black coauthored a report with Optiv Security called Modern Bank Heists, the Bank Robbery Shifts to Cyberspace. Among the surprising results were a marked increase of destructive attacks, which are "launched to be punitive by destroying data." The findings that 67% of surveyed financial organizations have reported an increase of cyber attacks over the past 12 months. 79% said cyber criminals have become more sophisticated and 26% reported being targeted by these destructive attacks. So what's to be done with all of these findings? What will financial institutions have to do not to get completely chewed up by these increasingly ferocious cyber criminals?

David: Isolate. You've got to segment your stuff out. You've got to isolate again, know who, who, what, when, where. When people are accessing data, where they're accessing it from. You've got to have those access controls around that stuff because if I get into one machine, I'm going to try to move laterally and that's huge. If you're not watching that or whether it's a SIM watching that or you've got logs. I'm real big on you've got to log this stuff, you've got to have it and be able to correlate it. Don't just have logs dumped into SIS log. You've got to have it into some type of SIM and don't say, well, I'm a small, medium business. I can't afford that. There's lots of free solutions out there. There's lots of low cost solutions that you can do and there are certain things you can do to mitigate your risk. Right? Make yourself a harder target. Have technology and people and processes in place with good training. That's an important part. That are looking for this stuff. Don't wait for something to happen. Be a threat hunter. Actually start looking through your network, trying to find stuff.

Chris: So what types of corporations, organizations or government outlets are looking for and hiring people with APT hunting experience?

David: Well, for sure all the three letter agencies are looking for people, right? As well as my favorites are going to be pin testing firms, IR firms, [MSSPs 00:23:56] are all looking for those kind of skills. And I would ask for an internship. So if I'm just starting out, man, I might be working help desk somewhere or maybe I'm doing something else, ask for an internship. Hey, can I shadow somebody for two weeks? Something. You'd be surprised how many of these companies will allow that to happen because they know you're getting free experience, but they know that hey, we could probably put that person later on. They're going to know it. So that's really good. And look for those open positions. I mean, get on your favorite job site of choice and find those for sure and apply. Even if you think that you can't do it, get in there, get used to doing those kinds of interviews. Because some of them are very technical, but even if you don't do great ask why you didn't do great. Oh, okay. That's what I need. That's the skill that I'm looking for.

Chris: Right. And yeah, I mean also a lot of those... We've talked to a number of people about the cybersecurity skills gap and a lot of those job postings are sort of looking for unicorn candidates that have 15 certs and 25 years experience and what have you. And so...

David: Yeah, don't believe the hype. Apply. Even if it doesn't say that. I've seen those too. They crack me up. And those generally not written by the security team, they're all written by HR. Oh what do we need? Oh this is what we're doing. I love the job postings where they says, Oh well you need to be a semantic expert and you need to be this and you need to be that. It's like, they just... Bad ops sec now you've told me everything in your environment. Wait until that you get people to come in and interview and then start talking, Hey, what are we actually doing? So you can do some vetting there from the employer perspective.

Chris: So if you work for an organization that could be the target of an APT, what aspects of your security program should you be strengthening? Or if you're not on the rung of the ladder to make those decisions, should you be telling your C suite that they need to invest in?

David: You know, don't go... You got to get the low hanging fruit. You know, for sure start there. But I would talk to... Before you start going to C-suite, because a lot of them are just not going to understand or they're super busy doing 50 other things, is get with your IT department, get with your security team saying, hey, I just saw this. I was playing with it at home. I think this is going to affect us. I noticed that other companies in my industry are also being breached by this particular attack, what can we do? And you'd be amazing how receptive security teams and IT teams are looking at that going, oh wow, I've got an employee here. And then ask for that internship again. You know, come back in and say, hey, I'm on the IT team, but I want to join the security team and this why. Oh cause I'm doing this, this or this.

But yeah, you've got to push it up. You've got to make it relevant for the C-suite to do something with. Make it based... a lot of people say, Oh, ROI, ROI. But what happens if we're shut down? What happens? Do a quick survey in your company. Figure out, you know what? What if we're shut down for 24 hours, what do we do? How much is that going to cost? What about a week? What about two weeks? C-suites are thinking, they're going to look at that going, hey, somebody did the research, looked at our company from the inside and saying, hey, we've got to do something with this and give some really good bullet points. Don't say, oh we need $1 million for security. Actually gives some pointed direction. Say I think we need VPN access to get rid of this threat because our web applications are not very secure. That kind of stuff.

Chris: So as we wrap up here, if people want to know more info about David Balcar or Carbon Black where can they go?

David: I could make a joke about just look on the internet, but no I won't. You can always get on Twitter. I'm at network two three two. You can go to CarbonBlack.com to learn more about what we do and like I said, catch me in a security conference. I speak at 50 plus a year around the world. So I would definitely encourage that and I'd also, not just me, but get involved in the community if that was my biggest takeaway from today is get involved in the community, get involved. I actually just got back from speaking at, BSides Vancouver. I'm going to do FSI SAC in a couple of weeks. Those are community events, especially BSides. Really encourage you. Most of them are free to get in for students. They're also very cheap, but they have some really good speakers and you can get a lot of knowledge. They have CTFs, they've got lock picking, they got IOT villages and stuff. It's really, really good. Look up for meetups. Go to like meetup.com and search under security in your area and then start going to those. Then you can start networking and then you see, oh, this is the other stuff that's happening and then you pick up all kinds of great knowledge.

Chris: David Balcar, thanks for joining us today. We could talk for hours. This is fascinating, but I really appreciate your time.

David: Thanks.

Chris: Thank you all for listening and watching. If you enjoyed today's video, you can find many more on our YouTube page. Just go to YouTube and type in Cyber Speak with InfoSec to check out our collection of tutorials, interviews, and past webinars. If you'd rather have us in your ears during your workday, all of our videos are also available as audio podcasts. Just search Cyber Speak with InfoSec in your favorite podcast catcher. To see curtain promotional offers available for podcast listeners and to learn more about our InfoSec pro life boot camps, InfoSec skills on demand training library and InfoSec IQ security awareness and training platform, go to InfoSec institute.com/podcast or click the link in the description. Thanks once again to David Balcar and thank you all for watching and listening. We'll speak to you next week.