What does a digital forensic investigator do in the government?
Digital forensics professional Ondrej Krehel talks about the work of digital forensics in federal and government locations, the things he learned during a months-long attempt at decrypting a well-secured Swiss bank file and why finishing the research beats any degree you could ever have.
– Get your FREE cybersecurity training resources: https://www.infosecinstitute.com/free
– View Cyber Work Podcast transcripts and additional episodes: https://www.infosecinstitute.com/podcast
- 0:00 - Intro
- 2:11 - Ondrej's cybersecurity journal
- 5:33 - Career stepping stones
- 9:55 - The Swiss job
- 16:02 - Chasing the learning and experience
- 20:01 - Digital forensics on a government and federal scale
- 28:07 - Forensics collaboration on a case
- 30:46 - Favorite work stories
- 31:33 - How to improve infrastructure security
- 36:01 - Skills needed to enter digital forensics in government
- 41:31 - Unheard activities of digital forensics
- 43:48 - Where do I get work experience?
- 47:05 - Tips for digital forensic job hunters
- 52:19 - Work with LIFARS
- 57:50 - Outro
[00:00:00] Chris Sienko: Today on Cyber Work, digital forensics professional, Ondrej Krehel, is our special guest. Ondrej talks about the work of digital forensics in federal and government locations, the things he learned during a multi-month attempt at decrypting a well-secured Swiss Bank file, and why finishing the research beats any degree you could ever have. That's all today on Cyber Work.
Also, let's talk about Cyber Work Applied, a new series from Cyber Work. Tune in as expert infosec instructors and industry practitioners teach you a new cyber security skill and then show you how that skill applies to real-world scenarios. You'll learn how to carry out a variety of cyber attacks, practice using common cyber security tools, engage with walkthroughs that explain how major breaches occurred and more. And believe it or not, it's all free. Go to infosecinstitute.com/learn or check out the link in the description and get started with hands-on training in a fun environment while keeping the cyber security skills you have relevant. That's infosecinstitute.com/learn.
And now, let's begin the show.
[00:01:06] CS: Welcome to this week's episode of the Cyber Work with Infosec podcast. Each week we talk with a different industry thought leader about cyber security trends, the way those trends affect the work of infosec professionals and offer tips for breaking in or moving up the ladder in the cyber security industry. Ondrej Krehel is a digital forensics and cyber security professional. His background includes time with special cyber operations, cyber warfare and offensive missions as well as being a court expert witness. His forensic investigation matters have received attention from Forbes, CNN, NBC, BBC, ABC, Reuters, The Wall Street Journal, and The New York Times. As you can see from this lengthy list, Ondrej has a deep background in digital forensics and ethical hacking, which are two topics that you the listeners have told us time and again that you love to listen to. Ondrej is going to tell us about the time he spent as a guest lecturer at the FBI Training Academy, a bit about the current state of digital forensics as a federal government context, as well as giving us some info about how that realm differs from similar work done in for-profit or private companies.
Ondrej, welcome to Cyber Work.
[00:02:09] Ondrej Krehel: Thank you, Chris, for having me today.
[00:02:12] CS: Since you have such an unusual and distinguished career and background, I wanted to start by asking you about your cyber security journey. How did you first get interested in security and computers and tech and what made you specifically want to get into forensics and incident response?
[00:02:28] OK: Chris, I really got into it because I have your style of a haircut.
[00:02:32] CS: Oh yeah?
[00:02:34] OK: I'm just joking.
[00:02:34] CS: Yeah, it comes with a territory, doesn't it?
[00:02:36] OK: It does. It does. Watching your videos a little bit different just for the listeners. I truly got into it where I was 19 years old and something very similar to your intelligence agency if you offer me a job. I was one of the individuals who managed a forum, in a Linux forum. I was a really big Linux guy at the time. I definitely didn't like anyone else. I’m not going to name, but I wasn't in favor of any other operating system. Very religious at the time, very religious Linux-based, likes the old tricks, like the lacks VMS systems. I really like that type of forum. And I was pretty good at it. Like really doing a lot of stuff with the Unix and I was helping other people to be good in the Unix. I was also a mathematical physics major. I was very interested in cryptography and a space, like in the mathematical space. At that time encryption was really run through most of a function. So like even like RSA is a function with the prime numbers and you have a ZN rings and then the numbers that basically are around these prime numbers, and it was really about the math that you had to create. They often call them factors. You have to break the space basically of the code and inject your own code either through initial vectors or some other means and try to break the algorithms by another function. I truly enjoy all of that being in Linux, being math, doing mathematical physics, doing some crypto and basically they came and they told me that I would probably make what my parents do make. I grew up in a family of a political dissident. My father was a politically prosecuted individual. So it was very appealing to me. I did not tell my father for two years what I worked for because of course we hated government. As a family that came from a background of persecution, right? Wasn’t that really —
[00:04:30] CS: Yeah, that’s some uncomfortable dinner table conversation I imagine.
[00:04:34] OK: Yeah. Yeah. That conversation would not go very well. Especially working for intelligence forces, they used to interrogate my father. But of course it was the curtain was – The iron curtain was behind it right. People didn't really change as much I would say very really care for it. That's how I got really into it and I feel very privileged that a very young age become a special forces of potential agent. So if you never became a sporting agent for the country, it took around like seven year process. Spent the four years working at the government. The facility I was deposed was the energy. At the time Europe had the program where they oversaw the nuclear power plant. So at a 20 years old basis like cover most of the shifts on the weekends and also one day during a week plus I attended all the trainings. So I became like a very invisible kid at a school.
[00:05:33] CS: Yeah. That leads nicely into what I want to talk about next which is some of your major stepping stones. It sounds like you had a lot of really unusual sort of jobs along the way and it's not necessarily like a linear path from where you are now to where you started. So what were some of the moments where a new job or a project or an educational path or just an aha moment helped you to grow into your next opportunity, your next step in terms of being a digital forensics person in this context?
[00:06:03] OK: I early realized that while I want to profile myself in ethical hacking and then looking at the reflection of that hacking. It's almost like in physics electromagnetic and you have something called electromagnetic shadow or a system what they call the entropy, like a state of unknown, and then you try to figure out like in that algorithm what would be the state of known and how you can maybe decipher that unknown into something. Almost like a predict something from that unknown. Almost like a prediction of the weather in it. And what were mostly stepping stone for me were through the dedication of the depth that I tried to acknowledge in everything that I was learning or working on every project just to go and continue deeper in that project itself and always master at as much I could the topic or area I was really interested in.
[00:06:59] CS: Okay. So whatever you were doing you want to be the very best at that thing you were doing at that moment.
[00:07:03] OK: Exactly, and gain as much knowledge. I truly believe the knowledge is power. Health is wealth, and that motto always pushed me forward. Get as much knowledge about what you're doing. Try to see what the best people that are actually best in this what they're doing. How they're approaching. How they're solving that issue. And from some of the notable cases – And I guess because you build that mindset over that dedication that you don't give up. You're always trying. You're always trying to do. And you could call it hustle. You can call being persistent. You can call that – You have a discipline of finishing. There many different acronyms. It was a special forces or intelligence committee that would assign to this. But if you build that persona that what matters to you is the task and you believe the derivation. The success really comes that you're doing this very well, then derivation is recognition of society. Then recognition is getting paid. Then recognition is to be something else. But if you have a belief that gaining the knowledge creates a duration of that success that you're seeking from recognition in society to monetary gain is something that I early acknowledge as I had as my own strategy to work on those tasks.
I end up working for example on Mr. Madoff here in the city on his personal stuff. And one of the reasons why I was maybe picked was exactly that. I was known individual who had that mental fitness and stamina, mental physical and knowledge in industry that I would be the one who can go. And if you ask me to go I'll just kind of go and do that. Another large case that probably very notable was a UBS investigation into privacy laws by Department of Justice. I was part of that whole twister that happened between the UBS and Department of Justice, and I was in Zurich for half a year right and working on a breaking the code, the crypto code, for Swiss privacy laws. And from two, three firms that came in and said it's non-doable, actually became doable. And we found a way to attack this and I come up with a way to attack this in the memory of the computer. It was very slow. I’m going tell you it was superfast. It was very slow. It has something called like almost like an enigma rotating hybrid cipher. But we were able to do that calculation in um in a memory and then basically attack and break the code. And then Swiss realized – The Swiss Banks realized that Americans not going to give up. They're going to break the code and they're going to get into the bottom of the documents. They give up. They sort of persistence and they realize that that's what it is. But that sometimes takes that level of being engaged and push it forward.
[00:09:56] CS: Yeah, those are some really cool highlights, Bernie Madoff, and this Swiss Bank hack here. And I like what you said that it's a sort of a methodical thing. Was that specifically something that you realized when you started it that the only way we were going to do this is it was going to have to be a sort of a slow methodical attack rather than like a quick brute force or something like that?
[00:10:18] OK: No. I was there for two three months and everything was failing literally. I'm glad, Chris, I you know I didn't subsidize that depression with any form of compensation outside what people sometimes do. But I was truly in the first two months in depression. I mean, almost everything that I knew of failed. Nothing was really working. And then I'm just looking at this and doing all these classical methods and all kinds of classical attacks. And then one of the forensic strategies occupied to me like, “Why don't we just dump the memory of the computer and start like a really popular memory?” I mean, this is a long-term shot but, what if this is going to work? What if that's the way we going to figure out and maybe we can read exactly how each buffer is allocated when the encryption happens. And maybe we can create our own algorithm how to decipher. Maybe we can take something out of the memory and then basically decrypt this document on the fly. And that actually did work.
And I've seen the same replication of the strategy in WannaCry. One of the tools for WannaCry actually replicate the same model where if you take a memory of the computer you actually get the prime numbers and calculations for the encryption key and you can decrypt the data. I kind of thought that's a call. 15 years later the same trig actually does work.
[00:11:37] CS: Yeah. Was that a completely new technique when you did it? I mean, did you have a sense of like we've cracked this in a completely different way?
[00:11:46] OK: Definitely. Definitely. I haven't seen anyone at that time. Well, I'm sure some other intelligence agencies were doing this type of work, right?
[00:11:53] CS: Okay —
[00:11:55] OK: Yeah. Yeah. But both of them is like something you would advertise, and we didn't advertise either. I didn't write a paper after that, after we did it, that this is the way this is schema. We basically keep it almost like a harsh. It's almost business confidential property at that point of a time that, “Hey, look if you have this enigma rotating ciphers and can't really attack them because static data is very hard to decrypt that.” There's one way to do that is basically do the memory based attacks.
[00:12:20] CS: How much of this was coming to this technique and this idea was collaborative? Because it means it sounds like you were sort of bouncing ideas off with other people in your team here. Was this something where – Because I want people to sort of know that maybe you will have that great spark of an idea, but I think a lot of times it comes down to having lots of people sort of thinking the same problem and at the same time like that. Was this something where it was it was sort of like you came to the idea collectively?
[00:12:52] OK: So I was leading a team of two more individuals that was exactly helping with the project, and they bounced on his idea too. I came up with the idea to take the snapshot of the memory and then one individual schematic idea that well it's good if we take a snapshot of memory, but why don't we take multiple snapshots when basically you slide a card, encryption card that we were provided, and then basically let's see what happens. Like a debug also the memory and also create a snapshot of that.
You often have – I would say that team effort really comes and says, “Well, it was my idea to do the memory.” That's great. But you know what? You also had an idea to create a shopping class the atomic race. But then you still have a factor you need a good sand you need it, right? So, yes, idea is great and you can claim ownership of idea. I'm a type of person that I'm not really – Yes, the idea is great that you have. But I'm true believer that a team effort mastered this. That's why in a special forces with two three people. And when you add them together, one person gets sixty percent the other person has another twenty percent other person said twenty percent. You mastered this. I'm a believer to a hundred 100 perfection. Versus that one person with because the pad he already took maybe get you that 60% to 80%. Almost like a part of the type of rule, right?
Yes, I had an idea, but I also had a lot of horrible ideas that I convinced them to do for first two months and all failed by the way and they also had a lot of horrible ideas too then. None of them work. And for two three months we didn't get anywhere in this case. Whenever I try to work on any problem I always try to find two more, two three more individuals like a group of us, right? So it will be group sell. Almost like intelligent sellers to three people with very deep knowledge, but very deep skill set in what they do. So for example one of them was top-notch programmer and debugger that worked on a team, right? I could do some of it too but he is doing it a 100% level. And another person, individuals who helped with all the integration APIs with this crypto because we basically had to take the documents, find the card readers, get the you know stuff from the Swiss and then they thought well you can't decrypt it because the keys already expire this and that right. Kind of replicate as much as possible come to plus to the conclusion and then I try to put that everything together and see how we can now attack this algorithm. What is the real algorithm? What are they using? How they're rotating the 60 ciphers? How are they rotating these algorithms? Is there some systematic approach they use or it's truly as random as they're claiming? Is it a basically random depict? And then go into it.
[00:15:27] CS: Yeah, that's a really good lesson to learn here is find people and pick people who can sort of cover your blind spots and whose blind spots you can cover as well. Everyone has a different area of expertise that all sort of come together and sort of a larger than some of its parts.
[00:15:43] OK: Yeah. And I would say in a special ops, if you do that, you always have – And I know in even in ceiling world you say have someone to watch your back. You do have to be 360. This is a job of 360. Two people, 180, is not as effective. Three people, 360, effective.
[00:16:03] CS: And again, just to sort of bring that back to the beginning, you were saying I feel like when you were working on this project you weren't thinking like, “Oh, this is going to like catapult me to the next level of my career and this is not going to bring me a big paycheck.” While you're busy doing this incredibly difficult thing, like all of the sort of uh acclaim as you said and financial enumeration is sort of happening behind the scenes, but you weren't chasing that. You were chasing the learning and the opportunity. And again, I think that's a really great lesson for aspiring professionals.
[00:16:38] OK: No. Exactly. And I tell everyone like when I came in here I made one of individuals former executive FBI called Richard Hudak and I was asking him like what is the meaning of being successful and he basically told me just do your job the best and people. And yes you do need to stay recognized. You do need to say what you do. If you don't open the mouth – Don't think that you don't open the mouth people are going to know. No. You have to do that. But you have to have something to back it up that story that you said. And often we get students here for interview and they ask me what advice would you give me? And I tell them if you had two, three papers from a school that you wrote and you published them and they are really in the area that you pursuing or they are somehow instrumental for that employer, trust me, the employer is going to read them and he's going to be very happy that you conducted some level of research into this topic and you have some level of understanding. The most challenging interviews are when the person doesn't have much to say and then –
[00:17:49] CS: Right. They sort of took ownership of this one. Even if it's a very small problem to be solved, they took complete ownership of it and followed it through.
[00:17:58] OK: Yeah. And then when we had individuals here at a life force, we always tell people say well I want to do this computer forensic. It's really cool what the life force does. Or I like how you work for on indictment the Department of Justice or I like the secret service interactions you guys get and all the love you have someone from agency on our board. I really like that. I said that's great. But this is not about that. So how about you start writing a blog about the knowledge that you are capturing right now and try to get into it? And we had a few individuals who after to start writing the blogs about parsing forensic artifacts, working with forensic artifacts and then they profile themselves. They said, “Well, I'm going to do the Windows forensics. I'm going to do network forensics. I'm doing the iOS and mobile forensics.” They kind of profile the areas on their own research and then they really can go and be more successful. And/or some of them realize that, “Listen I never realized how scientific this is, this forensic science.” I don't think that's a job for me. I'm getting back to development or I'm getting back to audit or I'm getting back to compliance, which is great. It's just that recognition of –
[00:19:06] CS: Yeah, it's good to know.
[00:19:07] OK: Yeah, it's good to know. And I told everyone that writing and writing and researching and being instrumental if one of the jobs you can get is just be the blog writer for a company that does it actually gives you an idea of what these people really do.
[00:19:21] CS: Yeah. Yeah. Yeah. And again, we hit on that so many times here that the communication is so key. Writing skills are so key because it's not just enough to break the case. You have to explain it and explain it to people who don't necessarily live this stuff 24 hours a day.
Yeah. As we noted in your bio – I’m sorry. Go ahead.
[00:19:46] OK: And also if you are first working on any of these cases with attorneys, there is a report that's coming out. Everyone says well there is a forensic report? And that's the document it's legally binding and discoverable. So writing skill set is very important.
[00:20:01] CS: Yeah. So as we noted in your bio you were a guest lecturer at the FBI Training Academy. And so we we've had guests talk about digital forensics in financial areas and law areas and we had Amber Schroeder of Paraben talk about some major forensic products that they make, but I don't think we really had that much experience with digital forensic kind of government or federal scale or in that area. Can you sort of tell me about how uh digital forensics, if there's any difference in the way that you work with government agencies? If it changes the nature of how you work? If there's additional sort of like security issues or clearance issues or confidentiality issues or anything that that make it a different type of work?
[00:20:39] OK: Yeah. Sure. If you will directly work at a government you do need to have a clearance. So that's normal. If you work in a commercial sector and you're working for the victims and government is parallel to it because it's criminal. Then you might no need a security clearance. Meaning that if all what you're doing is collecting the evidence and securing the evidence, you might get involved with something what we call the authentication of the evidence, that's the federal rules of evidence, and then you might actually call it a court to testify to basically present how you actually conducted that securing of dividends. This is not different than you and I tomorrow go and collect the fingerprints and then the FBI needs to get that fingerprints, but it's been one month later. So they have to rely on figure employees and you have to authenticate your method that you use for that case.
Now the federal government always have to open their own investigation because it's criminal. None of us is in criminal investigation business. Government is, right? So the government the good news is that the government will open its own investigation. Can they leverage anything from your investigation? The answer is yes. So they will learn what you have and where you are at the moment. So I feel very privileged that a lot of our work at the life force elect and that cooperation with the Department of Justice indictment. For example, individuals from Mazarus, on cryptocurrency hacking. We have a round of firms in the Google knife hash story and how they got hacked for 60 million. Another one coin check for close to half a billion, and if you Google the stories how we were involved in for example very heavy nice hash investigation, it's a very impressive investigation on a nation-state level and we're working on these nation-state threat actors.
Two Iranians, if you recall the samsam group that breached the hospitals. That's also we work with the Newark division here of FBI. That was a really uh great case. We work on cases for [inaudible 00:22:33]. That's another example. A few of the APT group out of the uh very large country in ASIA that everyone knows of. Not as important. But the groups was very important. And by the way the group around 10 years ago launched one of the most massive uh supply chain attacks in history. So these supply chain attacks have been ran by this military group for less than 15 years. That's exactly how the cyber warfare is being played. So it's not as direct. It's usually very indirect and the supply chain was always heavy focus of nation state and that's how it actually was conducted. But at the training academy I feel very privileged that I was one of the selected people to speak and the program has like 12 to 15 sessions. The program got partially canceled because of the COVID and when the COVID came in. And I had most of the executives and state police and head of the various FBI states and ahead of the various state police and various officials from various states and basically rotate so you can for secure reasons you don't put them into one location because then these locations are usually classified where you go to.
And then talk about the primary threat actors that we actually have seen on some of our investigations. Major focus of the federal law enforcement in the US is something called business email compromise and extortion, cyber extortion, and of course then you have a nation states this category. But from this criminal very large is something called business email compromise. Meaning someone gets into the email system and conducts usually fraudulent wire transfers out of the company. And ransomware when they get in they usually exfiltrate the data and they encrypt the systems. So there's like this double dipping momentum in it. And then really speak about the cases and experiences of people who we've done like, for example, [inaudible 00:24:40] that's been attacked various hospitals right in election time and basically explains these straight actors like how what are theirs what they call the techniques tactics and procedures and also what is the something called indicator of compromise. Like how you really think through it.
[00:24:59] CS: Right. Where did it come from? Yeah.
[00:25:01] OK: Yeah. And it's more on these lessons learned. You try to explain is the process to handling this ransomware way this cyber pandemic? And I would say the cyber is one of industries that trying to get hold of it very well because every strain of that virus is very different. There's no universal cure here it's meaning that you don't have one thing on one product you can cure all. Yes, the vendors tell you there's one product that does everything right on everything.
[00:25:29] CS: This is going to take care of all your problems.
[00:25:30] OK: Everything. Yeah, exactly. But any environment you come in they usually have a 40 products. Average company like a fortune 2000 is getting like right now hacked has 40 different products and all of them have 100% proof basically to do something that's purely magical, that's pure magic outside of it. It's more of like a demystify that, look, reality is we all going to get something. From a cyber call to cyber cancer, and there is no person on the planet that haven't seen doctor in whole life. We probably going to see the doctor and just the question is that if you focus let's say 50% of your strategy on detection and prevention, you should also plan the defense mission and that's how you're going to respond when something happens. And the companies who are very successful or our government also they're very successful is that they really play in the context of if something happens, if that phishing comes in, yes, it gets through it, the user is on it but we were able to remove the links user didn't click, or user click but we have this product that detects it now we act. We isolate the computer and we isolate the problem and then we stopped this pandemic from spreading because we had this early detection warning that yes we have this infected system on a network. That doesn't exist, that exactly you have a global pandemic, right? The company is at a global economy. Everything is a global pandemic and the straight actor basically laterally moves from a system-system and affecting the computers and basically the extra trading data.
The federal enforcement always seeking more of a wisdom of what really happened at the public sector? What would happen uh in a private sector? And in a private sector are they doing something maybe a little bit different or are they learning lessons. The government has a much higher budget of course than a private sector, but are they really maybe thinking about these issues like how they really – What a strategy? What is their strategy to be in a business? Stay competitive? Deal with a business in a compromise? Deal with the ransomware? And what are real stories that someone who is doing this type of forensic investigation actually seen and how we are really advising these clients going forward? That's the one piece.
Second piece is also demystify this conversation with the federal enforcement. Like something happens to you, or FBI is going to send two cars, eight agents going to appear in the office. The answer is no. Government is busy too. Government actually heavy life for firms like us to do that collection. The government heavily relies on the firm to provide them the evidence, and the government actually doesn't care as long as the firms know what they're doing that evidence is being provided to the government and then they can run their own investigation. They can come in and do their own work.
[00:28:08] CS: That's what I was going to ask next was how much sort of autonomy do you have in terms of the decisions you're making in the forensics that you're doing? I mean, are they basically giving you a set of directives? Find this? Enter this? Find the indicator of compromise here. Or are you sort of given the freedom of like we know something's happened here. Sort of find it in whatever way you see fit.
[00:28:32] OK: It's a very collaborative effort, meaning that when we have something we share it, but often when we stumble on a case, and let's say we have a conversation with the FBI over the secret service, often they have more than they do. Meaning that they have something under the TOP protocol, TMP amber, something they're already investigating. Like let's say there is an FBI Denver division investigating this case for last two years. They have a lot of material, like a techniques, tactics, procedures, IOCs and things they can share. So usually what we try to do when we get a commercial case that agrees to cooperate with the federal law enforcement basically get very close to the unit that is investigating this threat actors, this specific threat actor, and then the federal law enforcement, the FBI is really helpful to help that victim to get up and running with the knowledge they have.
Of course that knowledge has to stay concealed with the help of that victim only. So it doesn't go outside. They're still working on their case. They still want to close their criminal case. They don't want that knowledge that they share or the ideas they share really to go outside of that circle. But they want to help these victims. It's for the benefit of the victim and then often we play this conduit where we translate. Often we have something and give it to them and say, “Well, if we haven't seen this yet.” So we're going to add that into it. Then we don't know who they're sharing with. But the same way, they're sharing with the other victims and they said, “Well, go and scan this.” And someone asked me why is that important. I said, “Look, each virus for lack of better understanding has some scanning and detection process and we created this indicator of compromise. Whole idea is that I'll give you that set. If you scan on your endpoint, you scan your network logs, you scan your sim, you scan your thread intelligence. Then you actually will realize, “Wow! I had this type of cancer here half a year ago.” It's gone. Or, no, it's not gone. It's still here. That's the aha moment. Okay. What are we going to do now? Like it’s been here for half a year. Should we uh jump out of the window? No. Should we plug all the cables out?
[00:30:39] CS: Set it on fire. Yeah. Right.
[00:30:41] OK: So how do you react when you realize that, well, I have a nation state for half a year in my network.
[00:30:45] CS: Wow! Yeah. Yeah. Yeah. So I want to sort of pivot over from – I mean, do you have any particularly interesting? You mean you told me about [inaudible 00:30:53]. You told me about the Swiss Bank. Do you have any particularly interesting like war stories that you can share with us in terms of like an especially tricky case or a strange solution that you found?
[00:31:06] OK: Yeah, there was a one at the end of 2019, and there was a group that was deploying ransomware called double payment or bitpaymer and they were able to circumvent most of the EDR products. And we realized that that's exactly what's happening, we decided to actually develop something called cyber vaccine. It was before the pandemic event hit. We had this idea that we're going to develop this cyber vaccine and the way it's going to work is every time we have this new strain we actually do going to have someone from – We have four or five Ph.D’s on our staff. So we're going to have this Ph.D actually create something that can be attached to the endpoint product to basically do the removal like a containment eradication and recovery for that endpoint.
And we've wrote a really good paper about this and we get a lot of good traction and was used for many victims. What was also interesting about later on in performing this investigation, there's some penetration testing called Cobalt Strike. And Cobalt Strike is a product that helps you to create this memory type of resident beacons. Basically very limited footprint on a disk, but primarily live is in a memory. Chris, this is something like a virus living only of your DNA. You take it out of the DNA, it dies. It's very hard to detect. Very hard to do. By the way if tomorrow if you go to a doctor and tell him you have a DNA virus, you might end up at some mental institutions for half a year just check your sanity because no one's going to believe you, right? It’s like almost like, “Hey, a military from foreign nation is in my network.” Yeah. Sure. Sure. On your phone too. On your phone too, right?
It’s not as this really plausible story, but that actually do happen. So Cobal Strike mimic this a high-level of engagement just purely in the memory of the victim’s computers. And in the virtue of looking at it they realized that we stumbled on something on a trade actor called APT41 where they almost seems like compromise the ransomware people. Took their things, took the whole kit and pretended that deploying it while they were stealing the data and blaming them. So that was kind of like an interesting angle. So we're going from this big payment, double payment cyber vaccine to looking at this ransomware and now we're looking at this wonderful really large nonprofit organization. Something is not lining up. And we're looking deeper and we find this almost like a digital shadows all around and it seems to be, “Hold a second. This looks a little bit different.” And then we found out that it was a national threat actor who probably compromised the bad guys. The criminals took their kit and deployed it and blamed them while they're actually they're doing something else on the network. That happens too.
[00:34:09] CS: So someone was attacked and then the attackers were attacked by an attacker. Is that what I'm hearing?
[00:34:14] OK: That is correct.
[00:34:14] CS: Wow! Okay. That's incredible. Yeah. And you said you found that just by kind of looking at sort of like artifacts and shadows of things that were happening and you sort of deduced that based on sort of anomalous events and things?
[00:34:31] OK: It was more of a call from that company. They already had – A lot of these forensics now come from these insurance carriers and law firms working with insurance carriers. So this was already closed by an insurance carrier, the whole investigation. So they had a firm. It's almost like you come to and says, “Well, I want to fix this bumper,” and they're going to fix your bumper, but they don't look at it. They don't think you have a damage on engine, or a damage on engines would require like a really much more sophisticated operation. And they ran some basic checks. They don't see anything and they move on. This is one of those investigations that is already closed. The claim is closed. Everything's closed. But you have these two three individuals that don't believe the story. Just something is not lining up here I see. And then they call for independent review of the investigation. It's already closed by the way. It's already closed, already sealed. And they're saying, “Well, look, if something's not lining up. Can you just look at this? Let me tell you I think that this system, this system.” So then on one of the systems we did find the traces. Then we approached the firms that did the work and also the insurance carrier and of course they right away said, “No. No. We don't want to cooperate. We don't want to help you with this. Yeah, exactly. For us, it's a sealed case. And then we basically work with the victim and turned to be them to be exclamation state and they were happy really that they found the truth behind it because it just was not – The story was not lining up whatever presented.
[00:35:57] CS: Now, I mean that was an awesome story and I thank you for telling about that. But I want to pivot over back into the sort of work aspect of Cyber Work here. You mentioned at the beginning I think it's really good advice that if you're a new person entering the digital forensics realm that the employers are going to look at your research papers. They're going to see that you've taken ownership of a certain problem and you've solved it to the greatest of your abilities. But beyond that, can you sort of talk about particular skill sets or experiences that people should be striving for if they want to do digital forensics especially in a government or a federal sphere apart from like just being dogged and being obsessed? Like what should people be looking to sort of get into to make themselves appealing to employers?
[00:36:47] OK: So at the early stage of my career I was one of the individuals and probably the horrible mistake that many of us do make that I believe in this superior technical abilities are the only way to go, right? So superior knowledge into it. Later in our life I realized that one of the reasons I was picked is that – It never occurred to me, is that I'm one of individuals that if you put me in a lot of pressure I'm not going to break and I'm not going to cry. Unless Chris is going to play some really bad rap music. I don't cry. If you do that right. But everyone has this breaking. I'm trying to tell you. Everybody has this small breaking point you can find. We all do. Just the question is how big is this breaking point that you have and what is really will create this scratch on your shoulder that you don't like about it? And it's very important even to balance mental, physical and intellectual capabilities of the individual. You can have someone who is very strong technical, he is very strong anything, but it's very hard to work with him. Very hard to work with on a project. Very hard to explain. Can perhaps run – If you ask him to work 10 hours because you have to finish something, he can, but after eight hours he gets into very unstable emotional state. Or intellectually if you challenge him and you ask him to speed up, he starts making mistakes. So those are the qualities that if that work requires, then the person is staff.
In the forensic science work, I guess you might have a two type of employment. One employment is that you're working for a consulting company and the work is coming in and out and you're not bugged by this momentum that you constantly work from different cases and the speed is very high. So you're becoming this – It would be like really flattering like a formula one driver every day and you try not to crash and try to win the race, but you're trying to do that so many times in a month and you enjoy that. You enjoyed it right. The same way you keep your mental and intellectual and the physical unbalance as a shape.
For example, I go running every morning, there's 25 minutes, 30 minutes for three miles, and this is my meditation moment with myself and find myself and really stay energized for a day and stay healthy. And I think that's very important. And everyone who wants to get into really in this career should think about that. It's not only about being really good, but being a good team player, have a good intellectual skill set, how you approach individuals. Being developmentally balanced in a society. SO basically you are not of one of these awkward individuals that's just walking around and doesn't want to talk to anyone and can't present himself. And then also build this method of stiffness. I really build this method stiffness that if you need to put up with some serious load, it's not something – You understand that that's a part of a job. It's almost like imagine you go to hospital and the orthopedic surgeon in a middle of the heart surgery says, “Well, I have enough. I'm going home.”
[00:40:04] CS: Yeah. Yeah. Yeah. Right. Yeah. Yeah. I can't take this anymore look at all these hearts.
[00:40:09] OK: Yeah, like in the middle of the surgery. So it doesn't happen, right? So those individuals pick those jobs, yes, and then you can envy what they have and you can envy what they get paid. The question is would you want to have their job when that pager goes on and you have to go there? Would you have a job and work 12-hour surgery on someone with the people. And in the fact that you know that if you make one mistake the person can die. The good news is in forensics it doesn't happen at that aggressive level, but what we see for example from ransomware, like if you make an error in let's say in forensic investigation, things can go horribly wrong. There is a good case right now of the large data center company being compromised. There is a forensic firm who was pivoting. Very large forensic firm who literally underestimated the threat actor.
So for a month when they were doing the containment and eradication and isolation of the host, this threat actor went from 10 percent compromised company at a data center close to 80 percent behind their back. And I told everyone that doesn't look very good when you're forensic from higher to basically it happens to. And lesson learned here is don't underestimate the threat actors. Yes, you have to create a balance for yourself, but if they working 24/7 you have to be 24/7 opponent. If there are snipers, you have to be a sniper.
[00:41:30] CS: Yeah. Now I want to sort of drill into that a little bit because I use a metaphor of saying like you know a lot of people think they want to be you know a baker but all they think of when they think of a baker is selling a loaf of bread to a nice person at ten in the morning. They don't think about being up at three in the morning with the ovens or if they want to be a film director they just want to say action and not necessarily be up in the middle of the night choosing color swatches. But so can you give me some examples of like – Because you said like people can get burned out or as they hit a threshold in the day, what are some things about digital forensics that people who might be interested in getting into it aren't thinking about in terms of this is a real slog? This is more difficult than you think. Like what are some of the activities that you've seen that sort of break people down who maybe aren't ready to do the hard work of it?
[00:42:21] OK: So when the whole COVID started I really like the area of the New Hampshire. So I went for two months to New Hampshire and then in the middle of that I got called back to New York and we were asked to look at one of the prominent research and hospitals in the city for intervention on the foreign nations into the research. And I'm coming back to New York City I'm thinking that I'm going to visit all these COVID research facilities and I'm going to be doing these night missions. I have no – There's no waxy. There's nothing. And the team, I have to convince the team now to go with me and bust these computers overnight, every night and weekends for let's say for two months period. And how I'm going to do that to explain to people this is a good thing to do? To walk in, to like – Yeah, you isolate at home and you're happy and now I'm asking you, “No. This is a hospital. Three thousand people.” We are walking through it.
[00:43:31] CS: On a regular basis. Yeah.
[00:43:33] OK: Yeah and at night. It doesn't matter. But at the night, and then now you're not going to sleep tonight and this is what I want you to do and this is what you want to do the weekend too. No one can know what you're really doing there either.
[00:43:44] CS: Yep. Yeah, you have to be there quietly. Now speaking of COVID and traveling, for some of our listeners might be far from large tech centers or large cities and they want to maybe do this kind of work or maybe they're coming into it from a different industry all together, like what are some ways that you can sort of get the experience you need in a place where digital forensics is not happening? Is it the sort of thing where you can do it anywhere or do you really need to be sort of getting your apprenticeship in cities with sort of large consulting firms?
[00:44:23] OK: I do think that most of the work because of the cloud and the way the cloud system is being hosted right now truly can be done in a fashion acquisition and evidence collection remotely. And then you do need to have a forensic lab. So you and I on a camera. So what you see behind me is actually a forensic lab. It's a full-fledge forensic lab. So you do need to have a forensic facility. So you have to build your laboratory. And of course you have something what we call the road kit, but you have your laboratory.
So the same way people sending like let's say tissue blood samples to a laboratory, you have a forensic laboratory where the stuff is actually sent into. So the good news is that most of the work's always been done in a forensic laboratories. Not as much outside of these laboratories. The securing of evidence, meaning it goes to almost like a crime scene and securing the evidence was something that we were conducting. But now if that evidence is in the cloud, there is no more of that physical presence for anyone to go to. There is a process that us and let's say the provider Google, Amazon, Microsoft actually agrees on and they provide us this conceal image the best way forensically as it could be.
For example, Microsoft probably has the most advanced process. They have something called securing compliance center and they have fully discovery so they can actually go and conduct e-discovery directly and inside of the Office 365 platform. And they do have probably the most forensic type of capabilities. Meaning that you can acquire the evidence with the tool and then bring it to the forensics lab. You can just basically download.
So for most of the individuals I would say good news is that you can do a lot of it remotely. You don't have to be where you are. Also in our division called managed services they basically do what they call the level three managed detection response and endpoint. Meaning that you can be anywhere. You have access to the cloud console. You're reviewing these alerts. And based on metadata of these alerts the console lets you interact directly with the endpoint that you're trying to investigate. So now you're not at a customer site. You're not flying to customer site .You are actually very rapid response. And then if needed we call the customer says, “Yeah, we want you to unplug the computer from the network. We want you to take a memory image. We want you to take the full forensic image.” We give them a tool. So we tell them go to buy the hard drive. We give them a manual and we walk them through it how to acquire. If they are not capable of doing that, we only say dump the memory. They'll tell them pull the hard drive out. Give us the encryption keys and ship hard drives to this laboratory.
[00:47:05] CS: Okay. So with the world as strange as it is right now due to the pandemic, and we also have this the sort of skills gap in many cyber security fields, but also job hunting is a little over saturated. So you have all these sort of balancing contrasting forces. Do you have any tips for job hunters in the digital forensics field which might have a lot of openings? A lot of opportunities? But also lots of people applying and it still feels like you're having a hard time sort of finding your niche? Like what do you recommend in terms of making yourself stand out as a candidate from other candidates? I think probably going back to what you said about sort of your ability to complete research and so forth would be a good start, right?
[00:47:47] OK: So I would say the getting less and less traction as a forensic industry. Being the digital forensic examiner or being the digital forensic investigator is not as popular. Seems that most of the security is really shifting towards cloud and integration and also mobile devices and it is also that feeling of how much work you have to put in versus what the output is. And it's great for all us that cloud is happening, but the same way cloud companies or cloud security or mobile companies can attract the talent and they multiply on the products on the revenues and they offer better share units and phantom stack and options that can lead to very large payouts if the company is acquired. And we see that as a serious competition to basically more of a work being just a forensic examiner. Because being a forensic examiner means that yes you might stumble on something interesting. And we for example have our own um product, like a cyber vaccine. And we also create like an extension to the [inaudible 00:49:01] product just for forensic investigation. There are some products. They usually attach to other products. But they're not – Maybe not the mainstream, right? That there is a mainstream that you attach into it. Almost like you have a car and you just attach the wheels into it. So those viewers can come from different manufacturer. And that's not as lucrative for many candidates because the amount of science and knowledge they need to do in order to do the type of job it's maybe more significant.
The second piece I would say that's maybe more concerning for candidates is the fact that at some point they need to testify at a court. And do they really want to go and testify at a court. Or they want to really deal with attorneys. Do they really want to deal with the legal system? Do they really need the study federal rules for civil and criminal procedures and understand what really the construct is when working with the evidence. And that's a little bit more work.
So I would say that candidates who are seeking jobs right now in forensics are candidates who really like more of a scientific approach into solving something versus more of an automated software cyber security software type of approach where I'm part of a school tech startup that's doing XYZ. And they also seeking the drill from, yeah, I like this Tom Cruise core drama being at a courtroom and I really would like that. I would want to see myself working for a DA office and I'm going to do this forensic investigation and I'm going to be part of a courtroom and I'm going to do this criminal cases and –
[00:50:44] CS: Yup, bad guys.
[00:50:45] OK: Yeah, exactly. Bad guys and serve the justice. Like for us as a company one of our logos always been that we have something called discipline of finishing, meaning that we never actually asked the victim to pay for us to work with federal law enforcement. The opportunity on our own dime some of the investigations. Meaning we just get the signature from the victim and we work with the federal enforcement just so the justice could be served and some of the cases took between two to five years to be served. Some of it is also you can't expect just getting paid because you're not doing this because you looking to get paid. And, yes, those relationships with the federal law enforcement do matter. You value them highly. And it is very prestigious to point to let's say the Department of Justice and then says our evidence contributed to this indictment. No one is questioning that. But if there is no outside of that acknowledgement from society or outside an acknowledgement from the industry, that there is no really expectations of something monetary versus in some other cyber security fields and there is that aspect that you probably would get something out of the system. So that would be something for candidates to consider that it is definitely in my opinion much more work than just getting the other cyber security job.
[00:52:09] CS: Okay. So you got to really want to do the work if you're going to do the work.
[00:52:13] OK: Yeah, that's for sure.
[00:52:15] CS: This has been great. We're coming up on an hour here and I don't want to take much more your time, but as we wrap up today, can you tell me about your work with LIFARS LLC? Which, as I said, encompasses cyber security services including incident response, digital forensics, ransomware mitigation and cyber resiliency services. So what are some products or projects that you're excited about for your company in 2021?
[00:52:37] OK: So I'm really pleased we created with our – We have office here in US, Europe and India and we were able to pull some serious R&D especially in Europe and we have a few great individuals like [inaudible 00:52:51] like some of the top researchers right now Michael Julian with the Ph.D. people that create automation and in a managed detection response. And cyber vaccine is a product that we almost ready to launch and it's basically focusing on any ransomware, detecting ransomware through techniques, tactics and procedures. It's more behavioral type of approach versus, “Hey, I know this malware. I know this string and it's going to work,” because most of them already don't know. So we're looking at behavior how the ransomware actually operates and then we deploy this product at the end point that would basically detect it and kill the ransomware product itself. So that's what’s very exciting. We also have a tool called Gargamel that does the collection. So we are very pleased with that. We have another tool called log checker that we basically look and enrich the data and logs. We have another tools called virus checker that we actually go and check the systems for viruses through the API we kind of integrate. Almost something like a virus total, but on the endpoint. Then we bring that plugin in that we also developed over the last year. And we developing the mongoose which is the extension to forensic to the EDR, XDR products where it's more on the forensic side. So leveraging what the EDR, XDR tools are really doing and then basically helping us to extract the data.
Our company is based basically on the R&D and a product division and then we do digital forensics incident response. We have advisory practice. We have managed defense practice and we have offensive services or sometimes called technical services. So offensive services like red teaming penetration testing. And main reason why we have that is basically a life cycle half. Imagine when someone has this cyber injury. So when they're taken to the cyber hospital they are in a cyber emergency room. That's the DFIR unit, right? Then we move them into the hospital and in the hospitals that we have these managed defense. We put them all these monitors from the – Flip them from the emergency room to different type of monitors. Then we conduct a lot of the testing. The offensive testing needle comes in. And then when is this rehab, we do a lot of advisory. It is rehab type, post-trauma, cyber trauma type of environment. So we basically mimic this cyber health recovery for some of these other victims in a way where we take them into emergency room, we move them through the hospitals and we move them into the rehab and then if they still need some advisory practice, they become the family cyber type of individuals for them going forward. And that's been our business model. So most of the companies or enterprises that we work with truly came through that being cyber victim or not don't want to become a cyber victim mindset and then we replicated that life cycle.
Many of them want us to train them or give them education to listen, and I'm one of the individuals where I always said that one reasons I truly believe I am where I am because I always like to listen to older advisor people. I never been one of the individuals or I don't consider one of the individuals who makes very strong opinions without any context over data analytics or any rational and wisdom or not even understanding the subject. I always try to understand what that actually is. And I remember when I often worked with investors they told me that a lack of knowledge never precluded them from many strong opinions. But that's the risky investment mindset and it's not very applicable in digital forensics that we are actually carving as an industry.
[00:56:53] CS: All right. One last crucial question, if our listeners want to know more about you, Ondrej Krehel or LIFARS, where can they go online?
[00:57:01] OK: Feel free to follow us on LinkedIn. Please go to our webpage. It has many white papers and stories that we've done. We also have a GitHub repository that you can go into and see some of the products that it created and, for example, cyber vaccine, the first one in 2019 that we did. It's published. And also feel free to contact us, look at some PR press releases. We have a HR portal where you can send your CV if you're interested to become a writer or you seeking something or you have something that you would like to discuss with us. But I would say generally we are quite a bit all around the social media and channels.
[00:57:42] CS: Okay. And it’s L-I-F-A-R-S.com?
[00:57:45] OK: That is correct, L-I-F-A-R-S.com, lifars.com.
[00:57:50] CS: Very good. All right. Well, Ondrej, thank you very much for joining me today and providing your time and insights. I really appreciate it.
[00:57:56] OK: Chris, thank you for having me. And as one of the journalists pointed out at the special operations that culture basically eats the strategy for the breakfast. So one suggestion to everyone is that create a culture of yourself as a strong professional and create a culture around yourself that the other will perceive that is beneficial to the society. And that’s always been motto more for us, that we do have a strategy as a company, but our culture is a strategy for breakfast.
[00:58:29] CS: Culture is the strategy. We're going to leave you with that. So thank you very much again, Ondrej. And as always, thank you to everyone at home who is listening and watching. New episodes of the Cyber Work podcast are available every Monday at 1 pm central both on video at our YouTube page, on our website, infosecinstitute.com/podcast, or on audio wherever find podcasts are downloaded. And don't forget to check out our hands-on training series, CyberWork Applied. Tune in as expert info psych instructors teach you a new cyber security skill and show you how that skill applies to real-world situations. Go to infosecinstitute.com/learn to stay up to date on all things Cyber Work.
Thank you once again to Ondrej Krehel, and thank you all again for listening and watching. We will speak to you next week.
Subscribe to podcast
Free cybersecurity training resources!
Infosec recently developed 12 role-guided training plans — all backed by research into skills requested by employers and a panel of cybersecurity subject matter experts. Cyber Work listeners can get all 12 for free — plus free training courses and other resources.
Weekly career advice
Learn how to break into cybersecurity, build new skills and move up the career ladder. Each week on the Cyber Work Podcast, host Chris Sienko sits down with thought leaders from Booz Allen Hamilton, CompTIA, Google, IBM, Veracode and others to discuss the latest cybersecurity workforce trends.
Q&As with industry pros
Have a question about your cybersecurity career? Join our special Cyber Work Live episodes for a Q&A with industry leaders. Get your career questions answered, connect with other industry professionals and take your career to the next level.
Level up your skills
Hack your way to success with career tips from cybersecurity experts. Get concise, actionable advice in each episode — from acing your first certification exam to building a world-class enterprise cybersecurity culture.