Data backup in ransomware situations

[00:00:00] Chris Sienko: Cyber Work listeners, I have important news before we dive into today's episode. I want to make sure you all know that we have a lot more than weekly interviews about cybersecurity careers to offer you. You can actually learn cybersecurity for free on our InfoSec skills platform. If you go to infosecinstitute.com/free and create an account, you can start learning right now.

We have 10 free cybersecurity foundation courses from podcast guest, Keatron Evans, 6 cybersecurity leadership courses from also podcast guests Cicero Chimbonda, 11 courses on digital forensics, 11 courses on incident response, 7 courses on security architecture, plus courses on DevSecOps, Python for cybersecurity, JavaScript security, ICS and SCADA security fundamentals and more. Just go to infosecinstitute.com/free and start learning today. Got it? Then let's begin today's episode.

We did it folks. We got him. We got Mr. Backup. W. Curtis Preston aka Mr. Backup has been in the backup and recovery spaces since 1993. He's written four books, he hosts a podcast called Restore It All. He founded backupcentral.com and he's a tech evangelist for SaaS Data Protection company Druva. So, of course, we talk disaster recovery, the role of a good backup in ransomware situations, and why the data recovery person and the information security person in your company need to become fast friends and start sharing notes. Also, why we've always been completely wrong about tape backup systems. That’s all today on Cyber Work.

[00:01:40] CS: Welcome to this week's episode of the Cyber Work with Infosec Podcast. Each week, we talk with a different industry thought leader about cybersecurity trends, the way those trends affect the work of Infosec professionals and offer tips for breaking in or moving up the ladder in the cybersecurity.

W. Curtis Preston aka Mr. Backup has specialized in designing data protection systems since 1993, and has designed such systems for some of the largest organizations in the world. His lively prose and ride with real world approach has made him popular author and speaker. He's written for O'Reilly books, the latest of which is Modern Data Protection, published in 2021. He is also the host of backupcentral.com and the Restore It All podcast. He is now the chief technical evangelist for Druva, the only at scale software as a service provider of data protection. So, guess what we're talking about today, folks? Back up. We're talking recovery. We're talking Mr. Backup.

Curtis, thanks for joining me today. Welcome to cyber work.

[00:02:36] W. Curtis Preston: Anytime I get to talk about my favorite topic, I'm there.

[00:02:40] CS: Phenomenal. Let us get into it. So, your story goes way back. You've been working in data protection since ‘93. Where did you first get interested in computers and tech, and what first got you excited about this, especially considering that it was at the sort of start of the whole thing?

[00:02:58] WCP: Yeah, I don't remember like the very first thing, but I remember I was actually active duty in the Navy at the time. I remember, I actually took, and for those that go this far back, I actually took an NRI course, National Radio Institute, they used to advertise in the back of Popular Science Magazine. And I took this this correspondence course and computers while at sea in the Navy. So, they would send you books, you’ve got these books, and you would go through these books. And then you actually built a computer when you were done. I remember that it was in 8088, just prior to the 286 is coming out. And the 286 came out while I was while I was taking the course. And I was like, “Do I get one of the new computers?” And they were like, “No.”

[00:03:57] CS: You got to answer the one that brought you there. Obviously, we just mentioned that you've been in the backup and recovery space since 1993. Can you talk about some of the ways that the the practice whether technology or protocols or techniques have changed in the intervening 30 years? I mean, I'm guessing clouds a major sea change. But is there a fundamental difference in our understanding of the place and purpose of backup and recovery?

[00:04:21] WCP: I'm going to say that is a giant yes. So first off, for sort of scale of understanding, for a frame of reference, the entire data center – so I was at a credit card company, I was at a $35 billion company, our entire data center would fit inside my iPhone from a space perspective. It was a 300-gigabyte data center. And back then, it was just standalone tape drives. We were using native tools. It was a Unix Face world. So, we were using native tools.

At that world, the world of commercial backup and recovery grew, and it went through this massive peak, where what actually happened was, the tape drive actually got too fast for the job. I don't have time to go into it, but a lot of people think that tape drives were slow. And that's why we moved off of it. It was actually the opposite. Tape drives got too fast for the average backup. Literally, they were 20, 30, at this point, 100 times faster than the average backup. So, there was this move towards disk as the primary target for backup and recovery with tape. At that point, being just a salt mine copy. We're going to send it to Iron Mountain just in case a disaster happens.

And and then, we slowly migrated to more and more disk. And then yes, the cloud has changed data protection, in both the sense of how it's done, and why that happened. So, it's not just that the cloud came to be, and that it made possible a number of things like Druva, the company that I work for, is a cloud-based company. So, without the cloud Druva would not exist. But it's also the fact that there are so many ways, the easy thing to say is that the data center is no longer the center of data. So, the data center is no longer the thing that we back up. It might be one of the things that we back up, but you have the data center, you have your IaaS vendor, like an AWS or an Azure, and you have a bunch of SaaS providers, and you have something that didn't exist when I was in the industry. And that's this little thing called laptops and mobile devices. We didn't have those back then.

Yes. So, it's a giant yes that both in terms of what we back up, and from where we back that data up, as well as how we back it up. At this point, backup and recovery, data protection, disaster recovery, is if we set aside archive and long-term storage, is almost an entirely disk based practice. There are very few people that are still doing their backups to and from, directly tape.

[00:07:45] CS: Okay. It also sounds like those are almost different job – not quite job titles, but you're talking about, is the data protection that you're using just to make sure that you have a record of this thing? Is that different from the data that you're duplicating in case of like disasters and things like that? Because I might be misreading it.

[00:08:09] WCP: No, that's a great question. Historically, and again, this is another giant c change. Historically, if you had the need for what I would call real, as I make quotes in the area, if you needed, real DR back in the day, you were doing replication. You were replicating any mission critical systems and file systems and databases, to a hot site somewhere that you could fail over to in times of disaster. But that was really only the high end of the market. So, your trading firms, things like that, that could put a million-dollar price tag on downtime. So, they knew that they knew that every minute of downtime cost him a million dollars. And so, it was easy to spend $10 million on a replication system.

The rest of the world was doing tape back in a day, tape in a box and hope you never use it. The world has gone to the point where in general, the bulk of the industry does not accept tape in a box as a DR strategy for anyone. And the technology, again, due to the cloud, the technology has advanced to the point where you have products like Druva that can do both backup and recovery and disaster recovery while moving only a single copy of the data. And then just accomplishing those two purposes with the same copy.

[00:09:44] CS: I mentioned there's also a speed of recovery issue in terms of like the emergency copy as well. As you say, if every minute is losing you a million dollars, I mean, I have a backup system on my computer at home, but if you lost all that data, they offer you pay them $40 and they'll send you a disk with all your stuff rather than wait for it to slowly go back out over the Ethernet. So, how is this sort of technology of the recovery speed changed maybe with cloud or whatnot?

[00:10:15] WCP: Yeah, great question. The big thing that is possible today, that wasn't possible before, is cloud based recovery. So, if you truly have a mission critical system, or a bunch of systems, and you want to recover them as quickly as possible, back in the day, as I mentioned, you would replicate to a hot site. You would pay money for a datacenter, that was sitting there doing absolutely nothing, just in case you ever needed it, right? That was what you did back then.

Now, what you do is you contract with a cloud provider, and a company that is able to basically, you do a onetime setup upfront, you see these are the VMs, virtual machines, these are the VMs that I want included in this DR, and these are the parameters that I want. So, from terms of, I don't know if your listeners are familiar with RTO and RPO, recovery time objective, the amount of time that a restore is allowed to take, and then recovery point objective, the amount of data that we agree that we can lose. So, you decide those things upfront, and you configure the system to support that. And then what a modern system again, because of the cloud, what a modern system can do is actually pre restore a copy of your data, and keep that copy up to date, to your most current backup, so that when you go to do a DR, all you're really doing is spinning up the VMs. You might be doing a little bit of massaging of the data, but you're not actually doing a restore.

I can think all the way back to the early days of backup, when backup was still tape and replication for DR. I can remember companies with big companies that were saying, if you're reaching for your tape, at the time of DR, you're already dead. And that's still the case, if you're still reaching for your backup, your regular old backup and you're going to restore your entire environment after a disaster or ransomware attack happens, you're already dead. You find yourself being put in a situation of needing to or considering, “Gee, would it be better to pay the ransom?” Which is a horrible situation to be placed in.

[00:12:46] CS: For sure. Now, do you think there's still a benefit to having a tape backup, a physical thing in your hand that if all other fail safes fail that it you still have it? Or is it just going away?

[00:13:02] WCP: I think for most people it has gone away. What does need to happen is we need to get the things that take provides. And the big concept here is this thing that we use the term air gap, right? So, when a disaster happens, and again, it really doesn't matter whether this is a manmade disaster, like a ransomware attack, or a cyber attack, or a flood, or a hurricane, or whatever. You need to have a copy of the data, some other place that is electronically inaccessible, from the thing that you're protecting. So, that protects both from fires, floods, earthquakes and tornadoes, and also from a cyber attack.

So, when I look at, what kills me, again, as a person who spent my entire career helping people protect their data. When I read articles about ransomware attacks, and I read the phrase, and the backup was also encrypted. I just bow my head. Because how did you not create an air gap? How did you have your backup, sitting right next to, at least electronically speaking, sitting right next to the thing that it's protecting? And like some people would think, for example, by moving your backup server into the cloud. So, if you pick your favorite cloud vendor and you pick your favorite backup software vendor in the cloud, well, that's air gapped, right? No, the whole point of that is if that server is running a certificate backup software vendor, electronically, it's right next to your server. So, if your server is compromised, that server could be compromised. You need a system where there isn't a route from A to B no matter how good the hacker is.

Let me just to go back to your question. If you have that and the other thing that tape was really great at was cost. So, if you have that, and you do that in a way that isn't astronomically expensive, then you've provided the benefits of tape with cloud and disk. And I don't feel the need to have a tape copy. The other thing is making a tape copy is difficult, expensive, et cetera, et cetera. It adds additional complexity and risk to the system. I'd rather you spend that money elsewhere.

[00:15:37] CS: Yeah. Okay. So, I mean, that leads nicely into my next question. You've written four books on the subject, you host a podcast about it, you founded backupcentral.com and you’re a tech evangelist for Druva. And you've talked about the number one here, but can you tell me some of the main mistakes you see again, and again, with the way companies handle their long-term data backup? So, I guess, let's start with air gapping. I mean, you said that, if it's backed up in the cloud, that's not air gapped. Is there a way of air gapping your cloud recovery backup?

[00:16:08] WCP: Well, what I specifically said was that if you host your backup server in the cloud, that isn't necessarily air gapped. The question is, how you're storing it in the cloud, right? There are ways to store data in the cloud. For example, if you have a copy in an immutable storage tier such as S3. S3 offers, in AWS, actually all the major cloud vendors offer an immutable option that when you write your backup to it, and you specify, let's say, 30-day retention, or 60-day retention, nothing, including you as an authorized owner of the account, you're not allowed to delete that data.

So, that's one way. Another way is the way Druva has approached it, which is you don't own, manage, touch, ever see the actual infrastructure behind the backup. That is all managed by Druva. It's a different account It's a different process. There's no ongoing backup server that's running to be hacked. It's just an entirely different process. The data is stored in an account that our backups are actually stored in S3, and our S3 is configured so that only our backup process is allowed to write to that account. So, even if a hacker for example, got through every level of security, and somehow was able to get authenticated S3, they wouldn't be able to delete your backups. And also, of course, the data needs to be encrypted. Encrypted in flight, encrypted at rest.

So, there are ways, really good ways to protect your data in the cloud. It's just that you just need to think about the things like a ransomware attack, and make sure that you're protecting against them. I would add right next to a ransomware attack is because right now, all focus is on ransomware. Every data protection vendor is saying, “Oh, well, our backups are this or our backups are immutable. And so therefore, it's not subject to a ransomware attack.” I would say two things. One is, not every company that says their backups are immutable, are actually being honest about that. And number two, the other is ransomware is not the only problem. There are still just bad actors that might even work for your company. And you need to look at, for example, the ability of a backup administrator to log in, or someone pretending to be the backup administrator, to log in and just delete your backups via the backup system interface. That's why you're going to see more and more backup products that are adding actual immutability, where, even if you are a super user, you're not allowed to delete the backups.

[00:19:05] CS: Okay. Now, are there any other sort of things that make you want to pull your hair out that you see people do all the time? Or are those the main issues with recovery?

[00:19:16] WCP: Well, I mean, it depends on which part of the industry we're talking about. So, if it's an on-prem system, there's a litany of challenges that a person has to do when they're designing. What I generally saw back in the day was it was just this inability to understand how tape works, that everybody thought the tape was slow. And so, they were designing to fix that problem when that was the opposite of problem. So, they would buy more and more tape drives, more and more tape drives. I can't tell you the number of times where I went into a customer and I said, “Look, the problem is you have too many tape drives.” And they're like, “What?” And we would redesign their system, they would have to use fewer tape drives and get better performance. We don't really have that as much these days. I suppose the biggest challenge that I see with a with a modern computing environment is actually people that, that there's datasets that people think are protected from a backup and recovery perspective that simply aren't. And the biggest example of that are SaaS providers, like Microsoft 365, G Suite, Salesforce.

The vast majority of customers of these products do not back them up. We have example, upon example, of proving that if you or some bad actor does something bad to your data in Microsoft 365, G Suite, or Salesforce or similar systems, it doesn't come back. The stuff that Microsoft has, doesn't help. Most recently, and I don't mean to pick on Microsoft, the challenge with Microsoft is that they have so many good sort of backup like features, all over the product thbey talk about, they use words like restore. Like restore deleted emails. Well, you're not really restoring them, you're just pulling them out of the recycle bin, which means that they were still there, they just set a flag on them.

So, it's so good at sort of mimicking backup that a lot of people think they don't need it. But a perfect example is, the best example recently was sometime last year, KPMG. Someone at their site needed to do something, they needed to delete one user's private chats. And they were using something called Microsoft retention policies, which once you set them, you say, all users’ chats are kept for, let's say, 90 days. So now, if you want to delete Fred's chats, because he did something horrible, you can't because you said the policy was 90 days. It enforces that. So, the only way to do that is to create another policy that says one day, move Fred into that, and then this stuff will be deleted. That's what they went to do. And unfortunately, they did the opposite of what they intended to do. They moved the entire company into the new policy, deleted all private chats across the entire company. I think it was like 150,000 employees in a heartbeat, and then you got Microsoft, Microsoft is like – the product did its work as designed. You told us to delete everything, we deleted everything.

[00:23:00] CS: Was that a misunderstanding of the rule that they had created? They just like created it – it was just a human error?

[00:23:07] WCP: Human error. They meant they meant to move Stephen to the new thing. Instead of moving Stephen to the new thing. They moved everybody into the new thing. Left Stephen on the old –

[00:23:18] CS: Like getting greater than in less than backwards or something like that.

[00:23:21] WCP: Exactly.

[00:23:22] CS: Obviously, we're already well into discussing backups, and human errors and bad installations and stuff. But obviously, since this is about cybersecurity, we want to talk about, let's talk ransomware and so forth. But I want to talk about the limits of any of disaster recovery in cases where a network has been compromised, taken over lockdown or duplicated or stolen. I mean, is it realistic to say we just won't pay the ransom? We've got everything backed up. We talked about that a little bit so far that you might think you everything backed up, but there's more layers to it than that. What are some of the sticky situations around ransomware and backup?

[00:24:04] WCP: So, it's not as simple as just saying we have a backup or we have a disaster recovery copy. So, boom, we don't have to pay the ransomware. I would say, you need a ransomware plan. The difference between a DR plan and a ransomware plan is that the ransomware attack is ongoing. Even once the attack has happened and you've been sent the ransom, the attack is still present, right? With a disaster, typically a fire happens or a flood happened, so you come back and you just do what you need to do. With the ransomware attack, you need to do things that you wouldn't normally need to do in a restore, in a regular DR. And the big thing is you need to determine the extent of the infection, and you need to determine it in terms of both its breadth, in terms of how many systems have, and I'm going to differentiate between infected and encrypted.

So, an infected system is just it's got the malware on it. It may or may not have been deployed.

[00:25:17] CS: It could be there for a long time without doing anything.

[00:25:19] WCP: Exactly, exactly. The dwell time, the median dwell time is something like 60 days, which is in average is actually much higher. Anyway, you need to figure out which systems have been infected, and also which systems have been encrypted. And that process alone can be daunting, right? We had a really great guest, it was actually sort of, ironically, ironically isn’t the right word. But interestingly enough, it was Spectral Logic, the company, which is actually a tape backup company. They make tape libraries. They got attacked by ransomware and they believe in eating their own dog food. So, they have a tape-based DR system. But their challenge was not the actual restore. Their challenge was figuring out what was infected. And as I recall, he said it took them about two weeks to do just that process, because it's sort of like rip out all the wires, don't let anything talk to anything, and then just figure out who's infected. You got to be very surgical, because all it takes is one system.

So, there's that issue, that you have to do that before you begin your DR. The other is, we need to figure out how far back, what's the dwell time? When did we get this ransomware? Because two things, one is, I can't just restore – let’s say we have a simple ransomware attack, where it's Wednesday, and Tuesday night, they encrypted all my stuff, right? It's a simple ransomware attack. I restore to Monday, and everything's beautiful. I just need to find out where the where the actual ransomware malware was, and delete that. Easy-peasy. That's early days. What we have now are much more sophisticated attacks and they tend to do things like sit there in your environment, and silently encrypting stuff nobody's looking at.

So, if you have that situation, it's going to hurt to restore that. Yeah, so generally, this is a file server we're talking about, or someone's laptop. This isn't a database. You encrypt anything on a database server, and the thing's going to crash, and everybody’s going to know. But a big file server in your company or laptop, they start encrypting slowly the files that over time that no one's looking at, and then at some point, they get a collision with the encrypted file that somebody is looking at, and that's when they go crazy, and they start encrypting everything. That is a real challenge. I know that Druva, for example, specifically created a response to that challenge. I call it sort of a merge of archive and backup.

And by that, I mean, a typical backup, or typical restore, you say, I want to restore this system to last Thursday at noon. Boom, that's a restore. And archive is, I'd like all of the files from Fred, through his email system for the last three years. That's an archive. That's a retrieval is what we say in the archive space. So, what we did was we sort of combine the two and we said, alright, figure out when you got infected, and then specify, let's restore to the day before you got infected, and then restore up to the current point in time, and restore the last known good version of every file. And we figured out how to do that for file servers, most backup products have not been able to figure out how to do that.

So, with us, it’s a matter of clicking a few extra buttons during the restore, and then just letting the product take the time to do that. If you have a regular backup and recovery tool, you have two choices. Restore to today, and just know that you're restoring a bunch of encrypted stuff, and then just hope you could get rid of the malware. And then maybe do some onesie, twosie restores along the way. Or you restore to the day before you got the infection, and now you've got hundreds, maybe thousands, maybe tens of thousands of files that you have to figure out where they got encrypted the last known version. It's a nightmare.

There's that. The third issue is some of the more recent attacks are about exfiltration more than they are about encryption. I'll just be blunt if what your attacker has done has exfiltrated unencrypted sensitive data, there is nothing, that the greatest – design your greatest dream backup product. It can't help. Because they're just going to take your company secrets, or your sensitive data, or your customer sensitive data, and just post it all over the internet. You've got a very different business discussion. I want to come back to that in a minute, because that's much more of a cybersecurity discussion.

But the other is, they're also specifically targeting the backups. And we're going to come back to the thing we mentioned earlier, about – I can't talk about backups for very long without mentioning the 3, 2, 1 rule. Three copies of the data on two different media, one of which is somewhere else. That’s that air gapped copy that we talked about. If your only copy of your data is on a backup server in your environment, especially if that backup server is based on Windows, because that's, at this point, the primary attack vector. It’s not the only, they’re attacking Linux, they're attacking VMware, just just to a lesser degree. If you’re a Windows based data center, and you get a Windows based backup server, and your only copy of your data is there, and maybe on a replicated version that's also electronically accessible, what they're doing now is they're going in and they're actually exfiltrating your backup data, and then storing that off site, and then deleting your backups. And then coming to you and then hitting you with ransomware. They're saying, “We got all your stuff. We've already taken all your backups out.”

[00:32:03] CS: Yeah, go check your backups.

[00:32:05] WCP: Go check your backups, right? The only way to stop that is to make sure you have an air gapped copy. Now, just real quick, and again, this really isn't my space. But my opinion is that you need to be doing two things. You need to look at some kind of DNI system, where you have managed DNS, and one of the things that it should be looking for is the lookups for command and control servers. They're obvious to see if you know what you're looking for. If your product is designed for this. So, look for that. And then if you see that, stop it right away, immediately. It should be automatic.

And then the second is, you really should be using some sort of machine learning or AI to watch for exfiltration. Did you ever watch the TV show, Alias?

[00:33:11] CS: No.

[00:33:15] WCP: It was it was a big spy show, and it starred Jennifer Garner. That might have been one of the reasons I watched it. But anyway, it was a good spy show. There was this one scene where the IT guy, Marshall Flagman was his name, the IT guy comes running in. He's like, “They're downloading all the files off the server.” And he starts just flipping power switches to turn the power off all the servers. And the thing was, that was shoot, that was 15 years ago, and he was monitoring his outgoing data feeds. He saw a pattern that was unlike what he was used to.

Today, we can do that with software. If you're not using a DNI to watch for command and control servers, and if you're not using some sort of ML/AI to watch for exfiltration, I think you're falling down on the job at this point. Because if that happens, there's literally nothing anybody can do to help you.

[00:34:16] CS: Okay. Well, that entered my next question. I was going to ask about double extortion and then you basically laid it out there. I mean, it's a TV show, but it's not wrong, if you see it happening to turn everything off, right?

[00:34:30] WCP: Yeah. Not wrong at all.

[00:34:33] CS: It is kind of as simple as that.

[00:34:35] WCP: Go in there and slice all the cables on the back of the –

[00:34:40] CS: Yeah, we’ll deal with this later. Okay, so that's really great advice. I was going to say there's – yeah, so your primary recommendation is to have something that's monitoring whether data is leaving incorrectly. Is there any other sort of like encryption option where, you know, they try to exfiltrate the data, but they get it and it's just –

[00:35:10] WCP: That's a great question. We discussed this also on the Restore It All podcast. We had a host, who, by the way is a very tape, is a big tape advocate. By the way, I'm probably the most pro tape person at a no tape company, but tape has a place for me. It's really safe long-term storage. I just don't recommend it and have it for at least 10 years for day to day operational backup and recovery disaster.

My good friend Brian would would say that – that question you asked early, he wants a copy on tape. And again, it's not wrong. It's just costs money and time and effort and all that stuff. So, you were asking – darn it, I lost the thread.

[00:36:04] CS: I was just wondering if there was any kind of in terms of exfiltrating your data and then ransoming you for it, if there was like an encryption, any kind of encryption options out there.

[00:36:14] WCP: Basically, the only alternative that you have there is really strong item level, user level encryption, meaning that – because most of the encryption of data at rest, for example, the application sees the data unencrypted, right? Backup systems. See the data unencrypted as they're backing it up. They encrypted in transit usually using SSL, and then they encrypt it – they unencrypt it on the other end, and then they re-encrypt it using something like AES 256.

But in the on-prem world, and in the, I don't know, what would we call it, the live data, that data, if you're using on-disk encryption, like it's very common these days to turn backup or encryption on full disk encryption. But you turn on your laptop, you see the data unencrypted, right? The problem when we start talking about primary data, and we start talking about personal information, things that are subject to GDPR and CCPA, and things like that, is that if the application – if one application with one user ID can see all of the data unencrypted, then that means it can be hacked via that attack vector. So, what this one person was recommending is for that most sensitive of data to use item level or row level encryption in such a way that only – let’s say it is personal data, that it's got my personal data. Only if I am authenticated to the database with my user ID, is my data unencrypted for me. Again, I can't argue with that. But it is definitely an extreme measure.

[00:38:21] CS: It will be cost prohibitive and –

[00:38:23] WCP: Yeah, I'm sure there's software that will do this for you. But it's one of these things where is the cure worse than the disease? And then the other thing is, for everything that you do that for, you will, on the backup side, undo every advancement that we've done in the last 20 years. Why do I say that? Because all of the cool stuff that we do with backup, like it's based on this concept called deduplication. Where I look at the data, I chunk it all up, I run some cryptographic hashes, and I figure out what data is new and unique, and then I use that to significantly reduce the amount of traffic I need to replicate it. So, the problem is, when you turn on encryption, dedupe looks for patterns, encryption gets rid of them. So, that's what I mean.

[00:39:21] CS: That’s a little much.

[00:39:23] WCP: Yeah. If you feel that you need that, then use it on your most sensitive of data, and then just realize that backing that data up is going to be a real challenge, at least using modern tools. To which my friend, Brian said, “Well, we're just going to put it on tape.” And I said, “Okay.” I can’t argue with it.

[00:39:45] CS: Yeah, you can walk around all day in a suit of armor and you're not going to get punched, but it's not much of a life.

[00:39:53] WCP: Exactly.

[00:39:54] CS: So, let's sort of move over from the logistic have storage recovery into the sort of work and career perspectives. This is Cyber Work podcast, so let's talk the work of it. What are jobs like for people wanting to get into backup and recovery as a career? If a listener was hearing us talk and is excited about all these different options, and all these different ways to improve their backup systems in their workplace, what types of hard and soft skills should they be attempting to sharpen to make themselves a good candidate?

[00:40:27] WCP: Well, first off, I would say, welcome to a really, really small club. One thing that has been consistent in my entire career is that nobody wants this job. It's a real challenge, actually, historically. So again, back in the day versus now. Back in the day, it was really hard to get someone to be the backup person, right? The reason I got my first job, in IT, was because a guy by the name of Ron Rodriguez wanted to get done at the job. He said, “I don't want to do this anymore.” He’s like, “Who's the new guy? Oh, Curtis? Curtis, you're the backup guy now.” Honestly, I only ended up initially making this a career because through a series of happenstances that I did not control, I just never got out of it. And suddenly, I realized that I knew so much about something that –

[00:41:27] CS: Yeah, once you get good at something, you're like, “I might as well keep going.”

[00:41:30] WCP: Exactly. So, back then, it was a real challenge, but it was a good way to get a job in IT. Today, I would actually advise a person against making this like their career. For a lot of reasons, I would much more advise them to be focused in cybersecurity, right? Because that's never going away. Storage, management, and cybersecurity, and network management, these things are never going away. Backup and recovery, data protection, disaster recovery, as a specialty is becoming smaller and smaller.

One of the reasons for that is backup, if you use modern tools, you look at – so, I didn't grow up in the in the vendor space. I grew up in the end user and consulting space. I came to Druva, just a little over four years ago. And I remember making the joke that I came there, and I came there to do what I'm doing now, which is being an evangelist for the company. But I remember making a joke of like, all of the skills and stuff that I had built over the last 20 something years, they were all worthless, because none of them apply to our customers. All of this stuff about how to design a backup system, how to decide, how much throughput you need, and what type of tape drives, what type of disk drives, what kind of deduplication, all that stuff for our customers, they don't have to do that. They just have to tell us how much stuff they have, and buy the right license, and we do all that backend hard work.

I think that as the world is saasifying, if I can make up a verb. As as we saasify, more and more of the IT world, and as we cloudify, more and more of the IT world, the skills that make a good backup and recovery person become actually much less needed. So, I would actually advise someone against it. What I would say is, look at – but if you are in it, if you are deep in the backup and recovery world, I would say make friends with your information security professional and learn the things that can damage your backups and apply those and then just sort of slowly increase your skill set on that side of the world. I hope it doesn't sound like I'm sucking up to your audience.

[00:44:25] CS: Please do. It works out well for us.

[00:44:32] WCP: I'm just saying, it's not pandering. There will be more attacks, there will be more sophisticated. Just in the last four or five years, the degree to which ransomware has become much more sophisticated. The exfiltration thing simply wasn't a problem back in the day. And now it is, and it's just only going to get worse.

[00:44:55] CS: So, conversely to that, if listeners are in the cybersecurity space, is there a benefit to them learning on the other side of the fence? Getting some some knowledge about backup and recovery and so forth?

[00:45:09] WCP: Yes, I do think, because of the problem that I mentioned before, and I say it as a joke, but it's true. And that is nobody wants to shop, right? Nobody wants to be the backup person. Because of that, you have a continuous revolving door in the data center, if you still have one, of the person who's responsible for backups, which means that, that person is quite possibly a junior person, and quite possibly knows nothing about data protection. I'm sorry, about data security.

So again, I'll give the same advice, the information security professional, befriend the person whose job it is – first off, they probably need a friend, because they're new to the company. Befriend the person whose job it is to do backup and recovery, data protection. Help them without being a threat, per se, help them understand the risks that their data is under. Help them understand why. If they don't know what the 3, 2, 1 is, Google it, have them read it. If they don't know what a bad actor is, if they don't know what an insider threat is, if they don't know what exfiltration, encryption, and all of these things. If they don't know what those things are, help them understand what those things are, so that they know what's at risk in their backup and recovery system. Because quite possibly, they don't know any of those things, and they're doing really, really dumb things from an information security perspective.

[00:46:52] CS: Well, I mean, we're always going to talk up in favor of cross departmental collaboration anyway. So, that's a great advice right there. I mean, you always have to learn from someone who's doing something near to you. You pretty much said that storage and recovery and long-term backup and stuff is starting to phase out. Can you talk about what you think it'll look like in 5 to 10 years? Do you see a terminus point at some point where this is, people will talk about this, like they talk about Fortran or something like that?

[00:47:21] WCP: There's a lot of Fortran out there.

[00:47:24] CP: Yeah, we found that out last year. We got a whole lot of people come back into active service here.

[00:47:30] WCP: Exactly.

[00:47:30] CP: It’s a very old language.

[00:47:31] WCP: I think that there's a personality, [inaudible 00:47:38] his name's Mark Tumi and he makes a quote of saying that, “There's going to be a nuclear holocaust. And after the nuclear holocaust, you'll find a guy selling mainframes and tape drives.” They're going to be around an awfully long time.

I do think that the world of cloud-based data protection, there will be more and more things like Kubernetes and containers. If you're not familiar with Kubernetes and container, it does to VMs, what VMs did two physical systems. And it really, really – so VMs broke back up. If you didn't know that, when VMs came out, it broke back up literally overnight for anybody who started. Like, “This is great. We can have system. We can have like 20 systems.” Meanwhile, you had a backup system that thought it was still running on 20 systems.

Containers break backups if they were based on a VMware or a physical system perspective. The good news is that containers and Kubernetes and things like that, it allows an environment to define what they consider an application. Too many times over the last decade or so we thought of as a database as an application, a web server as an application. And it never really was. An application was a collection of things that do a bunch of things that accomplish a common goal. Kubernetes allows you to do that, allows you to define what an application is, which means that your data protection system, I think, over time, will more. If I know that Druva has worked on this, where when we look at Kubernetes support, we're able to protect an application and all of the things that comprise that application. And I think you'll see that more and more.

The other thing is that literally, the world is moving to cloud. The whole world won't, there will still be people and environments where it's more appropriate to be in the data center. But I think, the bulk of workloads will be in the cloud. And the way you do data protection in the cloud is extremely different than the way you do it in the data center. And there are a whole lot of backup products that were based in the data centers, that were built for the data center. And even in the last five or six years, there were backup products that were built for the data center. I can think of some of our competitors that built products to solve those problems that we had in the data center. But those those products are now data center backup solutions for what is increasingly become a cloud-based workload.

So, I would say that, I think you'll see more and more companies like Druva, that will be doing cloud-based protection for cloud-based workloads. I think, it already has become a lot easier. I think that backup and recovery, data protection, disaster recovery, and all of those will become much easier. I do just want to comment on one thing. You said it twice in a podcast, I'm just going to throw out this thing. Long-term backup. So, this sets off a bad place in my brain. So, long term means archive.

One of those things that like 3, 2, 1, backup and archive, or two, just like backup and DR are two different things, backup and archive are two different things. So, when you're talking about really long-term storage, most backup products are really bad at long term storage. Because just really quick, if you store a backup of your laptop, for let's say, 15 years. 15 years from now, if you needed something from that laptop, you're going to need to remember what that laptop was called 15 years ago. And since we're talking corporations, you're going to need to remember the name of the employee, and the laptop and the name of the server, and you're going to have to have access to the application that was running 15 years ago. That's, if you have backup.

If you're doing true long-term storage, then you should be using an archive system, which is designed to hold data for really long periods of time. That world is also changing as well. But it is a world where tape still reigns supreme, because tape is actually really good at holding on to data for really long periods of time, and it's also ridiculously cheap.

[00:52:52] CS: You don't need to tell me. I got a whole basement full of tapes. Some of them which I've had since I was a teenager.

I mean, we're getting close to an hour here, and I don't want to keep you all day. This been completely fascinating. So, as we wrap up, you want to talk a little bit about the Restore It All podcast? What are some topics you cover? And if a person wanted to dip their toe in, do you have a particular episode that you recommend as a good introduction?

[00:53:16] WCP: So, what do we cover? We cover backup and recovery. We also do talk about cybersecurity. We talk about archive, long-term storage. We talk about disaster recovery. We talked about barbecue, a couple of the episodes, we talk about barbecue. And oddly enough for a technology podcast, we actually covered COVID quite a bit. When COVID hit, we just felt it was weird to just keep on talking about technology. So, we actually had an expert on to talk about that.

From a favorite – you know what, I should have prepped. I shouldn’t have a favorite episode. I'm going to say, my favorites, I’ll answer two episodes. My favorite episode of the year was, I forgot the actual title. But it says the person, the admin deletes his entire environment and then tests his backups. It's a fascinating story of a guy. He's the head of IT for the the island of – darn it, it's in Alaska. It's the largest island in North America. It's escaping me. Anyway, he deleted everything. He had a reason why he did it. But he deleted it. And then he went to test his backups. It has a happy ending, but –

[00:54:46] CS: It took a while to get there.

[00:54:49] WCP: The best yet we had this year was we actually had on the guy who coined the term, 3, 2, 1 rule. His name is Phil Krug. He's a digital photographer. And since the 3, 2, 1 rule comes up in almost every episode, talking to the guy who coined the term was pretty cool. And then finally, I would say, in the beginning of last year, we had three or four episodes, where we talked about an event where everything went wrong. So, there was this thing called the OVH fire. So OVH is the largest cloud provider headquartered in Europe, and they lost a ridiculous amount of servers, and apparently, including data for companies where the company had paid for backup, and they're now just literally a few weeks ago, I saw that they're beginning a class action lawsuit against OVH. Those episodes are great, because we dig into the sort of the details of what was happening. And you learn things like how to sort investigate what your company is doing for you when they claim to be.

[00:56:13] CS: All right. So, as we wrap up today, Curtis, tell our listeners about Druva. What services you provide and what products or projects you're excited about in the new year?

[00:56:23] WCP: So, Druva does data protection, data resiliency, and DR for most of the modern workloads. So, data center, cloud workloads, SaaS workloads, laptops, mobile devices, including Kubernetes containers. And we do that as a SaaS service, which means that you don't have to install any infrastructure, either install it in a cloud or in your data center, you just basically, you put it an agent, and you authenticate us to the appropriate place, you authenticate us to your SaaS provider to your IAS provider, and then we just do all magic. You one bill, you pay – it’s per gigabyte, per month, and it's a deduplicated gigabyte. So, we do global dedupe across your entire environment. You pay one bill per month of – and obviously, you pay in advance and we credit your account. The beautiful thing is, you never pay for something that you don't use.

So, if you over provision, you have money left over at the end of the year, which is rolled out into the next year. Or if it's laptops and mobile devices, or SaaS services, typically, that's a proceed license, because that's the way that world likes to be charged. And we do DR in the cloud. So, as I mentioned, we can restore your entire VMware environment in 15 to 20 minutes with your RTO. We can make a 15 to 20-minute RTO with a one-hour RPO. But literally, you push a single button, and then everything comes up because we've already restored your data in the cloud.

And then I did mention about this enhanced ransomware recovery service that we have that again, it's an extra cost option. But if you do that, you can literally press, I think it's like three buttons. Tell us the beginning, tell us the end, and tell us to go and we'll restore data, the latest unencrypted version of every file.

What am I excited? Honestly, I got some stuff that I can't talk about. There are some features, and they’re cybersecurity based. I hinted at them, if you listen to me talk closely, I hinted at some of the features that –

[00:58:52] CS: Listen to this a second time for the clues.

[00:58:55] WCP: We want to make sure that, you know, we already have the data air gap. We already have this concept of the recycle bin which most backup products don't. So, if a customer comes in, and they accidentally delete their backups or a rogue admin actually delete their backups, we can actually get them back for you, for a user definable period of time. We want to take that the next step. So, I want to say much more than that. I will also say that Druva decided to sponsor my book, it's an independent book. It's an O'Reilly book, you should be familiar with that world. But your listeners can get a free eBook copy of it by going to a druva.com/podcasts. There you go. Modern Data Protection from O’Reilly.

[00:59:45] CS: Love it. Well yeah, if our listeners want to learn more about Mr. Backup and Druva, so you said druva.com/ podcast. Any other links you want to tell us about?

[00:59:53] WCP: Of course, backupcentral.com and you can search on, the full name is Backup Central’s Restore It All. We even have a theme song, which is a parody of Adele's Rolling in the Deep. You remember the you could had it all, in the chorus? So, in mine, it's you could restore it all, but you erased it. The sing is my daughter, by the way. She’s very good.

[01:00:25] CS: At least listen to the – even if you don't know anything about data recovery, listen to the beginning, listen to the theme song. Well, Curtis, thank you so much for joining me today. This was incredibly informative.

[01:00:35] WCP: It was a blast.

[OUTRO]

[01:00:38] CS: And as always, thank you to everyone listening to and supporting Cyber Work. New episodes of the Cyber Work podcast are available every Monday at 1 PM Central both on video on our YouTube page, and on audio wherever you get your podcasts.

I'm also excited to announce that our Infosec skills platform will be releasing a new challenge every month with three hands on labs to put your cyber skills to the test. Each month you'll build new skill ranging from secure coding, to penetration testing, to advanced persistent threats and everything in between. Plus, we're giving away more than $1,000 worth of prizes each month. Go to infosecinstitute.com/challenge and get started right now.

Thank you so much once again to W. Curtis Preston, Mr. Backup, himself, and thank you all so much for watching and listening. We'll talk to you next week.