DNS exploits, concerns and easy fixes

Chris Sienko: Hello, and welcome to another episode of Cyber Speak with Infosec Institute. Today's guest is Andrew Wertkin, CTO at BlueCat Networks. Among his many jobs and professions, andrew has extensively worked with problems of DNS security issues, a topic of great interest to many of our students. So we're going to talk today about some pervasive DNS security concerns as well as some easy fixes that your security department can take advantage of right now. Andrew Wertkin leads the product strategy, product management, and engineering teams across BlueCat's product portfolio. He is also responsible for BlueCat's research lab focused on innovation, technology roadmap, and vision.

Before joining BlueCat, Andrew built systems focused on the protection of intellectual property across multinational corporations and multi-tiered supply chains. He was Chief Technology Officer for PTC following its acquisition of MKS where he also served as CTO. Prior to that, Andrew was CTO and Co-founder of Synapsis Technology. Andrew holds a BA from the University of Pennsylvania and graduated the University of Pennsylvania's Neuroscience Graduate Program. Andrew, thank you for being with us today.

Andrew Wertkin: Thank you for having me.

Chris: So let's start out with how you got involved with computers and security. Is it true you started in neuroscience before you moved to tech? And what moved you from this fairly different path onto the one that you're on now?

Andrew: First of all, I didn't finish the University of Pennsylvania's PhD program in neuroscience, which is part of the story. I would like to say it was because I was doing a lot of work in neural networks and followed that to a passion in machine learning and it was a natural cause. But actually as much as I loved neuroscience, what I found myself just trying to solve everything with software, very much enjoyed the process of writing software, creating technology, and found myself guiding my research based on what I could write in software. And at some point I had an idea, and I decided to take a leave of absence from graduate school to see if I can do something with it. And through a series of fortunate and unfortunate events really carved out what I really reflect on as a great career in technology. So it was just about a passion for software and products.

Chris: What was it along the way that ... you said you just kept trying to solve problems using these sort of software methods. Like what were some of the problems that you were working on in neuroscience where you said, okay, this isn't working for me?

Andrew: Yeah. For sure I was working on things like computer vision and things that software was the main domain. But actually what my core research was around was neurodegenerative diseases specifically around Alzheimer's disease and Parkinson's disease. And I was working in a lab that had a great deal of funding, and we had some amazing laser microscopes. This is 1991, 1992, and you'd sit down at this incredible multimillion dollar microscope and manually move fields and then sit there counting amyloid plaques on the screen for instance. And there was a serial port and a manual and two weeks later and without sleep, I have wrote software to automatically scan through and then do some image analysis to do the work for me. And that was way more interesting to me then than what I was actually doing.

Look, the brain's an amazing thing, and I'll always be fundamentally just completely unsatiated. I just want to learn more about, but just love software. And then transitioned to, I thought I had a business idea. I was incredibly naive. I took a year leave of absence from graduate school. That idea did not work but I couldn't go back to graduate school. I was on a grant and the grant didn't kick in until the year was over.

So I took a contracting job at General Instrument, which was then bought by Motorola. And the next thing I knew I was in Taipei at their high volume manufacturing plant solving engineering to manufacturing cost overrun issues with the team there by writing software. And I was solving things quickly, as opposed to this world of research, academic research, which I loved. But there your hope over the span of your career that you make some sort of impact as opposed to you're at a party two weeks later because we've figured out how to deal with millions of dollars of excess and obsolete inventory. So I just liked the power, especially back then of how quickly you could use software to solve real problems.

Chris: Now moving along to that, what was your career path to CTO of BlueCat like? Going away from neuroscience and technology and security, you said you were in Taipei, you were solving problems over there and stuff. What were some of the other jobs and positions and career steps along the way that got you to where you are now? Did you have any other major career about faces between then and now, other than neuroscience?

Andrew: Yeah, a couple. So I stayed at General Instrument and then Motorola and built a lot of software there and built a team there. And I found that I was solving what I thought were some pretty interesting problems but I was part of the machine there. And what I really wanted to do was build something for many companies. I wanted to create commercial software. So I had come up with what I thought was a pretty good idea and was able to work out an agreement with Motorola many, many years ago to help fund a startup and then also be customer number one, which was a lesson that stuck with me my entire career. Ensuring that you're building technology for somebody, for a real user that you've ... Today we call it lean startup, but someone to continually test hypotheses with and hopefully many companies to continually test hypotheses with as opposed to assuming you're right.

So I ended up building a company that sold software to help customers mitigate the risks. Doing a whole lot of math around a lot of environmental laws that were creeping into electronics, like end-of-life vehicle and the restriction of hazardous substances, getting lead and hex chrome and cadmium and other substances of concern out of the supply chain, which is a fairly difficult thing to do, especially in electronics because most companies don't build their own products. Their different components are built all around the world. And that led to extending that to things like carbon footprint analysis and some other ways to analyze the supply chain.

I ended up selling that company to this company PTC in 2008-ish, and then ended up working with this other company that you mentioned in my bio, MKS, who was again focused very heavily on product manufacturers driving innovation via software. There was a major change in the amount of software engineers versus mechanical or electronic engineers hired at companies from ... It's Marc Andreessen's Software Eating the World. I mean when automotive suppliers hire 20,000 software engineers and no more mechanical, electronic engineers, you get a change in a dynamic. And so I joined a company where we very much focused on that. That company was also acquired by PTC. So at that point I became its CTO.

When I was at PTC the second time, I was always focused on building complex software to manage engineering processes and very focused on the protection of intellectual property and especially across global geographical boundaries and across supply chains. But the second time I was at PTC as their CTO, as the CTO of a public software company I had access to and the ability to have some great conversations with the CTOs of our customers, the John Deeres of the world and the Airbuses of the world, and understanding where they're trying to invest and where they were putting their R&D. And a lot of that was what we'd call today Industrial IoT. Back then it was just their product strategy, but it was John Deere moving from selling tractors to selling high-tech agricultural farming solutions where tractors are autonomous, and they're constantly measuring soil quality. Ultimately John Deere wants to sell crop yield, not tractors. And that requires bridging my enterprise with things I might not even own anymore. These tractors are IP connected. They're streaming data back. Who owns that data? How do we secure it? We can do over the air firmware updates.

And that's my break into this world I'm in now. But as I talked to these companies, they were struggling certainly on the R&D side and lots of opportunity on the R&D side. But the other area they were struggling heavily was the boundaries of my network, of what's my company, and what's the internet? Or what's my company and what's my customer's networks are starting to dissolve. They're starting to change.

And so how do I think about security now? How do I bridge IT and OT? How do I continue to participate with the product I sold a customer in a way where I can continue to gain intelligence about how it's being used, what it's doing so that I can build better data-driven strategies and provide better solutions to my customers? But how do I leverage IT? How do I leverage my network, my technology to do that and really start deep diving down in that?

Somewhere along the lines, the CEO of BlueCat who is a former colleague of mine, we worked together and sold this company MKS together. He was at BlueCat already and he was telling me about BlueCat and at some point what I was interested in and what he was trying to do aligned appropriately. And I love being at companies this size. I had a great time at PTC. I hope I had a great impact there, but big public company versus company where I can create a higher impact. This is what excites me. It's being able to help transform a company into different types of markets is very exciting for me. So I happily came here about four and a half, five years ago and have been here ever since.

Chris: Great. So one of the things BlueCat specifically deals with is DNS security, and that's mainly the focus that I want to have for today's interview. So let's start with a big philosophical question about DNS security. Why in your opinion has full adoption and implementation of the DMARC, the Domain-based Message Authentication, Reporting and Conformance anti-phishing standard been slower than anticipated or has it been slower than anticipated?

Andrew: Yeah sometimes I think if we can just stop using email altogether, my life would be easier and the lives of security professionals would be easier. But that's obviously not going to happen-

Chris: The horse is out of the barn on that one. Yeah.

Andrew: Yeah, exactly. You can ask the question about DMARC, you can ask the question about DNSSEC ... There's some level always of setting it up isn't specifically complex, but who's authorized to send mail on my behalf? What are my IPs? Set up policies ... What is the impact of inbound and outbound? There's a change management lifecycle there that has some level of complexity, and it's one of the systems where broad, broad adoption is what makes it work. And so I think some level of it is complexity or not.

Andrew: For instance, many organizations are moving to Office 365 or G Suite for mail. Why isn't it a mandatory process as I bring my domain on? If I'm just using Office 365 as adopting their domain, no problem. But if I'm bringing my domain there, there's work for me to do. Well, part of that work is in what they can't control, which is my DNS. But maybe they should enforce that I make those changes to my DNS so that it is more broadly adopted. So I don't think it's anything specific to the solution. I think it's more just around enforcement. And yeah, I think it's roughly around that.

Chris: Yeah. Yeah. Yeah. So to that end, what are some of the biggest DNS security weaknesses or maybe just the one biggest DNS security weakness currently being exploited by hackers? As far as you can tell?

Andrew: So I don't know that there's a one, and let me explain it this way. The one certainly the market is very interested in is for instance, DNS tunneling. If I can use DNS as a mechanism to have a conversation, exfiltrate data, do command and control, and I don't have any controls in my organization that's looking at DNS. I might have a well architected DNS so people can't go out to resolvers on their own. It all floods through some server in the DMZ. I might control the flow of traffic, but I may not be doing that inspection. A lot of our customers believe they've segmented networks for instance, but DNS ends up being a bridge with things like tunneling. So there's a lot of concern around tunneling, certainly in the customer base.

The way I look at it is a little bit differently though, which is DNS is being such a control point for connectivity and internet connectivity. It is used by those trying to compromise because they're trying to build highly available backend systems that are scalable, that aren't easy to take down. So they're using DNS the way it was intended to use DNS quite often. And so the biggest hole from my perspective, is not inspecting the traffic because again, it's being used the way it was intended in many cases. But we're just sort of naively sending it out or naively answering queries.

I would say DNS is like a chump, it tries its hardest to give the right answer to every question until its resources are exhausted, and it falls over. But it's doing its job, and I think that's the biggest risk with DNS is that it's a chump.

Chris: Hmm. Wow. That'll be the pull quote of this video. So I mean it sounds like you're moving in that direction already, saying using these services that use DNS the way it's meant to be used. But what would you say are some easy fix solutions to common DNS security issues that you're surprised more security departments don't take advantage of? Is that one example?

Andrew: One example for sure is just simply allowing DNS traffic out without any inspection or any policy at all. Or at the very least companies should have some sort of response policy. So if I subscribe to threat intelligence or I get some threat intelligence from my vendor, and I know certain domains are bad out there, then don't let people resolve those queries. It's a highly efficient way to block access. I haven't tried to create a TCP connection out somewhere. I blocked it with very little data exchanged on the network. Highly, highly efficient. So there's some bare minimums people should be doing for sure.

But wholly, a big part of it's around visibility. How can I access this data? How can I access the data correctly so I have client attribution and I know who is trying to do what? And if I can gather that, then how can I add more context? If I know it's a point of sale machine, and I know that point of sale machine should only be looking up 83 different DNS queries and the 84th is Google.com, pull it out of the wall. It's compromised because I know what it's supposed to be doing. So there's so much more that can be done with DNS, but at the very least enterprises, people should have some visibility into the DNS traffic because there's a such rich signals in that data both in terms of the intent of the devices and in many cases the actual communication of those devices.

Chris: Okay. So let's move one level up and talk about general strategies for your organization. Say you're an organization that worst case scenario has just been hit by one or a series of DoS or DDoS attacks, and you realize that your DNS security strategy is severely lacking. So what are some rudimentary solutions you can implement immediately for a low cost and quick implementation? Until you can harden your DNS strategy. What would something you could do today that would increase your security strength until you can build it up in a larger way?

Andrew: Yes, when we think of any sort of denial of service, we think of it inside out or outside in. So from the outside in perspective there are some distributed attack, denial of service attacks, specifically in this case targeted your DNS. Then you need scale, and you're not going to get scale if you're housing some DNS servers in your company. And there's many, many solutions out there to provide global scale to DNS, and we suggest a combination of using a service as well as have some servers because somebody might attack that service for a reason nothing to do with you, cause issue for that service. And you need two different strategies there.

So for sure you should have some cloud scale of your external DNS so that it can't be taken down so easily. There's lots of companies. We have a service but lots of companies who have services to do just that. The more interesting one, at least from our perspective, is the inside out one which is either to take you down or because some devices on your network got infected with, I don't know ... Mirai was a good example of it a couple of years ago.

And so now unwittingly you are participating in a DDoS attack on somebody else because some malware has been installed on one of your devices that's generating a ton of DNS queries. One, you don't want to be unwittingly part of that attack. But two, eventually that's going to take down your external recursor. Eventually you're not going to be able to send DNS queries out anymore so it's going to harm your company. And that's an area where we don't see a lot of people looking for that on their internal side. So there has to be beyond just some visibility, there needs to be the appropriate monitoring of DNS. It's not that complicated to do. Where are the anomalies? What am I seeing now that I didn't see before so that I can cut it off?

Chris: Now do you think in general, most enterprise organizations don't take the concept of DNS security seriously enough? And why do you think the time and resources aren't being applied to these problems the way we do to more glamorous or easily solvable issues?

Andrew: Well, I think one of it, and I guess it's one of the ways I look at this market, I mean we're a DNS company. That's what we do. We're not a security company. I mean we are a DNS company. We're highly relevant to cybersecurity. We're highly relevant to networking. We're highly relevant to servers and cloud deployment of technology. We're relevant across the board. And so you need us. Whether it's BlueCat, one of our competitors or open source or some free software. If you're running a company at any sort of scale, you have a solution to DNS. And so we believe wholeheartedly that you should leverage what you need already as opposed to putting a bunch of other things on your network that may provide similar capabilities, may provide different capabilities. But we find a lot of weariness in the buyers of buying thing after thing after thing, using a bit of each and not necessarily building a comprehensive defense and depth strategy, but rather a bit of a hodgepodge because vendors come in there potentially overselling what they can do.

So I don't know if this makes sense or not, but a bit of it is, it's a fairly simple and robust way to increase your security posture. But it's not the next best thing. It's something you already have, you know?

Chris: Right.

Andrew: So it's this odd position, and now look, there's mechanisms and more mechanisms now to encrypt DNS to [inaudible 00:22:04]. But by and large in enterprises, this stuff is not encrypted. It's right there. There's lots of companies out there that try to harvest this data passively inside of an enterprise to help a company do something with it. But DNS as a control point is very powerful. And finally, and I think very importantly, the number one requirement for DNS inside of an enterprise is reliability and uptime of service. And that's hard to achieve. It takes operators to achieve that. We believe we make it easier for our customers, but still, it's the number one requirement. If DNS is not available, compute stops working. That's bad. And so there's also a hesitancy to augment a DNS deployment architecture in any way that might disrupt its availability.

Chris: Well to that end, could you tell me a little bit about BlueCat's DNS Edge, which does DNS monitoring? How does it work and what processes does it use to remediate cyber attacks?

Andrew: Yes, so our Edge product is really all about the first hop. I mean we believe there's a lot of value that we can achieve the closer we are to clients. And so as that first hop in the DNS chain, we can one, harvest all of that data and harvest it in a way where we naturally have client attribution so that we can look for anomalies per device as opposed to in a fire hose of DNS data. So get it closer to the clients so that we can put policies around things like networks or types of devices. So Edge is a product that is a highly innovated DNS server that's able to process rules around the query stream coming through it so that we can look for, identify specific types of data that may or may not be threats. But also do stuff that's very DNS-y that has nothing to do with cyber, and basically synthesize responses based on context as opposed to simply just going back to a zone file and getting an answer.

And maybe that was too low level, but basically we're going to provide all that visibility to the DNS data. We'll make sure, I don't care if there's trillions of queries or billions of queries, we'll make all of that data available. And then we have a platform where we're doing the appropriate analytics on that data so that we can generate the value around cyber. And then also we can deploy policies, and those policies are critical to make DNS as a control point inside that network as well.

Chris: I believe you are about to host a webinar or maybe you already have titled Looking For Threats in a DNS Dumpster Dive. Could you give us a little sneak preview? As people harvest their DNS server log data in search of anomalies or red flags, what sorts of things should they be looking for?

Andrew: Yeah, it's part of what we're trying to solve for with DNS Edge was we have a lot of customers or companies I know that dump all of their data in their SIM or somewhere. They're trying to harvest the DNS data day to day, and they get overwhelmed because there's a ton of it. And most of it looks fine. So how do I start?

And so this will be the first in a series of webinars, and this one probably is going to be the most rudimentary. In other words, this one's more of a scatter. Here's the different places, start looking. And we go through some basic concepts like DNS query types, and what are normal query types and what are abnormal query types? Especially in the context like why are these devices on networks that are provisioned for end user clients looking up Amex records? They're not mail senders. They shouldn't be looking up those records, as an example.

Also though we do break down what is DNS tunneling? Go through actual tunneling exercises. Talk about what the hallmarks of those might look like, and we'll be talking about I think domain generated algorithms as well. So how domains can be generated algorithmically, and therefore malware can keep evolving as domains may or mayn't be blocked. And find that combination of domain that's live on the internet that isn't blocked so I can go communicate outside of the company, and look for some of the hallmarks of that.

But we'll be looking at real DNS data. I tried to keep the marketing slides rectangles with words in them, pointing to other rectangles with words in them to a minimum. But again, we'll follow this up with some deeper dives in some of those areas as well.

Chris: Very cool. So to wrap things up a little bit here, as DNS attacks get more sophisticated or complex or prominent or prevalent, where are the next wave of DNS attacks coming from? What should organizations be doing now to prepare for the future?

Andrew: Well, for sure, I mean it depends on the context of the attacks. We're big proponents of, especially on the external side, DNSSEC, [inaudible 00:27:24]. We obviously believe a great deal in making DNS data visible, making it harvestable so that it can be used as part of a cyber strategy, whether it's with our product or somebody else's product or your own data lake or you're stuffing it in Splunk. This stuff is important to look at and start modeling for anomalies. A average end user compute device does, I don't know, 2300 queries per day and over a month looks at 3000 different queue names. It's not hard to spot anomalies if you have access to the data. The averages, the norms become pretty clear.

Chris: So it sounds like a lot of it is intuitive as much as tech based.

Andrew: Yeah, for sure. And look, I'm skipping over ... I mean a proper enterprise DNS architecture is critical. Do I have a hardened infrastructure? Who is allowed to send queries outside the enterprise? There's a lot of hygiene that must occur, and in many cases we don't even see that hygiene. But by and large, large enterprises, enterprises, the hygiene will be there. They're not taking the next step. But absolutely look at the DNS data flows inside of your organization and understand if there's weaknesses in the deployment architecture. It's not hard to construct a DNS deployment architecture inside of an enterprise that is fairly fail-proof where there's available DNS servers for the clients and the ability to ensure that if any node fails, I'm still okay. Make sure DNS is survivable, make sure it's hardened.

So there's a great deal of basic policies, and we certainly work with our customers to help them create that hygiene. But outside of that, from our perspective it's all around visibility so that I can harvest that information. I can marry that information with other data sources. How old are these domains? How long have they been on the internet? Do these domains represent any sort of risk? There are some top level domains out there that are brutal. I mean they are where 90% of the seen domains are related to spam or malware. Take the basic steps. Look at your risk profile, and if your risk profile is anything other than Wild West inside of an organization, then lock those. So there's a lot of basic things that can be done as well.

Chris: Andrew, thank you for joining us today. This was very informative.

Andrew: Thank you for the opportunity.

Chris: All right, and thank you all for listening and watching. If you enjoyed today's video, you can find many more of them on our YouTube page. Just go to YouTube and type in Infosec Institute. Check out our collection of tutorials, interviews, and past webinars. If you'd rather have us in your ears during your work day, all of our videos are also available as audio podcasts, including this one. Please visit InfosecInstitute.com/cyberspeak for the full list of episodes. If you'd like to qualify for a free pair of headphones with a class signup, podcast listeners can go to InfosecInstitute.com/podcast to learn more about this special offer. And if you'd like to try our free security IQ package, which includes phishing simulators you can use to fake phish and then educate your friends and colleagues in the ways of security awareness, please visit InfosecInstitute.com/securityIQ.

Thanks once again to Andrew Wertkin of BlueCat, and thank you all for watching and listening. We'll speak to you next week.