Chris Sienko: Hello and welcome to another episode of the Cyber Work with Infosec podcast. Each week I sit down with a different industry thought leader to discuss the latest cybersecurity trends and how those trends are affecting the word of Infosec professionals, as well as tips for those trying to break in or move up the ladder in the cybersecurity industry.
Today we have a repeat guest, Andrew Wertkin, who is the CTO at BlueCat Networks. Last time Andrew talked about DNS security concerns, and if you’re interested in that topic, I urge you to go back and check out that episode as well. It was a really good one.
We’re talking about a new and hotly contested privacy technology called DNS over HTTPS or DOH. It’s backed by Google, Mozilla and Cloud Flare, and because it encrypts domain name system requests, it’s found approval from Internet privacy advocates. However, under regulations like the UK’s 2016 Investigatory Powers Act, DOH makes it harder to monitor terrorism and other illegal acts.
Today we’ll be talking about some of the ethical and procedural issues as well as ways in which DOH may change the way cybersecurity experts work. Andrew Wertkin leads product security, product management and engineering teams across BlueCat’s product portfolio. He’s also responsible for BlueCat’s research lab focused on innovation, technology roadmap and vision.
Before joining BlueCat, Andrew built systems focused on the protection of intellectual property across multinational corporations and multi-tier supply chains. He was chief technology officer for PTC following its acquisition of MKS, where he also served as CTO. Prior to that Andrew was CTO and co-founder of Synapsis Technology. Andrew holds a BA from the University of Pennsylvania and graduated the University of Pennsylvania’s neuroscience graduate program. Andrew thanks for coming back.
Andrew Wertkin: Thank you. Pleasure to be here.
Chris: To start with, could you walk us through the bare bones version of what DNS over HTTPS is. It’s a fairly new technology that could touch a variety of cybersecurity areas and workers in the coming years, but I don’t think that many people are familiar with the concept yet.
Andrew: Yeah sure. I mean quite simply it’s DNS questions and answers, instead of through the DNS protocol, through HTTPS. So, [inaudible 00:02:12] in HTTPS instead of DNS and DNS wire format. It’s … anything can be sent through HTTPS. In this case it’s a DNS question and a DNS response.
Chris: Hmm. So, what is DNS over HTTPS’ purpose? Breaking apart the what if’s and backdoor potential uses of it, what was it manufactured to do, I should say? There’s already a lot of talk about what the ramifications could be, but what is it actually for, according to its creators?
Andrew: Yeah, so DNS historically, with a couple of different options that weren’t used by the masses, is a clear text protocol. Therefore, anybody on the wire can listen, can snoop, can eavesdrop, can see what DNS questions are being asked, and what the responses are. And further, if, unless that domain’s signed in the DNS user, the [inaudible 00:03:15] is behind a validating resolver, they can be manipulated.
So, it’s a protocol that therefore has been leveraged to monitor. It’s been leveraged to … the service providers many, many years ago started monetizing misspelled domains when somebody went to a website to try to generate ad revenue. So, there’s been all sorts of use cases where that data’s been leveraged because it is clear text.
There have been other attempts to encrypt, and in fact, along with DOH, we often talk about DOT or DNS over TOS, which is another way, but in that specific case, it’s keeping with the DNS protocol and having a specific port. It’s still DNS as opposed to DNS over HTTPS, which just goes out through the [inaudible 00:04:10] for three.
Chris: I see. Could you talk to us a little bit about the evolving role of DNS, and as a result, the increasing importance of the security of the same. With everything being connected now, we can see what can happen when a DNS is under attack. This would seem to suggest that any change in the fortunes of DNS technology, is going to mean that cybersecurity skills are going to be more critical for professionals in this industry. Is that right?
Andrew: Yeah. I mean, DNS, as we talked about last time, and real briefly here, DNS is a super proxy for the intent of the user. If an address is being looked up, then one may infer that a web connection or different type of connections can be made to that server, which we looked up, and therefore it becomes part of the control plane of the Internet. DNS is a critical part of that control plane in the Internet, along with things like routing.
So, what cyber criminals or those who attempt to hack have figured out a long time ago, that DNS can be used not just to create scalable, resilient back ends to speak to client, for instance, for command and control, but we can also embed information, embed specific instructions, exfiltrate data. There’s all sorts of ways to leverage DNS from a cyber standpoint, and many organizations certainly most consumers aren’t taking steps to protect themselves from those.
Chris: So, we know that Britain’s national cybersecurity center, and assuming the U.S. government, have voiced specific objections over DOH, citing that encrypting DNS requests makes it harder to monitor illegal and/or terrorist activity. Are there legitimate concerns of malfeasance that could be committed under the protective cover of DOH do you think?
Andrew: Yeah, I think that the fact that DNS has been clear text has allowed it to be a simple way to monitor, and for sure, different … and it could be the UK or the U.S. trying to do it for, what they classify as good. It could be other countries where they’re trying to censor access to the Internet and access to Internet services, that might have a DNS firewall for the entire country, that is going to limit what can be looked up or change what can be looked up, or even change the responses coming back, and certainly log and monitor. And so, from a fair access to information, encrypting DNS through any protocol allows the removal of that simple command channel in order to monitor.
That doesn’t mean they won’t find others. Right now, there’s still, in many cases clear text headers of the host information in HTTPS, requests from a Web standpoint, for instance. And there’s mechanisms moving along in order to encrypt that as well, but right now there’s other ways to figure out where something’s going. Of course, you know the IP address it’s going to, and one can easily reverse lookup the IP address it’s going to, and start building out a blacklist that way.
So, there are other ways to do it. Blocking DNS, sorry, encrypting DNS makes it harder to use this command channel. And by the way, there’s other ways to get IP addresses other than DNS. So, there’s all sorts of other ways that people have used in order to get around potential censorship or monitoring with DNS throughout time as well.
So, I think it just makes it harder, and I think these governments are going to complain, because it’s harder. But at the end of the day, BlueCat works mostly with enterprises, and from an enterprise standpoint DOH is a bit of a nightmare. It’s a bit of a nightmare, because one, in this case, there’s less of a concern for privacy. I work at a company, and in many cases in most countries, I don’t have an assumption of privacy of what I use work-related devices to do.
But two, the network operators, not just the security operators, but the network operators leverage DNS for a healthy network. They leverage it to help them ensure the right service. They help them to make sure that people are getting to the right resources through the right data centers or through the right egress points to the Internet. It’s something that’s used from a monitoring health standpoint on the security side, but also very important on the net op side as well. So, it becomes a bit of a nightmare, because we’ve taken away the tool that’s already being used for that monitoring if everything is encrypted. But enterprises also do their darnedest to decrypt and are using all sorts of services today on the proxy side for web, and they’ll proxy these as well, and they’ll use the same mechanisms to figure out what’s inside of these requests.
So, it’s just more work and more decryption and more blocking and enterprises will block the IP addresses of most of the well-known DOH providers out there, regardless, because they don’t necessarily want to use the service.
Andrew: Right now, in enterprise, there’s a channel. If I issue … go to some website and therefore a DNS query is issued on my behalf by my browser through my operating system, I can’t go directly to the Internet. I don’t have a route to go [inaudible 00:09:56] to the Internet or go to Google’s 184.108.40.206 or anybody’s recursive layer. I have to go through the corporate services, finally hit a recursive server in the DMZ, and that will then go out. And that’s done because network operators are both securing their networks and also ensuring that the protocol is healthy and that web traffic is healthy.
Chris: Right, right.
Andrew: There’s an assumption that, because my ISP doesn’t see it or because my government doesn’t see it, I’m somehow private.
Andrew: But again, you’re not. It terminates and it terminates with the service provider who is leveraging this data for some use.
Chris: Yeah, I think this relates to, and I talked to Pete Zaborszky with BestVPN.com, and he was saying that a lot of people think that because they’re on a VPN, there’s just this complete anonymity, but there really isn’t complete anonymity anywhere, and as you say, if you’re going through this secondary service here, now your data is terminating at a less, possibly, reliable spot. Is there any way of shopping or finding out about your DOH plan to see whether they’re responsible, or is it just too new at this point?
And I haven’t reviewed that recently, so it’s more policies for consumers to review that they’re probably not going to review. But when I say … I don’t think … I think companies like Cloud Flare have very, really solid strategies. They want as much web traffic as possible, whether it’s DNS over HTTPs or anything else because they leverage that for their own business. Their business is trying to figure out the performance to the Internet so they can be a better deliverer of web content, of faster web applications.
So, even if they’re not … even if those for-profit companies aren’t leveraging it from a privacy standpoint, they’re providing a service and for that service, they’re gaining data that’s necessary for them to provide other good services. And I’m not saying that’s bad, but it’s … if everybody flips this on in Firefox and now … how much of the world’s normally de-centralized DNS traffic starts going to a single company. And form my perspective, just in a centralization of the Internet versus decentralization, that, to me, is a bit scary, because the choice is made for you and I don’t know how many people will turn it on, when Mozilla will turn it on by default. It may even be on by default.
But regardless, it’s a centralization that’s a little bit worrisome for me.
Chris: Okay, well my next question, you’re sort of answering it there, but I want to go a little further with that. An article on the Naked Security site called DNS over HTTPS is coming whether ISP’s and governments like it or not, we find this quote, “Not everyone was happy with this for architectural reasons, not least because it places a lot of trust and resolve for principle Google Cloud Flare and anyone else who adopts it. The other too, the Internet has been built as a compromise between what the user can do and what the service provider would let them do. DOH, some claim, upsets this balance.
So, you were talking about that a little bit, that there’s this sort of centralization, but what do you think about this new power dynamics, since Google’s already considered to be too powerful by some at the moment?
Andrew: Yeah, look, I think that the … if you go back to I’m trying to think of the name. Anyway, Google had, one of the route certificate providers was compromised years ago, and the Google Chrome browser actually sensed that this Google search wasn’t authentic, because the extra code in there … they know who their providers are, where other browsers would have seen it as authentic. And from that time, I think the browser vendors have been working hard to try to protect users against being hacked, and going to sites that are masquerading as the real thing. Man in the middle, whatever they can do. And part of that is a bit of a power grab of protocols that should, from my perspective, be run separately, like DNS.
Andrew: DNS is a control plane for the Internet, and if you have the control plane sort of merged in higher in the stack, then from my perspective, I see a lot of potential issues with that. I was not surprised about, I mean DNS over HTTPS has been around, it’s been in development. It exists, you know, and so now it’s not a matter of who benefits from it or should it remain. It exists and it will continue to exist, and so now the question is, how can we still ensure the appropriate amount of decentralization? How can we still ensure healthy networks? How can the service providers do what they need to do? And it’s just going to mean additional tools and different methods, and we’ll see from there.
To some extent, if DNS sec was more adopted then clients would better be able to trust the DNS answers they were getting, and it would be way more difficult for somebody to manipulate those answers. If DNS sec was wider implemented and protocols like DNS over TLS were used, then we can try to still keep these things separate and keep them less centralized.
So, it’s one of these privacy versus security things, because if you make it harder for me to see what’s going on, then it’s more difficult for me to use that data in order to provide a certain level of security. And again, corporations I don’t think will stand for it. Some cloud-based DNS firewalls are already blocking as many of these as they possibly can, and there will be well-known block lists of DNS over HTTPS sites, because security professionals need them, so they can block any connections going there.
If this is about ensuring a Internet that is freely accessible and privately accessible, there is so much more that needs to be done than just DNS over HTTPS. You’re taking away the easiest, lowest hanging fruit, but there’s many, many other mechanisms and many other mechanisms that will be defined. Unless somebody’s got a full-on VPN with a provider … there’s always some breadcrumbs somewhere, as your other [inaudible 00:19:06], but at the end of the day, that machine that you’re now using DNS over HTTPS to connect instead of just DNS, is going to make a web connection to an IP address and that IP address associated with that site.
So, there’s other ways to skin this, and so therefore, the cynical side of me, and I’ve got quite a cynical streak, the cynical side of me says here are some corporations that want this data for a variety of different reasons, and I don’t know that to be true. I know, though, that for very good reasons, and very good strategic reasons, there is a data grab. There should be a data grab. Companies should be looking to mine data to create better strategies, deliver better products.
Chris: I wanted to sort of pick out one thing that you mentioned in there, that the cyber work podcast, we like to talk about the issues of the day, but also the way that they can apply to the cyber security workforce. You mentioned that you think there needs to be an increase of DNS positions in enterprise. Is that the case?
Andrew: Yeah. I mean, I think in their zones. I don’t think there’s a question about it.
Andrew: And many still don’t. And there’s a variety of reasons they may not, but there’s lots of providers out there that make it simpler. This isn’t something that has to be complex, and so, signing good, it’s not going to solve … there’s a bunch of man in the middle attacks recently, and the DHS of the U.S. government issued some well-circulated advice to ensure that all zones were signed. And the hijacking really came about because the credentials to the external DNS provider were hacked, and so, somebody with appropriate credentials logged in and made changes to DNS.
Depending how the DNS controls are implemented, sorry the DNS sec controls are implemented, DNS sec may or may not have helped at all in those cases. In fact, in some cases, it wouldn’t necessarily have helped. If your credentials get hacked, your credentials get hacked. You’re done.
But in general, it certainly, for a variety of reasons other than somebody stealing credentials, for a variety of reasons, it makes for a more trusted answer. The other problem with DNS sec though, there’s two problems. One is companies that don’t sign. The other is the number of people that sit behind resolvers that don’t validate. So, if a company is Internet recursors aren’t validating the response, then it doesn’t matter if they’re signed or not. So, we also encourage companies to validate DNS.
From a consumer standpoint, you might be lucky enough to be behind an ISP, a service provider, that’s validating. Very few people launch their own DNS recursor, so therefore, when you’re at home you may or may not be sitting behind a validating or resolver, and if you’re not, then it doesn’t matter either. So, it’s DNS sec is, I think, though, that companies out there by default should be signing.
Chris: Okay, and for folks that might be interested in sort of going down that path, DNS sec, what types of skills and positions do you think people should be … if you want to study that area, what should you be learning and working on and doing now to prepare you for …
Andrew: Yeah. I mean DNS sec is all about cryptography and chains of trust, so I think it’s something for a security professional should be able to understand its architecture pretty quickly, and then it’s about assessing your current DNS, and then assessing your opportunities to start signing. I think it’s a place where the security teams can collaborate very effectively with the traditional DNS administration.
Chris: Are there any unintentional issues with DOH other than surveillance and criminal activity? Does the encrypted delivery do things to say break apart advertising algorithms or online sales devices or just other aspects of the Internet, or does it really come down to straight monitoring?
Andrew: No. Again, DNS has been used for a variety of things over time, right? So, because it’s clear text, anybody that was mining this data for their service might find it would be more difficult to do so. At the end of the day, once your DOH service gets the query, it terminates the SSL at that point, and at that point it’s going to be a normal DNS query to the authoritative server. So, a lot of the data that’s mined through passive DNS from the registrars, for instance, that stuff should continue for some time. But yeah, anybody who is using DNS, in order to understand … using that data for whatever reason, won’t be able to use it.
Andrew: But really, it’s really the first hop that matters, though, right? Because today, you’re sitting in your house, you issue a DNS query, let’s say you’re using your ISP’s DNS servers to recurse the Internet, they’re the only ones that are going to know it’s your IP address. Everybody after that will only know it’s coming from your ISP, so their ability to mine that data for you becomes a bit harder.
Andrew: Once you introduce DNS over HTTPS now, ironically, that service, in this case, let’s say it’s Google, and I’m not saying that Google has any strategy to do this, but today, now they’re actually getting your source IP address. So, I think the service providers hopefully will step up, and some of them are, I believe, are starting to provide similar types of services. But again, it’s been … it’s almost because it is plain text, anybody can see it that’s along the wire. It’s been used in ways that I think there will be several, several entities that are disappointed by not having this data, if it becomes prevalent.
Chris: There was also suggestion, and can you confirm that, DOH could potentially disable parental controls over in-home devices, for example?
Andrew: Absolutely, because if those parental controls rely on DNS, and a lot of them do. Schools, homes, libraries, any sort of communities that are using a simple DNS firewall to stop the access to pornography or gambling or whatever the case, can very simply be subverted by using DNS over HTTPS. Now again, if there’s no firewall involved, if there’s no mechanism to block traffic to an IP address, it’s only DNS, there’s a problem. If there’s a mechanism to block based on where it’s going, then those services can simply implement blocks to the well-known DOH sites, and you shut down DOH.
So, there are controls that can be put in place that will force, I would assume, the browser to, if I can’t get out via DNS over HTTPS, then I’m going to go and try to get out via normal DNS. That will be possible. Obviously, somebody can set up a DNS over HTTPS service that’s on their own IP address. You just set up an Amazon host and you can launch one in a few minutes, and now that IP address is not going to be on some block list, so now I’m going to have to inspect traffic or look for fingerprints of the protocol in order to effectively block it. But yeah, all of those services are going to either need to change a bit or they will provide less service for their intent.
Chris: So, with DOH it seems like kind of an all or nothing affair. It blocks everything. It’s causing all these extraneous or intentional problems. Is there a possibility, and I know, like you said, that the horse is out of the barn. It’s out there, but is there a possibility for a similar type of thing that allows law-abiding people to use the Internet with a bit more anonymity while not completely closing the door on the possibility of it being used as a shield against illegal activity?
Andrew: No, in a word.
Chris: I thought so, yeah.
Andrew: Right. And I mean that by … I don’t personally, I really like, I think privacy is quite important, and I try my darnedest from a hygiene standpoint on the privacy side. None of us can do, I’m not going to go completely out of my way, but I think that between GDPR and DOH, a lot of these things exist because of how our privacy has been monetized and used, and so in general, these things aren’t bad, but something on its own, just DNS over HTTPS is just one piece of the puzzle. It doesn’t provide an anonymous Internet experience at all. It simply allows you some anonymity from your ISP, or … it’s, you’re blocking one path and there’s plenty of other paths out there, so I think it would be naïve of somebody to think that using this, they would be anonymous.
Those that have tried to become very, very anonymous have long engineered solutions that would make it difficult for somebody to understand that those DNS requests are coming from them, DNS over HTTPS or not.
Chris: Right. So, in talking to you today, my initial impression of this was that DOH was really this bullet-proof wall that agencies couldn’t get through but it seems like it’s more that it’s just taking away a particular, as you said, a low-hanging fruit or a particularly easy method of DNS collection. So, how, if at all, will these cyber security professionals skill sets change to address these new levels of privacy with DOH?
Andrew: Yeah, I think what it comes down to is the same thing as HTTPS in general, right? The amount of traffic that is encrypted is going up and up and up and up and up, and with encryption, it becomes way more complicated to understand the content of the message. You still know where it’s coming from and you still know where it’s going, but the content is becoming harder and harder to understand, which brings all sorts of skills around, behavioral type analytics. It brings in, certainly there’s, companies are implementing technology and buying services to decrypt and installing the right certificates to encrypt or decrypt I should say. So, it becomes more of a data analytics role as well, and I think everything ends up leading to analytics in some way, shape or form.
If I can’t inspect everything or see anything, then I need to look at the meta-data. I need to look at the length of the connections, the number of connections, the likelihood that the pattern of connections is made from a human, the fingerprint of the way the stuff works in different applications so I can fingerprint connections to specific applications. There’s all sorts of ways I can look at what’s happening without looking at the content of the message, because that content and message is going to become less and less clear to me as a security professional. And I think that’s pretty critical.
Also, it comes down to … there’s this assumption in the software community that the hardest thing to protect, obviously, is the endpoint. The easiest thing to hack is the endpoint. And so, how do I protect the network? How do I protect back end data? How do I protect what’s leaving my network, as opposed to the endpoint itself? But this is one of those areas where endpoint technology can help with DNS over HTTPS as well, because it’s on the endpoint that these requests are being made in general. And so I think there will be some endpoint technology as well here.
But regardless, as users who certainly don’t like additional endpoint technology placed on their devices, I think … sorry, and by the way, I should have said before, there generally probably won’t be DNS over HTTPS for valid internal, we deal a lot with private DNS. Inside of a company they have their private DNS. We wouldn’t necessarily expect there to be DNS over HTTPS. We’re certainly prototyping from the perspective of being market ready and understanding and testing what sort of uses there might be for that, inside the enterprise. But the control chain of … part of trying to maintain reliable services internally, I need to understand what queries are bound internally, what queries are bound externally. If all of my queries end up getting sent out, but really this was a private query internally, that could be problematic as well.
So, I think security professionals should continue to learn more and more about DNS, and understand that the tools that they have to either ensure that these protocols aren’t used or if they are being used, the tools that they have to still do some level of monitoring.
Chris: Okay, so as we wrap up today, you mentioned it a little bit, but if you want to talk about, a little bit about how BlueCat is working with enterprises and organizations, what your strategies are to come to grips with DNS over HTTPS, and what sort of services or consultation or whatever you’re providing.
Andrew: Certainly it’s a lot of consultation. A lot of our customers had a lot of questions about it and we talked to them quite a bit about it. Like I said, we’re prototyping up some product ideas around it as well, and see if we can leverage it to help our customers, and the answer may indeed be yes there. But right now, we’re also providing the right advice for our customers, in terms of how to configure their systems and what to block and what not to block from the DNS over HTTPS standpoint.
I think, from a enterprise deployment standpoint, especially where desktops are more locked down in general, it’s easier for them to deal with it. But from a BlueCat standpoint, changes in DNS and advances in DNS like these protocols, often offer up opportunities for us to innovate in ways that maybe weren’t even the original intent of the protocol, like DNS over HTTPS. So, there’s things we may be doing there, but right now it’s really just in the research mode.
Chris: Okay. So, if you could wave your hand over the crystal ball here a little bit, in five years from now, do you think that DOH is going to have fundamentally changed security or privacy, or is this sort of a bump in the road that’s going to be subsumed or in other ways sort of gotten around, in the meantime?
Andrew: Yeah. Look, I think that, again, this was an obvious place where things aren’t normally encrypted. I think, from an HTTPS standpoint in general, SNI is another example, where there was plain text identifiers that can be used that will be encrypted now. I think what we’re going to see over the next several years is more and more opportunity for … it will be more accessible for people to be more private, without having to be a knowledgeable for these specific protocols, which most people aren’t knowledgeable of these specific protocols. And generally I think that’s a good thing.
Now, how that works, part and parcel with security, I think that’s become a … I think what we have found is, no matter what we do to try to block, somebody’s going to exploit something that wasn’t thought of before or come up with some new, innovative way, and it becomes, not whack-a-mole. I mean the sophistication of well-done security architectures today certainly don’t allow for whack-a-mole, but it gets to the point where, there’s a presumption, a correct presumption, that something will be around the corner that’s going to change our posture, that we weren’t familiar with ahead of time.
And so organizations are going to have to make sure that they’re designed and architected to make changes based on what they’re seeing. It all becomes around some level of visibility of what’s going on in the network, and that’s going to be more around the meta-data of these connections than ever before if it’s encrypted.
But from a crystal ball standpoint, it’s difficult to tell what sort of adoption there will be of this protocol. There’s a hot, heated debate out in the DNS industry over the merits and problems with these changes. What I fear, frankly, is that this wonderful distributed world of recursive DNS that we have, that gets abused, ends up being highly centralized and therefore, we’ve created more single points of failure, more … It also will be interesting to me to see what consumers out of the U.S. are happy and welcome to prefer a U.S. company to send their DNS to. So, I think a lot is going to happen, a lot is going to change. I don’t know what it’s going to look like, but I’m happy to be part of the journey.
Chris: Okay, and if people want to hear or learn more about BlueCat or you, Andrew, where can they go online?
Andrew: www.BlueCatnetworks.com, and give us a call, come and see us somewhere. We love talking about DNS.
Chris: Andrew Wertkin, thank you so much for your time and your copious information today.
Andrew: Thank you.
Chris: That was very fascinating. And thank you all for listening and watching. If you enjoyed today’s video, you can find many more on our YouTube page. Just got to YouTube and type in Cyber Work with Infosec to check out our collection of tutorials, interviews and past webinars. If you would rather have us in your ears during your workday, all of our videos are also available as audio podcasts. Just search Cyber Work with Infosec on your favorite podcast app of choice. See current promotional offers available for podcast listeners, and to learn more about our Infosec pro live boot camps, Infosec skills on demand training library and Infosec IQ security awareness platform and training platform, go to infosecInstitute.com/podcast or click the link in the description.
Thank you once again to Andrew Wertkin and BlueCat and thank you all for watching and listening. We’ll speak to you next week.