Root Causes 375: What Is Name Space Lifecycle Management?

Episode Transcript

Lightly edited for flow and brevity.

• 

  Tim Callan

  And I say two industry veterans, but it's really three because this is the guest day. We love guest days. And we're very excited to be joined by Geir Rasmussen. Geir is the founder of NodeZro and I think Geir will tell us a little bit about what NodeZro does but you guys are in the domain name security space. Isn't that right, Geir?

• 

  Geir Rasmussen

  Yes. Thanks for having me first, Tim. So yes, we are in a specific area of domain security, which we call name space security and that is looking after identifiers, not just at the primary level, where google.com and ibm.com, etc. but all through the name space of a company.

• 

  Tim Callan

  And we've covered this in a recent episode at a real high level, where we talked about this idea of sort of abandoned things like abandoned subdomains that never really got properly shut down and how those can be vulnerable for members of attacks and one of the things I think, Geir, and part of what led up to this conversation, was your comment to me, your friendly comment to me, that that is just the tip of the iceberg. That it's really much more complicated and I think sophisticated than simply I have an old subdomain. Is that right?

• 

  Geir Rasmussen

  That is correct. That is correct. We quite often see organizations find what I think you referred to as dangling CNAMEs, which is the concept where you point the DNS to somewhere else, you point the domain name to a CNAME.

• 

  Tim Callan

  Yes.

• 

  Geir Rasmussen

  But reality is that these underlying vulnerabilities can occur in many different ways, not just for CNAMEs.

• 

  Tim Callan

  Right. And I'd love to get into that. I think it's a deep topic, and it's a rich, and maybe a little bit complex topic but before we get into that, I think you personally had an interesting journey that led to this space and it might be useful for us to hear about that, and what sent you down the path of investigating and understanding this area of name space hygiene, let's call it.

• 

  Geir Rasmussen

  Well, I've been in the domain industry for a very long time, and I started back in the late 90s, with a company that owned well over 100,000 primary domain names, and we had all the challenges of managing those that any organization will do, you know, making sure that they renew every year and all the things that a brand protection registrar will do to domain names. So I went from that to founding a TLD, which is on the other side of the table, where you run a top level, in the 2000s and saw this space from the complete opposite site. And, having done that, again, then for a very long time, I ended up as the Domain Name Manager for the UK Government and looking after a very complex name space.

  And through this journey, it occurred to me that domain names at the primary level, let's say, a google.com, again, are very well managed. There is a good supply chain there. There's lots of great brand protection registrars that help you look after these brands, making sure they renew every year, making sure you protect them. But when it comes to internal in organizations, there is very few companies and organizations that we have come across that have any processes or policies in place for managing their subdomains and that is where the underlying issue occurs. It's a lifecycle management and an asset management issue, much more so than a technical issue.

• 

  Tim Callan

  And you can tell me if I'm not getting this right. One of the things that Jason and I discussed, because we see this a lot in the certificate space, is what we'll call a rogue certificate or a rogue CA. And that is, there's somebody in the organization with enough technical acumen to get certificates or set up their own CA and run it and they do it completely on their own. Nobody knows about it. There's no internal centralized management of it. There's no approval process. There's no kind of go up the chain required. They just do it. And those things cause problems. I would imagine that a similar thing occurs with subdomains. Is that correct?

• 

  Geir Rasmussen

  Yes. You would be correct. And I think it's the same fundamental issue just on a much larger scale. So one of the things we very often come across is we call it shadow IT where you have someone in an organization setting up a service with a SaaS provider, or some other digital service. They pay for it with their own credit card, and they link it into the name space. They use a domain name to point to it. Three years down the line, no one has any records of this. They are decommissioning the service, and then it becomes a problem. So it's the lifecycle management, which is the fundamental issue here.

• 

  Tim Callan

  And these things can linger indefinitely, right?

• 

  Geir Rasmussen

  Well, we came across last week, a company where they had a supplier they decommissioned in the mid-2000s. 2005, I think. And it was still in their name space being used and obviously, the brand name of that supplier had been recycled a number of times and now they were subject to pointing their domain names to someone who had no relationship with them. So that is a significant risk.

• 

  Tim Callan

  So what is the specific risk here? What happens? Like what's the bad thing that can occur?

• 

  Jason Soroko

  Well, if you think about it, if you have a domain name, which you point to a supplier, and it doesn't have to be a CNAME, although that's what we normally see bug bounty hunters, etc. find. It could be any, any link within the DNS. It could be name servers, mail exchangers, it could be the domain names that you put in an SPF record, and so on, and so on. And quite often, what we see is that these domain names end up being not renewed by the company that used to use them. They then get data and expire, get picked up by someone else, and either get used for something else or get parked, and anyone can buy them. And you know what, if you have a name server which points to one of these domain names that are available for purchase, your domain name is trivial to steal. And, one of the things that, now that we have looked into this on a on a global scale, one of the things that I think now is that if I get an email or see a website from most large companies, I am no longer sure that it is actually them because I know how easy it is to go up these identifiers.

• 

  Jason Soroko

  Geir, would you say that there's a problem of inherited trust, the way that the DNS settings are or the CNAME settings are because of this problem? We've talked here about shadow IT, but I think the scale of this problem, the scalability problem of managing domains and domain spaces of subdomains over time, becomes a gigantic scalability problem, even if it was just one or two employees keeping track of things through time. The hygiene has just never been there before. I don't think people realize the risk of this inherited trust between these domains and just would love to hear really two things from you about what are the full set of risks that people are just unaware of with respect to this inherited trust between these domains that are not well managed right now. The lifecycle is not well managed. But, as well, what do you think the future is in terms of enumerating this problem and understanding the scope of all of these orphaned subdomains that people have forgotten about, and every other issue that comes with what you've described here?

• 

  Geir Rasmussen

  So I think the main thing that organizations should do is that they should try and understand the complexity that they have organically created over a very long time, and which quite often gets even more complex with M&A. So you roll another organization into your name space, and you don't know. So first of all, I think it's important to not think of this as a technical problem, which you assign to a System Admin, and then hope that he can deal with it. This is an asset management issue.

  So first, try and get visibility on what you actually have and then you need to try and figure out how you instill some lifecycle management in your name space. And that is complicated. We work for organizations which have hundreds of thousands of these identifiers and fortunately and unfortunately, the DNS is a little bit too good at what it does. So, it doesn't have to be a domain name which no longer works anymore. We quite often see domain names that have name servers. I'll give you an example where the name servers should have ended in .net but they have forgot to put in the t and now the name server sits in a domain name in the share and that may be available for registration. So everything works fine. No one's seen this for years until someone comes along and goes, hang on a minute, I am gonna take that name server, and then I control your identity. Because these identifiers, they're like your digital passport because they are the ones you use to authenticate your company.

  When you issue certificates, the concept of domain validated certificates is used all the time. If I control the domain name, I can, with most suppliers get a certificate, but I can also, I can also authenticate to Google that I control that domain name and set up Gmail or whatever else it is, and the same to many other companies. So that's the underlying issue.

• 

  Jason Soroko

  It's amazing that we've been dealing with domains for a very long time. Your introduction from the late 90s onwards is just proof point that we've all been dealing with this for a long time. What do you think the reasons are for this exploding now? Tim and I were talking about this in previous podcasts, but now that you're here with us, why? Why now?

• 

  Geir Rasmussen

  So the problem has been there for a very long time but what has happened now is that ironically enough, digital transformation, which is supposed to make things more secure, has the unfortunate side effect that the DNS is being used to connect a company's name space with their cloud providers and SaaS providers. And now that you move from one provider to another, you have cloud infrastructure that you spin up and take down. Unless you have lifecycle management for your name space, there's just, there'll just be more of these issues. So ironically, it's exploded with digital transformation. Even though the problem has been around for a long time, it is getting very, very large indeed.

• 

  Jason Soroko

  I just cannot help myself here to make the parallel between Certificate Lifecycle Management and domain and subdomain lifecycle management. Some of the words you use, Geir, in terms of how to solve this, were about getting a handle on visibility and, to me, all of a sudden, if I'm having to cobble together perhaps even potentially multiple DNS systems - you mentioned, if there's M&A - a lot of these problems are very similar to what happens in the certificates that are often related to these domains. It's such a parallel topic. Right now, of course, there is a certain level of maturation of Certificate Lifecycle Management tooling out there. Where are we at, with respect to enumerating these DNS settings and accounts, and bringing it all together and discovering what you might not know you have and bringing together visibility and then trying to solve these for these orphans somehow? What does that look like? What's the current state and where do you think it's going?

• 

  Geir Rasmussen

  So the reason why we did this company was that there wasn't a specific product to deal with just this problem space, but what is happening is that there's a lot of providers out there that are providing services at the edge of it. So you will find many cybersecurity firms that will tell you about some of your dangling CNAMES, etc. They'll tell you about some of the other issues and that is getting more and more common. What there is no one really doing now, though, is trying to solve the underlying problem, which is the problem of lifecycle management. I'll give you an example.

  So one of the companies we’re looking at may have 3,000, 4,000 primary domains. Google.com would be a primary domain. And then inside of each of those primary domains, they may have – it’s a very technical term, but in the DNS, it's called a zone, or sometimes a delegation. And each of these primary domains may have 200 or 300 of these delegations, and quite often each of those delegations or - - basically, a delegation is handing control to someone. And each of those quite often have different teams managing them. So trying to figure out who runs what, where is a very complex process. And we quite often see companies come and say, well, we'll do this ourselves. We’ll try and figure out who runs what and then four weeks later, they come back, and we have no idea who runs this. The person we have on record left five years ago. So that is the that is the challenge.

  First, you need to map this out and you need to find out who is actually in control of these different zones or areas of control within your organization. And then you can start remediating maybe some of the more critical issues where your brand is at risk. And you can imagine, if you can take control of the brand of a major organization, that is a problem because that can be used to attack not only the organization but also the stakeholders of the organization.

  So first of all, get that visibility. And then second, you need to find the individuals or teams that are in charge or in control and then that is - - the lack of tools. We don't have the tools to do so. And we're trying to do something about that but that is the toolset that is required to solve this issue.

• 

  Tim Callan

  Yeah and it's more than just standing up a website. There's lots of ways that these things are used. Like you mentioned SPF records earlier. That allows me to send email now and pretend to be somebody that I'm not. We all kind of imagine there's this page and we've all been trained to look at the domain name and I look at oh, it's really the company. It must be real. But there's more to it than just that.

• 

  Geir Rasmussen

  That is correct and one of the things scarily enough that we come across very, very often is SPF records that points to companies that are no longer providing services to an organization and quite often, these companies have brand names or domain names that are now owned by someone else. And that is actually a delegated control to someone else allowing them to send email on your behalf. And just that is a problem.

  Obviously, we only classify that as a medium risk but that should illustrate what a critical risk is. So this is very, very common. And that's an issue. And again, lifecycle management. That's the problem.

• 

  Tim Callan

  And it sounds like for these organizations, you're talking about, I guess, if I were starting a company up today, you could say we could put some proper lifecycle management in place and as long as it runs properly, that would be the right path forward. And that's a path forward for anybody but at the same time, there is this massive set load of technical debt that needs to be dealt with that I think can be pretty shocking and, intimidating, and very difficult to deal with, right?

• 

  Geir Rasmussen

  And, you add to that, that the DNS is truly not that well understood. Most developers, most SysAdmins, even marketing teams quite often know how to register a domain name, and maybe do some basic things with it but what it actually does is not well understood. And again, it's that these identifiers, they are your digital self. They are the ones that identify your company today because we're in a digital first era. So, just understanding of what losing control of these identifiers actually means I think is one of the important things that we try to communicate.

• 

  Tim Callan

  So all right. So I'm CIO of an enterprise, hypothetically, I recognize that this is a risk. What do I do?

• 

  Geir Rasmussen

  Well, the first thing you need to do is to try and figure out what are the processes and policies we should put in place here, and then once you understand what you need to do, then you need to discover what you have. And only when you discover what you have, which I guess will surprise you, then you can start dealing with it. It's interesting.

  So ask yourself, do I have a list of all the subdomains that I have in my brands? And do I know who manages them? And I think the answer that nearly every organization will have to that is unfortunately, no.

• 

  Tim Callan

  No and no. So then the first thing you do is you've got to get that under control. If you can discover them, now you have the list and then there's some kind of process that might be time consuming and difficult to understand who is managing them. You can probably get a lot of them real easy, right? But then there's probably a long tail, that isn't so easy.

• 

  Geir Rasmussen

  Yes. And for larger organizations, this is often globally. You can imagine that a large corporation delegate, they may be based in the US; they may delegate parts of their namespace to Europe, which then may again, delegate to third party suppliers, and on and on and on it goes. So that discovery piece is actually surprisingly hard. It sounds so simple. So what we normally hear is, oh, we'll just go to the DNS admin and get him to give us the song file, which is where the records are kept, but then quickly finds out that he only has the song file for the top level or the primary domain, and then he doesn't have the rest of them. So and then the challenge starts. Where do I go? Who do I contact for these other things?

• 

  Tim Callan

  And you could have a disconnect between technical owners and business owners so you might not be able to just do it that way. You might not say, oh, well, this belongs to marketing. I'm gonna start there because they don't know what's going on with DNS. They don't do that. You could have a disconnect. And again, you talked about the M&A situation. You could acquire brands and domains in a name space through a company, and there's no record or memory whatsoever of what's gone on. It might be that nothing comes across in the acquisition. It might be that the people who didn't are long gone, you don't even know where to go to start.

• 

  Geir Rasmussen

  Correct. So, it is deceptively complex. And, like every rabbit hole and I can tell you that this rabbit hole is very, very deep. It's much deeper than most people assume. But it is that trying to cut through that complexity and not the worst thing you can do here is to go and say this is a technical problem. We'll let the cybersecurity team deal with it because it really isn't a technical problem and then the other thing is we quite often see that it gets put in the this is too difficult pile, and we don't even know where to start. So, try and when we've seen organizations where we have raised maybe 100,000 findings for that organization and where do you even start when someone comes and tells you that, oh, we've found 100,000 things for you. So it's trying to figure out, how do you attack that?

  So what we try to do is to give you a workflow for where you should start. Maybe you don't need to deal with 100,000 issues. There may be 100, or 200, which puts your organization at critical, real critical risks. If you deal with those, you have really taken away a big chunk of your overall exposure.

  So there are ways of doing this piecemeal, but ultimately, the fundamental fix to this, all of these cybersecurity issues, and where your brand may point to content, which is not authorized, etc., those are symptoms, the fundamental, those are all symptoms of lack of lifecycle management. So, figuring out what is my process to get to a place where I have lifecycle management around these things, that is the ultimate underlying outcome that you want.

• 

  Jason Soroko

  Geir, in the Certificate Lifecycle Management business, one of the big things that we point to are of course, outages of web properties, websites, etc. For what we're talking about here, I love the way you're putting this. It's not a technical problem, right. It's a process problem. It's a governance problem, perhaps even. And we have tried to say that in the Certificate Lifecycle Management space for a long time, the way to get this to the notice of Risk Officers and CEOs, CISOs, everybody else who understands risk within an enterprise is to really point out what can happen. I think we need to do a better job of actually explaining here is the true risk. Your identity as a company, your brand as a company, is tied to these complex technical pieces, and they are not managed well. And the news we're hearing recently, Tim and I are podcasting about this, you're now explaining it very deeply and it's great, I would like to then ask you, some of the enumeration tools and inventory tools and discovery tools that you're offering are, frankly, they're pretty, I mean, they're just great. They're what's needed at the moment. What do you think the rest of the industry needs to do? From the public clouds to the DNS people, the registrars. These are folks you know well from the world you've been living in for many years. What's the role they have to play for ultimately making it easier for people who not been used to just getting their hands dirty in the DNS records and understanding and listing of their subdomains? What does the rest of the industry have to do on top of your discovery tools?

• 

  Geir Rasmussen

  So Jason, I think the first bit of that is that we are talking about corporate identity theft. It really is identity theft that is one the underlying outcomes here.

  The other thing is, I think it's important that there's a very solid mature industry out there of what is often called corporate registrar, so brand protection registrars, that look after the primary domains, and we come across companies all the time who think they're protected by their supply chain, but there really isn't, and it's not like your supply chain has sold them something. That is not the case. It's just there's a lack of understanding. You may have gone through a corporate registrar and they may go, oh, well, we'll look after your domain names for you and then we will register them, make sure they're renewed every year and if someone infringes on your brand will take down that content, blah, blah, blah, all of these things. But sometimes that message ends up being someone's looking after all our domain names. So there's a disconnect between what people think they're buying and what they're actually buying.

  The other thing is that we're seeing when it comes to traditional cybersecurity, there's a lot of great companies doing attack surface management, and they do service things like dangling, CNAMEs and cloud assets that can be compromised, etc. but it's looked at from a pure cybersecurity angle. These are vulnerabilities that your techies need to deal with. No one's really looking at, well, all of these things are nice, but they're all symptoms. What you actually have to do to secure yourself for the long term is a boring management process. It's very different. It's not some sexy cybersecurity thing. So that is, I think there's a messaging issue here where we need to talk about these things for what they actually are and they're pure old boring asset management.

• 

  Tim Callan

  Have you seen examples? So you talk about these companies, they have hundreds of thousands of domains, they have 20 years of legacy, and they have this ridiculous quantity of old stuff that's never been cleaned up. Have you seen examples of companies that actually do manage to get this in a headlock and get it under control, and they have a much cleaner name space than when they started?

• 

  Geir Rasmussen

  Well, interestingly enough, I can tell you that there are a couple of industries that are doing surprisingly well. And I don't think I'm outing anyone when I'm saying that the energy industries have done very well in this space. Why that is, I am not sure. One of the offices we have is that there is still a lot of on- premise infrastructure in use, which means that they haven't used the DNS to connect cloud services yet. So it may be that this is a problem waiting to happen with digital transformation, but it is really interesting that energy specifically have been doing well.

  The other thing we're seeing is that there is a difference from geography to geography, with some countries having less of these issues than others. Interestingly enough, that also seemed to be tied to digital transformation.

• 

  Tim Callan

  Gotcha.

• 

  Tim Callan

  Simpler infrastructure, just less digital in general, and therefore there's just less opportunity for this kind of mess to occur.

• 

  Geir Rasmussen

  Correct.

• 

  Tim Callan

  That makes perfect sense. But then to your point, you're right, which is, we know that these industries, or GIOS, or segments that haven't digitally transformed, we know they will. They're not just going to stay in a 2010 architecture. They're going to update their systems and as they do, the same risk occurs for them, too. But they have a chance to get in front of it right and exercise hygiene from the beginning and at least minimize the scope of the problem.

• 

  Geir Rasmussen

  Exactly. So if you start now, there's some very simple things you can do. When someone delegates or spins up a domain name asset, make sure you record it. Make sure you look after it through the use phase when it's actually being used and when it's no longer needed, all you need to do is decommission it appropriately. That is the three things. There's no rocket science involved here. And if you do those things, then you will be in a much better place.

• 

  Tim Callan

  That last one's the biggest problem. Is nobody bothers to sunset anything. They just leave it and move on.

• 

  Geir Rasmussen

  Correct.

• 

  Tim Callan

  Gee, I feel like we have barely scratched the surface here.

• 

  Jason Soroko

  Tim, this would probably be the third or fourth time and I feel like we're digging deeper and deeper. I tell you, if you're a Director of IT, if you are a CISO/CIO, you need to be aware of this. So glad we had you on today, Geir.

• 

  Tim Callan

  Thank you so much, Geir.

• 

  Geir Rasmussen

  Thank you for having me.

• 

  Tim Callan

  So that's, again, Geir, you're the founder of NodeZro. Just so you know, that spelled N-o-d-e-Z-r-o. That's important. And thank you, Jason, as always.