Root Causes 287: GoDaddy Private Key Breach

Episode Transcript

Lightly edited for flow and brevity.

• 

  Tim Callan

  At the top of the show, I always say that we talk about all matters digital certificates and PKI and in this case, I think we are talking about a matter of digital certificates. So, I’m looking at a headline. This is from February 20, 2023 from InfoSecurity. Author is Alessandro Mascellino and here’s the headline:

  “GoDaddy Announces Source Code Stolen and Malware Installed in Breach.”

  So what happened here and what does this have to do with certificates?

• 

  Jason Soroko

  It looks like, Tim, in this breach – of course, GoDaddy being a gigantic website hosting company, right – they are one of the big guys out there. It looks like if you read into the article and I’m completely dependent on the journalism here because, you know, this is second-hand information but we want to report it because the juicy bit for us is kind of buried.

  And we want to bring out the root cause of the issue. Basically – and I’ll read right out of the text. It says, “however, in some cases, customer’s SSL private keys were exposed and if abused, this key could allow an attacker to impersonate a customer’s website or other services. While GoDaddy has reset customer WordPress passwords and private keys, it is currently in the process of issuing them new SSL certificates,” Tim.

• 

  Tim Callan

  Ok. So, if there is a private key exposure for a public SSL cert, which is what we are talking about here in this scenario, right, because they are for hosted websites, then there is a 24-hour revocation required on any exposed private key. That means that in the event that GoDaddy has become aware of the private key, that they actually have 24 hours to revoke that cert. So, you know, this might be just a matter of how things got reported to the journalist but the process of is a little surprising to me unless this is being reported in real time. Because that’s the kind of thing that needs to be dealt with very, very quickly or you are out of compliance with the CA/Browser Forum rules and the Root Program rules.

• 

  Jason Soroko

  It looks like this was a November 2021 incident, Tim.

• 

  Tim Callan

  Ok.

• 

  Jason Soroko

  And according to the article, 1.2 million of their customers were accessed. Because, you know, just to get back into the root cause here, it says that GoDaddy is revealing that those customers – who many of them were still active. Most of them were still active – had their FTP credentials, usernames and passwords, for their WordPress databases stolen. So, with those FTP credentials, the SSL private keys were indeed exposed and therefore, you know, as soon as this, you know, the moment - - as you say, what triggers that 24 hours, I’m sure they had at least a million or so certificates that they probably had to renew.

• 

  Tim Callan

  Yeah. That’s interesting. And it does need to be done in 24 hours. So, I guess the other thing that’s not clear to me, and I don’t know if you have a viewpoint on this or not, is if this was a November incident, why are we learning about it now? Did GoDaddy just reveal that this occurred or is there new information or what happened with that?

• 

  Jason Soroko

  Um.

• 

  Tim Callan

  Ah! They filed their 10-K. That’s what happened. And I guess this was covered in their 10-K. So, presumably all of those certificates were dealt with long ago and there’s probably a tense problem in the way we are discussing it because you can’t let exposed private keys sit around for months. That’s just utterly unacceptable.

• 

  Jason Soroko

  That’s right. It probably was dealt with. It’s just, you know, it’s just interesting that this November 2021 incident, not even last November but the previous November, as well as an incident that happened in March 2020 is what’s being referred to here and it’s hitting the news now because of that government filing.

• 

  Tim Callan

  Yeah. It was in November 2022. There is a quote here, “In early December 2022 we started receiving a small number of customer complaints.” So, when this incident occurred and this actually was monitored and covered in the public community and the CA/Browser Forum and that occurred at that time. I remember that. There was no real conversation at that time about them not dealing with any exposed private keys that they had. I recall that there wasn’t a lot of clarity in the beginning about whether or not private keys were affected because they had this major breach and people started asking questions to say, look, were private keys stolen, and there was a period of time where I don’t know if people at GoDaddy knew the answer, but the community at large did not know the answer. It was definitely a big question mark that we had. So, at this point, we can say that according to this journalism at least that private keys definitely were and let’s hope that all the certs were already replaced

• 

  Jason Soroko

  Correct. I’m gonna assume they were but, you know, I don’t have the information. This is what gets me is that, you know, Tim, you live and breathe this stuff. If there was a breach of that size that affected a hoster that was using Sectigo certificates, you’d be all over it and your transparency about it would be really, really good. Nobody would have to be waiting for a government filing to find out what happened and what you did to revoke and replace. This is one where you gotta choose your vendors carefully and if it’s - - you know, I’m not saying that there was anything hidden because I don’t even know, you know, maybe I just wasn’t listening carefully enough at the time. You know, your comments about the fact that it hit the message boards that people are looking at, but it was unclear about what was breached, unclear about private keys, that kind of stuff shouldn’t be unclear.

• 

  Tim Callan

  What’s interesting in this case was there’s different rules—and this is probably an interesting thing to highlight. They are very clear rules about transparency when it comes to public certificates. We’ve talked about this ad nauseum in the past. The enforcement mechanism is that there is a small number of browsers that really control how public certs are treated in the world. We say browsers, but it's operating systems and software. We call them browsers but they are mostly not browsers anymore. And as a consequence of that, there’s a great deal of strength of enforcement for public CAs to be very, very transparent, and public CAs have to go through a great deal of exposure of any incident that occurs by the rules, and if they don’t do it, that can be a very serious problem. In some of the distrust events that have occurred in past, at least one of the contributing factors was lack of transparency. Like in every case I can think of at least one of the contributing factors was lack of transparency. There were always others as well, but that was one of them. And so, the CAs have this very high bar that they are held to get over. Now that doesn’t mean every CA always does but, in general, you see a large quantity of compliance. Now if you turn around and you look at another industry space, there is seldom that level of forced transparency on just about any other industry space. Certainly in the hosting space. Like there’s no equivalent of the CA/Browser Forum and there’s no equivalent of a browser root store program for web hosting services. So hosting services are policed by what are vaguer, less attentive and laxer requirements like, you know, laws around things like fraud that are just much more general and less focused. As a result, you see the difference, which is there is a breach in GoDaddy and they’ve got some breach notification laws they have to follow and I’m sure that a bunch of people got a letter and maybe some people got some kind of make-good service that they had to offer but it was nowhere near the level of codified prescripted reporting and scrutiny and obligation to answer questions from any member of the public who knows how to use a web browser. Like, all of that that exists for public CAs just isn’t there in the hosting space. So, you know, it’s interesting to see, to observe the difference between a breach that occurs in a hosting platform that affects a very large number of people and once that same breach may have taken their private keys. Right? And all of the sudden, a whole different set of rules kicks in. And, you know, this is, I think, illustrative of the very high degree of transparency that a public CA is held to as opposed to almost any other technology space in the industry. I won’t say any other, but almost.

• 

  Jason Soroko

  Absolutely. Absolutely, Tim. And, of course, GoDaddy themselves being a CA, right, they are a player in this. So I think a lot of people think of them in this case as being the hoster when in reality they are also playing the role of the CA.

• 

  Tim Callan

  I’m glad that you brought that up because the other thing that came up in the public discussion around this is, well, what’s the definition of a CA? It is the same company and if it’s different people at different desks, can they argue that they are not a CA? Because, well, these people at these desks that are dealing with it don’t work for those people at those desks who are the people who are the CA. And how do you draw those lines and where do you draw those lines? You know, a lot of companies are in a lot of spaces, right? GoDaddy is a CA. They are also a large hosting provider. Microsoft is a CA. They are also a huge operating system provider. Google is a CA. They are also an operating system and web services provider, etc. Apple is a CA. And so you get into these weird, nebulous questions about well, is this, now is this particular thing, is this particular issue or vulnerability or failure part of the CA’s obligation to report and remediate or not? Because if you are one of these providers and you can find a way to cleverly and creatively construe that this is not occurring in your CA operation, then all of the sudden the requirements in what you do drop dramatically. So you get into that whole interesting conversation that does not have any kind of clear answer and doesn’t even have a clear referee.

• 

  Jason Soroko

  Do you want a really short and sweet way of - - if you are procuring these kinds of services, I’ll tell you the way you gotta think then because while people are having these nebulous conversations, you can’t be doing nebulous procurement. You gotta be protecting yourself. So, therefore, Tim just told you and Tim is right is saying that the CAs who call themselves CAs and who call themselves CAs are very transparent and take this unbelievably seriously.

• 

  Tim Callan

  They have to.

• 

  Jason Soroko

  They have to.

  Your existence depends on it and for everybody else who is acting as a CA because they’ve got root certificates in the Trust store of your browser, ok, to me that’s a CA. It’s a short conversation. You got that, you are a CA. And therefore, GoDaddy is a CA and so, therefore, if you are procuring SSL certificates from that vendor and they are not going to be super transparent about these things, well, you gotta think twice about where you are getting your SSL certificates from.

• 

  Tim Callan

  I think that’s a valid point, Jay, which is to some degree the market does have an opportunity to have an enforcing function here which is to say, look, you might come back and you might argue this is here or this there but at the end of the day, if I can’t trust your overall company to, you know, maintain extremely high standards then how can I trust you as a CA? And that absolutely is a valid approach and I think there is some strength to that. I think it’s less strength than if Mozilla says you need to straighten up or we are gonna cut you out. So, that’s kind of the challenge. But, you know, to your point, maybe to some degree that is what we have to work with and so maybe we just need to use that as our lever, as our tool.

• 

  Jason Soroko

  That’s it. That’s how to think about it, folks because there’s a lot of us who listen to this podcast who mess around with websites and host websites or know somebody who does, but friends don’t let friends deal with vendors who aren’t gonna be on your side. Look, there are a number of - - the premier CAs out there, it’s worth working with them and using their certificates.

• 

  Tim Callan

  Ok. So there you have it. A simple story but it turns out there is a decent amount there when we unpack it.