Root Causes 385: Failed Revocation and Wildcard Certificates

Episode Transcript

Lightly edited for flow and brevity.

• 

  Tim Callan

  Okay, so we have over the last two months, had a lot of conversations about this constellation off interesting events that's going on right now, that in the grand scheme of things we refer to as the Bugzilla bloodbath, and also real specifically, in this phenomenon, this epidemic right now that's sweeping the world of public CAs about incapability or unwillingness to do revocations on time.

  And I wanted to quote, a specific line from a specific CA’s Bugzilla incident in which they are failing to get their revocation done on time, and I'm going to just read this passage, this pair of sentences from the CA's bugs. Again, this is a bug where the point of the bug is they're not getting the revocation done on time. So let me read this quote. “The remaining two wildcard certificates involve multiple hosts. Most hosts have either completed the replacement or switched to certificates from other CAs. However, some hosts still require configuration changes to be made by vendors. We will allow an additional two week extension.”

  So basically, what the CA is saying is that they have some certificates that they failed to revoke on time and their rationale for this is that these are wildcard certificates and there are a bunch of different people using the wildcard and not all of those different people have done their work to swap out the certs. So that even though some of them have, they're still not revoking the cert because some of them have not. So before I go further, Jason, do you have any comments on that?

• 

  Jason Soroko

  Let me translate my thoughts into really plain English.

• 

  Tim Callan

  Yeah.

• 

  Jason Soroko

  If you're not completely sure who you're issuing, like who - -

• 

  Tim Callan

  Right.

• 

  Jason Soroko

  Oh, my God. Okay. So, yeah. So there's so many things in my head because, first of all, right, the whole, not revoking on time thing, which is the core. That's the spine of the subject we're talking about here.

• 

  Tim Callan

  Yeah.

• 

  Jason Soroko

  But wildcard usage? Do we need to have another two podcasts on wildcards and why they're a problem, Tim, because this to me really points at true heart of what's going on with wildcards, which is the overuse of wildcards. I can totally see how this can happen, Tim, because if you have a big website, you are creating subdomains, you have partners that are creating subdomains, consultants, maybe you have third party applications who are creating subdomains, and you're sharing certificates amongst them, and you have to revoke that cert and now you have multiple third parties that you don't even have a list of. Oh my God! Tim, this isn't just the need for certificate lifecycle management, this is the need for some other kind of lifecycle management. You need to maybe rethink why you're using wildcards at all.

• 

  Tim Callan

  Oh, yeah. This whole thing is just so fragile. First of all, it's the opposite of agility here, right? This is ossified. I mean, you can't, if you can't even revoke your own certificate because you feel you're going to break somebody that you don't want to break because they're sitting on your wildcard and you can't rely on that somebody to do the things they need to do, all I can say is yikes. Like, what would have happened if one of these individual hosts was trying to get the certificate revoked? Are they going to be disallowed from revoking their own certificate?

• 

  Jason Soroko

  I bet you the conversation would be had and isn't that incredibly scary?

• 

  Tim Callan

  Isn't that incredibly scary? And, I mean, this is obviously gross misuse of a wildcard. You can't. Obviously, technically they can, but it's a crazy idea to use a wildcard in this way where you're taking your wildcard cert and you're just plunking it down on a set of these properties, these servers that you have no visibility or control on whatsoever. Like it's just, like this is spread across these different hosts and they're not all - - they don't seem to know each other. Right? It’s just - - what a terrible practice and how completely - - first of all, you're putting all of your eggs in one basket. You are completely anti-agile. Right? You're just incapable. There's no agility and flexibility whatsoever and I'm hard pressed to imagine that that kind of architecture is necessary.

• 

  Jason Soroko

  It's not.

• 

  Tim Callan

  If these are different hosts and they're running they're different properties then give them different certs.

• 

  Jason Soroko

  This is just - - it’s just sloppy.

• 

  Tim Callan

  This is what’s certs are for. Yeah.

• 

  Jason Soroko

  No, no. Tim, look. I’ll even add to this. Folks, 90 day certificates are going to be a reality. And it really should be a reality. Okay.

• 

  Tim Callan

  Yeah.

• 

  Jason Soroko

  Like Tim, has just uncovered, you know, a situation where people have done a poor job of managing what certs are for, you know, even what the purpose of a cert and overuse and complete inappropriate usage of a wildcard certificate. Let’s just call it that.

• 

  Tim Callan

  Right. Yeah.

• 

  Jason Soroko

  And don't want to get away from your mass revocation story, Tim, because that's the core of it, but my God, folks, when 90 days here and heck, one day when you're going to want to move to 10 day certs, this exact example, how in the world does that even work? If you can't revoke the cert, how are you going to renew it every 90 days or 10 days?

• 

  Tim Callan

  Oh, I agree. I agree. Like this is just so broken in so many ways. And what if, what if it were a 24 hour revocation event? What if it were an inflexible sorry, ain't given you no extension 24 hour revocation event? What if it were a DCV error? What if it were a private key compromise? Then are these various, random, seemingly large number of who knows who they are hosts just going to just break and have to deal with it? I think the answer is yes. That's what's going to happen.

• 

  Jason Soroko

  Tim, you've heard the expression security as an afterthought. This is security as a never entered the mind.

• 

  Tim Callan

  Right. Yeah. And so like, what about, if I'm one of these hosts, I should not want to be in this relationship either where I'm depending on some certificate that I have no visibility or control over to keep me up and running and somewhere along the line, someone else might make bad decisions and the consequence of that person's bad decisions will be that my services go down.

• 

  Jason Soroko

  Tim, you and I are managers at Bad Corp. Let’s just call it Bad Corp.

• 

  Tim Callan

  Sure. Bad Corp.

• 

  Jason Soroko

  You and I both run a whole lot of websites because we're a consulting firm and we basically are like, ah, I don't want to get 200 certs. I want to have one cert. I'm gonna go order it from my CA of choice. And I'm gonna go nah, Jay and Tim told me not to get a wildcard but I'm just gonna go get a multi-cert with the 455 domains that I manage and I'm just going to share that one cert right across there. And then I'm going to dust off my hands and not think about it anymore.

• 

  Tim Callan

  Oh, we had - - So we told an anecdote on this podcast. This was probably four years ago at this point. I can't even tell you what the episode was but where we had had a revocation and it was a single - - it was one off. It was something like a private key compromise and we went in and we revoked the cert and next day, this angry customer is putting something on his Twitter saying my cert just got revoked. Now I have 300 businesses that are down. Thanks Sectigo! And my first response to it was so you had 300 disparate separate businesses, who had nothing to do with each other, who didn't know each other all sitting on that same wildcard cert? Really? Like I'm sorry. Sectigo is not the bad party here. It is that hoster.

• 

  Jason Soroko

  Yeah. So you know, Tim, the hoster should know better. But you know, I bet you there's a lot - - there are at least a proportion of people who are listening to this podcast, fully well-meaning people who might not be fully aware of what we're talking about. I think that's worth recapping in another episode. I'm sorry for taking it in that direction. But yeah.

• 

  Tim Callan

  No. This is the direction. Right? So, yeah. And it's just interesting. Like, it's this constellation of problems, right? If we look at some of the things that make, you know, the web PKI and public SSL certs less than their potential, overuse of wildcards is one, this failure to revoke problem that’s going on right now is another, and just seeing those kind of intersect and collide in this way, was really interesting.

• 

  Jason Soroko

  Yes.

• 

  Tim Callan

  And kind of showed and, and then they exacerbated each other, right? The overuse of wildcard problem is aggravating the inability to revoke problem. And so it's kind of this negative, uh, what’s the word I'm looking for? It's kind of this anti-synergy that's occurring.

• 

  Jason Soroko

  It definitely causes a fragility with any of the systems. Look, folks, you know, a lot of you out there might be procuring SSL certificates. You know, we're talking - - the podcast here today is about a specific, we can't revoke because x story which, we've been covering intensively here on this podcast over the last little while, because it really has been a major subject matter. But it's just an example of a much bigger problem as well.

• 

  Tim Callan

  Yeah, absolutely. And this overuse of wildcards thing has been going on a long time and I fear that we're going to be discussing this one for a long time.

• 

  Jason Soroko

  Total certificate agility, Tim. Something you coined a while back. You do not have total certificate agility if you are sharing multi or wildcard certs across third parties who are not aware of each other. That is just crazy.

• 

  Tim Callan

  Yeah, you don't have any certificate agility. I’m sorry. Total certificate lack of agility.

• 

  Jason Soroko

  Yeah.

• 

  Tim Callan

  Anyway, that was it. It was just, that was such a jaw dropper, I thought we had to talk about it today.

• 

  Jason Soroko

  Wow. Great One, Tim. Thank you.

• 

  Tim Callan

  Thank you, Jason. This has been Root Causes.