Root Causes 456: 2024 Lookback - Bugzilla Bloodbath

Episode Transcript

Lightly edited for flow and brevity.

• 

  Tim Callan

  We are doing 2024 lookback episodes. I don't know how we could do a 2024, look back for the Root Causes podcast without discussing the Bugzilla bloodbath. What a crazy year!

• 

  Jason Soroko

  I just got to say this, and then I'm gonna let you rip.

• 

  Tim Callan

  What? You think I have opinions on this, Jay?

• 

  Jason Soroko

  I think you do. Maybe only seven years ago in my - I'm getting close to 30 years in this industry. You're right there with me. Only once before did we experience a year that kind of approached this level. But there's no precedence to 2024.

• 

  Tim Callan

  Insanity. So we covered this in a number of episodes. The one called the Bugzilla Bloodbath. The one called Drama on Bugzilla. Those are both great places to start. Several episodes about revocation in particular, which are also good ones to look at.

  Capsule summary is that endemic problems and sins that had been committed, that were going on for a long time that I think the CAs, a lot of CAs, had become blasé about came into the spotlight, and as they came into the spotlight, it became really clear that a lot of things were really bad and so we saw just numbers of posted bugs that were off the charts. That were multiples of what we're used to. We saw activity on these posted bugs, just large numbers of posters, huge engagement from the community that were off the charts. That were very different from what we were used to. We saw bugs dragging on for a long, long time because they weren't being successfully resolved, which partly was the community and other people, like browsers, not accepting a kind of hand wavy thing from CAs, and partly it was CAs who weren't fixing their problems and weren't stepping up and weren't providing answers and weren't doing the right thing. All of this led to two distrust incidents.

  First one is a company called Ecommerce Monitoring, which is a very small German CA, but the second of which, of course, was Entrust. This was a shocker. If you had said to me on February 1, Entrust will be distrusted this year, I would have said no way. And yet, here we are. And not just, by the way, for TLS. Entrust also had distrust events around S/MIME and VMC and time stamping. That also is very noteworthy, just because of the size and longevity of that particular CA. When these CAs get distrusted, and they're small and they're regional and they're quasi- governmental, and they have less than 1000 active certs, I think it's easy to sort of be dismissive about it in terms of its scope and scale on the web PKI, but when you see a distrust event around a large global, very well- known CA with the large number of large enterprise customers in important industries that affect a large number of people, then that's a big deal.

• 

  Jason Soroko

  It's a huge deal. It's a huge deal. The customers get affected, subscribers and their end users get affected. I think that for me, what was interesting out of all of it was to watch the tightening up of - and we talked about this in 2025 predictions recently - talking about how I think in 2025, after looking back right now, what you're doing in 2024, the look forward is that I think that more discipline will be put on us.

• 

  Tim Callan

  I agree with that. So you saw this kind of aha moment where it became clear how widespread and cavalier the failure to follow expected norms was. There was this initial response around, analyze what's going on, put pressures on the CAs. Some CAs changed. Some CAs at least made vows to change their behaviors. We'll see if they walk the walk on that but at least made promises and put up good action items and seemed to be trying real hard. A bunch of them didn't and still don't. I’m honestly a little surprised there weren't more distrust episodes this year. I think more might be coming because there's some CAs that just didn't seem to get their act right yet. But then now coming out of that, you say, well, what happened later was you saw real scrutiny on the rules. So we saw a much more specific proposal from Mozilla about how to deal with delayed revocation and the consequences of delayed revocation that had ever been proposed before, and a proposal that I think would be a vast improvement over what we have now. We saw a shortening certificate lifespan proposal from Apple that in part was driven by this just recognition of the difficulty and the recidivism of getting the right thing out of CAs. Then there has been other discussion in kind of the CA/Browser Forum root program world about further restrictions and tightening, and it really owes itself to all of this Bugzilla bloodbath stuff.

  There have been real things. There's a passed ballot that will go into effect early in 2015 that will require pre-issuance linting. That is a direct consequence of the Bugzilla bloodbath, very directly. So you see this increased scrutiny, and this increased action to force a certain minimum level of quality and competence and compliance out of CAs in recognition of the fact that absent that, you can't count on getting it.

• 

  Jason Soroko

  It's really something to watch how the process happened in 2024. For those of you who are not as inside baseball, I'm going to attempt - this is usually my role when we talk about public - attempt to kind of oversimplify what's going on.

  I think there was a point in 2024 - it was a fairly dramatic moment. It's almost like in a western where there's a standoff between the sheriff and the bad guys. The bad guys seem to own the town, but the sheriff is a tough dude. I think what we saw was a question being asked, do the rules matter? And we found out. The rules matter.

• 

  Tim Callan

  I think that's fair. I think there was a big, giant question mark about saying, is there a bit too big to fail concept? Is it possible that a CA could have a certain scope and impact to the point where they essentially would have a different set of rules than other CAs that were below that threshold? It was a serious question. I, for one, was not at all clear on what was going to happen. Then it turned out that that was answered, and I think we saw a very strong statement made, first by Chrome, but then followed up by Mozilla, Apple and Microsoft, to say, no, every CA is going to be expected to meet our minimum standards, and we can deal with any distrust event that we need to deal with. And they weren't cavalier about it. They were measured and they were considered, but they were also firm, and I think if you didn't get that message, and you're in our industry, then you just weren't paying attention.

• 

  Jason Soroko

  Yes. For those of you who are in the periphery of public or you have an interest in it, but it's not your daily day job. I think that that's one of the things to note, is that the trust in Certificate Authorities remains, and in fact, it's been enhanced because I think that part of what I'm reading into the root stores and comments that were made and you can read all Tim's comments and listen to our podcast. I think that the CAs are put under enormous scrutiny for good reason, because it's the basis of trust. Because there is a community of CAs who follow those rules, the root stores are like, that's why you are in our root stores. Because you follow procedures. Things like misissuance are not the problem. It's how you deal with it and how you deal with procedure.

• 

  Tim Callan

  I think that's a good point, Jay, which is, another thing if you look back, and you look back at the Bugzilla bloodbath, you look back at what the transformative year that 2024 turned out to be in the web PKI, it was much less about the errors than the handling of the errors.

  Which to put it another way to say, when you scrutinize CAs, I tend to divide it into two broad camps, which is, there is a competency question and there is an integrity question. Are you able to do the right thing? Are you prepared to do the right thing? And in this case, in 2024 while we saw big aspects of both, it was the competency problems that started the conversation, but it was the integrity problems that ended it and that's an important point, which was at the end of the day, a CA that makes well-meaning errors but doesn't continue to make the same well-meaning errors over and over again, and earnestly addresses them and does the hard thing when the hard thing is required, probably does not need to fear distrust. It is the CA that doesn't learn from their mistakes, doesn't earnestly do everything they can to get better and obfuscates or obscures or attempts to dodge responsibility. That is the CA that needs to worry about whether they'll still be in business this time next year.

• 

  Jason Soroko

  That is how we are exiting 2024. That's the big note.

• 

  Tim Callan

  So that's the big note. So it was a big year. Lot of conversation about it. Go back and listen to those episodes and consequences of Bugzilla bloodbath, far from over, and we're going to see those into 2025 and beyond.