Root Causes 390: Chrome Boosts Its Distrust Agility with a New Root Trust Deprecation

Episode Transcript

Lightly edited for flow and brevity.

• 

  Tim Callan

  We have talked a lot about the Bugzilla bloodbath, and which by the way, has just continued to increase and we've seen a new record in terms of the number of open bugs at the same time. New bugs are being opened all the time. It's just continuingly. But we have also seen the first casualty. So one of the things that you and I have hinted at, at least in our discussion of this is, is this going to wind up in the distrust of any CAs? And the answer now is a definitive yes.

• 

  Jason Soroko

  So we have a CA distrusted. Interesting, Tim. Can't wait to hear about who it is, where they are on the long tail and what the significance is.

• 

  Tim Callan

  This CA is called e-commerce monitoring GmbH or E-commerce Monitoring Corporation, often called EMC, and also operating under the brand of GLOBALTRUST. So these are all words for the same thing. I'm just going to call them either EMC or e-commerce monitoring for the remainder of this podcast. But this is a relatively new CA. It was included in the major root stores, I'm going to say around 2021. 2020. Something in that ballpark. So it's not super old. It's relatively small. It seems to be very niche, and very specific. It is a regional CA, in Germany, as you can tell by the GmbH. And on Friday, the 23rd of March, 2024, an announcement came to the public list from the CCADB.org, which is we've talked about what CCADB is in the past. It's the centralized CA database. And it came out the sender was a member of the Google Chrome root program. And maybe I'll just read directly from this email. What do you say, Jay?

• 

  Jason Soroko

  Tim, thanks. I think you said March 23. You meant May, 23^rd.

• 

  Tim Callan

  Sorry. May 23. So a week ago as of taping, and it reads, “The Chrome root program policy states that CA certificates included in the Chrome root store must provide value to Chrome end users that exceeds the risk of their continued inclusion.” That's a good point, we'll go back and unpack that. “It also describes many of the factors we consider significant, where CA owners disclose and respond to incidents. When things don't go right we expect CA owners to commit to meaningful and demonstrable change, resulting in evidence continuous improvement. On numerous instances over the last three years, e-commerce monitoring GmbH fell short of the above expectations.” Then they have a list of links. “In light of this, we have reached the conclusion that the GLOBALTRUST 2000 certificates suffer from a loss of integrity and action is required from the perspective of ensuring web security for Chrome users. To safeguard Chrome's users, we are taking the following action.” And then they describe the action, and I'll get into the details of that but the gist of it is that after the end of June 2024 this CA is distrusted.

• 

  Jason Soroko

  There's at least two questions forming in my mind, but I'll let you keep going. This is good.

• 

  Tim Callan

  So, there's a number of things here that are noteworthy and different and those are worth discussing. I'll hit maybe the most obvious one, which is if you've been following this podcast, and you've heard about other distrust incidents that have occurred, at least since as long as you and I have been doing Root Causes, Jay, they all have been initiated and have run through Mozilla. So the Mozilla community has had a public discussion about a CA and at the end of that time, there was a choice by the Mozilla community to distrust them in the Mozilla browser and then coming out of that, other browsers move forward with distrust as well.

  This is different. This came from Chromium, not Mozilla. And there was no public discussion, because Chromium is not an open source project. Well, Chromium is, but the Chrome root store is not an open source project the way that Mozilla is. It's not a community run project. And so as a result, the hierarchy, the command and control there, made a decision and announced its decision to the world. So in that way, it's very different.

• 

  Jason Soroko

  So, Tim, the Bugzilla bloodbath that we've been talking about, is, as you say, public discourse through the Bugzilla forums. And, there's all sorts of legitimate events that have been dealt with properly by CAs and others that we've podcasted on that might be questionable.

• 

  Tim Callan

  Have been dealt with very badly, I would say.

• 

  Jason Soroko

  Right. And so has there been public discourse on this? Or is this coming out of the blue from the Chromium root program?

• 

  Tim Callan

  So there has been plenty of discourse with e-commerce monitoring. So e-commerce monitoring has had a wealth of bugs and bad bugs, and badly handled bugs. So for example, one of their bugs earlier this year, the Mozilla prescribed formula, if you will, template for a write up, includes a series of sections, and you're supposed to provide that information in those sections. And one of those sections is called root cause analysis. And we all know what root cause analysis is. You and I would, in particular. We were on a podcast called Root Causes. But the ecommerce monitoring write up for this bug that they had said, roots were unaffected. That was it. And it was kind of jaw dropping. Like, no. That's not what root cause analysis means. It has nothing to do with your CA roots.

• 

  Jason Soroko

  They were confusing a root cause analysis with their root server analysis.

• 

  Tim Callan

  And then you're just like, okay. Like, on the one hand, maybe that's kind of a funny error. But on the other hand, that's not an error you should be able to make in earnest when you're doing something very important, where the stakes are very high. And then there were just any number of things. It was clear as these bugs were going on that this CA did not understand, in any way, what was expected of them. That they didn't understand the expectations for receiving a certificate report, acting correctly on a certificate report, responding correctly, writing themselves up correctly, providing transparency and accurate information, dealing with problems, dealing with revocation. All of that was just a mess. It was failure after failure after failure. And so, I don't like to get on here and talk ill about people. I don't know how to avoid it in this case. It was just mismanagement top to bottom. And, not really improving and almost a tin in terms of the people who were representing EMC on this public forum didn't, almost didn't seem to get it. Didn't understand that they had messed up so badly. Weren't trying to fix it, weren't trying to get better, weren't learning the lesson. And all of that was just very discouraging when you think about the fact that this is one of the places that a bad guy could go and attack and get a certificate and be trusted by the global Web PKI.

  And so all of that was bad. Did this come out of the blue? Well, it came out of the blue in that there was no public discussion because that's not how Chromium does it. It didn't come out of the blue in terms of I wasn't surprised.

• 

  Jason Soroko

  So then I would then say, anytime we have these distrust events, we look very closely at the true root causes of why the distrust happened. Since there's so much activity going on, Tim, is this a one-off that's in its own category or do you think that this - -

• 

  Tim Callan

  That's the big question, Jay.

• 

  Jason Soroko

  So are there other distrusts coming, Tim? Or not?

• 

  Tim Callan

  That's the big question. And that's an extremely important question. You can imagine two kind of scenarios.

  One is, we are going to try to scare everybody straight. We are going to pick the slowest antelope, and we're going to distrust and then all the other antelopes will get hurrying up. That is one possible interpretation.

  Another possible interpretation was, when you looked at the field, like, it's hard to see how someone else would deserve distrust prior to EMC. So then you go, okay, well, if someone's gotta go, it's got to be EMC. But for all we know, Chromium or another browser, has more distrust they're planning on and they might not necessarily do them all on the same day. There might be good reasons to stretch it out, especially when I talk about some of the other things that are special about this particular event. And when we do that - because there's more here that's going on that’s a little unusual - when we do that, we can back up, and we can talk about why this might cause them to space these kinds of events out.

  So and I'll remind you also that Google is not the only possible source. Mozilla, again, has been the major source of distrust over the years. Mozilla has published a Wiki page about the failure and offenses of one CA, which is not e-commerce monitoring and they have stated in a bug that they're preparing such a Wiki page for another CA, who is also not e-commerce monitoring. So it is very real possibility that at least two CAs are due to have the public dialogue on the Mozilla page about whether or not distrust is coming. That hasn't happened yet. And it may be that it won't. But in every time that there was one of those public dialogues, it started with one of these Mozilla Wiki pages. And so just having that page written about you, that's a bad thing. You don't want that. And that absolutely may progress. So it is possible right there that we're seeing daylight on to distrust events from Mozilla. And then we don't know what else. The root programs that don't do things on a public forum, like Chromium and Apple, they can do whatever they want, and we don't know what they're going to do.

  So it is really possible, Jay, in response to your question, which is a good one, that this is not the whole event. That this is the start of the event. But we're just going to have to wait and see.

• 

  Jason Soroko

  Stay tuned to this channel. You are right, Tim. This probably is the beginning of things.

• 

  Tim Callan

  So now let me tell you, let's talk a little bit about how this distrust is happening because this is the other part that's really interesting. And I'm gonna go back now to the verbiage on this email. Okay, I'm gonna pick up where I left off.

  Upcoming changes in Chrome 124. and higher, goes as follows:

  • TLS server authentication certificates, SSL certificates, validated to GLOBALTRUST 2020, whose earliest signed certificate timestamp, SCT, is dated after June 30, 2024 will no longer be trusted by default. TLS server authentication certificates validated GLOBALTRUST 2020 whose earliest STC is on or before June 30, 2024, will be unaffected by this change.

  So this is a new thing. They've basically put a timestamp monitor in here. So what they can do is they can turn on this monitor, which they have in this case. And what this means is if you have a GLOBALTRUST 2020 certificate that you're using now on your website, it's fine. Keep right on using it. Use it until it expires. There's not going to be a problem. In fact, if you buy a GLOBALTRUST 2020 certificate, and it is issued to you, not after June 30, 2024, you're fine. Keep using it. Use it for its full lifespan, and you have no worries. But any certificate that's issued starting July 1 isn't going to work in Chrome.

• 

  Jason Soroko

  So Chromium, they're trying not to be disruptive here.

• 

  Tim Callan

  This is new functionality. Nobody's ever had this before. How it's worked in the past is the browsers say, okay, it's gonna be distrusted as of this date. Then as of that date, it's distrusted and other people go, okay, better get off my certs. And they go in to get new certs.

• 

  Jason Soroko

  So the whole system was kind of binary in the sense that if your CA’s public cert wasn't in the trust store of the browser, that's it.

• 

  Tim Callan

  Just didn't work. That was it. It was in or it was out.

• 

  Jason Soroko

  Now you're saying the browser is going to have additional functionality to check for - -

• 

  Tim Callan

  The browser has new functionality so they can put in a date, which is the last date they will accept a trusted cert, which from a disruption perspective is much better. It means that you don't have a bunch of subscribers who aren't reading the websites I'm reading, who don't suddenly wake up one day and their cert doesn't work anymore. And at the same time, you can distrust. It also means you can pull these things forward. So they announced this thing in late May and the distrust day is the end of June and they never would have done anything this fast. Nobody did anything that fast on prior distrust events, for exactly that reason. For the disruption reason. So it gives them more agility. It lets the move quicker.

• 

  Jason Soroko

  I would imagine that for the other major browsers, I don't think that the barrier to entry to do the same kind of logic within the browser is high. In other words, the other browsers could also do that but now we know Chromium is doing it. And that's very interesting.

• 

  Tim Callan

  Absolutely. Now, let me read the very last line of this email. “This approach attempts to minimize disruption to existing e-commerce monitoring GmbH subscribers, using a new Chrome feature to remove default trust based on the SCTs and certificates.” So they're saying right now, this is a new feature. Then they're also saying this is the first time this feature was used. So this goes back to - and this is just me just what-iffing. But if Chromium did have other distrust events that they were planning on, if you were in charge of Chromium, and you had this new capability, and you wanted to use this new capability because it made distrust better, and you had other distrusts, you had multiple distrust events you're planning on, what I would do is, I would do one, and I would take it all the way through, and I'd take it through to the distrust day, and I would watch it all work as expected before I moved on to the second one. So is that happening? Is that not happening? I don't really know. But if they were planning on another distrust, they're smart people over there at Google, you would think that they would let one of these run its course. It's not that long. It's gonna take a little over a month. Watch it all work correctly, at a minimum, before they went around and they went on to the next distrust event.

• 

  Jason Soroko

  There you go. That makes a lot of sense, Tim.

• 

  Tim Callan

  And so now you see there's kind of a built in spacer. At least for the first one, there's a built in spacer. So now you say, okay, when you asked me that very astute question, are there more coming? Well, if there were more coming from Chromium - and again, I don't know that there are. I don't know anything you don't know. - But if there were more coming from Chromium, this is how they would behave. So what they're doing now does not tell us that there are not more coming from Chromium.

• 

  Jason Soroko

  I always like to vastly oversimplify and fall on my face, Tim. So I'm going to do that, again. What you're saying, and this is really, really interesting to me. Google is allowing themselves to potentially accelerate the pace of distrust, without over disrupting. The end users.

• 

  Tim Callan

  Very much, Jay. They have enhanced their own root store trust agility. That's what this new functionality gives them. It lets them make moves more quickly. And the other thing it does is - you and I have been talking a lot in our Bugzilla bloodbath episodes over the last three months about the need for more granularity and control by the browsers in the world of trust, that this single on off switch situation they have is very blunt, and it's causing problems. This is certainly the first example, maybe not the last, of browsers giving themselves more granularity of control.

• 

  Jason Soroko

  Wow, the root store moves beyond 1990.

• 

  Tim Callan

  And doing it in a way, by the way, that I didn't predict. People in the community had been throwing out ideas. They could do this. They could do that. Nobody said this one. This is the one they wound up doing. So, and this also emphasizes that we very much don't know. We can speculate. We can guess. We very much don't know. But certainly at a minimum, I feel very comfortable saying that Chromium is giving themselves the ability to do more distrust events in the future more rapidly, more easily and with a lower threshold, even if they have no intention to use it now, it is crystal clear the only reason to build this functionality is to be able to use it. And it is crystal clear that this was deliberate on their part to give themselves the ability to have more control over who is in their root store.

• 

  Jason Soroko

  I think it tells us something important, Tim. I think there's a signal in this. You don't do these kinds of things for nothing. You don't do it for singular events. I think you're onto it but as you say, no guarantees, but I would stay tuned.

• 

  Tim Callan

  And you know what, they're decent people over there. They probably would prefer that they never have to do this ever again. I'm sure what they would prefer is that every CA gets its act together and flies straight and they never have to do it. I'm thinking that they recognize that the only way that that's a possibility is if the reality of distrust is there. And then they probably don't ever have to use it.

• 

  Jason Soroko

  The Bugzilla bloodbath, we've speculated that it is going to have to have some kind of consequence. I think that you're about to see it at some point. You're right, Tim. Right on.

• 

  Tim Callan

  So anyway, very interesting. First of all, anytime a distrust event happens, they're rare enough that we want to cover them. But second of all, some real important factors in this one, that point to how the industry is evolving, that I wanted to make sure the listeners were aware of.