Root Causes 308: E-Tugra Root Deprecation

Episode Transcript

Lightly edited for flow and brevity.

• 

  Tim Callan

  We have reported a few times on this podcast, including earlier this year, about a public CA distrust incident and we now have another one to talk about today.

• 

  Jason Soroko

  My goodness, Tim. You and I, we are gonna have to take back some of our words because these things don’t usually come up too often and we’ve said that in the past, but here we are.

• 

  Tim Callan

  Here we are. Yes. So, normally, this is a once every several years kind of occurrence and this is literally the second one this year. I guess it was last year. The second one in the last 12 months. Let’s put it that way. And in this case, the CA involved is called e-Tugra. I think I’m pronouncing that right. It’s a Turkish CA and e-Tugra has been operating as a CA with cross-signed certificates from other public CAs and this is a fairly common practice when people are trying to get spun up as a CA as a business entity. They start with a cross-signing deal. They work on getting their own roots included. Eventually, they get their own roots included, they move to their own roots, and they stop using the cross-signed roots. And in this case, e-Tugra started as that. It has roots that are being considered for inclusion in root stores. They are in some and they are not in others and due to this incident, which we will get into, those roots are now being deprecated from at least one of the root stores that they are in, which is the Chromium Root Store.

• 

  Jason Soroko

  You know, I’m always curious, Tim, there’s a lot of smaller CAs that are out there. Some of them are fairly regional. Some of them have very specific purposes. What do we know about this CA and what they are about?

• 

  Tim Callan

  I believe they are in general what I would call a regional CA. There’s a fairly large number of CAs that focus on a geographic region that isn’t one of the major computer markets. So these are people who are not in the U.S., in Canada, in Great Britain. They are somewhere else and they really focus on that region. Their competitive differentiator tends to be, first of all, local, language and cultural understanding for starters because, you know, you gotta know what they call business entities and what abbreviations are acceptable and what are not and things along those lines that are easier to know if you were born and raised in that place. And they are easier to do business with if they are native speakers of a language that isn’t English.

  Then the other thing that they usually have is market advantages just because they have feet on the ground in terms of connecting with customers. So it’s easier to do local events; it’s easier to do local marketing, use local media and things along those lines. So you see a lot of these in a lot of different countries and you can understand why that might be beneficial for the local populace to have something they can use that’s a local business and it’s easier to do business with. At the same time you can understand where you can carve out a business that way. I think e-Tugra is of that class. They are not particularly focused on another niche other than the fact that they are Turkish and they are really doing business in Turkey.

• 

  Jason Soroko

  Ok. A regional publicly trusted certificate CA. Right on.

• 

  Tim Callan

  And so all of this started on November 13, 2022 when a researcher named Ian Carroll – and this is a name we’ve heard before. Ian floats around the world of SSL and pops up once in a while with something interesting and noteworthy and new. Ian Carroll reported that the e-Tugra systems basically were pretty fraught with security vulnerabilities and Ian obtained information and included screencaps and did all of it in a responsible disclosure and fashion. But at the end of the day made I think a pretty convincing case that a lot of security errors occurred that were unbecoming of a public CA and were worrisome within a public CA. That led to a Bugzilla bug which I believe was published by e-Tugra themselves on the 18th of November. So, again, we are back to November 2022 and then from there, there has been a seven-month long thread on Bugzilla with many people asking detailed questions. Just to put things in perspective, I’m going to go down to the bottom of it. I’m scrolling and I’m scrolling and I’m scrolling. This particular Bugzilla bug – when I finally get there – will have - - hear we are. It has 53 comments in it and some of them are very long and some of them include screencaps and there’s a rather large number of attachments to this particular bug. A lot of bugs have no attachments and this has one, two, three, four, five, six, seven, eight.

• 

  Jason Soroko

  Yes. Eight.

• 

  Tim Callan

  So, there’s been a long dialogue and the focus of this dialogue has kind of been on two main things. Well, I guess three. The first one is, is e-Tugra running a secure CA that’s reliable and secure because remember, CAs are especially pivotal in our system. If you can compromise a CA and own a CA, you can do all kinds of stuff because that’s the root of trust. So, first of all, are they running a secure CA?

  Second, are they responding to the community in a reasonable way? Are they responding quickly enough? Are they responding accurately and are they responding let’s say completely enough or expansively enough? There was a sub-dialogue about that because some people were finding fault with the responsiveness and the forthcoming nature of e-Tugra’s dialogue. Then the third thing that I kind of focused on was what was e-Tugra’s relationship with the CAs that they were cross-signing from – that they were sub-CAs of - and were those CAs in any way also part of this problem? And all three of those things I would say were pretty thoroughly examined over a long period of time. So the end of that occurred – and let me find the date on this – on the Google group, not on the actual Bugzilla bug around this but on the Google group as of June 2, 2023, a representative of the Google Chromium Root Store has said – I’m just gonna exert a little bit from this message – “After full consideration of the available information relating to the vulnerabilities disclosed in blah, blah, blah, including that of incident reports and other public responses we have decided that in order to appropriately protect and safeguard Chrome users, the following e-Tugra root CA certificates will be removed from the Chrome Root Store,” and it is two authentic e-Tugra roots that are there in the Chrome Root Store. So that’s important. Other root programs have not necessarily gotten to the point of including them in the first place but Google pulling them out essentially means those roots are not really gonna be usable and it means that e-Tugra can continue to be kind of a white label reseller like it is now but doesn’t have a reasonable prospect of being a full-fledged CA anytime in the foreseeable future.

• 

  Jason Soroko

  Great, Tim. Thank you. I’m really glad. A lot of us listening to this podcast will not have to scroll through the entire Bugzilla list.

• 

  Tim Callan

  It’s a lot of reading.

• 

  Jason Soroko

  Yeah. Thank you for reading so the rest of us don’t have to get that far.

  So, let me ask you, Tim. I want to get, you know, let’s get to the root causes, right? So, you were saying cybersecurity issues. I don’t think that was your word but that was the way I heard it in my head. So, we’ve heard about distrust issues because of other things. I can’t recall a general lack of cybersecurity posture of a CA, you know, coming up first and foremost. Am I wrong in saying that?

• 

  Tim Callan

  I think if you go all the back to DigiNotar, that was definitely a cybersecurity problem in a pretty big way.

• 

  Jason Soroko

  Right.

• 

  Tim Callan

  I’d have to go back and 100% refresh my mind but since DigiNotar, I agree with you. I do not think that cybersecurity has ever really been the root-cause problem. I think there’s definitely been questions about integrity. And I don’t believe integrity was really seen as a concern or an issue here. I don’t think people were questioning e-Tugra’s honesty or commitment to privacy or commitment to security or anything like that. There definitely have been concerns about integrity and if we think about the TrustCor deprecation last year, it’s a perfect example. There also have been concerns about just general technical acumen and that comes up a lot. We saw that with Certinomis. We saw that with Symantec. We saw that with all of those. In this case, you could say this definitely falls into the technical acumen camp because the question is do they have the technical capability to maintain a secure operation. But this particular focus on security and the potential of being owned by a bad guy definitely was the lead story here in a way that it hasn’t been with something else since DigiNotar.

• 

  Jason Soroko

  So, could I compare and contrast to DigiNotar for a moment then since it may be the closest analogy?

• 

  Tim Callan

  Sure.

• 

  Jason Soroko

  My recollection is that DigiNotar was breached and I haven’t studied this particular case enough to know whether or not there was evidence but you haven’t mentioned it so I suspect there wasn’t which makes this a little bit unique in a way.

• 

  Tim Callan

  I would agree. The white hat researcher Ian Carroll himself breached their security and presented evidence.

• 

  Jason Soroko

  Wow.

• 

  Tim Callan

  Again, in keeping with responsible disclosure practices, did it in a way where, he wasn’t compromising anybody’s data or anything along those lines, but provided convincing evidence that he was in and he was in places that he shouldn’t have been in. Did exactly what you are supposed to do – gave that information first to e-Tugra so that they could actually rectify the problem before he told the world so other people couldn’t come exploit it. Exactly the way you are supposed to do it. From what I can see did everything correctly but he had it. He got in. He got them. And he detailed the problems and, again, a lot of what this dialogue was about is there’s a recognition of the fact that look, there’s some very serious foes out there and even really strong people get breached. And we see this all the time, you know, major banks, leading cloud providers, like very, very good cybersecurity people sometimes still lose that battle. A lot of the point of the dialogue on this was that this was pretty straightforward stuff. This was the kind of stuff that should have been protected. This was the kind of stuff that should not have been a hole and a vulnerability. So, it wasn’t even just that someone got in in their outside pen testing exercise. It’s that someone got in in ways that just really shouldn’t have been possible.

• 

  Jason Soroko

  Great, Tim. That’s a great explanation. I think I got a better understanding now of exactly what’s going on here. But it does show you - and I think sometimes you and I at this point in any of the podcasts like this will reflect a little bit and the fact that this has been going on for seven months shows you these kinds of decisions are not taken lightly, and thankfully the process works. It’s diligent and it works and good on the industry for making the hard decisions when they have to and taking all the steps necessary to make sure that the facts and figures are known.

• 

  Tim Callan

  This one was definitely diligent. I mean, the number of words here, I don’t have an estimate on it but just tens of thousands for sure were written on this topic between various sources. There was a lot of diligence, a lot of questions, a lot of opportunity for response to the questions, and I think this is important. As a general rule - I don’t know enough to know that it’s truly universal but I can’t think of an incidence where the browsers were not fundamentally very careful to give the CA the opportunity to explain and defend themselves and present evidence and talk about why perhaps something that might look bad doesn’t really turn out to be bad if you understand the whole picture. They really do try hard for that. We’ve said this in previous podcasts. I do not believe that anyone from a major root store browser is looking to deprecate anyone’s trust ever. I think they are deprecating trust when they feel like that’s what they must do for their users and you see them giving the CAs plenty of opportunity to state their case, present their evidence, explain what’s going on, and they really try to feel like they’ve exhausted that process before they move forward with the trust deprecation event.

• 

  Jason Soroko

  Right on, Tim. I agree with you completely and I really hope this is the last one in 2023.

• 

  Tim Callan

  I agree.

• 

  Jason Soroko

  Hopefully we are not reporting back on this but this was a good one, Tim. You’ve explained it really well. I feel like I understand it now. So, I appreciate that.

• 

  Tim Callan

  Like again, wow! These are happening fast.

  This is also a little unusual, Jay. I hinted at this in the beginning in that this was a set of roots that hadn’t really been fully adopted by the community. Apart from test certs, there aren’t any leaf certs. There aren’t end user subscriber leaf certificates in production today for these roots because they were still in the process of getting included. They were in Chrome but they weren’t in Apple. I’m not sure what the status was of the other major programs, but they were not fully adopted yet and so because they are not fully adopted nobody is really gonna buy that, right? You are not gonna buy a cert, stick it on your website, and someone says it doesn’t work on the MacIntosh and if it doesn’t work on the Apple stack who really cares. You’d say, well, I care and I’m gonna wait. So, there aren’t end businesses or end subscribers that are affected by this. And this is another way that’s unusual. If they are going and blowing up somebody else’s CA then that Certificate Authority has subscribers and those people need to get new certs. And they are gonna have things that stop working one day if they don’t get the message and they don’t change them out and surely some of them don’t and find out when things stop working. This is different in that there are no subscribers affected because they kind of got it before it made it that far. In that sense I can’t think of another example of CA distrust happening before they were in the world of issuing leaf certificates.

• 

  Jason Soroko

  Yeah. Right on, Tim. Listen, that is really good context. So we are not talking about something where it’s widespread catastrophe. This is something that the white hats - - Ian Carroll’s name is very, very well-known. He obviously did a great job here well before any kind of an event like that. The fact that it was early on is great context.

• 

  Tim Callan

  The last thing is there’s nothing here, nothing has been said publicly to suggest that e-Tugra will stop being a reseller of other CA’s services, which it can be, because then those other CAs do the essential critical stuff that people are scared about. They had cross-signs with a couple of CAs. The one that’s been most involved in this dialogue was SSL.com. SSL.com did a cross-signing event with e-Tugra and there were questions asked, and it was looked into, and it certainly appears to me that the widespread consensus is that SSL.com themselves are without blame and operated everything correctly, and nobody is really worried about that.

• 

  Jason Soroko

  That’s good.

• 

  Tim Callan

  As a result, there is nothing here in terms of technically or the rules to suggest that e-Tugra can’t go right on with the relationship they have with SSL.com into the indefinite future and that that combination itself does not appear to be suspect or worrisome to anybody in the community. So if I had to predict, I would predict that that’s what we are gonna see happening and that they are all just gonna move forward that way.