Root Causes 260: CA TrustCor Deprecated

Episode Transcript

Lightly edited for flow and brevity.

• 

  Tim Callan

  So, we want to talk about a recent public certificate industry development. These things don't happen very often, so when they do, they're newsworthy, and they're always something we try to cover. There recently has been deprecation of trust for the roots of an active public CA. And as I said, this doesn't go on very often. Seem to occur roughly every two years on the average. So, when it does happen, it's news and we make sure that we discuss it.

  The CA in particular in this case is called TrustCor. And we'll get into the background and exactly what went on, but very recently, TrustCor, as of this recording, has been distrusted by Microsoft and Mozilla, which will essentially make their certificates commercially unviable and we may see follow on from other browsers, but whether or not we do, their certificates basically aren't really usable anymore.

• 

  Jason Soroko

  Thanks, Tim. I definitely want to know what happened. My first question always with these distrusts, though, is how big are they? I haven't done any personal searches in the CT logs to see just how many certs they've issued. But relatively, are they a small player, Tim.

• 

  Tim Callan

  Yes, TrustCor is a pretty small player and a pretty niche player and that's part of what we'll get to. TrustCor in a lot of ways was the in-house CA for a couple other companies that appear to be associated with TrustCor. That wasn't 100% of its volume but it was a big piece of its volume. And so TrustCor, unlike a different company, like let's say Sectigo, which is trying to provide certificates to hundreds of thousands of companies all around the globe, TrustCor was focused on a very specific subset of use cases, and specific subset of actual end users and customers. That was part of the dialogue as well. So, here's what happened. Let's back up to the beginning.

  So early in November, an article came out in the Washington Post, that detailed, let's say, a large number of uncomfortable connections between public CA TrustCor and a couple other companies that were, for all intents and purposes, let's say that were in the spyware category. And we can get into the details of those companies and what they were doing specifically, but they were companies that appeared to be there for the express purpose of allowing people to see, spy into digital communications or usage without the users knowing it. And that became a lot of the core of the dialogue and the discussion, which is, are we okay with this? Public CAs kind of as part of their mandate are supposed to be there to serve an open and let's say, unexploitative Internet, and spyware fundamentally is just philosophically in opposition to that idea. And so, in addition to a specific set of rules and regs that are covered in their programs in the CA/Browser Forum rules, also the browsers have this standard that they say in their root program requirements, which is that CAs are expected to be aboveboard and impeccably clean and beyond reproach. And if you're seen to be aiding and abetting spyware, then it invites the conversation, does this qualify as beyond reproach? And, you know, again, we'll get into the detail but the long story short was at the end, the conclusion was, no, it does not.

• 

  Jason Soroko

  So that's interesting, Tim. Were there any known cases of the certs being used? Or is it just a suspicion?

• 

  Tim Callan

  Maybe let's get into what's going on.

  What's interesting is, and let's clarify this. There was no indication that any TrustCor certificates were either deliberately or accidentally mis-issued - issued to the wrong party or abused in another way. So, there's no direct worry about the certificates themselves. As a consequence of that, there have been things like no revocation event was demanded. No backdated distrust for old certs that are already in the market. None of that stuff went on, which is the kind of thing that could have happened if it had been determined that the company was, for instance, wrongfully issuing certificates.

  Let's just get into what happens. This article comes out from the Washington Post, and it details a whole lot of connections between these three companies. One of them is called Packet Forensics. Packet Forensics, essentially, is in the mobile spyware application category. They will take their spyware and find various ways to put it onto apps that are on mobile devices and then from there, you will be able to spy on those mobile devices in certain ways. The one example I'm aware of—and this will tell you, I think this will indicate a lot right there—there was a Muslim prayer app. So, an app that's there to help you know when to do your prayers, and monitor and track them and things like that, that contained the Packet Forensic spyware. Now you can start to imagine, hmm, if I'm in the business of selling my spyware services or the information that I'm harvesting, who might I be selling the information from users of a Muslim prayer app to, right? You can start to imagine government, military, people like that. So, you start to get a sense just based on that one example for perhaps how this is being used, and perhaps who that end customer is.

  That's Packet Forensics. Packet Forensics has been around for a long time and appears to have been in this business for a long time.

  To give you another example, at one point, Packet Forensics was an exhibitor at an event that they called the Wire Tappers Ball. Now, just by virtue of the name of that event, you can imagine who is going, who the target market is, and what sorts of content is being discussed, what kind of wares are being purveyed. That's the world that they're in. And, you know, there was no discussion of whether this is legal or illegal, or any of that. And certainly, the people involved wouldn't probably be experts in the nuances of that. But it's pretty clear that this is not a business that corresponds to the concept of secure private online usage.

  Another company that was associated with with this triad was a business called MsgSafe.io (“Message Safe”) and Message Safe bills itself - I think still does. Did as of a couple of days ago – as encrypted, end-to-end, email communication, which means to say, I'm going to log into my account, I'm going to put in an email, and that email is going to be encrypted all the way through until it shows up at the desktop of the person that I'm communicating with.

  Now, this is important because a lot of people don't know this, but email goes in the clear. And so anything you put in an email could be read on the way to its destination. So encrypted end-to-end email certainly could have some value, and for people who are dealing with certain specific kinds of information could have a whole lot of value. Unfortunately, it has been definitively proven beyond any shadow of a doubt, because these things are often unambiguous in the world of computer science, that any email that you send through MsgSafe.io at least can be read by MsgSafe.io. So even if it's encrypted when it leaves your desktop, and even when it's encrypted when it shows up on the other end, and even if a random person who is trying to spy on you could not read your email, definitely MsgSafe.io at least could. The reason that that is unambiguous is because the people who do the “encrypted end-to-end email,” don't hold their own private keys. They don't generate a private key. They don't have a private key. But, Jason, you and I know how this works. If it's encrypted, does there have to be a private key somewhere? Yes, there does. So, where's the private key?

  Well, I think it's on MsgSafe.io servers, and so does everybody else because otherwise the system could not work. email would be permanently encrypted, undecryptable.

  And so, these are both deemed to be problematic. Packet Forensics was deemed to be problematic because it's fundamentally in the business of spying on people when they don't realize they're being spied on in their digital usage and communications. And MsgSafe.io is deemed to be problematic because it's a supposed completely private communication that isn't completely private. And that someone somewhere in the system has to know. There's no way that nobody at that company knows enough about computers that they don't realize that this is the case. And so, when you look at those two things in conjunction, these are considered to be pretty problematic.

  Now, these companies are not TrustCor, right? This is Packet Forensics. This is MsgSafe.io. So where does TrustCor enter into it?

  Well, it turns out that the overlap between the three companies is huge. That the founders are all basically the same. That the ownership is all basically the same. Or basically the same, let's say, at least up into the year of 2021. A TrustCor representative stated that ownership was completely separate as of 2021, which I see no way to prove or disprove that, but at least up through 2021, the ownership was basically the same. The founders were the same. And the key executives were basically the same. There's a huge amount of overlap. We had CEOs and CFOs and other C-level people and board members that were in roles in two or three of these companies all at the same time. So, you had senior leadership that’s steering it that was the same people, ownership that was the same people.

  Furthermore, TrustCor, which is also a registrar was the registrar and CA of choice for both of these other businesses. So, if you looked at not only SSL certs, but also the S/MIME certs that were used by MsgSafe.io, they were almost all or all issued by TrustCor. If you look at the domains that were registered, they were almost all or all registered by TrustCor. If you looked at the WhoIs on domains that were owned by all three of these companies, it was some of the same individuals, you know, same email addresses. So, it's, it's very difficult to make a case that, at least for more than a decade, these companies weren't completely intertwined. And even if you're going to claim that for the last year or so, they've been entirely separated, it's very, very difficult to make a claim that extends earlier than that. And meanwhile, these activities, the basic activities, the supposed encrypted email that isn't encrypted, the spyware stuff, all of this has been around for ten years plus. So, when you look at it that way, it winds up looking pretty bad.

  So, the Wall Street Journal article comes out. There is some dialogue on m.d.s.p, which is mostly where these things occur. We've covered this in the past. If you want to hear more about that, look at our episode about Mozilla specifically and its place in the ecosystem. And then Mozilla comes out with a statement that says, basically we're going to give TrustCor - - we have a very well-written statement with a variety of questions. We need these questions answered. And they are things that come out of the article. We're going to give TrustCor time to respond and as of this date, we're going to have a distrust event put in place if we're not satisfied. Then what happens, what occurs, is a great deal of writing on this thread on m.d.s.p. And it gets really frustrating for community people because the general sense from the community is that the TrustCor part of the thread feels to a lot of people like it is obfuscating, you know, picking at knits while avoiding the big issues, sometimes picking on things like grammar and tense while avoiding the big issues. People feel like a lot of questions are just flat out not being answered. And when they come back and ask and say, “Uh, but you didn't answer this question,” the response might be, “Well this was addressed in an earlier message,” which indeed it wasn't. Things like that go on. And there's a general sense that a lot of people are getting that they're not getting an open, transparent dialogue. And it's very frustrating, right? Because I think Mozilla and others went into this with a sincere desire to know the truth, whatever it was. And I don't believe that anybody at Mozilla was married to a position or that anybody at any other browser was married to a position.

  There were also these kind of vague hints from the TrustCor crowd that people who levelled accusations against TrustCor would be seeing them in court, which didn't help their case either. Because nobody wants that. We're just trying to talk about an open safe internet, and earnest actors who are really trying to get to the bottom of that don't really appreciate being threatened when they're doing that.

  This went on for probably not quite two weeks. And honestly, it wasn't going anywhere. Mozilla actually gave them an extension. And then eventually,what happened was Microsoft pulled the trigger and Microsoft did a distrust of the TrustCor roots. So, I gather that the root store program at Microsoft, the people who operate that root store, felt that they had learned enough. And then shortly thereafter, we hit the deadline that Mozilla had set, and Mozilla went ahead with its distrust as well. So, at this point, these three roots are not usable on the Windows stack, which among other things, means they're not usable on Chrome on Windows, because the Chrome Windows app still uses the Windows root store as of today. And they're not usable on the Mozilla platform, on Firefox. So that's it. Like, it's hard to imagine the scenario where these certificates are really, practically usable, and TrustCor is, gone. It's no longer a CA.

  So that's the long and the short, the existing certificates are fine. It was a notBefore distrust, which means that if you got a cert on October 31, it'll be fine for the duration of its lifespan. But basically, they prevented TrustCor from issuing any new certificates that are going to be usable

• 

  Jason Soroko

  Tim, it sounds like this was - - obviously, the way you described it, part and parcel of this set of companies, some of which might have been involved with some things that were shady, who knows? This question is, is the history of TrustCor long enough that it predates any of…

• 

  Tim Callan

  Not really. It’s a great question, Jay. It’s very much a good question to ask. No. It's not like TrustCor was this existing CA that became used by these, and these are important things to clarify. These weren't just customers.

  If you were a CA, and you had a customer and the customer is buying certificates from you, and it turns out the customer is doing something skeevy with those certificates, there might be no way for you to know as a CA. And it would be hard under those circumstances for a responsible community to turn around and say we're going to distrust the CA because the CA’s customer without their knowledge did something bad. It would be like saying if I sold you raw materials. If I sold you raw materials and chemicals, and you use those raw materials and chemicals to build a bomb, I didn't build the bomb. Right? And so, in that regard I think there would be a much different approach. I think the point that was so important was all of these companies really were very connected. There was a complicated set of shell companies where there were companies in Panama, and there were companies in the Caribbean, and there were companies in Canada, and how the ownership passed between these things was, one might think, built to protect themselves from certain legal actions.

  The other thing is, and this wasn't, this would not have been enough for distrust on its own but there were differences, let's say irregularities, behind some of the operations that were published in places like these companies’ websites and their WebTrust audits and what actually occurred. So, there was a significant American datacenter in Arizona, by the way, that TrustCor was using that did not appear in all of its documentation to be covered by its WebTrust. I think that was in the report not in the Management Assertion, I believe. I might have that wrong. It might be backwards, but there were gaps in the paperwork as well. Interestingly, that particular datacenter that was missing from some of the public CA paperwork was the exact same datacenter that was also used by MsgSafe.io.

  Now, again, multiple companies can sit in the same datacenter. This is a thing that happens. But when you put all of this together, and when you put in the inability of earnest people to get good explanations of these things out of the actors involved. So, a lot of questions were asked. And these vague assertions would be made like, “Well, there is an operational firewall between MsgSafe.io and TrustCor that is complete and absolute.” And then some people came back and said, “Great, show me that. Show me the documentation. Show me how I know that to be the case.” And nothing ever came out. So, there is a problem with things like that. There was a community that was open-minded and prepared to learn that these uncomfortable coincidences were just coincidences, or were properly safeguarded, or things along those lines, and the community couldn't get that. And that was part of as it well. But no, that's a great question, Jason.

  And another question that’s connected that is you'd say, well, do they have a significant set of customers beyond these? And the answer is not really. I'm not saying there were no customers, but it was not very big at all. When you look at that, it certainly, number one, it means it lessens the impact on let's call them innocent parties, if the CA is taken out. Because people who are just buying from the CA who don't know what the CA is doing, and they just want to live their lives, you think about the impact you have on them. It's a valid consideration. But in that case, that was a pretty small subset.

  And then the second thing was that it reinforces this idea, this hypothesis, that these companies really are part and parcel of the same activity. And so, when you put all of those things together, I don't think anybody's level of discomfort really went down in any way over the course of this investigation.

• 

  Jason Soroko

  Right on, Tim, I think that's a super summary. And I definitely understand more now than I did before.

• 

  Tim Callan

  Yeah. So anyway, it's distrusted. The other thing that's probably worthy of commenting here is we do see these distrust events in the past. Usually, they are about technical competence and operational competence. If you go all the way back to DigiNotar or if you look at some of the more recent distrust events, you know, Certinomis, and CamerFirma and some of these distrust events that happened more recently, ultimately, what it came down to was the community. And you know, in those cases, Mozilla, saying we are not prepared to trust that this company is technically and operationally able to maintain the standards that we demand of a public CA.

  That was not what this dialogue was about. There was not really any question raised at any point about TrustCor’s operational capabilities. Like I said, it was clarified on multiple occasions through the thread that there's no reason to believe that they mis-issued certificates. The concern was an entirely different thing. It was a concern about the role they're playing in the digital ecosystem versus privacy and spying, and all of that. And so that was, in that sense, also a very unusual. It was a very unusual event in terms of it was really about a different fundamental concern than other recent distrust events have been about, which was about competence and capabilities and operational excellence. And so, in that way, this one also jumps out as kind of an unusual case and it's something that's worth noting.

• 

  Jason Soroko

  Great, thank you, Tim.

• 

  Tim Callan

  These things don't happen very often and it'll probably be awhile before there's another one but you do see that the root programs I should say are actively monitoring the roots that are in their stores, and that they do view themselves ultimately as the stewards of the safety of their end users, their relying parties. Which in the case of consumer facing browsers is just ordinary internet users. They believe that they're the place where that has to be enforced, and that is how they behave. You see that at work here as well. We see a lot of how the CA ecosystem really operates going on in this particular event. And certainly it's an interesting one to watch, and if you're interested in getting more details, there's lots of reading, but you can pretty easily find the thread online. It's all happened out in public. Someone you can go read all of it, and there's a lot of reading to be done. But, you know, it's a fairly interesting read.

• 

  Jason Soroko

  I can't recall a distrust that was quite like this one that wasn't about, as you said, technical issues, competence issues, you know, even just being hacked issues. I definitely think this one is a little bit unique, especially in terms of the more recent distrusts.

• 

  Tim Callan

  There was a little thread of what's called an integrity argument in the Symantec distrust by Google, because one of the things that Google didn't like was the fact that Symantec had issued certificates to Google domains improperly. And the other thing that Google didn't like is that they felt that Symantec’s reporting of what had been done early on very much mismatched what turned out actually to be the case and the only way that the public found out it was the case is other people went and investigated, and Google investigated, and they discovered things that originally Symantec had not stood up and owned. And nobody got so far as to say Symantec lied versus Symantec couldn't figure it out. But either one of those is bad, right? I don't know that it was openly stated, but when you watch that dialogue, there was a sense of saying, we're not sure we trust your capability and we're also not sure we trust your intentions. This one was much stronger. Again, there was no discussion of their capability. I can't comment on TrustCor’s capability. It could be there were flaws there that nobody looked at because that wasn't the point or could be or not. But that was never what was being investigated. And in that sense, this is also very unusual.