Root Causes 90: An Analysis of Distributed PKI

Tim Callan

How you doing today, Jason?

Jason Soroko

Doing great, Tim. Thanks for having me.

Tim Callan

We are also joined by our many times repeat guests, Alan Grau. Alan is VP of IoT and Embedded Systems. How are you doing today, Alan?

Alan Grau

I'm doing great, Tim.

Tim Callan

So, thank you for joining us today and we are here today to talk about distributed PKI. I think both you gentlemen are pretty familiar with and understand the concepts behind distributed PKI and for starters, maybe before we get going, Alan, Jay one of you can you just tell the listeners what people mean, when they say distributed PKI.

Alan Grau

Sure, Tim I'm happy to jump in and cover that.

Tim Callan

Great.

Alan Grau

So, when people say distributed PKI, generally they're talking about a PKI solution that's based upon using the Blockchain protocol or a Blockchain protocol for storage, as well as some of the communication requirements that go into implementing a PKI solution. So, at the very highest level, that's what people are talking about. Putting a traditional PKI system or putting a PKI implementation on the Blockchain for storage of certificates, for enabling requesting certificates as you would with a certificate, a CSR certificate signing request, for enabling checking of revoked certificates. So those are the elements that would then be implemented using the Blockchain for storage of issued certificates, as well as a protocol element for handling the issuance and revocation checking.

Tim Callan

So how many blockchains are we talking about here? Is it one? I'm guessing - - is it a Blockchain for the certificates so we know that there? Are the keys - - but the keys aren't on the Blockchain because they can't be, right?

Alan Grau

Right. So, it would be one Blockchain where certificates that have been issued would be stored. So, you could go look up certificates and then certificates - - if they were revoked, they would also be stored on that same Blockchain. So, everything would be stored on the same Blockchain. And I guess one of the things if we kind of take a diversion here for a second, when you say you've got a Blockchain implementation, you know, there's a couple of - - there's more than one element in any Blockchain implementation. There's the storage itself, right, the distributed storage, where you’ve got the blocks of data but then there's a host of protocol elements that are also included, that allow people to add records to the Blockchain, to determine - - there usually has to be some sort of a consensus mechanism to determine, you know, what records are added or if there are conflicts between information being added to determine what gets added and what doesn't and those protocol elements can be leveraged to handle the certificate issuance, certificate revocation and certificate checking pieces.

Tim Callan

Gotcha. So, I'm thinking about the high certificate volume implementations that exist in certain scenarios, you know, there are an awful lot of IoT certs out there in the world by way of example, there are certainly a lot of public SSL certs, does your Blockchain get infeasibly large? Is this something that's kind of limited in terms of how many certs you can have in your ecosystem?

Alan Grau

Blockchain solutions are actually highly scalable, you know, if you look at the public Blockchains that are used for cryptocurrency, right, there are, you know, vast numbers of transactions occurring on those and some that are used for applications other than cryptocurrency. So, you know, while I see some limitations with the Blockchain-based PKI solution, scalability isn't really one of the limitations that I'd be worried about.

Tim Callan

So, and is this a real thing or is this kind of an idea that's floating around? Like is anybody doing this?

Alan Grau

There are some folks that are actually active in the space. There's a protocol called Remme. It’s R-e-m-m-e, is the name of the protocol and there's a company called remme.io. that is behind that specific implementation. So, they're utilizing an open-source implementation that they've created. It's called PKI ID, or distributed PKI, that they've implemented, and made open source along with some other tools for doing things like certificate discovery and management and some identity and access management utilizing their protocol and, you know, it an early stage company, but they do have some traction and some users and are seeing some growth in their - - at least the number of people that are utilizing and trying their solution out, just based on information I've been able to uncover on them. And there are some other folks that are experimenting with similar solutions. But they're all very, very early stage and all utilizing or implementing private PKI solutions. This isn't something that's crept into the OpenSSL or the public SSL space yet.

Tim Callan

So, this a fairly new concept and a fairly new initiative then in the grand scheme of things I gather?

Alan Grau

Yeah, definitely.

Tim Callan

Why would somebody say, I prefer a distributed PKI strategy over I guess, what I'd call it a hierarchal or a centralized PKI strategy, why would they that?

Alan Grau

Well, there's a number of reasons and, you know, whether they're great reasons or not is an open question.

Tim Callan

Okay.

Alan Grau

So, I believe it was 451 Research said that, you know, this is a great solution searching for a problem. You know in that they weren't seeing clear, definitive, compelling use cases, at least at this point in time, for having a distributed PKI. Now, if you go read some of the literature that the people who are doing this put together, they certainly have their arguments as to why the distributed PKI, you know, is a preferred solution and I think it's probably worth spending some time on what their arguments are and whether those are valid or not.

Tim Callan

Yeah, so definitely worth it. So, like what's argument number one?

Alan Grau

Well, so, you know, they make a kind of a generic blanket claim that kind of Blockchain-based solutions solve problems with what they're calling old school PKI, or the traditional centralized PKI and you know, one of the things that they describe, as they say, well, the some of the traditional PKI solutions are hard to manage. And, you know, one of the things that we spent a lot of time working on at Sectigo, I know you guys have spent a lot of time on this podcast talking about, are all the automation capabilities that people are building to make the newer PKI solutions easier to manage, easy to utilize. So, you know, there's no question that, you know, the PKI of 10 years ago was not easy to manage, but newer PKI solutions actually are much, much easier to manage and that's not anything that's inherent in it being a distributed PKI solution, or a centralized or traditional PKI solution. So, to me, that's a bit of an argument that—

Tim Callan

Right, and we run into this a lot, right? Like people who ventured into the PKI world a decade ago, and hated it, come away and say, oh, that's not - - that's all really hard, right, and don’t notice what went on in the interim. It's kind of like the analogy I use a lot is it's sort of like, you know, so you wrote a word processor in C++. and now you think word processors are too hard for people to use, right? Well, you know, don't write them in C++ buy - - get Microsoft Word and you'll find that that's not the case. And, you know, I think that's a lot of what we see here.

Alan Grau

Yeah, I wrote one in college for a Vax in Pascal. We ended up calling it - - we call it Mangle because every once in a while, it would mangle up your document pretty bad.

Tim Callan

Right. So, you've had that experience firsthand. Yes, exactly.

Alan Grau

So, the next thing that they argue is that there are challenges in scaling PKI solutions to support kind of modern distributed applications. And this, you know, kind of falls into that same category as well.

Tim Callan

Yeah.

Alan Grau

The newer solutions are much more scalable.

Tim Callan

Just based on the evidence, that doesn't seem to hold. I mean, I suppose you could imagine a scenario that's orders of magnitude bigger than anyone's done today and ask if it's going to scale to that level and maybe Blockchain would and that won't and maybe that's an argument but in terms of what's really happening in the world, like PKI is everywhere. If there's a chip, there's PKI and it's scaling just fine.

Alan Grau

Yeah, no agreed. And, you know, modern computing infrastructure is providing greater scalability. So again, you know, it's not something that to me rings true is a fundamental flaw or a fundamental advantage of distributed versus traditional PKI.

Tim Callan

Would distributed scale - - like if we talked about an extreme scenario, where hierarchical PKI couldn't handle the scale, would distributive be a solution in that kind of scenario do you think?

Alan Grau

Potentially.

Tim Callan

Okay.

Alan Grau

You know, I think that devils in the details on how you handle the authentication mechanisms, the validation mechanism and things like that so.

Tim Callan

Sure.

Jason Soroko

So, Alan, I got a question for you the—

Alan Grau

Yeah.

Jason Soroko

There's some obvious stone throwing against traditional PKI. They brought up terms, you know, old stories such as DigiNotar as a CA that fundamentally messed up. The CAs that are around today, you know, are nowhere near having that kind of a track record and yet they throw stones at CAs like that. And some of the other things as you're saying, Tim and Alan are really all about, you know, they're almost like they're casting stones at PKI 15 years ago and 20 years ago compared to what it is today. I'd really like to get into what is more fundamentally interesting about distributed technologies and Blockchain in particular that could make it interesting and then seeing if we still have an argument against distributed PKI and so, therefore, you know, the scalability aspect, I think is par. Right? We know that scalability is not a problem for traditional PKI. That's been solved. I think one thing that is interesting, if you just go back to good old Alice and Bob, trying to do a transaction together and if there's, you know, the comp - - the whole concept of revocation, in a traditional PKI sense which is Alice and Bob are exchanging keys and Eve comes along and Eve's certificate has been revoked because of the fact that the private key has been revoked, because of the fact that, you know, somewhere down the road, Eve has been known to be a bad person. Alice and Bob don't know this yet so they have to go and check a centralized system to see whether or not a revoking of Eve has occurred. And so, something like an OCSP response occurs and then all of a sudden Alice and Bob now know that there is a problem.

Problem is, of course, there is a level of latency to that, it means that there has to be some sort of double check and there has to be work done. With something like a public Blockchain something like a distributed PKI the presumption is that that step is almost inherently part of the design, meaning that if Eve has been booted out of the trusted league of people, that is known inherently by Alice and Bob as part of just how the system is set up and that is an advantage.

Tim Callan

But that work still got to be - - had to be done right, Jay?

Jason Soroko

Work still had to be done.

Tim Callan

It done earlier.

Jason Soroko

Completely right. So, Alan, this is where I'd like you to comment is really, my sense is that especially in the public trust sense, the scalability of OCSP is still, you know, is massive. I mean, as a public CA we respond to billions of OCSP requests per day. And in terms of IoT, even more interestingly, the problem of revocation really looked difficult at the beginning, whereas the constraint to the devices, the constraint of the networks meant that traditional PKI revocation looked like it just wasn't a good idea but what came to the rescue was the concept of shortening the lifespan of the certificate itself. So just curious to know any comments you have about that thinking, Alan?

Alan Grau

Yeah, no, I think when you when you start to look at these topics, you really have to look deeply at these use cases to understand do the claims hold true or not. So, if you look at OCSP, I mean, those systems are actually going to be distributed systems, right? It's not, you know, just one single point of failure type system, right, running on to kind of your typical high performance cloud infrastructure. So that is, in a sense, already a distributed system. With distributed or PKI or a Blockchain- based revocation, or PKI system, again, you still have to do the same work, you have to revoke the certificate, somebody has to make that decision that has to get added to the Blockchain and then the device has to do a check on that certificate when it's using it to make sure that it's still valid. And it just ends up being a different mechanism. So, it ends up being part of the Blockchain protocol, right, there would be an element in the Blockchain protocol to check the status of one of the other certificates. Now, depending - - and this is where the implementation becomes really important. That could end up being a lighter weight protocol, allowing the visit small footprint IoT device with limited resources, allowing it to more easily check the status and it go to the OCSP or to check it in a way that doesn't require as great of compute intensity on the device. So that is one use case where distributed, PKI could hold an advantage. It's not entirely clear that it does and a lot of it will depend upon the implementation details, but that's certainly one to pay attention to.

Jason Soroko

Alan, I have another thought. Just - - that's thinking along lines of private trust. But, Tim, this one's for you. In public trust, traditional PKI SSL has gone towards certificate transparency logs, CT logs, which in itself is a public distributed system, right?

Tim Callan

Right.

Jason Soroko

That's one of the ways in which that that problem was solved, and therefore some of the slings and arrows being shot by the distributed PKI guys - - these problems have already been solved.

Tim Callan

Yeah, CT log I think isn't actually a Blockchain, but it is rather Blockchain-like, right - - like, right, it is similar.

Jason Soroko

It's a Merkle tree. It's a hash Merkle tree.

Tim Callan

Exactly. But um, you know, it's interesting that you bring that up because I was sitting here and the next question I was gonna ask is, well, are we talking about public trust or are we talking about private trust? Because on the one hand, you know, this whole argument about, well, it's distributed, and it's wherever it is and it's all real fast and efficient and scalable. All that screams public trust kind of scenario, right? If I have my little walled garden and I have my 500 employees and I need 500 certs, scalability isn't really a matter to be considered. On the other hand, though, if we're talking about public trust then you've got a whole world of mechanisms. You've got standards, you've got CT logs, you've got browsers that work in certain ways, you have back-end infrastructure and servers and services that work in certain ways and retooling all of this stuff for a fundamentally different strategy, where now we're looking at Blockchains and writing to Blockchains – wow! Like that is nontrivial.

Alan Grau

Yeah. And that's actually really kind of a good, an important point, because there are a number of functions and capabilities that it's not clear to me yet have been addressed when you look at distributed PKI. One of them is the validation process, you know, if I want to get a certificate issued to me, I need to be validated.

Tim Callan

Right.

Alan Grau

And if it’s in the public SSL world, right there's a specific defined process.

Tim Callan

Sure. Or you use someone used the example of Eve got revoked because Eve was a bad actor. Well, who decided Eve was a bad actor and revoked Eve? Right now, in the public world - - right now, the answer is the Certificate Authority. Right? In the public world, it's the public CA at Sectigo are one of our peers in the private world. It's whoever stood themselves up at the top of that hierarchy. It is hierarchical in its nature. Once you take the hierarchy away what is the mechanism for deciding that Eve is a good actor or a bad actor and all that has to be worked out too.

Alan Grau

Right and the consensus mechanisms that would be required for this, I think are probably not at a level to handle those sorts of questions yet on a distributed PKI.

Tim Callan

Right. The consensus of mechanism is never gonna say, oh, this cert is being used to distribute malware. Right? That's not what the consensus mechanisms do. The consensus mechanisms say this has trusted status or this has untrusted status. By what is the evidence trail and what is the decision-making process that leads to someone deciding to take away trusted status? I - - maybe you guys are better at this than I am, but I can't fathom how a Blockchain would be solving that.

Alan Grau

Yeah. You start to get into questions like, you know, having - - with Blockchain, you do have mechanisms for smart contracts that can implement certain decision-making processes but the level of complexity of the decisions that need to be made are probably well beyond what most smart contract mechanisms could handle today.

Tim Callan

Yeah. If you're imagining something that goes through, like a simple flowchart, that's also valueless for purposes of deciding that we're not going to trust Eve, because Eve is a bad actor, right? I mean, at the end of the day how does that go, right? Somebody is caught using a cert on a phishing site and then you go, and you revoke the cert. Well, how am I going to build that into your simple flow chart?

Alan Grau

Right. Exactly. So, as we look at distributed PKI, I do think it's still very, very early days. And there's a number of questions that I would, you know, if I were going out and researching, implementing a solution like this, I would need to spend quite a bit of time and get quite a bit of detailed to understand how do they provide protection for root private keys? I mean, if it's truly distributed, who's doing the signing to validate to, you know, when you have a certificate signing request, who signs that? And how is that private key protected, and, you know, CAs such as Sectigo, are doing that using a hardware security module and we keep the root offline in the data center with both high levels of cyber protection as well as physical protection. So, how do you map that into a distributed environment? So, there's some really fundamental questions that, you know, they require a lot more research. And perhaps people have thought about that I haven't yet dug into this into that level of detail. But yeah, it's still very early and there's a lot of integration questions and use case questions that would need to be addressed before I see this technology as being ready for kind of mass adoption.

Jason Soroko

Yeah, Alan, some of the questions that I definitely have is, you know, do I have to buy coins to use this service? Who is hosting my nodes? Are those nodes mature enough? Like a major publicly trusted CA, with 20 years of experience of how to protect, you know, the actual root? The questions go on, and on and on.

Tim Callan

Is there any regulation? Are there any standards?

Jason Soroko

Exactly, yeah. So, I guess final comment for me, guys is, you know, Tim, you and I, we've explored the Blockchain concept a lot. We bring Alan in to help explain the technicals and, you know, we both agree that there probably is killer apps out there for Blockchain. I don't think PKI is one of them, at least at this point in time.

Tim Callan

Or there might be a niche, right. I could imagine, someone comes up with a scenario and says in the following circumstances this is the right way to do it for these reasons, assuming we answered all the questions that we just sort of started to ask, which wasn't even remotely a comprehensive list, I'm sure. But if all that gets worked out and all those questions get answered, then you might say, look, under these circumstances, this makes sense. But it feels to me like that that's the direction this technology is probably going as for very specific use cases and circumstances. What do you think, Alan?

Alan Grau

Yeah, the one that occurred to me and I'm not even sure that this fully works, but if I want to implement a full PKI solution from digital wallets for cryptocurrency that's running on a Blockchain, then perhaps that's a scenario where utilizing the Blockchain to store certificates and do certificate revocation checking and some of those things would actually be valuable. You still need a centralized store for the private root key, you know, for the issuing CA and a few centralized components. So maybe it's a little bit of a hybrid model. But if you're implementing a Blockchain based system, then for some other application, such as cryptocurrency, then perhaps it fits. But yeah, no, at this point in time, that would be my analysis. You know, it'll be really interesting to see if we came back in, you know, 10 years, when this technology is really evolved to see if it really has evolved into a niche solution or if it's matured in a way that it becomes much more broadly applicable. But at this point, I'm not seeing it as being as broadly applicable.

Tim Callan

We'll look into that for Episode 900. When we get there. We’ll return and find out where we have been. Thank you so much for explaining this. This is, you know, I think an important thing for us to be aware of and keep our eye on, you know, obviously not really happening in the real world today. But you know, that's how every technology starts. So, you know, it's good to tell the listeners what's going on and keep them current. Any final thoughts? Either of you, Alan, Jay, before we go?

Alan Grau

I think we've covered everything that was on my mind, Tim.

Tim Callan

Great. All right.

Jason Soroko

Thank you, Tim.

Tim Callan

Thank you very much. As always, I love our conversations. Thank you, Listeners. This has been Root Causes.