Root Causes 353: Why Isn't PKI Everywhere?

Tim Callan

So, Jason, you and frequently talked about, in one way or another, about the advancement of PKI everywhere. It’s one of the things that we discussed in our 2023 Lookback Episode recently. You know, especially with new innovations like passkey and web – I don’t want to say innovations, but new rollouts of things like WebAuthn and aka FIDO2, etc. and, you know, I think we both agree that PKI should be in all digital interactions and systems and part of all digital identity. And yet, I’m saying should, not is. So, it’s 2024. Like it’s later in history than it’s ever been. What the heck is taking it so long?

Jason Soroko

Let’s break down three of the most important things that PKI does. Right? When you are talking about asymmetric secrets, you are either encrypting or you are authenticating or you are signing. So those are the three things you are doing and what I think is true, Tim, that we can say is that it is very, very prevalent in society. It is far more ubiquitous than people think.

So I think what you are saying, Tim, is let’s call out some of the surprising areas where it still isn’t and probably should be. I think that’s what’s interesting to me.

Tim Callan

Sure. I agree. So go ahead.

Jason Soroko

Tim, I think there’s a pattern being filled in here. Like we can come up with the anecdotes that either fit this pattern or break the pattern. I think the pattern is any kind of consumer-level technology, it’s a mix.

It’s a real mix. And it’s surprising. Because your smartphones, for example, consumer-level technology. But it’s very advanced and, my goodness, it is incredibly prevalent within the iPhone environment and any of the devices that typically host Android, etc. There are keys and certificates all over. In other words, asymmetric secrets are doing all three of those things.

Right? All over. But, and yet, there’s all kinds of consumer-level systems that are, you know, burgeoning IoT systems, for example.

Tim Callan

Sure.

Jason Soroko

Where the amount of asymmetric secrets going on there is so low. It’s just so low in consumer-level IoT. It’s almost, you know, it’s amazing just how infrequent it is employed and so part of the pattern there is perhaps it’s some kind of cost sensitivity is the excuse. I don’t think it’s the real reason, but it definitely is the excuse. I definitely think the cost of designing it in or the difficulty of designing – the perceived difficulty in designing it in is the real problem.

Tim Callan

Yeah. And I think there is a reality of the supply chain problem when it comes to a lot of IoT devices which is there’s a couple of things going on. One is just speed to market. Right? These are fundamentally fast-moving categories and you need to come to market with new products fast if you want to stay ahead of the game. And so, as soon as you are building PKI into it, you are adding complexity to a project. You are adding engineering overhead and they don’t want to. Right?

The second thing I think related to that is flexibility in use of chipsets. All these things have chips. There tends to be a relatively small number of fairly standard chips that are used for many, many, many, many use cases. And manufacturers want to be able to source these chips out of Taiwan and they want flexibility. They want to be able to go from manufacturer to manufacturer, get a quote. How many can you get me? What’s it gonna cost? And when can you deliver it? And they want to pick the quote that best suites those three. And trying to suddenly lock yourself into some kind of PKI algorithm at the chip level just hurts their flexibility in creating their bomb. And again, they choose that over the security. And why is that? That’s because the market doesn’t reward security.

Jason Soroko

That is true.

Tim Callan

And if the market did reward security then they wouldn’t skimp on it. Right? But it doesn’t.

Jason Soroko

I find it interesting that not even laws – and you and I had some podcasts, Tim, where we talked about North American laws, Australian laws, European laws.

Tim Callan

There’s laws. Right.

Jason Soroko

That, you know, minimum levels of security had to be put in place and you know what? As far as I can tell that’s pretty much been ignored and not many people went off and enforced any of that.

Tim Callan

Right. And then so why doesn’t the market reward security? There could be a bunch of reasons for this from category to category but at a high level, I think it’s just because consumers don’t understand enough. That the average person with a few hundred dollars in their wallet who is going out to buy something for their home, a smart door bell, just doesn’t have the level of security education to appreciate the difference between a high security and a low security product. It’s just not what they do all the time, and I don’t mean that as a slight and a smear and the say way that I don’t know particularly much about dentistry, your average dentist doesn’t know particularly much about IT security. It’s just not their area of expertise. And so, as a result, they make different decisions. They pick the cheapest product. They pick the product with the cool new feature. They don’t pick the product that’s secure.

Jason Soroko

So, Tim, let’s come up with the list of patterns then. The two that we agree on here in this first set of anecdotes is tough to design in. Right?

Because there is some effort that’s required. There’s some effort that’s required. Like, for example, Apple and Samsung have the capability to design it in. And so they do. Right? They do.

Tim Callan

Right. Yeah.

Jason Soroko

But other types of consumer-level electronics, they don’t have that kind of the trops to do that so it’s tough. And also, there are some cost features to the reasons. I would say, Tim, let’s choose another area and pick it apart here and that is, let’s just for the sake of just making it very general, critical computing. And so that’s everything from a car. Right? There’s computing systems in a car. And if the computing systems in a car go down, well, that’s bad news. Healthcare systems. Right?

Tim Callan

Sure.

Jason Soroko

In the operating room there’s all kinds of systems in there that are computerized and could benefit from asymmetric secrets and then we could go all the way to very large scale either manufacturing or the really critical stuff, uh, nuclear plants, water chemical plants, etc.

Tim Callan

Infrastructure, the power grid.

Jason Soroko

The power grid.

Tim Callan

The traffic lights. Yes. Absolutely.

Jason Soroko

And it is incredible, there’s been lip service, but it’s never been much more than that and hey, if somebody wants to get ahold of this podcast and talk to us about your phenomenal usage of asymmetric secrets in critical infrastructure, we would love to talk to you.

Tim Callan

I would love to do that. I would love to interview you and make you a hero in front of all of our listeners. For sure. But go on, Jay.

Jason Soroko

Yeah. But my point is, I’m thinking we are probably not gonna get a lot of knocks on the door because there’s not many people doing it and I think – let’s call out the pattern. Whether it’s a reality or excuse, I think reality is part of it here is, geez, these systems were set up years ago.

Not only is there technical debt and it’s tough to refactor but, also, people who set up critical systems are experts in uptime. And we’ve talked about this before. True, true heroes of making things that have to work always work. And by introducing rule sets around asymmetric secrets, they risk that.

Tim Callan

Right. An unrenewed certificate can make the lights go out. Or an unrenewed certificate could make the brakes stop working. And so, that is viewed as a risk factor that they don’t particularly have an appetite to take.

Jason Soroko

Absolutely. I’ve seen proposed PKI systems for automotive systems overall and some of the most complex PKI I’ve ever seen and I’m sure that car manufacturers, the OEMs, when they look at that, they think to themselves, ok, (a) expensive. But (b) what happens if I gotta recall this?

What do I even do?

Tim Callan

Or what happens if I somehow get it wrong and someone is standing in the 20-degrees below weather hitting their key fob and the car won’t unlock? Yep. Absolutely.

Jason Soroko

All of it. All of it. And especially once you start to get into the guts of the car. You know, so many things now are computerized. We talked about it years ago in aircraft. We were all amazed by concept of flyby wire. Well, flyby wire is in cars now for the most part and when you press that gas pedal, you might think that’s it’s engaged mechanically but in reality, like your shifting knob right now is probably flyby wire.

In just about all modern vehicles. And so, imagine if you were encrypting the communication between the systems, between the different modules within the vehicle and, you know, whether there’s an authentication sequence or even just making sure, Tim, that even if you are not encrypting, even if you’re not authenticating, is every single computer system within the car, does it have a signed firmware so that it doesn’t get changed by some sort of malicious over-the-air update event.

Tim Callan

Right. So, this is what I start thinking about with the critical systems is we say, ok, well, you know, we don’t want the lights to go out because we didn’t – we couldn’t manage our certificates. But we also don’t want the lights to go out because bad guys turned the lights out!

Jason Soroko

I gotta tell you, what you are talking about there – the grid. That’s one where I gotta shake my head and go, guys, I realize that uptime is it’s almost a religion. And I don’t say that in a bad way. I’m glad that it’s a religion amongst people who do operations for our grid. But being luddite level in terms of how you are protecting those systems from a computer standpoint, computational standpoint – goodness gracious. You guys need encryption and authentication and signing all over your systems in order to protect yourselves in the same way that an average enterprise does.

Tim Callan

Right. And then the other thing probably connected to that when you start talking about these critical systems is these critical systems are old. These same organizations and businesses that have been providing water or processing sewage or building automobiles have been doing that since before there were computers. And so, you’re building on this long legacy platform and one of the things we see with legacy platforms is they’re slower to change. They have trouble keeping up.

Jason Soroko

Very much so. And refactoring these systems was never going to be easy. I actually have been involved in architectural discussions of these kinds of critical systems and the problem I see though, Tim, is net-new systems, brand-new computing systems that are being put in also – even though the computer may be new – the types of authentication going on and the types of protocols being communicated on these systems are all circa 1970 in a way. Sometimes you could say to yourself, hey, this core system that’s from the 1960s, it’s sitting there and its going to be sitting there for the next 40 years at least and anything we are putting in new, hey, let’s at least think about what we could do to protect these things and I don’t see a lot of effort for that right now.

Tim Callan

Yeah. And then that goes I think also to, again, part of the thing with these critical systems being in these sort of different organizations is I think you have a cultural difference and you’ve kind of talked about with this preference to uptime over security. You have people, these aren’t born in the cloud, born tech native organizations. They just aren’t. And so people think about technology in a different way. They attract a group of people that are perhaps a little less computer-y. The people who have risen to the top who have spent their whole career at these places are perhaps a little less computer-y and in general, it’s just not the same culture of engrained IT security thinking that you see at a place like a pure technology company.

Jason Soroko

Tim, I almost wonder if this bias against what is looked at as being off-the-shelf, you know, consumer-level or people who look at computer systems and think blue screen of death. You know, 1990s blue screen of death. Like they are mortal enemy of just computer systems that are just gonna go down that I can’t trust. What I’d like to say to them is, yeah, but there are critical industries that did embrace asymmetric secrets that are ticking along just fine, and I’d like to point to finance as an example.

Without finance systems sloshing money back and forth across the world all day long every day, our society would probably come to a standstill. And that’s the truth. And yet, there’s very few systems that have more encryption, authentication and signing in those systems.

It’s completely ubiquitous and prevalent within finance and so what I’d like to say is there are examples of critical systems that do use it.

Tim Callan

Yeah and I’ll contend that in the case of finance systems probably the reason that occurred is because there was an obvious immediate existential need that if they didn’t do it they just plain wouldn’t have a bank.

Jason Soroko

That’s right.

Tim Callan

And so they had no choice. And this reminds me of a conversation that you and I had about – and this was probably about a year ago on one of our episodes – where we talked about one of the effects of COVID was that buy online/purchase in-store which was claimed to be an impossibility by bricks and mortar retailers for 20 years suddenly occurred in three months when it became an immediate existential need. And it turns out that no! We didn’t need 20 years to do it. We actually needed three months. And you see the same thing with banks. Any bank that did not use PKI and encryption just ubiquitously across every aspect of its business would just plain go out of business because people would come steal all the money. And therefore, it’s an immediate existential need and what do you know? Surprise, surprise. It turns out they can do it. And I think maybe in some of these other functions, even though you and I talk about the terrible risks of shutting down the infrastructure and making cars crash on the freeway or other really bad things, it isn’t perceived to be the same level of absolute annihilation level existential threat than it was for financial institutions.

Jason Soroko

Right on, Tim. So, for all these patterns that we just talked about, I’d like to just talk about what some of the solution may be and I think for any of you who are in those other industries where asymmetric secrets are just not ubiquitous, not out there was much as they could be or should be and you guys know it. You guys know it. It would be an ideal world if you were using asymmetric secrets but what needed to come along was something Tim and I have been talking about now for a long time on this podcast, which is certificate lifecycle automation – meaning automation being a big part of it. In other words, before the days of ubiquitous and very well running automation, it’s difficult to be able to renew those certificates. Difficult to be able to get key management systems to be able to get – get the secrets to where they needed to be from the source they needed to come from and we are now living in an age where that’s possible. We are now also living in an age where secure elements and the ability to interact with them – in other words, the place to put the secrets. So, automating the secrets, where to put the secrets, and even some of the critical hardware that you guys are gonna be needing to work with. Tim mentioned at the top, the chipsets, the supply chain has also evolved greatly as well. There are now chipsets from all the major chip manufacturers that are able to have built-in secure elements built in ability to provision certificates from where they need to go to where they need to go and do it on in a timely manner with just as much uptime as any of your other computing systems will offer you. Things aren’t the same as they were 5, 10, 15, 20 years ago. It’s time to look into these things and I would really like for asymmetric secrets to be truly ubiquitous in every computing system, not just a handful of them, as you said Tim, where it was either an existential threat to enterprise, to finance systems, etc. I’d just like to see it everywhere and the excuses for it are becoming less and less.

Tim Callan

Yeah. So, all that’s well and good but let me throw one more at you.

Logins. Freakin’ logins. And I don’t even mean logins on your little consumer site where I don’t know anything about computers, and it’s deemed to be a low stakes kind of environment. And you and I have talked about the horror show that is passwords more times than we can count but logins at the enterprise level. Logins for people who work for large companies or government or places where they absolutely have the knowledge and the acumen and the resources to get it right and the stakes are high and yet I’m still logging in with username and password. You know, basic shared secret strategies. What’s going on there?

Jason Soroko

That’s just good old-fashioned. Look, it really comes down to if you are architecting the systems and you don’t know that better technologies exist, well, that’s an error of what?

Tim Callan

That’s just an error, you are saying? Yeah. Ok.

Jason Soroko

So, I have spoken to a lot of architects, good and bad, and what amazes me is I’m always - - some of the people are gonna say, look, I know those things exist but I’m under a time crunch so I’m always just gonna choose the easier way and then there’s also a group of people who will select things like multi-factor authentication because, hey, it ticks the audit box and it’s good enough and then there’s the flat-out people who think of computing systems without thinking of its integrity or security and that is unfortunately a large population of not just people who write the code but also architects. I’m not badmouthing anybody, but I have seen people who are like it just never occurs to them.

Tim Callan

Yeah. So, that’s one. I mean that also seems like just an obvious one. Right? Like you’ve got the resources, you’ve got professionals who are here to do this; this is how your job is and what’s the cost of a breach? And what’s the cost of a breach compared to the cost of putting something in place that is just fundamentally more secure?

Jason Soroko

It’s actually amazing to me how it used to be you could turn on the news on any given day and the latest breaches were all over the place and there used to be charts about oh, here’s all the breaches this year and you notice how you don’t even see that anymore, Tim?

Tim Callan

Yeah. It’s too big. It’s too many to put in a chart.

Jason Soroko

There it is. There’s the answer. So, guys, you know, architects of computing systems, it’s not that anything has gotten better. Things haven’t gotten better.

Tim Callan

Yeah. I mean, yeah. There’s this thing that happens with scale. Right? Like you live in a relatively small city, right? If there’s a fatality accident in your city, it probably makes the news. If you live in New York City and there’s a fatality accident, it doesn’t make the news. And so, as the just total scope and scale of these kind of breaches has gotten up, it’s gotten to the point where it doesn’t make the news anymore and we don’t hear about it, but we shouldn’t make the assumption that it’s not happening. Right? It’s just happening more than ever.

Jason Soroko

It’s happening more than ever, and I think that over the past few years we’ve had enough other stuff going on from other news stories that have crowded out a lot of these things. You know, from COVID to other kinds of world events and things seem to have ramped up. I know in Canada we have major inflation problems and people are unfortunately, losing their livelihoods and losing their homes. It’s just it’s not a normal time right now and so a lot of these stories which might have come to us in a slow news day, you know, what was just normal news cycle, we’re just not seeing any of it. So, I just want to call people’s attention to, you know, the whole idea behind this podcast was there are areas that are quite strong right now because of the use of asymmetric secrets. And thank goodness it has become ubiquitous. But we wanted to call out some of the areas that it isn’t and now what we are discussing here, Tim, is general computing systems and the ways in which we do authentication. Thank goodness for some of the more advanced methods that are using asymmetric secrets but these shared secret methods, Tim, are everywhere. IoT, they’re in your car. They’re in your operating room. And you know what, it’s not benefitting any of us. These systems really need to be rethought out. Come on, guys. It’s time.

Tim Callan

Yeah. I agree. Come on, guys. It’s time.

Jason Soroko

It really is. It’s 2024, folks. It’s time.

Tim Callan

It’s 2024. Alright. Well, thank you, Jason.

Jason Soroko

Thank you, Tim.

Tim Callan

This has been Root Causes.