Root Causes 297: Certificate Expiration Creates Starlink Outage

Episode Transcript

Lightly edited for flow and brevity.

• 

  Tim Callan

  As so often occurs in the world these days, this story starts with a tweet by Elon Musk.

• 

  Jason Soroko

  Tweets from Elon Musk. Well, they are always interesting to read.

• 

  Tim Callan

  Yes. There’s always a lot.

• 

  Jason Soroko

  That’s where I get all my investing advice and etc.

• 

  Tim Callan

  Right. A lot can come out of a tweet by Elon Musk. This one is from April 7, 2023 and the first tweet reads, “Sorry, slight glitch with the SpaceX Starlink. Coming back online now.” And then there was a follow up shortly thereafter that reads – also by Elon – “Caused by expired ground station cert. We are scrubbing the system for other single-point vulnerabilities.” So, of course, this is something that you and I always notice when these kinds of outages occur, but in this case, Starlink is taken down by an expired certificate in the ground station. And, Jason, I think you felt this one personally. Did you not?

• 

  Jason Soroko

  I did. I did. And in fact, I didn’t even see Elon Musk’s tweet, mostly because of the fact that I didn’t have access to the internet. I am a customer of Starlink and I have one of his satellite dishes at a rural property that I happen to have and I depend on that because of the fact that there really is no other kind of internet service out there and I enjoy the service. So there it is. The problem is, of course, when it goes down, it goes down. A lot of times it’s due to who knows, maybe there’s wind that’s shaking my satellite dish or some other crazy thing. But in this case, the outage was a lot longer than what was normal and myself and a few family members were kind of looking at each other going, geez, I hope this thing comes back. And sure enough, it did. And then it wasn’t that much long after, Tim, when you shared that tweet with me and I was like, my goodness, I know that outage. It affected me very personally and what was it, Tim? What was it? An expired certificate.

• 

  Tim Callan

  An expired certificate. In April of 2023, an expired certificate. That was probably an SSL certificate. I guess we can’t know for sure, but an expired ground station cert, you would imagine that’s what it is. Connecting to something else sitting on a server, connecting up to the satellite itself and then, of course, this ties directly back to a topic that I think is just gonna come up over and over and over again, is this is still in the era where people can get a 398-day cert. What’s gonna happen to situations like this once we are down to 90-day certificates?

• 

  Jason Soroko

  Yeah. You are exactly right, Tim. Some people could say, hey, who knows? Maybe it was a private TLS certificate we are talking about here, but it doesn’t matter. That had some kind of an expiry date that was not captured. The certificate was not renewed on time and guess what? It affected people. It even affected me. A lot of times when we talk about these things, Tim, you know, sometimes it’s European outages or people are in gaming systems. I don’t have time to game, so it doesn’t affect me. This one affected me and so to me, I don’t care. I don’t care what kind of cert it was. It expired and it wasn’t handled. And isn’t it amazing, Tim, Elon Musk, with all his technical resources, all the people who work for him, you know, his SpaceX, very tech leaning company and…

• 

  Tim Callan

  Yes. SpaceX is not a technically naïve company. This isn’t your local law firm that has some IT infrastructure because they just have no choice. This is a company that is born in technology. It is about advanced technology and it is led and founded by somebody who is all about advanced technology, and even here we have this kind of problem.

• 

  Jason Soroko

  This is a company and a person who has the capabilities of regularly launching hundreds of satellites into space and then having the spaceship come back to Earth and land on its feet. But they cannot renew their certificates. Wow!

• 

  Tim Callan

  Sigh. So, how long are we gonna keep telling this story, Jason?

• 

  Jason Soroko

  We talked about this just before we came onto this podcast and said, ah, just another one of these, you know, expired cert stories but the difference being, of course, it affected me and I hate to say it but here we are – April 2023. You and I looked into our crystal ball and predicted we’d see more of these in the future. Here it is. We are seeing these again.

• 

  Tim Callan

  Here it is.

• 

  Jason Soroko

  And, Tim, I’m gonna throw it in just because I think this is the call to arms and the big red light warning. If you haven’t heard it on previous podcasts, Google is probably going to at some point, either passed through CA/Browser Forum or force it themselves, maximum of 90-day certificate lifespans for SSL certificates, for publicly trusted certificates. Which means this kind of site outage due to an expired cert is now gonna be running the risk not just once every 398 days but every 90 days. I can’t imagine.

• 

  Tim Callan

  Alright. So, Jason, I’m gonna take a little side jog here and you and I are gonna both make a prediction right now. I’m gonna put you on the spot. And for what it’s worth, I haven’t thought about this at all. So, we are both in the same boat because I thought of this right this second. It is the middle of April. We are going to do almost two podcasts a week for the rest of the year. So, let’s say that is roughly 65 podcasts. That’s probably the ballpark. What’s your over/under? Think of a number. I’ll think of a number and we’ll both say on how many podcasts we will mention 90-day certs between now and the end of the year.

• 

  Jason Soroko

  I’d be surprised if it wasn’t at least -

• 

  Tim Callan

  Ok. I got a number

• 

  Jason Soroko

  Sure. I’d be surprised if it wasn’t at least three. It’s gotta be at least three in that many.

• 

  Tim Callan

  At least three podcasts? Between now and the end of the year?

• 

  Jason Soroko

  That’s right.

• 

  Tim Callan

  My number is 45. I think we’re gonna mention it almost every episode. That’s my prediction. I think we are gonna mention it two-thirds of episodes for one reason or another. My number is 45. So, we’ll see. Now of course, this is bias now because I can just make sure I mention it 45 times. It’s unfair.

• 

  Jason Soroko

  My problem is that I would like to maintain our listenership who doesn’t want to just tune in and listen to the latest outage. But the reason why we wanted to bring up this one - -

• 

  Tim Callan

  No. Not outages. 90-day certs.

• 

  Jason Soroko

  Oh. How many are you gonna hear about 90-day certs?

• 

  Tim Callan

  I predict we will mention it 45 times.

• 

  Jason Soroko

  Oh. My apologies. I heard you wrong. Yeah. So, I would say at least three major outages by the end of the year.

• 

  Tim Callan

  Sure. I would agree with that.

• 

  Jason Soroko

  I would say though that, geez, how many times are we gonna mention 90-day? I’m gonna say - - I’ll tell you what. I’ll be really generous to the audience here and say I bet you it’s 35 to 40. Let’s give it 40.

• 

  Tim Callan

  Alright. So, we are predicting between 30 and 45. I think that’s a reasonable range because this is the story. And the point I’m going to with this – not to belabor it, this is the story. Like, this came out of nowhere. It is the story for the rest of this year. It is definitely the story in the next year for some period of time until it becomes reality, at which point it continues to be the story. So, this is gonna be the story for years to come.

• 

  Jason Soroko

  It is going to be. Yeah.

• 

  Tim Callan

  And we are seeing it just showing up over and over again.

• 

  Jason Soroko

  You are not wrong at all, Tim. I completely agree. And, in fact, for those of you who hear us on the next 30, 40 episodes and mention it, it’s gonna be on purpose. It is absolutely on purpose. You need to hear more about this than you realize and until it totally sinks in to everyone, we are going to continue to talk about this because we do not want to enter a time in the deeper future, 2024, right, next year – we would love for people to have a soft landing with 90-day certificates and if we don’t push it hard now, it’s not gonna be a soft landing.

• 

  Tim Callan

  And I think there’s that point about educating. I think that’s a very valid point, Jason. The other thing about you and me, we kind of like to analyze and explore. I do not believe that the industry or the IT community actually has wrapped its head around the full set of implications of this change.

• 

  Jason Soroko

  Correct.

• 

  Tim Callan

  And I think that one of the reasons you and I are gonna discuss it a lot this year is we are gonna take different angles on it. Think of all the different angles we’ve had on Post-Quantum Cryptography. Right? Say, well, what about the Z-date and what does harvest and decrypt really mean and is China gonna outspend the U.S. and all these different kind of sub-angles that we took. I think we are gonna see the same thing with 90 days. We are gonna keep thinking of new implications and new reasons why it’s important and as we do, we are gonna want to explore them and that means we are gonna be returning this topic over and over again.

• 

  Jason Soroko

  We absolutely will. So, stay tuned, folks. The 90-day announcement from Google, go back to a couple of previous podcasts that we have published on this – you are gonna hear a lot more.

• 

  Tim Callan

  Yep. So, this was a certificate outage reported by Elon, impacted Jason who is among the most technically astute people I know, so if you can be taken down by a cert expiration, Jay, I think anybody can.