Root Causes 208: Automotive Information Systems Bricked by HD Radio Error

Episode Transcript

Lightly edited for flow and brevity.

• 

  Tim Callan

  I always start this the same way. I say we talk about all matters digital certificates and PKI but I think today we are going to discuss something that actually has nothing to do with digital certificates or PKI but it is an article you sent me and I think both of our jaws dropped so much that we need to talk about it here. Would that be a fair assessment?

• 

  Jason Soroko

  I think it’s fair, Tim. Let’s share it with the audience.

• 

  Tim Callan

  So, first of all, you found this. I’ll give you credit. It’s an Ars Technica article from February 9 written by Johnathan Gitlin. Thank you, Johnathan. And the title of the article is Radio Station Snafu in Seattle Breaks Some Mazda Infotainment Systems. That gives you plenty to look it up and read the article so, you can read Johnathan’s original article. Was the gist of it, Jay?

• 

  Jason Soroko

  Think about this for a moment. You could be using a modern infotainment system in a car and for those of us who were born in the ‘70s or earlier, the idea of infotainment systems being as complex as they are, it’s almost Star Trek now.

• 

  Tim Callan

  Your 8-track player was your modern infotainment system.

• 

  Jason Soroko

  You’ve got AM and FM radio. Then all of the sudden, a lot of years later, FM stations started to have information modulated on them so that you could actually see what song was being played and - -

• 

  Tim Callan

  I would look at my radio and it would say the name of the artist and then was obviously built in. That was digitally built in what was being broadcast.

• 

  Jason Soroko

  And for those of you who listen to satellite radio, there’s even more content is given. Just the richness of what’s modulated on top of the signal. So, a really modern infotainment system. Don’t forget, all the car makers are trying to up one another with how fancy their dashboards are in terms of what they can do. Well, they are just computers and, Tim, computers are hard.

• 

  Tim Callan

  Bugs happen. A bug happened in this case. So, it turns out that these systems – let’s see if I can paraphrase this. Correct me if I get it wrong, Jay. But it turned out that there are a bunch of these infotainment systems that basically had a – I’ll call it a bug – had an error case that probably should not have occurred, which is that if the files that were broadcast by the terrestrial radio station, the terrestrial HD radio station, were missing extensions then basically it put the system into a permanent reboot loop that it could never get out of.

• 

  Jason Soroko

  Exactly. It was an error that it couldn’t handle and it kind of had a bit of a freak out and so, the computer was not taught how to deal with that condition.

• 

  Tim Callan

  How it dealt with it was very badly and then, of course, Part 2 of what you need is you need a radio station to make this particular mistake. And so, this station, this NPR station – perhaps ironically – in Seattle – also perhaps ironically – was the one that made this error and so all of these NPR listeners in that very computer savvy market all of the sudden had their auto infotainment systems bricked. Permanently disabled by this error.

• 

  Jason Soroko

  So, Tim, think about what a lot of people think hacking is. A lot of people think it’s like, oh my God, you got a guy like Charlie Miller who is using extreme math and fuzzers to find the exact combination of A’s and B’s and C’s, that can be fed into a system at high speed so that it creates a raised condition which causes memory to be moved around and then the computer basically goes, uh-oh, goes into freak out and then the next thing Charlie Miller has got himself a root shell. When most people go to Black Hat, that’s what they are thinking all the hackers are doing. That’s one way of doing bug bounty work but what happens if you end up finding out that it’s simply the software wasn’t able to handle a fairly simple condition such as an extension on the end of a file. And because, simply, the people who developed the system never ever expected that condition to occur.

• 

  Tim Callan

  Apparently, they didn’t think of it. Apparently, they didn’t test for it. There’s two interesting things in this article around the date range of the systems that were affected. I just want to look this up real quick because I want to get it right. I know the most recent was 2017. 2014 to 2017. So, I notice two interesting things about that. One is, apparently, presumably, that means we went from 2014 to 2022 and this circumstance never occurred! Anywhere!

• 

  Jason Soroko

  That’s just how rare it is.

• 

  Tim Callan

  In the whole globe. Then suddenly it did. So, that’s interesting point number one.

  Interesting point number two is systems after 2017 are unaffected, which presumably means that somewhere along the line somebody figured it out or somehow updated their system so it didn’t have that vulnerability anymore, but nothing was done for the old ones that were all still out there.

• 

  Jason Soroko

  So, wouldn’t it be nice to live in a world where over-the-air software updates would have solved this problem? You are right, Tim. I’m making your point for you which is this problem didn’t happen for a long time because it is an incredibly rare condition but people who have a slightly older vehicle are running older versions of the software. Wouldn’t it have been nice for these people just driving around doing their daily business to then be able to say - -

• 

  Tim Callan

  We know they have the ability to accept data because they are accepting data all the time. It’s not an old-fashioned FM radio. They are getting digital content. That is what this is. Absolutely.

• 

  Jason Soroko

  So, and over-the-air software update – and that could be done through various channels. Maybe the FM modulation is not enough. Maybe it has to be done through a wi-fi connection or through a 3G, 4G, 5G connection. There’s all kinds of ways of doing things. To do it properly, you need a few things, which is just like when you connect to a website, Tim, you hope that that’s an encrypted connection. So, that over-the-air connection, it’s gonna get the file from somewhere.

• 

  Tim Callan

  I need an encrypted connection to a known source. I also need signed code. Just like when I get a software update. When the new version of my OS comes down to my iPhone or my laptop, that’s code signed.

• 

  Jason Soroko

  You also need – and this is where a lot of people forget – the electronic control unit or the infotainment system computer has a kernel. And so itself should have code signed bootloader, code signed, whatever the booting sequence of that computer is everything should be code signed and then even the user land experience. In other words, where the software sits, the software itself being checked, code signed, everything against a complete unbroken chain of double checks. And by the way, that’s PKI all the way.

• 

  Tim Callan

  By the way, the stakes are really high on this. I mean in this case, again, going back to the article, these people are stuck. Their infotainment system won’t work because they can’t get a new one because there are supply chain problems. They can drive down to the Mazda dealership and hang around as long as they want. It doesn’t matter. There isn’t a unit to install. So, they are stuck and they are saying six months before they get a unit. And by the way, originally, they were gonna be charged $1,500.00. Now Mazda said, no, we are gonna make that available for free, which is a smart move but originally they were gonna be charged a whole bunch of money. And now let me go a step further – and I know you are trying to get a word in, Jay, but this is just the infotainment system. Nobody is gonna die. What if it was the brakes?

• 

  Jason Soroko

  Tim, now you are making my point. You got it. We are laughing because it’s an image or something in the infotainment system. We are laughing about radios. And we are entering a world, Tim, where computers are essentially laptops with wheels and all these things that we struggle with in the enterprise world all of the sudden becomes reality in your car. Your connected car. And wow!

• 

  Tim Callan

  I agree. So, we had to talk about this and then you had to go and ruin it and make it a PKI thing. But, when you sent me this article literally my chin hit my sternum. I was just so amazed. And so, I had to talk about it today.