Root Causes 116: Ripple20 Exposes TCP/IP Vulnerabilities for IoT

Episode Transcript

Lightly edited for flow and brevity.

• 

  Tim Callan

  We are fortunate once again to have our guest Alan Grau, VP of IoT here at Sectigo. Alan, we want to talk today about the Ripple20 I guess I would call it a vulnerability set. Right? Ripple20 is the name ascribed to a set of 19 vulnerabilities that were revealed for the Treck TCIP stack and tell us a little bit about this. This was from July I believe and what happened here?

• 

  Alan Grau

  Yeah. We are actually joking ahead of time, it should probably be called Ripple19 since it was 19 vulnerabilities, but Treck is a software company that has been around for many, many years. One of the products that they developed, and sell is a lightweight TCIP stack that is used in a large number of embedded and IoT devices. So many small lightweight devices utilize this stack.

• 

  Tim Callan

  And they are constrained and so they like it cause it’s small and it lets them use their very limited computing power in a better way, right?

• 

  Alan Grau

  Yep. Exactly.

• 

  Tim Callan

  Ok.

• 

  Alan Grau

  So, I was actually familiar with this company back in the late ‘90s. We were doing a lot of work in the embedded space clear back then and they were one of the people provided a TCIP stack and so in some of the early versions of their TCIP stack, going back it looks like as early as 1997 researchers starting, well they announced here just a couple of months ago, a few months ago, some vulnerabilities that they found in this stack. So, they were going back 20+ years and looking at a version of software that was developed at that point in time and just found that there were some vulnerabilities in the TCIP stack that are exploitable.

• 

  Tim Callan

  So, if you are going back 20 years to look at a TCIP stack, is that meaningful? Like if I went back to a 20-year-old version of Windows I could show you all kinds of vulns but who cares? So why is this important?

• 

  Alan Grau

  Yeah. This is where the IoT world is challenging because there are a lot of devices that were built 10, 20 years ago that are still out in the world doing things and we are talking critical infrastructure, medical devices, all sorts of manufacturing electronic devices really across all the various vertical markets where you would see embedded devices. So, yeah, again, the advantage of the Windows world is any things, you know, just fall off the usage quickly.

• 

  Tim Callan

  Right. Everybody throws out his PC after five years anyway. Right. So, it only goes back so far before it doesn’t matter. But are there a lot of 20-year-old devices that are still in use?

• 

  Alan Grau

  Well according to this article, I mean they are saying there are potentially hundreds of millions of devices that are being used that could potentially contain some of the software. So, it’s a staggeringly high number.

• 

  Tim Callan

  And those possibly are rally important things too, right? This is one of the things we’ve talked about in earlier episodes is often that really old stuff is the real foundational stuff, right?

• 

  Alan Grau

  Yeah. I mean they are talking printers – well, that’s not quite as critical, but things, you know, commercial aircraft devices, transportation systems, industrial equipment, healthcare, you know, we’ve got really, really, critical stuff.

• 

  Tim Callan

  So, if these are 20-year-old devices that are out there somewhere in the world with presumably 20-year-old ecosystem support around them, are they updatable? Like what do we do?

• 

  Alan Grau

  Yeah. So, if you really dig deeply into this the vulnerabilities were in some of the early versions of the software stack from this company. Their newer versions of the software stack they’d solve the problems, the vulnerabilities are no longer there. I don’t have details on the exact date when those solutions were, you know, when those problems were resolved, and solutions became available to address that but even that was still a long time ago right? We are talking certainly 10+ years ago where the fixes were available. I don’t know exactly, again. But the problem is of the devices out there very few 20-year-old devices had remote firmware update capability.

• 

  Tim Callan

  Right.

• 

  Alan Grau

  So, chances are that if you are using one of those old devices there isn’t a way to update it or even if there is, you know, is the printer company that made my printer 20 years ago or the medical device that was built 20 years ago, is the company still in business? Are they still providing firmware updates for 20-year-old systems?

• 

  Tim Callan

  Sure. Is this unsupportable? Could they even do an update? Could they do a build if they wanted to? Absolutely.

• 

  Alan Grau

  Well, maybe. Right? I mean because you are talking about you need to go get a compiler that will work on a chip that was 20 years old and that’s probably on that laptop that got thrown away 10 years ago.

• 

  Tim Callan

  Yeah, exactly.

• 

  Alan Grau

  So, it’s a, you know, this is what one of the big challenges with security in the IoT world is these devices live a long time and early on security wasn’t a huge concern and certainly being able to remotely patch the device was not commonplace.

• 

  Tim Callan

  So, are we going to have this same problem? Are we all going to be sitting when we are on episode 1000 of this podcast 20 years from now, are we going to be having the same conversation about things that people are making today?

• 

  Alan Grau

  Yes and no. I am sure that there is software being released today and developed today that have vulnerabilities that have not yet been discovered. Some of the things that are changing though is security has become a front of mind, is huge for developers as has been a strong recognition that you need to be able to patch these devices.

• 

  Tim Callan

  Yeah. That’s a big difference. Right? Is again, we always come back to this concept of agility. Crypto agility, certificate agility. Where the agility is just so much higher today than it was in the year 2000 or even for that matter the year 2010.

• 

  Alan Grau

  Yes. Very true. And we are seeing regulation and standards requiring software update and patching capability for a variety of embedded devices.

• 

  Tim Callan

  So fair enough and it’s hard and they are out there, and they are being used in large numbers still and there are probably a lot of them are foundational important, maybe some of them are only printers but maybe some of them are keeping someone alive. At the end of the day though, I feel like from I read it seems like some of these are very big vulnerabilities. Like are pretty important. So, something has to be done?

• 

  Alan Grau

  Yeah. And it is a challenge, right? I think the good news is people are aware of it. So, if you are a company that’s operating systems and devices that are fairly old there is at least some traceability to go back and say hey, you know, what’s the legacy of this device. Where did it come from and some ability to determine whether it is potentially vulnerable and then you are left with a couple of choices. Either you can try to make sure that that is on an isolated network and provide protection for the device by isolating it through using kind of traditional IT tools, but as we know that’s fallible or the other alternative is, you know, a rip and replace, which is expensive.

• 

  Tim Callan

  Yeah. And may be difficult. And maybe not all that easy. Maybe the device that you were going to must rip there isn’t something obvious to replace it with.

• 

  Alan Grau

  Well, I’ve worked on software that’s orbiting our planet right now. Right? They’re in satellites. It’s kind of hard to rip and replace those.

• 

  Tim Callan

  You can shoot them down, I guess.

• 

  Alan Grau

  Yeah.

• 

  Tim Callan

  That’s what it would take. Yeah. Or I suppose you brick them and put a new one but that’s gotta be, that’s horribly expensive and difficult and has a space junk consequence and all kinds of other things. So, wow. Where does that leave us?

• 

  Alan Grau

  Well, I think there’s two main points. One is where you can take some action to address the problem, you need to do so and if it’s a matter of changing some firewall rules or putting some network segmentation or ensuring that some of these devices are run in isolated networks then that’s critical. If you can update the device. Right? Maybe you can’t update it today but as you are looking at plans for new capital expenditures and network equipment that suddenly must be a much higher priority. So, there’s the mitigation factor which is critical and then there’s the, you know, what are the lessons learned and how do we not make these same mistakes.

• 

  Tim Callan

  Right. Very important.

• 

  Jason Soroko

  On that topic, Alan, what do you think are some of the major trends to avoid problems? There’s a lot more code analysis going on than there was 15, 20 years ago. We also have things up now that are automatically built with micro segmentation in mind. Languages such as Rust which inherently do not have things like buffer overflow problems. I don’t think it’s impossible, but it definitely is reduced because of the way that the language is actually designed. What do you think are the main trends towards avoiding these problems in the future?

• 

  Alan Grau

  Yeah. I mean there are many, as you said, and the things that you touched on are certainly important parts of that and in the embedded world, you know, starting to have security as a front of mind issue and not an afterthought is critical and is starting to really change. How that plays out is in many ways code analysis tools are an important part of it, you know, security best practices. Ensuring your device has Secure Boot built into it. Ensuring that your device is using strong authentication methods is an important part of it. Ensuring that you’ve got an ability to update the firmware on the device is a very, very critical piece of it.

• 

  Jason Soroko

  And don’t roll your own TCP stacks.

• 

  Alan Grau

  Yeah. I think that’s one that we’ve gotten away from. I mean, you know, we are talking 1997, right, when TCIP was fairly new and the idea of a lightweight TCIP stack was something that wasn’t readily available, and customers were starting to get away from it and probably everybody’s first version of TCIP stack had vulnerabilities. I mean we certainly know that Windows did and when people first started building these stacks, they hadn’t been battle-hardened and that’s true of most any software. Right? The first version of anything is likely to have some vulnerabilities. It takes time. It takes security testing. It takes penetration testing to work through those issues and the key is that that first version or second or third version that hasn’t yet been battle-hardened isn’t in a piece of critical infrastructure equipment that is still operating 20 years later.

• 

  Jason Soroko

  Yeah. Satellites and pacemakers come to mind for sure.

• 

  Alan Grau

  Exactly. I mean and this article lists satellite communication equipment as one of the devices or device types.

• 

  Tim Callan

  Yeah. It’s very different than servers and desktop machines and mobile phones. All of which have a much shorter lifespan and are much easier to deal with.

  All right. Good to know. This is a fairly recent news item so I’m glad we covered it and is there anything we want to add to this topic, gentlemen?

• 

  Alan Grau

  No. I think we’ve covered it. I mean the only thing I would say is the story when you first read it sounds bad and it is, but it is to recognize that there are companies that use this software that upgraded their equipment to newer versions. The company did address these vulnerabilities a decade plus ago. So, it wasn’t as flagrant as the article may at first sound but it was the long-lived nature of these devices that really made this a problem.

• 

  Tim Callan

  Yeah. The point is more the inflexibility of this kind of device rather than the individual decisions that were made by companies or developers at any stage in history. Right?

• 

  Alan Grau

  Exactly.

• 

  Tim Callan

  Right. All right.

• 

  Jason Soroko

  Yeah. So, Tim, taking inventory of the devices you are using within critical infrastructure is just so important. I think that there are movements in the industry to do that, especially now because of things zero trust architecture principles where we are just not trusting what’s behind the firewall and part of not trusting is not just addressing everything with a secure identity, it’s also taking inventory of what certain kinds of devices are capable of doing, how they are privileged within your network, etc. because the fact of the matter is we could probably be reporting on bugs like this if not on a weekly basis then fairly often at the very least and so it’s probably reasonable to assume that most critical environments are gonna have some kinds of devices where there is known vulnerabilities of some level and you are just gonna have to live with it because there’s no way to swap it out so how do you deal with that. That’s perhaps a whole podcast unto itself down the road, Tim.

• 

  Tim Callan

  Good. Good one. I love when we come up with new podcasts while we are doing existing podcasts. That’s always a great development.

  Alan, thank you for joining us and walking us through Ripple20 and what it is and what it means.

• 

  Alan Grau

  Thanks, Tim.

• 

  Tim Callan

  Jason, always a pleasure to talk to you, of course.

• 

  Jason Soroko

  Always, Tim.

• 

  Tim Callan

  And Listeners, this has been Root Causes.