Root Causes 95: Cryptographic Key Vaulting

Episode Transcript

Lightly edited for flow and brevity.

• 

  Tim Callan

  So, today, we are going to talk about key vaulting.

• 

  Jason Soroko

  For those of you who work in DevOps, this is probably a subject that you already know a heck of a lot about, maybe too much, and really, it's about where do you put your secrets in order to automate. So, Tim, a lot of our podcasts talk a lot about automation. Automation, especially with respect to cybersecurity, and really particularly around key management. And so, key vaulting or secrets vaulting, or just simply, you know, the vault, is a concept that's used quite often in automated systems so that the system that is running autonomously, without a human being can grab its necessary secrets to log into APIs, log into, you know, TLS sessions, basically any form of authentication that's necessary. There's some credential that's needed. Where do you get those credentials? The best way to get it from is a vault.

• 

  Tim Callan

  Right. So, you have those secrets, they're stored somewhere consistent and safe and you know where they are. And, again, the main thing is safe. There's something about that vault that makes it, shall we say, more difficult for someone to come and steal those secrets?

• 

  Jason Soroko

  Yeah, because the alternative is pretty rough. Let's say you've got an Ansible script, and you're automating some sort of cloud implementation of your application and you are logging into - - your automated system is logging into databases, might be logging into the cloud system itself. It might even be provisioning SSL certificates. That's a lot of systems to have to log into, and so, instead of hard-coding those credentials, quite often using passwords, perhaps for database systems, legacy database systems, and instead of putting private key information into your Ansible script, you can store those things in the vault. And, unfortunately, still, to this day, we are seeing hard-coded credentials within automation scripts. We're talking about Chef, Puppet, Ansible, you know, name, your favorite flavor of automation scripting, and you're definitely seeing a lot of credentials stored in there. Even Python languages.

• 

  Tim Callan

  And when the credentials are in the script, obviously, first of all, anyone who has access to the script can go find those credentials, right?

• 

  Jason Soroko

  As soon as the bad guy gets a hold of your automation script, it's absolutely game over because you've given them the full recipe of how to get into everything.

• 

  Tim Callan

  And on the other hand, with the vault, what the script does is the script calls the vault and indicates which secret it wants, but the secret itself isn't in the script.

• 

  Jason Soroko

  Correct.

• 

  Tim Callan

  Therefore, if you get the script, you don't get, you might get other information that's useful to you, but you're certainly not going to get those credentials.

• 

  Jason Soroko

  That's right, Tim. So, like everything else in computer science, it comes down to an abstraction layer is a good idea. And so, you want an abstraction layer between your scripting logic, and the actual credentials themselves. The credentials themselves should be safely ensconced within a vault, and called when necessary, and never, ever should be hard-coded within your actual logic.

• 

  Tim Callan

  So, what are the main? I mean, you kind of touched on a couple of them already, but like, what are the main - what do I want to say? Environments. Use cases. What are the main circumstances in which secrets vaults or key vaults or credential vaults are being used?

• 

  Jason Soroko

  Sure. I would say the number one is in DevOps, as we just said.

• 

  Tim Callan

  Right.

• 

  Jason Soroko

  And that's particularly obvious, because if you are implementing automated cloud systems, then those automations will necessarily request credentials of some kind for all kinds of purposes.

• 

  Tim Callan

  Right.

• 

  Jason Soroko

  And I would say number two, and this is interesting, Tim; this is not a topic we've brought up yet on this podcast, but robotic process automation.

• 

  Tim Callan

  Ok. Ok. That's a topic we need to - - that's one of the things I've been meaning for us to hit, so can talk about that.

• 

  Jason Soroko

  You can just imagine, you know, back, you know, it wasn't that long ago, let's say yesterday, which wasn't that long ago, you might have a fair amount of your staff doing all sorts of repetitious computer commands. So, in other words, your HR department, your finance department, these kinds of departments like that, perhaps which in order to say onboard a new employee in the HR department, or in order to close the books at quarter end, as a finance employee, you might have dozens of commands.

• 

  Tim Callan

  You have a set of predictable steps that you must always follow.

• 

  Jason Soroko

  100%. There's just tons and tons of predictable repetitious steps, that what happens if you just got rid of that?

• 

  Tim Callan

  Yeah.

• 

  Jason Soroko

  What happens if your employee just triggers one event, which then causes an automation of the ten other things that had to happen in order to complete the task.

• 

  Tim Callan

  Right, which is essentially a script. Like whether or not it's done through a scripting language or something else, at the end of the day, that's what it is, right? It's a recipe book. And guess what? That automated task may have to access various systems and that means it's going to need credentials.

• 

  Jason Soroko

  Well, think about it, Tim. I could be a DevOps employee who in order - - I could choose to manually implement my cloud application, which is like, you know, let's say 100 steps. Or, I could close the book as a finance employee at the end of the quarter, which is another 100 steps. Each one of those things absolutely could be codified and in each of those steps, there probably is some sort of log into this, log into this, log into this, right? And every time you have to log into a new system, you have to grab a credential from somewhere. Well, for a human being, either you are using a client certificate from your computer, or your mobile device, or you're having to remember a username and password, which you're typing in manually and in the automation scripts, right, in robotic process automation, or in DevOps automation, those credentials have to come from somewhere. Ideally, they're coming from a vault.

• 

  Tim Callan

  Gotcha. Ok. So that's another that's another good one. When you go through this process, it strikes me that probably for single sign-on services, you have to have a key vault or credentials vault of some kind, right? Because they're doing the same thing.

• 

  Jason Soroko

  Yeah. That's it. That's it, Tim. It really is a very, very similar kind of concept. But the thing is, robotic process automation technologies have some vaults. There are vault technologies, say in HashiCorp. We have our own with our DevOps tools. And so, therefore, you know, the vault that you will use will probably be specific to the tasks that you're trying to solve.

• 

  Tim Callan

  Which means you're using multiple vaults. So, you used this example, I've got some DevOps use cases going on and I've got a vault for that, or more than one and I got a vault for my Ansible and my vault for my Puppet, possibly and then I might also have a vault for, like we said, some other application like my single sign-on apps. I might have a vault for keys I'm using on the public cloud service, like Azure or AWS. So, do I have lots of vaults? Is that what really happens in today's enterprise?

• 

  Jason Soroko

  No. Well, I would say you as a DevOps employee are probably running with one vault. You as somebody who might be triggering a robotic process automation set of steps, you're probably dealing with just one vault. And enterprise may have a very small handful, one or two, and depending on what use cases are being used but you as an employee are probably just dealing with one.

• 

  Tim Callan

  So, tell me if I'm crazy, Jay, but this strikes me as a potential problem. You and I have talked in the past about concepts like rogue certificates, or even rogue CAs, right? Somebody in the company stands up a CA, they're running that for some number of certs, other people don't know that CA exists and because of that, problems arrive. There's no governance or nobody keeps an eye on it, and the patches don't get deployed, etc. Is there the same potential here? Can we have rogue vaults?

• 

  Jason Soroko

  Tim, let me let me scare you. This is worse than a horror movie.

• 

  Tim Callan

  Ok. Please do so.

• 

  Jason Soroko

  So please, anybody who's a CIO, CSO, listening right now, this is where you got to take a note and maybe do some homework because there's a really, really good chance that your DevOps employee might have set up a Certificate Authority without you knowing it and the employee might not even understand the full implication of what setting up a Certificate Authority actually means. They might have just set up, you know, an OpenSSL instance, which is pumping out certs happily, and goodness knows what certificate profile is being used and where that CA is actually residing. I've taken a look, Tim, at robotic process automation software documentation, and guess what? The instructions are very similar. If you just want to, you know, set up a root certificate, just fire up OpenSSL, here's some Linux commands, you know, there's no guidance in terms of what an appropriate bit length is, you know, RSA versus ECC, etc. And just oh, this will just make it work. So, on you go. And you and I know what kind of a nightmare that potentially is in terms of governance and, you know, what kind of door that opens to the bad guy. Because what we're talking about here is these are the keys to your kingdom and when you're setting up just rogue CA's all over the place - and that's what your employees may be doing if they're following the instructions. And heck, Tim, I wouldn't even count on the consultants you have in to be doing this correctly. They might not be PKI experts.

• 

  Tim Callan

  Right. And even if they are, they're not going to be here in a year. They’re consultants.

• 

  Jason Soroko

  Did those consultants spend the time to set up full blown visibility from the CA they set up for a vault for either RPA or for DevOps and give you visibility into your main PKI governance for your enterprise? I doubt it.

• 

  Tim Callan

  And did they document and did they train the appropriate people and did that training and that knowledge continue to perpetuate correctly inside of the organization without the telephone game after those experts have left?

• 

  Jason Soroko

  Good question. I bet you - - these are the questions that you as CSO, if you can't off the top of your head, know the answer that question - that's homework for you.

• 

  Tim Callan

  Alright. If there's a vault and the vault has all the material in it, all the really good stuff, it seems like that now becomes a really, really, really good target inside the enterprise. So, how do we know that our vaults aren't being stolen by bad guys?

• 

  Jason Soroko

  The way that you implement the vaults, and every vendor is going to have their own instructions for how to do things. But I would say that the most - - it probably at this point in time, it's probably important for us just to say, make sure that your credentials are not hard-coded and just sitting out there and you don't have some OpenSSL, rogue SSL, CA just sitting out there. In terms of how to protect your vault, Tim, how about we have that as another full-blown podcast?

• 

  Tim Callan

  I love it. So, I think and I think that's a really good point, Jay, and we talk about this a lot, which is perfect isn't - - perfect shouldn't be the enemy of good, right? So, if we can sit and cook up with some hypothetical and ask hard questions about well, what if the blah, blah, blah, blah, blah, well, gee, it's better than sitting in your Ansible script. Right? So, if that's what we're choosing, then let's choose the one that's unambiguously better. And I think that's a good point. That's a great place to start and we can pick up in a future episode and we can get into more depth on this.

• 

  Jason Soroko

  Perfect, Tim, and that looks like one we should put on our to-do list and we'll be happy to have that podcast soon.

• 

  Tim Callan

  Excellent. So, thank you very much, Jay.

• 

  Jason Soroko

  Thank you, Tim.

• 

  Tim Callan

  Thank you. This has been Root Causes.