Root Causes 226: The Six Benefits of SSH Certificates

Tim Callan

So, this is the third in our series of podcasts walking through the concept of SSH keys and SSH certificates. So, a little bit of background for the listener. In our Episode 221, What Are SSH Keys?, we do exactly that. We define what SSH is and what SSH keys are and how they work and how they’re used. In our Episode 224, we discussed the five problems with SSH keys, and we go into some depth in explaining what they are and how they happen. And now, in this episode, we want to describe SSH certificates and the six benefits of them which will also be another way of saying how they directly mitigate those five problems. Isn’t that right, Jason?

Jason Soroko

That’s right, Tim. The list we’ve come up with here is – I don’t even think it’s completely exhaustive, but I think it hits the main points, and also, I don’t even think if we chose a specific order that there’s a particular priority. I think they’re all just very important.

Tim Callan

Well maybe we’ll do them in the order I have them written down then, if that’s okay?

Jason Soroko

That’s the way we’ll do it.

Tim Callan

These are six. That works. Again, if you need to, if you want any background on this, go listen to those other episodes then come back and listen to us. If you listen to those other episodes or you think you get SSH, let’s jump into it. So, first of all, we talked about the difficulties that come with raw keys and a certificate, of course, is essentially keys in an envelope with rules. Is that a fair, if over-simplified, explanation?

Jason Soroko

That’s right, Tim. Policies that you define basically are the way that I like to say it. Remember, this particular use case – you can use cryptographic key pairs for a lot of different reasons. You can encrypt, you can sign, and you can authenticate. We are talking very specifically about the authentication use case. And so, if you think about the authentication use case, especially when you’re dealing with human beings and servers that are remote this list could be different depending on the use case. I would say that the number one thing when you’re dealing with authentication is, who is this that is trying to log into me, if you’re the server. If you’re the user, is this the server I intend to be talking to.

I’d like to talk about this in context of authentication and the number one benefit I would say, Tim, is trust. In other words, if you go back to SSH just with the raw key pair, we talked about one of the problems earlier, I’ll connect this, that trust on first use, where you get that big notification that – it’s not an error message, it’s essentially a declaration of, there’s no guarantee that what you’re connecting to is what you think it is. You better do your homework. Whereas, with certificate-based SSH Authentication, you’re not having to do that homework. That homework has been done for you, presumably by the centralized trust system. In other words, if the thing you’re connecting to possesses a public and private key pair, and it’s going to trade that private key pair with you at the point of trust and first use with this new scenario authentication, you have immensely more confidence that you are speaking to the actual server that you’re actually intending to speak to. And so, no more big warning message at the beginning, Tim. That’s a huge thing for SSH.

Tim Callan

Not to be in any way obscure about it, what we’re talking about here is by virtue of the fact that it has a certificate, that means that it has a certificate chain. At the top of that chain there is a Certificate Authority, and your system can be set up to know what that CA is, to know what that root is. So, it won’t just take just any cert. It’s going to take a cert with a root that you’ve chosen to trust. And that’s monumentally different. That’s complete different from a public or private key pair, which has no chain at all. There’s nothing, there’s no meta information attached to it in any way to say where this came from, who is this from, who is vouching for it. All it knows is, do the keys match, and that is the only thing it knows.

Jason Soroko

How is that technically done? Essentially because it’s a certificate and not just a raw key pair, the certificate itself that’s been issued is signed. So in other words, think about this. Isn’t this interesting, Tim? We’re dealing now with a few different cryptographic features. One is obviously the certificates are used ultimately for authentication, but when they are issued out, they are signed by the CA’s private key.

Therefore, it’s the certificate itself that allows for itself to be signed, and that signing is part of what you could call a policy. In other words, you shouldn’t trust it unless it is signed by that central authority. And so, therefore, your trust of the centralized authority is essentially what allows trust to be spread across all of the different parties.

Tim Callan

And that CA in this scenario, in this use case, operated and managed and controlled by the enterprise that ultimately is using these keys for access. And you, like your IT department would own that CA?

Jason Soroko

That is correct. This is a walled garden, and the walled garden is defined by whoever it is that controls the CA. And isn’t that nice. That is what you want in an enterprise.

Tim Callan

Absolutely. It’s perfect. Otherwise, again, there’s just none of that is there. It’s just, does this key work? That’s the only thing you know. Trust. Extraordinarily important. What’s the second one on the list?

Jason Soroko

Tim, we must have brought this up, I don’t know how many times, on the last podcast and even the first one we did not this series. It’s about the expiration, the fact that you can customize the expiration.

Tim Callan

Expiration. So, if you’ve listened to the other podcasts, in my opinion, the worst problem with SSH keys is that they last forever. Like, these other things you’ve listed, they’re bad, but just the fact that they last forever, they’re like these bombs that are out there in the world. They’re like these landmines that no one ever found, and you never know when one day you’re going to be walking through a grassy field, happy as a clam, and you’re going to blow off your foot. And the fact that they never expire - -

Jason Soroko

Tim, how many podcasts have we talked about expiration of certificates, whether it’s in publicly trusted certificates in the SSL arena for web servers or for any of the other use cases such as private authentication which is what we’re talking about now. And yet, the world of traditional SSH, just, it’s just not there. It’s just not there. We obsess over it in normal PKI with certificates, and yet, people who have only ever used raw key pairs never think about it. And yet, my goodness, should they ever because these are the keys to the kingdom. You want – there’s enormous amounts of benefit to having an expiry date to these things and sometimes very short expiry dates. I would say, there’s a lot of SSH certificates that really should not live long.

Tim Callan

I was just going to ask, what do you think is a sensible expiry date for an SSH certificate? Like, if you were setting up a PKI, and I know it might, very much might depend on context and usage, but just give us some thoughts. If you were setting it up, what kinds of expiry dates would you be thinking about?

Jason Soroko

I would say, Tim, the ideal expiry date wouldn’t even be something that’s set ahead of time. Let’s talk about just perfection if it could be achieved.

Perfection would be as soon as I’ve done my job - whatever my task is that I used this SSH certificate for, I would actually like this certificate to no longer be usable as soon as I’ve done my job. And so, therefore, what is ideal in that case? Because you may know how long it’s going to take, you might not know how long it’s going to take. I would say what a lot of people are doing is saying, ok, well, is that a day, is that two days? And what the problem is, some people, within a 24-hour period might still be – they may have the certificate issued to them 24 hours ahead of time, they’re doing a job, and then right at the 24-hour mark, they might still be doing their job, and the certificate expires. So that’s the big chicken and egg problem that we run into, and so therefore, I don’t think it’s a one-day type of scenario.

Tim Callan

But you’re getting us into the ballpark. It’s days, not weeks, not months, certainly not years. It’s days.

Jason Soroko

I think that’s right, Tim. And therefore, if your work cycle is kind of a like a week – in other words, if you’re not doing tasks with that certificate over a weekend, then make it the week. And that’s just an idea.

Tim Callan

Even then, if you’re making it a week and it turns out that a lot of time you only needed two days - I understand you have some exposure time, but, compare that to just everything is good for a year. And, under those circumstances, the amount of exposure you have is just so much, so vastly less than it would have been otherwise.

Jason Soroko

I think, Tim, one of the things that you can control is the point at which the certificate is issued. In other words, because of the fact that this is now going to be a centrally managed Certificate Issuance System issuing certificates, one of the biggest things that you can – instead of just having keys laying around to use, one of the best things you can control – if you can’t control when you’re done with the job, you can control when you start the job and so, therefore, issue the certificate as soon as possible to the point at which you’re about to start the job, and that really helps.

Tim Callan

If we start to imagine that this whole thing is being automated by a system with sophisticated capabilities and rules, then you can exactly imagine that. When you’re going to start the job, that’s when it’s issued that way you maximize your time. The other thing you could do under those circumstances is it could vary based on circumstances. It could be based on the user or the task or the environment, and for all those things, you might say, for this set of criteria we’re going to issue a one-day cert, for this set of criteria we’re going to issue a one-week cert because we think, work probably takes more time, and the risk is lower. That’s the kind of rule that you could build in if you think it through, and you’ve got some sense for why you would have things set at those different times.

Jason Soroko

I think the biggest consideration is what’s the level of privileges being given to that credential? You covered that. You covered that. I’m just putting a real fine point on it, which is, if this is a root-level, odd, certificate, well, my goodness, that really should not be lasting a long time.

Tim Callan

If I’m giving you God rights, I’m going to limit you to a day, and if you’re not done, you get to come get another one, and I’m sorry if it’s inconvenient. If on the other hand I’m giving you access to something that isn’t all that important, I just consider the risk lower. I love it. What is Number 3?

Jason Soroko

Tim, we talked a lot about just the amount of work it took. We talked about, do you manage raw keys and spreadsheets? Do you keep track of them? Where are you putting these things, how are you maintaining them? Because even though they are just raw key pairs, you should be doing some sort of maintenance, and I hate to say this – this is a little bit of sidebar, Tim – here’s the dirty secret - I don’t think anybody’s doing much maintenance at all on their, on their - -

Tim Callan

Probably not.

Jason Soroko

And so therefore, should you be? Heck, yes. And so, therefore, one of the big benefits is, with certificate-based authentication, it’s kind of just built in.

In other words, when we say, lower maintenance, it’s at least giving you the capability of doing maintenance in a much, in a frictionless way, because it is – that capability should be built into your Certificate Lifecycle Management System.

Tim Callan

Because back to your previous point. They clean themselves out. The main maintenance is getting rid of the ones you don’t want. And they will do that on their own, if you have built-in expiration. And that’s most of maintenance.

Alright. I agree, lower maintenance. But otherwise, think about what is the nightmare. You’ve got these keys, and they’re just keys. They work, I mean, what you really have to do, your version of expiration when you're using raw keys is somehow, prove beyond a shadow of a doubt that this key, this private key, is no longer in use anyway.

Jason Soroko

That’s the biggest piece, and there are others.

Tim Callan

It’s a Black Swan problem.

Jason Soroko

Absolutely. We could actually probably have a whole podcast just on what is maintenance of authentication key material. But suffice to say, with a certificate, it just, it’s just there.

Tim Callan

I love it. What’s Number 4?

Jason Soroko

So, think about all the other form factors of authentication you have. Think about the other digital identities you have just in general. We talked about some of the other use cases, Tim. We talked about signing. We talked about encryption keys. We talked about publicly-trusted SSL certificates.

Tim Callan

I think I see where you’re going with this.

Jason Soroko

S/MIME certificates. And isn’t it interesting that once you now have SSH as a certificate rather than a raw key pair, you nowhave the potential to put this as part of a Certificate Lifecycle Management System as a whole. And therefore, you have, it has a bit of buzzy term, but, a single pane of glass, all of it, and so I think we’ve got to call out really what this is. This is visibility. And you really cannot have governance without visibility, Tim.

Tim Callan

Lump it in with all the other like material, and use best practices to manage to deploy, manage, track, understand, and ensure governance for all of that like material. Once it’s a certificate, there is no good reason not to treat it the way you treat other certificates.

Jason Soroko

Well said. You got it.

Tim Callan

Absolutely. That’s excellent. Then you can start imagined reporting and compliance reporting and all sorts of things like that. All just gets so much easier. What’s Number 5?

Jason Soroko

Number 5. Well, this is a really – that very last point we were talking about, Tim, really kind of speaks to the CSOs and the Risk Officers because that’s their job. To worry about governance compliance. This next one is maybe all the way up to the CIO level. And beyond. Because we all gotta worry about our bottom line, and if you’re doing all of this properly, it shouldn’t cost you a fortune. In other words, if you tried to do governance properly with raw key pairs, a raw SSH key pairs, it going to cost you a lot. If not in just money spent on tooling - there really isn’t a lot of tooling - so it’s going to cost you a lot in time. And therefore, you might have to hire more staff. Well, that’s not going to happen anymore.

Tim Callan

That’s still TCO. That’s your total cost of ownership still went up.

Jason Soroko

Your cost of ownership of this technology is way better when you organize it in a way that it can be managed much, much easier. And I think that’s been the theme of a lot of things we’ve just said. It’s not just the enhanced security, it’s the fact that this is a lot easier. You don’t need to do extra work, and therefore, you don’t need as much time or people or tooling. And therefore, by going to a certificate-based authentication for SSH, you’re reducing your costs.

Tim Callan

And on the topic of tooling, and this goes directly back to the previous bullet single pane grass manageability, you’re probably using tools that you have to have anyway, that you have to own and that you have to know how to use, and you just apply those to this situation as well.

Jason Soroko

You're bringing it into the fold of handling your authentication key material, certificate material. Yes. Absolutely, Tim.

Tim Callan

You're bringing it into the fold of handling your authentication key material, certificate material. Yes. Absolutely, Tim.

Jason Soroko

So, when you have any kind of credential form factor that expires by definition, you need to renew it, once it expires, and you need to issue it, and heck, sometimes you actually want to say, I no longer trust this thing. I should revoke it.

And there’s a lot of different reasons to revoke, Tim, and we’ve talked about that on previous podcasts. But at least you get those capabilities because with things like SSH, replacement processes and revocation, well, you could say that it – replacement kind of exists, but revocation does not.

Revocation actually is – if you think about what would it take to do the equivalent to a revocation in an SSH key pair, we could list out what it would take here. It’s technically possible, but it’s a lot of work. And it’ s almost, really in reality, I think the reality of it is, it’s almost impossible because where are all the copies of the keys, Tim?

Tim Callan

I was just going to contend that. It is impossible because, and this is the Black Swan problem I was talking about earlier, you have to somehow prove to your satisfaction that this key, this private key, is not in use anywhere. How do you do that?

Jason Soroko

Unless you can absolutely guarantee there are no other copies, you cannot. That is the right answer.

Tim Callan

On the other hand, certs can be revoked. That’s one of the great things about certificates. Ideally if you have these short-lived certs that you and I are talking about, the exposure is low, but still isn’t necessarily zero. Among other things, let’s say I’m issued that one-week cert, and I’m done with my job, and I know I’m done with that job, maybe I can request a revocation right then and three hours later get on with my life. Now there’s no exposure.

Jason Soroko

So, Tim, if you're already running that Certificate Lifecycle Management System and now you’re bringing SSH key materials into the fold of that CLM, running it with certificates, you can also do what you’re probably already doing, which is, if those certificates are tied to a specific employee, well, if the employee leaves the company, you probably have a governance process that’s handling the revocations for that.

As well, if you determine that there’s an automated system that needs to be reissued, have a renewal process for certificates because after a week or two or a month, whatever it is you decide for you that your certificate lifecycles are, that renewal process can actually be automated by your Certificate Lifecycle Management System. SSH raw cannot do that.

Tim Callan

I’m almost contrasting this to the opposite scenario. So let’s suppose that I have a developer, an employee who’s deep in the bowels of my systems, and let’s suppose that for some reason I have to let this person go and maybe this person’s disgruntled. Well, gee, I mean, how do I ever know that that person isn’t going to use an SSH key to come in somewhere and do something? On the other hand, in the other scenario you just revoke the cert or the certs, and it’s done.

Jason Soroko

Tim, it’s so important. It so important. I’ve resisted the urge to repeat this a hundred times, so I’m saving this for the end. If you wrap up these six points we just made, keep in mind it’s for anybody listening to this podcast. If you’re an organization of just about any size, you probably have some kind of Linux system in the cloud. Whether it’s your own private cloud or more than likely, you’re using public cloud for something. Just about everybody does. And you will, more than likely, have a Linux operating system out there doing something, at the very least, web servers running Apache or Nginx on, on Ubuntu or whatever your flavor or Linux distribution is. And who is the technical person in your company who actually does the administration of that server? I could put at least a few dollars on the table and bet they’re using OpenSSH to get there. And so, Tim, I just have to make the point. It is so ubiquitous, it is so ubiquitous, and this is such an important step in the lifetime, the history of SSH to go toward SSH certificates and finally bring it into the cryptographic management fold. Bring it into the arena of Certificate Lifecycle Management, and you get all of these benefits, and a lot of the old problems that we talked about just go away.

Tim Callan

Let’s just say these benefits one more time as recap: trust, customizable expiration, lower maintenance, single pane of glass manageability, reduced costs and simple revoke and replace processes.