Root Causes 76: Implications of COVID-19 for PKI

Tim Callan

How are you doing today, Jay?

Jason Soroko

Doing great, Tim, thanks for having me.

Tim Callan

Yep. Staying safe and hopefully, we are both staying away from global pandemics.

Jason Soroko

Don’t you hate global pandemics?

Tim Callan

Yeah. We talked about this, which is should we do a podcast? Should we talk about COVID-19 and I think we decided we kind of had to. It’s just too important in the way it’s changing how people work and there are just matters there that need to be addressed.

Jason Soroko

I don’t want to make light of any of this. It’s almost trench warfare that we are fighting at this point with this virus, but, you know, obviously, health and family it all comes first. But for a lot of us, business continues and it’s continuing from our homes, isn’t it?

Tim Callan

Yeah. And it’s important that business continues. Like a lot of the negative affects of this virus will be collateral damage because of the action that people are taking to slow its spread and so there is a big difference for the health of the global economy, for everybody’s well-being, if business can continue. So, it’s not even crass to want to make sure that when we all shelter-in-place that we are doing it in a way that allows businesses to continue to operate. Those supply chains flowing is important. Those deliveries arriving at people’s doorsteps is important. Business has to flow for everybody to weather this as well as they can.

Jason Soroko

Yeah. And a lot of you information workers out there who are the backbone of how all this works, instead of commuting to work in your car for a few miles down the road, you might be commuting a few feet in slippers to your computer, and I guess what we are trying to do with this podcast is say that security still applies perhaps just that much more so.

Tim Callan

Yeah. That’s exactly right. Right? So, first, none of the old security vulnerabilities go away and you can rest assured that the bad guys are licking their lips right now saying disruptive factors are always great for the people who are trying to breach your systems. They like that and so, you can rest assured that this will be viewed as an opportunity by many people around the globe and so we want to talk about (1) just making sure that your security fundamentals are still in place but (2) is we are going to be changing a lot of things about our processes and let’s make sure about how to be smart about how we do that so we introduce vulnerabilities where we didn’t have them otherwise.

Jason Soroko

Right. So, for all the folks in IT departments that are tasked with perhaps imaging new workstations for people to work from home. All those tasks of making sure that your patching cycle is good, backups, you know, all that regular IT/security stuff. We can’t forget it. We just must be right on top of it. What we are gonna be talking about I think in this podcast, Tim, is more along the lines of how to make it easier to provision access control essentially is what we are coming down to. Because you want to be able to give access, secure access to remote employees perhaps that might be coming online at scale. So, a lot of businesses might have already had, you know, the sales department might have been on the road a lot, you might have issued out some VPN to them, but they may be special cases in your organization. What happens when most of your organization becomes remote employees? This is what we are gonna be talking about.

Tim Callan

Yeah. So, let’s dive into that one. So, if I’m scaling up the number of employees that have VPN access from let’s say 10% to almost 100%, what are the potential pitfalls? What do I need to be careful about?

Jason Soroko

For smaller IT departments who may not be really experienced with this, boy oh boy, Tim said it right. That the bad guys are licking their lips because they know some of the mistakes that you are gonna make. Some of the mistakes you could be making, provisioning RDP very incorrectly within your organization and just doing things the old way or in a weak way and quite often just in terms of authentication, the simple way to test yourself, whether you are just the user or are the IT person is kind of simple. Is the user who is logging in remotely using some system that only has one form of authentication? In other words, just a username and password.

Tim Callan

Yeah. Username and password. Right.

Jason Soroko

If that’s the case, pause for a moment and then really think hard. Maybe I could be doing this differently and the answer will probably come out in this podcast about different ways that you can try to approach that.

Tim Callan

Yeah. And, of course, a lot of people think, when they think of different, they are thinking of MFA, and we did an episode on that not too long ago. MFA is better than username and password but it’s not perfect either.

Jason Soroko

It’s not. And, in fact, how long ago did we hear – it was maybe just a few weeks ago – that Google’s authentication, OTP application, which is used by a lot of systems had actually a backdoor in it and it was made known to Google a long time – I forget how long it was, but it was a very long period of time and it was only recently that somebody then realized that being able to screenshot that particular application perhaps from a remote attack against a mobile device would be possible. It’s just things like that where - - really there’s two problems with MFA. And I’m a proponent of MFA. I really am because it’s better than just username and password, but the problem is, probably the worst aspect to it is not even the security. It’s the fact that it’s one other thing that the user must do to log in.

So, if you have staff that are not savvy IT-type thinking people, it could be anybody on your staff, those are the people who might get, it might be too big of a speed bump for them to be able to get into your operations securely, Tim.

Tim Callan

Right.

Jason Soroko

And then, additionally, I think is what we were saying, which is MFA is not, in terms of the spectrum of security, it’s a speed bump to the bad guy but there’s so many examples of weaker forms of authentication that have fallen short.

Tim Callan

Yeah. Like for somebody who is a very persistent bad guy, for a spear phisher or an APT, where the stakes are high, you know, there are also dangers like SIM swapping that can really sort of crush your traditional phone-based two-factor identification approach. So, it has some vulnerability. As opposed to, of course, you know, certificate-based authentication. Right, Jason?

Jason Soroko

Absolutely, Tim. So, those are the two criteria, right? Which is, is it easy for the user? If it wasn’t easy, we shouldn’t even be talking about it because it’s gonna fail. And number two, is it secure? Well, the beauty of PKI certificate-based or client certificates being used, and we are talking about client certificates being used potentially for VPN or Desktop as a Service, um, these are ways to very carefully restrict access so that only the device and the person who possesses that key material, that certificate, is gonna be the one who can authenticate in. So, it’s very secure in that sense. The only way it’s gonna work is if we can automate the provisioning of the client certificate and then, therefore, eliminate the friction from the user standpoint. So, for any of you who have been provisioned, who were lucky enough to have your mobile device provisioned with a client certificate for your wi-fi access point at work, for example, you know the joys of being able to just show up at work and fire up your mobile device and then you are automatically authenticated. No username and password, you are not worried about anything and by the way, incredibly secure and that certificate is stored inside of a secure element that’s incredibly difficult for the bad guy to get at. Isn’t it interesting that now, Tim, we are talking about the same concept but at the workstation level. So, therefore, having a workstation provisioned with a PKI certificate is then able to, without a password, be able to log into a Cisco any connect type of remote connection to work or, what’s increasingly becoming common is Desktop as a Service. And, a lot of people don’t realize but Desktop as a Service whether that’s Amazon Workspaces or the equivalent from Azure or any of the other independent clouds out there, a lot of them can be provisioned with a PKI user client certificate so that the device that basically has the client for the Desktop as a Service is the only one going to be allowed to log into that remote desktop and that’s incredibly important, Tim, because those remote desktops are essentially the edge of those systems is the public internet, which is a scary place.

Tim Callan

Right. And, you know, automation for provisioning of these things obviously is great because it helps with the workload problem. It also helps with the error problem, right? If we got people doing this individually there is always a chance that you get something wrong and with automation, if you automate it right the first time and you QA it right the first time, you can have a pretty high degree of confidence that it’s being done correctly every time. So, you know, it’s not just the time and the effort and the overworked employees that now have to do all this additional stuff, it’s also just not leaving holes in your perimeter by getting things wrong.

Jason Soroko

That’s exactly right, Tim. Exactly right.

So, beyond that, we still must do some hygiene which is if you are in fact using a VPN, the biggest problem with VPN, you are essentially creating an encrypted tunnel between your user’s computer and some other endpoint which is hopefully within a secure network such as your enterprise network. The problem with that comes in when, let’s say that user is tied to an Active Directory credential and that Active Directory credential is overprivileged. In other words, perhaps you’ve given that user admin privileges simply because it was easy. The problem is if that credential happens to be stolen in some way or form, that bad guy now is over prepared.

Tim Callan

Yeah. They have everything.

Jason Soroko

They get everything. Enter the world of APIs. So, if you think about that Desktop as a Service idea, what a great idea because that user is no longer having to do a VPN and be overprivileged into a corporate network. That user is now able to get their job done in a specifically tailored cloud environment. That’s fantastic. They are not overprivileged. In other words, that principal of least privileges comes in. But if you don’t have that option and you are not that lucky, that’s still no problem. What it does mean is you’ve gotta do your homework, get into your Active Directory settings and make sure that the users who may never have been typically remote in the past, you have to ask yourself the question what would happen if my HR employee, my Finance employee, you know, a developer who usually works in the office, that credential somehow gets out into the wild and a bad guy is able to use it to come in. Make sure that your privileges are locked down such that the minimum amount of damage would occur. And Tim, here’s the other aspect to that when you talk about PKI specifically and the certificate very particularly. Make sure whatever system that you are using to automate the provisioning of that certificate to the user is taking advantage of a secure element on the laptop. If you’ve got Windows 10 users with Windows 10 workstations such as a laptop, you can take advantage of a TPM chip on the motherboard to put that key material in a very safe place. I think that’s a key, key, key idea here.

Tim Callan

TPM stands for Trusted Platform Module, and I think we’ve mentioned in earlier episodes that all Windows 10 systems by contract with Microsoft must include a TPM. So, that winds up being a very good option for a lot of the desktops that your users are gonna be using.

Jason Soroko

So, if we tie up the whole concept, Tim, automate the provisioning of that client certificate to that user’s workstation, whatever it’s gonna be – probably a laptop – make sure that that certificate is in the TPM nice and safe and then the beauty is if you’ve done the right due diligence with that laptop and you have, you know, a secured BIOS, you’ve encrypted the hard drive and you are using the TPM, even if that employee is not used to working on the road, they happen to leave their laptop at the coffee shop or on the bus or whatever it is, you can be reasonably assured that the bad guy is not going to be able to take that certificate out of the TPM, as an example.

Tim Callan

So, the loss in that scenario is the value of the laptop rather than the value of the access and the value of the laptop is always much, much less than the value of the access. So, yes. That’s a good scenario for sure.

Jason Soroko

Yeah. And I don’t want to sound too much like a salesman for Desktop as a Service but imagine a workplace where you can hand out inexpensive laptops that have no data or very minimal data on them except for a highly protected PKI certificate which is then able to ensure secure access to Desktop as a Service in the cloud which is where all the data can reside in a secure manner. So, if I were starting from scratch and I was running an IT shop and I quickly had to take 50-100 people remote, boy, what a nice option.

Tim Callan

Let’s segue a little into more thoughts about automation. So, I think you talked about automation as a great part of making this remote happen correctly. So, we talked about the scenario where everybody is going off to work remotely and that’s great, but what about the scenario where their productivity is negatively affected? Let’s say some of them are sick or let’s say that they don’t have kids in school, and they are 70% of the employees they normally were. Under these circumstances, it’s just getting harder and harder for the old-fashioned management by spreadsheet scenario to work. Right? Again, this is where there is risk. We’ve got a disruption to our business. We are doing things in a different way. Our employees are distracted. Our employees are overworked. Our employees are being asked to fit other things in because they are taking everybody remote and doing all the things you talked about and during all that, that’s when those errors can occur.

Jason Soroko

Tim, we’ve already seen the effect of people being disrupted in exactly the way you said when the U.S. government had a bit of a blip in its services where all kinds of very important government websites had their SSL certificates expire because they were not automated.

Tim Callan

Yeah.

Jason Soroko

And the employees who were meant to take care of them simply were not able to for whatever reason. I can see that happening.

Tim Callan

Yeah. And those employees, you know, if you go back to that scenario, those employees – the government shutdown was 35 days long and those employees at the beginning of that were fully expecting to be working 30 days from now. Right? They didn’t have any vacation scheduled and they were going to deal with that certificate when the time came. And they couldn’t because this factor came along and prevented them and that’s a real risk here. Like there will be people who for one reason or another are not able to give the full focus to the IT tasks that they were doing a month ago.

Jason Soroko

Well, Tim, even in the best of times, Microsoft Teams – and I’m not picking on them because it can happen to anyone and it has happened to a lot of people – Microsoft Teams went down because of the fact that, you know, a certificate or one or more expired in the background which caused a backend system to not be able to authenticate essentially shutting out the entirety of Microsoft Teams, which was a really unfortunate situation because it probably could have been avoided with automation.

Tim Callan

Yeah. And when we did our own self-assessment - we mentioned this in an earlier podcast, we did a self-assessment to understand how our business continuity plans applied to the specific risks that we identified around the COVID-19 virus. When we did our own self-assessment, one of the big things that helped us feel good about operational continuity was the presence of automation. Right? Obviously, we are big automation fans, and it helps because, you know, there is no one individual who if they suddenly stopped showing up to work tomorrow would prevent any process from running. And that’s really the situation that you need to be in if you want to be confident in your business continuity.

Jason Soroko

Even wrapping it into what I was talking about at the first half of the podcast is, you know, for those of you who are responsible for building say laptop images, as part of your job, if you could automate the provisioning of client certificates into that laptop at that point in time and also provide your users, you know, a very easy portal to be able to recover a credential or whatever it is that they need, you know, you don’t necessarily get the chance to put your feet up on the desk and declare yourself done but at least you are not gonna get overwhelmed and I think that’s the risk that we might be facing with this huge change that nobody was expecting because of this virus.

Tim Callan

Yeah. Yeah. And, again, who knows how many other unexpected IT tasks are going to come up because of these massive changes we are making in the way that we work. So, you know, even if you think you’ve got a handle on everything that your team is going to be asked to do, and maybe you don’t, and so, whatever help we can get to ensure that everything is operating correctly is help we should avail ourselves of.

Jason Soroko

It’s true and when it comes to security, it’s almost imperative because you really can’t watch every corner, but the bad guy is.

Tim Callan

Yeah.

Jason Soroko

You only need to make one mistake and the bad guy is in. So, you really, your defensive posture you need to take advantage of everything possible and that means not screwing up the simple things by not - - by simple human error and I think the chance of human error just grows and grows as time goes on during what we are going through right now.

Tim Callan

Yeah. So, um, so, yeah. I mean kind of the takeaway is automation, automation, automation. Like, for people who haven’t gotten the memo yet, the benefits of automation are evergreen but now this might be a very illustrative time to think about the value and I guess the last point I’ll make on that is, it isn’t too late. So, we’ve talked to customers who say, gee, I wish I had been having this conversation with you two months ago because at this point, I’d be all buttoned up. But I don’t expect it to be business as normal months from now. So, if I get on something now then I know I’m gonna be getting the benefits of it before we’re out of the current crisis that we are in. So, if you think about it in those terms, even now you can act on an automation initiative and be glad that you did it – not just for the long-term health of your company but even for the short-term needs that you are experiencing right now.

Jason Soroko

Maybe, Tim, I’ll give you a good example of where I think some of the low hanging fruit might be for some of the listeners which is the uptick of phishing emails, spear phishing, because of the Coronavirus and COVID-19 key words that are of high interest to people right now. A lot of people are opening those emails to get their information - - the people who run spear phishing attacks know this and they are gonna be taking advantage of it. So, therefore, you want to make sure that when the HR department or whoever it is, you know, the boss sends out a COVID-19 or Coronavirus titled email that you know where that email came from. We’ve talked about S/MIME a lot and we’ve talked about how the difficulties of S/MIME caused it to not be a popular option for many, many years but automation solved it, Tim. Right?

Tim Callan

Yeah.

Jason Soroko

And, so, provisioning S/MIME into your workstations and to your mobile devices and being able to search on encrypted emails, those problems have been solved. The ability to know that an email did come from your HR department and your CEO, you know, and you want to read about these important topics, boy, that’s low hanging fruit for people right now.

Tim Callan

Yeah. And it’s not just COVID-19 subject matter. Of course, that is an obvious one. Phishers like to grab whatever is in the news and whatever creates anxiety, and this does both of those. So, in that regard, COVID-19 subject matter and titles and things for sure is happening but also just in general, again, when there is a disruption of process, all the sudden people are in different places, they are not at their normal desk, they can’t lean over the cubical wall and ask somebody did you really want me to do this. You know, that’s the kind of circumstance where spear phishers start to get their way. They get their chinks in the armor. They fool somebody into thinking that oh somebody is on a different device and, you know, therefore, therefore it’s different and weird. Or, gosh, I’m at my home office and I don’t have access to the following. Can you send it to me? Or, what’s the log in? And that sort of thing is how spear phishers succeed. So, just also the disruption in the normal process becomes a big vulnerability and, again, signing the email saying I know for sure that this email really came from my CFO makes a big difference in combatting that kind of thing.

Jason Soroko

Tim, that’s very well said but I’m gonna add one more thing just to show you how dangerous this is becoming.

Tim Callan

Please.

Jason Soroko

So, I’m lucky enough, you know, I have resources to look at the CT logs daily and it’s interesting just to watch them and it’s not surprising that all the different Coronavirus, COVID-19 terminologies are showing up in new domains. That’s a not a surprise because, you know, either businesses are setting up subdomains for informational purposes or it could be just people parking a domain because they think they can make a buck off having a domain with those terms. That’s not surprising at all. The issue I have and what I want to bring to light in this podcast is there’s so many, so many new Coronavirus domains, how many of those do you think might be used for phishing attacks?

Tim Callan

Sure. Absolutely.

Jason Soroko

So, I just wanted to, you know, everybody who is too busy to pay attention to those things, I wanted to bring that to light. Please pay attention. Pay very close attention. Spear phishing is gonna be bad and it’s absolutely despicable that anybody would do it but we just have to expect it.

Tim Callan

Oh, and they will. We know they will. Every time there is a major disaster there is a wave of spear phishing that comes immediately after and a wave of general phishing that comes immediately after it to take advantage of the disaster. So, we should not expect any less in this case and, you know, again, we are doing everybody a service - ourselves, our coworkers, our employers, our shareholders, and our society a service if we prevent those bad guys from getting their way. So, this is good for everybody if we are vigilant, and we take care of this stuff.

Jason Soroko

Yeah. So, anyway, Tim, I think automation, automation is the major theme. For those of you having to provision a bunch of remote users, you have my sympathies, but you will get through it.

Tim Callan

You’ll get through it. And that’s probably a great place to leave it today, Jay. There was a bunch of implications of COVID-19. There may be more. We may return to this topic if there are but, in the meantime, we have a lot of other evergreen topics that we want to talk about and we will keep going with those as well. So, thank you, Listeners, today, for joining us. Thank you, Jay. It’s always a pleasure even if it’s under unusual circumstances.

Jason Soroko

Stay safe, everybody.

Tim Callan

Stay safe. Thank you. This has been Root Causes.