Root Causes 304: Your 90-day SSL Certificates Checklist

Episode Transcript

Lightly edited for flow and brevity.

• 

  Tim Callan

  We actually have a trio of industry veterans today because we are joined by Henry Lam. Henry is Sales Engineering Manager here at Sectigo. How are you doing today, Henry.

• 

  Henry Lam

  Not too bad, and yourself, Tim?

• 

  Tim Callan

  Good. Thank you for joining us. The reason you’re here is Jason and I have been talking a lot in the last couple months about 90-day certificates and the announcement that they’re on the way, and we speculated a timeframe and things along those lines, and one of the messages that we have been trying to communicate is, you need to be preparing in advance. Like, the day that all of a sudden, all certificates go down to 90 days, if you haven’t done your homework it’s really too late. What you have to do, is get ready now so that when that occurs, you’re not going to be damaged by that. And you deal very directly with a lot of enterprises who have to work on this kind of thing. Right, Henry?

• 

  Henry Lam

  Correct.

• 

  Tim Callan

  And I’m given to understand that one of the things that we’ve done is we have created what I heard described as a 90-day preparedness checklist. Why don’t you tell us about that.

• 

  Henry Lam

  Not a problem. So, since that 90-day certificate is coming, to your point, Tim, we have prepared a checklist for our customers to prepare themselves to actually move towards issuance of 90 days. This checklist makes up of four major points. And the idea is to get an idea of what you have inside your environment today so that you could actually prepare for this. So all-in-all, the four checklist is, find out what certs you have inside your environment today, whether it’s external or internal, so then you have full list of what is currently in use. That’s first and foremost. Every organization should be doing this.

  Then, the second part is generally the harder part. What vendor technologies are you currently utilizing within your environment? Whether it is internally facing or externally facing. And most organizations may not have that list. Once you get the two datasets, you can then move on to say, let’s go to do an exercise of, you want these given systems with the certificate, and you want to automate it. Let’s go to mapping exercise, and say, Sectigo has x automation utilities. You have a specific set of your systems and the certificates, and let’s do a mapping exercise to figure out what automation utilities you could utilized from Sectigo to minimize and to reduce manual work that you may have today or reduce outages. Once all of that is done, the last step would be to create a deployment plan in terms of setting priorities for how you want to tackle to implement all of the data you collected so you can methodically take care of line-by-line or system-by-system to be able to automate the certificate effectively, put a checkmark beside it to say, it’s done, it’s automated, you have proof to then go down that list. So therefore, once you have it set, you’re most likely or more ease of mind, saying that your environment is pretty much automated.

• 

  Tim Callan

  Got it. Let’s just rattle those four off again real quick so they’re fresh in our minds, and then I think you gave us a lot to dig into here. So the first one was inventory your certificates.

• 

  Henry Lam

  Yes.

• 

  Tim Callan

  The second one is?

• 

  Henry Lam

  Vendor technology list mapping within the customer’s organization.

• 

  Tim Callan

  Vendor technology list mapping. The third one is?

• 

  Henry Lam

  Vendor technology list to automation mapping.

• 

  Tim Callan

  Gotcha. And then the fourth one is automation implementation. So, like, when you say a vendor technology list, what sorts of things are we talking about?

• 

  Henry Lam

  Let’s take the general, most organizations have. When it comes to the public side, they may have a firewall. Could be a Citrix, could be an F5, could be a Palo Alto. Those are publicly facing. You may have systems like an Nginx, an Apache, maybe a Node.js or you could be running Kubernetes with Docker. Those can happen. Inside your environment, depending upon what type of organization you are, you could have a network attached storage that requires certificates. You could have inside an internal access point that may need an SSL certificate that you can automate. So it’s anything and everything that an SSL certificate could be put on and secure. And most organizations generally buy and sell specific pieces or not renew specifically a piece of software within their org and sometimes they overtime lose track of that and lose track of what certificate could be utilized.

• 

  Tim Callan

  That strikes me as another interesting point. There’s a lot of directions we can go with this, which is, I do this inventory and I do this mapping exercise, but our technology stacks are not static. People are deploying, updating, patching, changing; so is there also kind of an ongoing maintenance exercise that’s part of this to ensure that my understanding is still current and correct.

• 

  Henry Lam

  Absolutely. We should always keep up with that. Once we ever let anything go lax, things generally go sideways pretty quickly and then, systems could break, outages could happen, and then you run into another set of problems that you never anticipated.

• 

  Tim Callan

  Then connected, you know, this is a big thing you just said. It’s inventory of all your certificates. Right there that sounds big, but then, inventorying all your affected technology products is probably bigger. And then, the next two, do your automation map and implement your automation –these are big, big tasks. How much calendar time, how much effort – I don’t even know how to ask the question. Like, how do people deal with this?

• 

  Henry Lam

  It’s a monumental task. There’s always a timeline , but to peg a timeline for any, a one-size-fits-all is pretty hard. You have to slowly divide and conquer. If you’re a smaller organization, it’s probably pretty straightforward because you have a limited set of applications, products and resources that know what’s in the environment. If you’re a larger organization, it mostly has to be divide and conquer to get that dataset. If you expect one person to get it all, you could take forever.

• 

  Tim Callan

  Likewise on that, it sounds like probably everything doesn’t have to be on the same train, right? If I can take some set of my operations and I feel like I’ve got a good handle on my inventory exercise, I might move ahead to the next step, while in another part of my operations, I’m still trying to understand all the vendors that I have. And they wouldn’t all have to move forward to the next stage at the same time, so long as they all get there by the time they need to get there.

• 

  Henry Lam

  Absolutely. They could all run in parallel. They’re not sequential.

• 

  Jason Soroko

  Henry, I think you’re getting at one of the reasons why people have certificate outages anyway. If you haven’t already had your inventory done, if the tasks that you spelled out on this checklist aren’t already completed within an organization, then it really does explain why certificates go missing and go rogue and outages happen. So therefore, the time to do this has always been now, but now with the 90-day certificates, it really, really is now. And I’m so glad you’re spelling this out in a real clear way. One of things I want to point out here - Henry, tell me if you agree - the beauty of this checklist is it doesn’t matter if you’re really large or really just a small organization just with a few webservers. I think your checklist really works from large to small organizations. Would you agree?

• 

  Henry Lam

  I would agree. It’s supposed to address or help every organization. The list could be as long as you want, it could be as short as you want, based on your organization’s needs and based on your organization’s size as well.

• 

  Jason Soroko

  Henry, tell me about some of the technologies underlying this because we’ve got a lot of people listening to this podcast who live in the weeds, and it’s an interesting place to be cause it’s a rich technology environment. You talked about some of the webserver technologies and the load balancer technologies and other things that are in this thing that a lot of listeners would be familiar with. Maybe what they’re less familiar with are the automation technologies. That’s something that you touched on. Can you give some color as to what’s available and what kind of flexibility that a company like Sectigo could offer because that’s ultimately what people are trying to map between, alright, well, I’ve got these certs, I’ve got these technologies that consume these certs, and now I need to automate them. What are my automation options?

• 

  Henry Lam

  The primary one we are utilizing is ACME. ACME is the protocol of choice that Sectigo has been looking to utilize for customers to automate certificates. ACME is the standard protocol. There’s a lot of different ACME clients out there that could be utilized to automate a certificate from your given application to get a certificate from Sectigo. If that somehow does not satisfy your criteria, Sectigo will have their own sets of utilities that we have potentially built that could also help with automation of certificate requests from Sectigo that could be put on to the end system as well. But ultimately, the goal is, let’s use the standard protocol to utilize rather than try to add as many potential custom application out there.

• 

  Jason Soroko

  Henry, while we’ve got you, can you characterize just for people who may not be as aware of Google’s announcement around the 90-day maximum certificate lifespan? The fact that you're going to now have to update the certificate, renew the certificate every, you know, at least every 90 days. It can’t be 91. It’s got to be 90 days or less. I think, Henry, for those people listening to this specific podcast about this problem, this is the reason why you’re coming up with this checklist, this 90-day checklist. This is the reason why we’re talking about these automation technologies. For those people who are thinking, I’m just going to suck it up, and I’m going to do a manual renewal every 90 days, can you spell out what you think are the big risks in trying to do it the hard way?

• 

  Henry Lam

  Well, the big risk, you missed that window. Gone are the days where a certificate is two years, three years, heck, even one year. When it goes on, in the one, two or three-year mode, you only have to worry about it once every calendar year at a minimum. But when it comes to 90 days, you’re not thinking of only 4 times a year. You have to think about doing it five times a year, and when someone has to repeat that five times a year, one cert is okay, but you have to repeat it for 50 certs, that’s an astronomical task. And not to mention the amount of times that same certificate could live on several systems simultaneously, which is then, a massive headache, and you will run into human error along the way.

• 

  Jason Soroko

  So, it’s a difficult task to get that mapping and inventory done, but it just seems like it’s worth it. It’s something that will help to reduce your outages regardless of your choice of automation, is my take. So even without automating, Henry, I think that your checklist, even if you don’t complete the final task, I think that the checklist you’ve come up with is valuable in that it will help to reduce outages, just regardless of whatever the final choice of remediation is for this particular problem. So, thanks for giving us that, Henry.

• 

  Tim Callan

  And, Henry, the last thing is you touched on this idea of prioritization. How do you advise people to go through that prioritization exercise?

• 

  Henry Lam

  There have been two prevailing thoughts when it comes to prioritizations. Some organizations will prioritize their most highly prized assets within their environment. So, for example, if you are in the e-commerce space, you will prioritize the system that brings in money. When that system goes down, you will lose x thousands or x millions of dollars for every minute it is down. That is one prevailing thought.

  The other prevailing thought is if an organization starts early enough, they will do the low hanging fruit so that they will at least see some of the technologies being implemented to automate the certificate to give them the warm and fuzzies until they move to the more difficult ones. Both options are viable. I generally tend to say customers will help us drive which priority is the best, with some guidance from Sectigo to help out to determine if it’s the right course of action after some proper discussion.

• 

  Tim Callan

  Cool. Alright, well, thank you very much, Henry. I think that’s great. As Jason said, I can see where that’s a useful tool, and you know, we’re trying to unpack the implications of this as the industry and everything else is discovering it, and I think as somebody who is dealing directly with enterprises and having these conversations, your perspective is very important in that regard.