How to Avoid Another Let's Encrypt-Like Meltdown

The Internet should not have broken as badly as it did when one of Let’s Encrypt’s root certificates expired on Sept. 30. Although the expiration date had been known well in advance and Let’s Encrypt had taken steps to prepare users, many websites, services, and older devices ran into trouble, leaving people unable to access their applications and online tools. Among the organizations caught off-guard were Palo Alto, Cisco Umbrella, Catchpoint, Shopify, Fortinet, and Netlify, according to Scott Helme, founder of Security Headers.

"Let's Encrypt really did do everything possible to help avoid this issue," Helme says. "Root certificates have expiry dates."

The life span of a certificate is usually between 20 to 25 years and, according to Helme, we'll experience more events like this in the near future. Several root certificates from multiple major certificate authorities (CAs) are scheduled to expire in the coming years.

Organizations that underestimate the impact expired root certificates can have on their environments are risking another crisis when the next certificate expires. And those that were lucky this time might not be so lucky in the future.

"Companies are still at risk for future problems until they develop a robust and complete plan to account for the [sometimes unexpected] need to replace any of their certificates at any time," says Tim Callan, chief compliance officer at certificate provider Sectigo.

A Primer on Root Certificates
Most people are familiar with SSL certificates, which they install on their servers to make sure the data of those accessing their websites is secure. Root certificates, also called trusted roots, are more powerful and are used to issue other certificates.

Every device has a collection of preinstalled root certificates called a root store. Some of the best known are Microsoft, Apple, Google, and Mozilla. Android devices use Google's store while macOS and iOS devices rely on Apple's.

While end user SSL certificates last up to two years, root certificates live for decades. Still, in theory, root certificate expirations shouldn't pose any problems because new certificates are created to replace the old ones. They are "distributed via updates to all of the clients out there, often many years in advance," Helme wrote on his blog.

Updates Are Critical
The process for updating root certificates is different from the way SSL certificates are handled, says Sectigo's Callan. Root updates must occur on the client side because that is the machine that ultimately needs to be able to establish trust, he says. Root stores exist on the operating system or application level and are usually updated automatically. However, older systems and devices may require explicit action by the user, Callan warns. For some very old or constrained devices, updates are simply not possible under any circumstances.

Older Android devices fall in the category of devices that can't be updated to get new root certificates. Let's Encrypt wound up advising users of Android 5.0 Lollipop or earlier versions to download the Firefox browser. Most browsers, including Chrome, Safari, and Edge, trust the same root certificates as the OS they are installed on, but Firefox has its own root store, which updates automatically. Installing Firefox gave those users the new root certificates.

Google plans to have its own Root Store in the future so that Android users won't face this issue next time.

Helme noticed that some devices with the latest versions of iOS and macOS still had issues because they were relying on an intermediate certificate, which sits between the root certificate and the SSL certificates, even though they had expired around the same time.

“Having an Intermediate Certificate be treated like it's a static part of your configuration is extremely bad and it will eventually cause issues. The whole certificate bundle should be updated each and every time you get a new server certificate!" Helme wrote.

When organizations learn that a new certificate is going to replace a soon-to-expire one, they need to update them. And as Helme noted, the whole certificate bundle needs to be updated.

Know Your Certificates
The Let's Encrypt episode shows the critical importance of certificate agility — the ability to respond in real time to events requiring the replacement of certificates in an organization's environment, says Callan.

"Lack of certificate agility can come from rigid and outdated design decisions, like certificate pinning or hand-curated root stores, but most often it comes from the simple failure to understand and maintain the full set of certificates the organization depends on," he says.

If a full inventory of all certificates used in their environments — with information such as where they are physically deployed and how they affect the systems and processes that depend on them — doesn’t currently exist, this is the time to create one. The inventory should consider all the root, intermediate, and end-user certificates since everything breaks if even a single certificate in that chain becomes invalid. Some certificates have a different update process than others, so knowing what that process looks like beforehand is also critical.

"This inventory must include knowledge of when each individual certificate is due to expire and any technical, security, or compliance requirements for its eventual replacement," Callan says.

The best way to deal with expiring certificates, according to Callan, is to have an automated system of certificate renewals. This needs little action from the user’s part.

"The days of 'management by spreadsheet' are behind us," Callan added.

Do Not Do This
While scrambling to address the Let's Encrypt problem, some organizations made the decision to allow untrusted or invalid certificates so that people would be able to access their applications and tools. This is not a good idea because it undermines digital trust.

"If we were to forego the chain of trust, we would expose ourselves to any number of malicious-actor-in-the-middle and other attacks that presently are blocked by the cryptographically secure mechanism of public-trust certificates," Callan says.

When Is the Next Expiration Date to Be Aware Of?
Let's Encrypt's new root certificate, ISRG Root X1, is scheduled to expire on June 4, 2035, which gives users plenty of time to prepare. Yet other certificates will come to an end sooner.

"Over the next few years we're going to see a wide selection of Root Certificates expiring for all of the major CAs and we're likely to keep experiencing the exact same issues unless something changes in the wider ecosystem," Helme wrote.

Determining which expiration date administrators need to pay attention to next requires “a bit of sleuthing and even speculation,” says Callan. After looking at all public roots in the Microsoft, Mozilla, and Apple root programs, Callan found that roots from several major CAs and former CAs, including GlobalSign, GeoTrust, and Cybertrust, will expire next year.

However, just because they are expiring doesn't necessarily mean they will have widespread impact. The root certificate would need to be heavily relied on to have that kind of effect.

The root certificates listed in very old root stores are the ones where problems are more likely to occur, Callan says. An analysis of root certificates stored in Android 5.1’s root store found the same roots as the ones in the Microsoft, Mozilla, and Apple root programs, suggesting that when they expire, Android devices will encounter similar issues as what happened with Let’s Encrypt.

“While that’s not a guarantee that any of these three will turn out to be a significant expiration, they are all worth keeping an eye on,” Callan says.