top of page

Posts

    • Oct 12, 2021
    • 4 min read

How To Avoid An SSL Certificate Error For An Old OS Caused By Expired LetsEncrypt's Root Certificate

Updated: Apr 29, 2022



If you are using a root certification from LetsEncrypt, you might discover that an SSL certificate error occurring on access from an old OS. LetsEncrypt's root certificate was changed to a cross-root certificate with a certification authority "ISRG Root X1", which is valid until 2035, due to the expiration of "DST Root CA X3" whose expiration date was on September 30th, 2021. After "DST Root CA X3" was expired, SSL certificate errors have been occurring in some environments when accessing sites using LetsEncrypt certificates.



Causes and solutions by different OS


For CentOS7, Amazon Linux, Amazon Linux2

This is most likely due to the fact that the root certificate installed in the OS is outdated and the "ISRG Root X1" root certificate does not exist in the root certificate tree. You can update the ROOT certificate with the following command.

yum -y update ca-certificates

The problem also occurs when the OpenSSL version is old (1.0.1 or older). In that case, you need to upgrade OpenSSL to the latest version as well.



For CentOS6 and common errors

Cross root certificate validation requires OpenSSL 1.0.2 or higher version, but CentOS6 is no longer supported and only versions up to 1.0.1 are available. OpenSSL 1.0.1 or older versions do not have the partial chain function for SSL certificates, so the certificate cannot be verified correctly so that an SSL communication error will occur.

The following are examples of common error patterns.


Example of wget

wget https://aWebsiteThatUsesACirtificateFromLetsEncrypt

Execution result

--2021-10-12 12:06:18--  https://aWebsiteThatUsesACirtificateFromLetsEncrypt/
Resolving aWebsiteThatUsesACirtificateFromLetsEncrypt... xx.xx.xx.xx
Connecting to aWebsiteThatUsesACirtificateFromLetsEncrypt|xx.xx.xx.xx|:443... connected.
ERROR: cannot verify aWebsiteThatUsesACirtificateFromLetsEncrypt’s certificate, issued by “/C=US/O=Let's Encrypt/CN=R3”:
  Issued certificate has expired.
To connect to aWebsiteThatUsesACirtificateFromLetsEncrypt insecurely, use ‘--no-check-certificate’.

Example of OpenSSL

openssl s_client -connect aWebsiteThatUsesACirtificateFromLetsEncrypt:443 -servername aWebsiteThatUsesACirtificateFromLetsEncrypt < /dev/null

Execution result

CONNECTED(00000003)
~ommited~
    Start Time: 1633378075
    Timeout   : 300 (sec)
    Verify return code: 10 (certificate has expired)
---
DONE

Example of a PHP program that uses the OpenSSL library (e.g. libssl.so.10) to communicate

If a program uses the get_file_contents function as shown below example, it will fail to validate the certificate.

<?php
$url = "https://aWebsiteThatUsesACirtificateFromLetsEncrypt/";
$output = file_get_contents($url);
var_dump($output);

Execution result

PHP Warning: file_get_contents(): SSL operation failed with code 1. OpenSSL Error messages:
error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed in "executedProgram" on line 3
PHP Warning: file_get_contents(): Failed to enable crypto in "executedProgram" on line 3
PHP Warning: file_get_contents(https://aWebsiteThatUsesACirtificateFromLetsEncrypt/): failed to open stream: operation failed in "executedProgram" on line 3
bool(false)

The problem does not occur if you are using the curl command or the PHP curl function. This is probably because the curl command uses not only the OpenSSL libssl.so.10 library, but also the NSS libssl3.so library.

ldd /usr/bin/curl | grep ssl

libssl3.so => /usr/lib64/libssl3.so (0x00007f43fc44b000)

libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f43fb1c5000)

ldd /usr/bin/wget | grep ssl

libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f79a04c6000)

ldd /usr/bin/openssl | grep ssl

libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f2872a3a000)

rpm -qf /usr/lib64/libssl3.so

nss-3.44.0-7.el6_10.x86_64

rpm -qf /usr/lib64/libssl.so.10

openssl-1.0.1e-58.el6_10.x86_64


How to solve the problem You will either need to modify your program to use the curl command or a curl-based library, or you will need OpenSSL version 1.0.2 or higher. However, as mentioned earlier, CentOS6 does not provide versions higher than 1.0.1, so you need to get SRPM from the CentOS7 repository, modify it for CentOS6, then build and install the right version if it is difficult to modify the program.


*This is a procedure that has been proven on our servers, but please note that we have not been able to confirm the impact and operation of all OpenSSL functions.

*Fundamentally, it is recommended that you update your OS.



How to forcefully install OpenSSL 1.0.2 on CentOS6

Create an rpm package from srpm

  • Obtain the OpenSSL 1.0.2 rpm package from the CentOS7 repository

cd /usr/local/src
wget --no-check-certificate https://vault.centos.org/7.9.2009/updates/Source/SPackages/openssl-1.0.2k-21.el7_9.src.rpm
rpm -i openssl-1.0.2k-21.el7_9.src.rpm

*Move to a directory named rpmbuild which is created under the home directory of the user who executed the rpm command. The source code will be extracted in the directory.

cd ~/rpmbuild

  • Fix SPEC file

Comment out "%patch68" as it is not necessary.

sed -i -e "s/%patch68/#%patch68/g" SPECS/openssl.spec

  • Fix patch file

The glibc function "secure_getenv" is not available in CentOS6, so replace it with "getenv".

sed -i -e "s/secure_getenv/getenv/g" SOURCES/openssl-1.0.2a-env-zlib.patch
sed -i -e "s/secure_getenv/getenv/g" SOURCES/openssl-1.0.2j-deprecate-algos.patch
sed -i -e "s/secure_getenv/getenv/g" SOURCES/openssl-1.0.2a-fips-ctor.patch

  • Build the rpm package

If you don't have the necessary tools for the build installed, install them and then rpmbuild.

yum install rpm-build krb5-devel zkib-devel lksctp-tools-devel zlib-devel gcc
rpmbuild -ba SPECS/openssl.spec

Install the rpm package

  • Install the packages required for the dependency (perl-WWW-Curl)

yum install perl-WWW-Curl

  • Go to the directory where the rpm package was built, and install it with the rpm command

cd RPMS/
rpm -Uvh openssl-*

  • Check if the version has been upgraded

openssl version

*You are good to go if the version is 1.0.2k



Deactivate the DST Root CA X3 certificate

The problem will not go away if the DST Root CA X3 certificate is still in the root certificate of the OS. It is an expired certificate anyways so let's make it invalidated.


  • Update the root certificate of the OS

yum update ca-certificates

  • Enable the update-ca-trust feature and update the root certificate with the update-ca-trust command

update-ca-trust enable
update-ca-trust

Right after updating the root certificate with the "yum update ca-certificates" command, the trusted root certificate (/pki/tls/certs/ca-bundle.crt) file in the OS will contain not only the contents of the certificate but also comments such as Issuer and expiration date.

However, if you run the "update-ca-trust" command without arguments, the trusted root certificate of the OS will be reorganized into a file with only the contents of the certificate and comments.


Be sure to run the command in the next step to ensure that only the certificate part is extracted.


  • Extract only the "DST Root CA X3" certificate and save it as a file under /etc/pki/ca-trust/source/blacklist/

perl -e 'while(<>){last if $_ =~ m/DST Root CA X3/;}print $_;while(<>){last if length($_)==1;print $_}' </etc/pki/tls/certs/ca-bundle.crt > /etc/pki/ca-trust/source/blacklist/DST_Root_CA_X3.pem

If the file named /etc/pki/ca-trust/source/blacklist/DST_Root_CA_X3.pem is created and the content looks like the following, you are good to go. You can also create the file manually if it does not work.

#DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

  • Disable "DST Root CA X3" certificate

update-ca-trust extract
grep "DST Root CA X3" /etc/pki/tls/certs/ca-bundle.crt

You are good to go if you don't see anything.



grep "ISRG Root X1" /etc/pki/tls/certs/ca-bundle.crt

You should see "# ISRG Root X1" as a result.



After the update, run the program that previously provided the error to see if the problem has been resolved. It is recommended to restart the service that provides SSL such as Apache if the server is running the service.




This blog post is translated from a blog post by Seiken Sato on our Japanese website Beyond Co..

2,671 views
bottom of page