DigiCert Revocation Incident
(CNAME-Based Domain Validation)

Executive Summary

DigiCert will be revoking certificates that did not have proper Domain Control Verification (DCV). Before issuing a certificate to a customer, DigiCert validates the customer’s control or ownership over the domain name for which they are requesting a certificate using one of several methods approved by the CA/Browser Forum (CABF). One of these methods relies on the customer adding a DNS CNAME record which includes a random value provided to them by DigiCert. DigiCert then does a DNS lookup for the domain and verifies the same random value, thereby proving domain control by the customer.

There are multiple valid ways to add a DNS CNAME record with the random value provided for this purpose. One of them requires the random value to be prefixed with an underscore character. The underscore prefix ensures that the random value cannot collide with an actual domain name that uses the same random value. While the odds of that happening are practically negligible, the validation is still deemed as non-compliant if it does not include the underscore prefix.

Recently, we learned that we did not include the underscore prefix with the random value used in some CNAME-based validation cases. This impacted approximately 0.4% of the applicable domain validations we have in effect. Under strict CABF rules, certificates with an issue in their domain validation must be revoked within 24 hours, without exception.

Customer Action Required

Impacted customers have been notified and must replace their certificates within 24 hours. Please login to your DigiCert account to view impacted certificates and reissue/rekey certificates.

How do I reissue certificates in CertCentral?

1. Login to your CertCentral account and view the CNAME Revocation Incident banner when you first login to see impacted certificates.
2. Navigate to the Certificates > Orders page and locate your impacted certificates.
3. Generate a new Certificate Signing Request (CSR).
4. On each certificate’s Order # details page, in the Certificate actions dropdown, select Reissue certificate.
5. Complete any additional required validation steps.
6. Install your reissued SSL/ TLS certificate.

If you use a certificate management solution such as Trust Lifecycle Manager, please refer to its instructions on how to automate replacement of impacted certificates

For any questions, please contact your account manager or reach out to DigiCert Support using the information provided in your CertCentral account. You can also reach our Support directly at +1 801-770-1718.

Technical Details

Browsers require Certificate Authorities to verify each domain included in a TLS certificate request before issuing a certificate. One of the allowed methods of DCV is called “Method 7” or “DNS-based verification”.

The CABF Baseline Requirement state: “3.2.2.4.7 DNS Change Confirming the Applicant’s control over the FQDN by confirming the presence of a Random Value or Request Token for either in a DNS CNAME, TXT or CAA record for either 1) an Authorization Domain Name; or 2) an Authorization Domain Name that is prefixed with a Domain Label that begins with an underscore character.”

Method 7 verifies control of a domain by having the domain owner add a DNS resource record with a random value or request token. Domain owners can add the value in the CAA, TXT, or CNAME record. When using a CNAME record, there are multiple ways to add the record. For example, if a certificate is being requested for “foo.example.com”, a valid DNS CNAME record can be added in the following three ways:

1. “_randomValue.foo.example.com CNAME dcv.digicert.com”
2. “foo.example.com CNAME randomValue.dcv.digicert.com”
3. “_dcv.foo.example.com CNAME randomValue.dcv.digicert.com”

Note that in Case 1, an underscore (‘_’) prefix is required with the randomValue, whereas in Case 2 or 3, it is not required. The underscore prefix requirement in Case 1 is based on RFC1034, which requires that domain names start with an alphanumeric character. Including an underscore means the subdomain used for validation can never match with an actual domain. Failing to include the underscore is considered a security risk because there is potential for a collision between an actual domain and the subdomain used for verification. Although the chance of a collision is extremely low because the random value has at least 150 bits of entropy, there is still a chance. Because there is a finite chance of collision, revocation is strictly required per CABF rules.

The documentation provided by DigiCert to customers for Case 1 did not explicitly specify that the underscore prefix is required. As a result, domain validations that used this method where customers did not add the underscore prefix were non-compliant.

According to CABF Baseline Requirements, any non-compliance with domain validation requires 24-hour revocation of issued certificates:

“4.9.1.1 Reasons for Revoking a Subscriber Certificate […] With the exception of Short‐lived Subscriber Certificates, the CA SHALL revoke a Certificate within 24 hours and use the corresponding CRLReason (see Section 7.2.2) if one or more of the following occurs:

5. The CA obtains evidence that the validation of domain authorization or control for any Fully‐Qualified Domain Name or IP address in the Certificate should not be relied upon (CRLReason #4, superseded).”

Any issue with domain validation is considered a serious issue by CABF and requires immediate action. Failure to comply can result in a distrust of the Certificate Authority. As such, we must revoke all impacted certificates within 24 hours of discovery. No extensions or delays are permitted. We apologize if this causes a business disruption to you and are standing by to assist you with validating your domain and issuing replacement certificates immediately.

Root Cause Analysis

In August 2019, we began modernizing our domain and organization validation systems towards a service-based architecture with a goal of improving performance and simplifying workflows. Legacy code in CertCentral (our public TLS certificate issuance portal) automatically added an underscore prefix to random values if a customer selected CNAME-based verification. Our new architecture redirected all validation through separate services instead of the legacy monolithic code structure. The code adding an underscore prefix was removed from CertCentral and added to some paths in the updated system. The underscore prefix addition was not separated into a distinct service. One path through the updated system did not automatically add the underscore nor check to see if the random value had a pre-appended underscore.

The omission of an automatic underscore prefix was not caught during the cross-functional team reviews that occurred before deployment of the updated system. While we had regression testing in place, those tests failed to alert us to the change in functionality because the regression tests were scoped to workflows and functionality instead of the content/structure of the random value. Other paths through the system either added underscores automatically or required customers to manually add the random value before verification completed. Unfortunately, no reviews were done to compare the legacy random value implementations with the random value implementations in the new system for every scenario. Had we conducted those evaluations, we would have learned earlier that the system was not automatically adding the underscore prefix to the random value where needed.

On June 11, 2024, engineering completed a user-experience enhancement project that collapsed multiple random value generation microservices into a single service. This service began including an underscore prefix before each random value, regardless of which validation method the user chose. This project allows DigiCert to simplify the random value generation process. This also reduced customer support calls related to the manual addition of underscore prefix, fixed a bug in CertCentral’s display of validation status, and inadvertently ensured that every CNAME-based verification included an underscore prefix to each random value. As before, we did not compare this UX change against the underscore flow in the legacy system.

Several weeks ago, someone contacted our problem report alias over email asking about random values used in validation. Although the reporter did not provide serial numbers for any certificates, DigiCert conducted a preliminary investigation. This initial investigation did not uncover any issues with random value generation or validation. After the reporter requested additional reviews (still without providing any certificate serial numbers), DigiCert sought guidance from external CABF participants, who suggested DigiCert conduct an additional review. Upon further review, DigiCert discovered an issue regarding the underscore prefix for random values. DigiCert then initiated this incident management process.

Preventive Actions Taken

We recognize the impact an incident like this can have on our customers and partners. To prevent reoccurrence of such incidents, we have taken or will take the following actions:

1. Consolidation and review of all random value generators across DCV [Completed]
2. UX simplification so customers do not need to know about specific random value formats based on their choice of DCV method [Completed]
3. Compliance team members will be embedded in all Certificate Authority (CA) and Registration Authority (RA) sprint teams (including design/architecture reviews) and will review all applicable changes [Completed]
4. Increase test coverage beyond functional testing in all validation workflows with compliance-based automated test cases [In Progress; ETA 5 August 2024]
5. Open source DCV for community review [In Progress; ETA 1 November 2024]