The SSL certificate chain is different; how to check?
Short version . I see the SSL certificate chain which is different depending on how I access the https server. What is happening and how can I verify the certificate under these circumstances?
A slightly longer version:
I am trying to use libcurl to validate SSL connection certificate. The server I am connecting to is Amazon S3.
When I access Amazon S3 in Firefox I get this certificate chain:
- VeriSign Class 3 Public Primary Certification Authority - G5
Serial Number: 18: DA: D1: 9E: 26: 7D: E8: BB: 4A: 21: 58: CD: CC: 6B: 3B: 4A- VeriSign Secure Server Layer 3 CA-G3
Serial Number: 6E: CC: 7A: A5: A7: 03: 20: 09: B8: CE: BC: F4: E9: 52: D4: 91- *. S3.amazonaws.com
Serial Number: 43: FB: BA: C2: 66: 27: E0: 97: 1E: 1C: 11: E0: 30: C3: 6B: 66
- *. S3.amazonaws.com
- VeriSign Secure Server Layer 3 CA-G3
When I access the same URL using the OpenSSL command line tool, I get this certificate chain:
- *. S3.amazonaws.com
Serial number: 43fbbac26627e0971e1c11e030c36b66. - VeriSign Secure Server Layer 3 CA-G3
Serial Number: 6ecc7aa5a7032009b8cebcf4e952d491 - VeriSign Class 3 Public Primary Certification Authority - G5
Serial Number: 35973187f3873a07327ece580c9b7eda.
"* .S3.amazonaws.com" and "VeriSign Class 3 Secure Server CA - G3" certificates look the same, but different after that! It is named "VeriSign Class 3 Public Primary Certification Authority - G5" in both chains, but the serial number of the certificate is different. (Other information is different, please let me know if you want a longer dump.)
I believe this difference is the reason why I cannot get libcurl to validate the SSL certificate. The certificate with the serial number "18: DA: D1 ..." is in the CACERT.PEM file, but the certificate with the serial number "35:97:31 ..." is not.
Obviously a simple fix is to add the "35:97:31 ..." certificate to the CACERT.PEM file, but I want to make the correct change here, not just a quick fix.
- What does this difference in certificate chains mean?
- Is it possible for the SSL server to return different client based certificate chains (Firefox vs OpenSSL / libcurl)?
- How do I get libcurl to validate this SSL certificate?
Really long version and background information:
I am using libcurl with OpenSSL to boot from Amazon S3. Libcurl returns "SSL Certificate Problem: Cannot Get Local Issuer Certificate" which I know means the root certificate is not listed in my CACERT.PEM file. (I'm using the one downloaded from the curl site that's converted from the Mozilla certificate store.) I can check the certificates against a different SSL, so I know my libcurl setup is correct.
To see what is happening and why the certificate is not validated, I stopped the same URL in Firefox. Firefox was not showing any SSL certificate warnings. The Firefox certification path is shown here. The root certificate, "VeriSign Class 3 Public Primary Certification Authority - G5", is listed in my CACERT.PEM file and the serial number of the certificate matches what is shown in the screenshot.
Here are the serial numbers of all three certificates in the chain:
- VeriSign Class 3 Public Primary Certification Authority - G5
Serial Number: 18: DA: D1: 9E: 26: 7D: E8: BB: 4A: 21: 58: CD: CC: 6B: 3B: 4A- VeriSign Secure Server Layer 3 CA-G3
Serial Number: 6E: CC: 7A: A5: A7: 03: 20: 09: B8: CE: BC: F4: E9: 52: D4: 91- *. S3.amazonaws.com
Serial Number: 43: FB: BA: C2: 66: 27: E0: 97: 1E: 1C: 11: E0: 30: C3: 6B: 66
- *. S3.amazonaws.com
- VeriSign Secure Server Layer 3 CA-G3
On a different platform (different OS, different version of OpenSSL, etc.) I tried to access the same URL using the OpenSSL command line tool and I got a different certification path!
$ openssl s_client -showcerts -connect stackoverflowtest.s3.amazonaws.com:443 CONNECTED (00000003) depth = 3 C = US, O = "VeriSign, Inc.", OU = Class 3 Public Primary Certification Authority verify return: 1 depth = 2 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = "(c) 2006 VeriSign, Inc. - For authorized use only", CN = VeriSign Class 3 Public Primary Certification Authority - G5 verify return: 1 depth = 1 C = US, O = "VeriSign, Inc.", OU = VeriSign Trust Network, OU = Terms of use at https://www.verisign.com/rpa (c) 10, CN = VeriSign Class 3 Secure Server CA - G3 verify return: 1 depth = 0 C = US, ST = Washington, L = Seattle, O = Amazon.com Inc., OU = S3-A, CN = * .s3.amazonaws.com verify return: 1 --- Certificate chain 0 s: /C=US/ST=Washington/L=Seattle/O=Amazon.com Inc./OU=S3-A/CN=*.s3.amazonaws.com i: / C = US / O = VeriSign, Inc./OU=VeriSign Trust Network / OU = Terms of use at https://www.verisign.com/rpa (c) 10 / CN = VeriSign Class 3 Secure Server CA - G3 ----- BEGIN CERTIFICATE ----- MIIFOTCCBCGgAwIBAgIQQ / u6wmYn4JceHBHgMMNrZjANBgkqhkiG9w0BAQUFADCB tTELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2Ug YXQgaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykxMDEvMC0GA1UEAxMm VmVyaVNpZ24gQ2xhc3MgMyBTZWN1cmUgU2VydmVyIENBIC0gRzMwHhcNMTQwNjI1 MDAwMDAwWhcNMTUwNjA1MjM1OTU5WjB6MQswCQYDVQQGEwJVUzETMBEGA1UECBMK V2FzaGluZ3RvbjEQMA4GA1UEBxQHU2VhdHRsZTEYMBYGA1UEChQPQW1hem9uLmNv bSBJbmMuMQ0wCwYDVQQLFARTMy1BMRswGQYDVQQDFBIqLnMzLmFtYXpvbmF3cy5j b20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC + mjGH4PrN0TDNmTsF my2fHpHpTuySRwBMe / nuGIkL3cvKIxFKLHvkK9kx1UpO0skdQTdCY55LywhubLNO fD19IzJdoRGdlqgkDAYC8vz3LRYj8WWsGnROfS / YFtgj25YaHPnsNp6lWrff4 / qi ctbojJpMxm + 9Q0A4nTzrZymHEUkRbx6AVVUBVKH3uZi / w0aV + i4cp2bs + CYIK3DL Qp584DJ9bOImgUhDfz19 + Wtv64zIezE0Uz9eOkqgQ1X // XumyZWyD6N6 + h / XqTnc YTvIer / s83T / IngGMbfPRqjpQCay6ySXCNbJ5izMgo + gwN84t7JhaI + EYcxf1dDN w0mXAgMBAAGjggF9MIIBeTAJBgNVHRMEAjAAMA4GA1UdDwEB / wQEAwIFoDAdBgNV HSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwZQYDVR0gBF4wXDBaBgpghkgBhvhF AQc2MEwwIwYIKwYBBQUHAgEWF2h0dHBzOi8vZC5zeW1jYi5jb20vY3BzMCUGCCsG AQUFBwICMBkaF2h0dHBzOi8vZC5zeW1jYi5jb20vcnBhMB8GA1UdIwQYMBaAFA1E XBZTRMGCfh0gqyX0AWPYvnmlMCsGA1UdHwQkMCIwIKAeoByGGmh0dHA6Ly9zZC5z eW1jYi5jb20vc2QuY3JsMFcGCCsGAQUFBwEBBEswSTAfBggrBgEFBQcwAYYTaHR0 cDovL3NkLnN5bWNkLmNvbTAmBggrBgEFBQcwAoYaaHR0cDovL3NkLnN5bWNiLmNv bS9zZC5jcnQwLwYDVR0RBCgwJoISKi5zMy5hbWF6b25hd3MuY29tghBzMy5hbWF6 b25hd3MuY29tMA0GCSqGSIb3DQEBBQUAA4IBAQBlXrn1FTPjVIFbcuQNbBesrAMI NV4L7jS1mobEwFrb7UrqZ7kHvuvoR / BpDygATyqLvPihs7nUc2TUHsw / 41EAHKoq QBVfRTOH0yWaTC6SYSx7fiElL + k55Pvrz4 + 7gLRy5zUlVX3iUw93zr95ka / LPuCL 7PQOFPeQDOgveDjcSNVtLcTfQfvog / rMSu / 4XPFHu7zaZwUEurt9CzLeVbdB6O25 bHuHTaZLP0wmjCIbwgXu8bFWqOTnAjG70EtYrbIiQhl / ISJU6HioFzLiy5Ibp07r RbV4ir5EI2EPKxIy30YDvpCQ0WQvYLWFV0qQOuOXkMC2M2IsBmVn2 / 9GQ7eP ----- END CERTIFICATE ----- 1 s: / C = US / O = VeriSign, Inc./OU=VeriSign Trust Network / OU = Terms of use at https://www.verisign.com/rpa (c) 10 / CN = VeriSign Class 3 Secure Server CA - G3 i: / C = US / O = VeriSign, Inc./OU=VeriSign Trust Network / OU = (c) 2006 VeriSign, Inc. - For authorized use only / CN = VeriSign Class 3 Public Primary Certification Authority - G5 ----- BEGIN CERTIFICATE ----- MIIF7DCCBNSgAwIBAgIQbsx6pacDIAm4zrz06VLUkTANBgkqhkiG9w0BAQUFADCB yjELMAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQL ExZWZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJp U2lnbiwgSW5jLiAtIEZvciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxW ZXJpU2lnbiBDbGFzcyAzIFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0 aG9yaXR5IC0gRzUwHhcNMTAwMjA4MDAwMDAwWhcNMjAwMjA3MjM1OTU5WjCBtTEL MAkGA1UEBhMCVVMxFzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZW ZXJpU2lnbiBUcnVzdCBOZXR3b3JrMTswOQYDVQQLEzJUZXJtcyBvZiB1c2UgYXQg aHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL3JwYSAoYykxMDEvMC0GA1UEAxMmVmVy aVNpZ24gQ2xhc3MgMyBTZWN1cmUgU2VydmVyIENBIC0gRzMwggEiMA0GCSqGSIb3 DQEBAQUAA4IBDwAwggEKAoIBAQCxh4QfwgxF9byrJZenraI + nLr2wTm4i8rCrFbG 5btljkRPTc5v7QlK1K9OEJxoiy6Ve4mbE8riNDTB81vzSXtig0iBdNGIeGwCU / m8 f0MmV1gzgzszChew0E6RJK2GfWQS3HRKNKEdCuqWHQsV / KNLO85jiND4LQyUhhDK tpo9yus3nABINYYpUHjoRWPNGUFP9ZXse5jUxHGzUL4os4 + guVOc9cosI6n9FAbo GLSa6Dxugf3kzTU2s1HTaewSulZub5tXxYsU5w7HnO1KVGrJTcW / EbGuHGeBy0RV M5l / JJs / U0V / hhrzPPptf4H1uErT9YU3HLWm0AnkGHs4TvoPAgMBAAGjggHfMIIB 2zA0BggrBgEFBQcBAQQoMCYwJAYIKwYBBQUHMAGGGGh0dHA6Ly9vY3NwLnZlcmlz aWduLmNvbTASBgNVHRMBAf8ECDAGAQH / AgEAMHAGA1UdIARpMGcwZQYLYIZIAYb4 RQEHFwMwVjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cudmVyaXNpZ24uY29tL2Nw czAqBggrBgEFBQcCAjAeGhxodHRwczovL3d3dy52ZXJpc2lnbi5jb20vcnBhMDQG A1UdHwQtMCswKaAnoCWGI2h0dHA6Ly9jcmwudmVyaXNpZ24uY29tL3BjYTMtZzUu Y3JsMA4GA1UdDwEB / wQEAwIBBjBtBggrBgEFBQcBDARhMF + hXaBbMFkwVzBVFglp bWFnZS9naWYwITAfMAcGBSsOAwIaBBSP5dMahqyNjmvDz4Bq1EgYLHsZLjAlFiNo dHRwOi8vbG9nby52ZXJpc2lnbi5jb20vdnNsb2dvLmdpZjAoBgNVHREEITAfpB0w GzEZMBcGA1UEAxMQVmVyaVNpZ25NUEtJLTItNjAdBgNVHQ4EFgQUDURcFlNEwYJ + HSCrJfQBY9i + eaUwHwYDVR0jBBgwFoAUf9Nlp8Ld7LvwMAnzQzn6Aq8zMTMwDQYJ KoZIhvcNAQEFBQADggEBAAyDJO / dwwzZWJz + NrbrioBL0aP3nfPMU ++ CnqOh5pfB WJ11bOAdG0z60cEtBcDqbrIicFXZIDNAMwfCZYP6j0M3m + oOmmxw7vacgDvZN / R6 bezQGH1JSsqZxxkoor7YdyT3hSaGbYcFQEFn0Sc67dxIHSLNCwuLvPSxe / 20majp dirhGi2HbnTTiN0eIsbfFrYrghQKlFzyUOyvzv9iNw2tZdMGQVPtAhTItVgooazg W + yzf5VK + wPIrSbb5mZ4EkrZn0L74ZjmQoObj49nJOhhGbXdzbULJgWOw27EyHW4 Rs / iGAZeqa6ogZpHFt4MKGwlJ7net4RYxh84HqTEy2Y = ----- END CERTIFICATE ----- 2 s: / C = US / O = VeriSign, Inc./OU=VeriSign Trust Network / OU = (c) 2006 VeriSign, Inc. - For authorized use only / CN = VeriSign Class 3 Public Primary Certification Authority - G5 i: / C = US / O = VeriSign, Inc./OU=Class 3 Public Primary Certification Authority ----- BEGIN CERTIFICATE ----- MIIExjCCBC + gAwIBAgIQNZcxh / OHOgcyfs5YDJt + 2jANBgkqhkiG9w0BAQUFADBf MQswCQYDVQQGEwJVUzEXMBUGA1UEChMOVmVyaVNpZ24sIEluYy4xNzA1BgNVBAsT LkNsYXNzIDMgUHVibGljIFByaW1hcnkgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkw HhcNMDYxMTA4MDAwMDAwWhcNMjExMTA3MjM1OTU5WjCByjELMAkGA1UEBhMCVVMx FzAVBgNVBAoTDlZlcmlTaWduLCBJbmMuMR8wHQYDVQQLExZWZXJpU2lnbiBUcnVz dCBOZXR3b3JrMTowOAYDVQQLEzEoYykgMjAwNiBWZXJpU2lnbiwgSW5jLiAtIEZv ciBhdXRob3JpemVkIHVzZSBvbmx5MUUwQwYDVQQDEzxWZXJpU2lnbiBDbGFzcyAz IFB1YmxpYyBQcmltYXJ5IENlcnRpZmljYXRpb24gQXV0aG9yaXR5IC0gRzUwggEi MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCvJAgIKXo1nmAMqudLO07cfLw8 RRy7K + D + KQL5VwijZIUVJ / XxrcgxiV0i6CqqpkKzj / i5Vbext0uz / o9 + B1fs70Pb ZmIVYc9gDaTY3vjgw2IIPVQT60nKWVSFJuUrjxuf6 / WhkcIzSdhDY2pSS9KP6HBR TdGJaXvHcPaz3BJ023tdS1bTlr8Vd6Gw9KIl8q8ckmcY5fQGBO + QueQA5N06tRn / Arr0PO7gi + s3i + z016zy9vA9r911kTMZHRxAy3QkGSGT2RT + rCpSx4 / VBEnkjWNH iDxpg8v + R70rfk / Fla4OndTRQ8Bnc + MUCH7lP59zuDMKz10 / NIeWiu5T6CUVAgMB AAGjggGRMIIBjTAPBgNVHRMBAf8EBTADAQH / MDEGA1UdHwQqMCgwJqAkoCKGIGh0 dHA6Ly9jcmwudmVyaXNpZ24uY29tL3BjYTMuY3JsMA4GA1UdDwEB / wQEAwIBBjA9 BgNVHSAENjA0MDIGBFUdIAAwKjAoBggrBgEFBQcCARYcaHR0cHM6Ly93d3cudmVy aXNpZ24uY29tL2NwczAdBgNVHQ4EFgQUf9Nlp8Ld7LvwMAnzQzn6Aq8zMTMwNAYD VR0lBC0wKwYJYIZIAYb4QgQBBgpghkgBhvhFAQgBBggrBgEFBQcDAQYIKwYBBQUH AwIwbQYIKwYBBQUHAQwEYTBfoV2gWzBZMFcwVRYJaW1hZ2UvZ2lmMCEwHzAHBgUr DgMCGgQUj + XTGoasjY5rw8 + AatRIGCx7GS4wJRYjaHR0cDovL2xvZ28udmVyaXNp Z24uY29tL3ZzbG9nby5naWYwNAYIKwYBBQUHAQEEKDAmMCQGCCsGAQUFBzABhhho dHRwOi8vb2NzcC52ZXJpc2lnbi5jb20wDQYJKoZIhvcNAQEFBQADgYEADyWuSO0b M4VMDLXC1 / 5N1oMoTEFlYAALd0hxgv5 / 21oOIMzS6ke8ZEJhRDR0MIGBJopK90Rd fjSAqLiD4gnXbSPdie0oCL1jWhFXCMSe2uJoKK / dUDzsgiHYAMJVRFBwQa2DF3m6 CPMr3u00HUSe0gST9MsFFy0JLS1j7 / YmC3s = ----- END CERTIFICATE ----- --- Server certificate subject = / C = US / ST = Washington / L = Seattle / O = Amazon.com Inc./OU=S3-A/CN=*.s3.amazonaws.com issuer = / C = US / O = VeriSign, Inc./OU=VeriSign Trust Network / OU = Terms of use at https://www.verisign.com/rpa (c) 10 / CN = VeriSign Class 3 Secure Server CA - G3 --- No client certificate CA names sent Server Temp Key: ECDH, prime256v1, 256 bits --- SSL handshake has read 4624 bytes and written 399 bytes --- New, TLSv1 / SSLv3, Cipher is ECDHE-RSA-AES128-SHA Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE SSL-Session: Protocol: TLSv1.2 Cipher: ECDHE-RSA-AES128-SHA Session-ID: B642ED1E6FE32F7A374B5A62847DEC63C2F37DCA7A18FD669B8F0FCDC98C49BF Session-ID-ctx: Master-Key: EE2D31F43D341A0895B36E0BCCE7557B221F1469AC1B7B0BA22D843C75F25F949822B0D0E22E967A1F373F034E9624E4 Key-Arg: None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1430871883 Timeout: 300 (sec) Verify return code: 0 (ok) --- closed
When I decode the certificates it lists, I get a different certificate chain than Firefox gave me.
- *. S3.amazonaws.com
Serial number: 43fbbac26627e0971e1c11e030c36b66. - VeriSign Secure Server Layer 3 CA-G3
Serial Number: 6ecc7aa5a7032009b8cebcf4e952d491 - VeriSign Class 3 Public Primary Certification Authority - G5
Serial Number: 35973187f3873a07327ece580c9b7eda.
The certificate used and its immediate parent certificate are the same, but the next one has the same name but a different serial number.
Here's my version info:
> curld --version curl 7.40.0 (i386-pc-win32ce) libcurl / 7.40.0 OpenSSL / 1.0.1e Protocols: dict ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps tftp Features: NTLM SSL
> curld --version curl 7.40.0 (i386-pc-win32) libcurl / 7.40.0 OpenSSL / 1.0.1c Protocols: dict ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps tftp Features: AsynchDNS Largefile NTLM SSL
$ cat / etc / redhat-release CentOS release 6.6 (Final) $ openssl version OpenSSL 1.0.1e-fips 11 Feb 2013 $ yum list openssl Installed Packages openssl.x86_64 1.0.1e-30.el6.8 @updates
Firefox: 37.0.2 running Windows 7 x64.
I looked at this question: SSL certificate - browser certification path different from the certificate chain file , but my problem looks different: in this case the certificate chain in the OpenSSL command line tool went 1 - 2 - 3 - 4, and in IE it is was 1 - 2 - 3. Since IE thought "3" was the root certificate, the chain stopped earlier. In my case Firefox reports 1 - 2 - 3 and OpenSSL reports 1 - 2 - 4; The chain is different.
source to share
The server sends the same chain certificates to firefox and s_client:
- CN = .s3.amazonaws.com SAN = DNS: .s3.amazonaws.com, DNS: s3.amazonaws.com
- CN = VeriSign Class 3 Secure CA Server - G3
- CN = VeriSign Class 3 Public Primary Certification Authority - G5
But the way certificates are verified differs depending on the SSL stack and the client's trusted root certificates. And in case of curl, you are faced with the problem of validating OpenSSL. Details:
- Firefox has a trusted root certificate, similar to certificate # 3 sent by the client. This means it is a different certificate, but it contains the same public key, so the signature for certificate # 2 remains valid. Since Firefox, the underlying TLS stack (NSS), has thus found a useful trust anchor, it will treat the chain as verified and ignore certificate # 3 sent by the server.
- But the version of curl you are using uses OpenSSL as the TLS library. OpenSSL tries to get the longest match so it tries to find the trusted root certificate that signed certificate # 3. If it fails, it won't try to use the shorter chain of trust, but it just fails. This is a long-standing bug that is responsible for many strange problems like this, and it looks like the problem has only been fixed in the latest development branch (not released) so far.
Perhaps the solution is to use a CA store for curl which still includes the old legacy 1024 bit CAs so that it contains the trust anchor "/ C = US / O = VeriSign, Inc./OU=Class 3 Public Primary Certification Authority"
For more information on this issue, see
- more detailed record
- OpenSSL bug # 2732 from 2012
- workarounds with the patch
- newer Bug reports that indicate a bug fix for the current development branch (after 1.0.2)
source to share