Re: armhf: buster: TLS / HTTPS partly broken
Hi Tomas,
> > Yes, "curl -k https:/www.google.com" succeeds.
>
> Then it's quite probable that the problem lies with certificate
> resolution. Either it doesn't find a trusted root cert to validate
> the server against, or the validation fails.
>
> You might try curl's -v option (with and without -k) to see whether
> it sheds some light.
# curl -v https://www.google.com
* Expire in 0 ms for 6 (transfer 0x109d880)
* Expire in 1 ms for 1 (transfer 0x109d880)
* Expire in 0 ms for 1 (transfer 0x109d880)
* Expire in 2 ms for 1 (transfer 0x109d880)
* Expire in 0 ms for 1 (transfer 0x109d880)
* Expire in 0 ms for 1 (transfer 0x109d880)
* Expire in 2 ms for 1 (transfer 0x109d880)
* Expire in 1 ms for 1 (transfer 0x109d880)
* Expire in 1 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 2 ms for 1 (transfer 0x109d880)
* Expire in 2 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 3 ms for 1 (transfer 0x109d880)
* Expire in 3 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 3 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 4 ms for 1 (transfer 0x109d880)
* Expire in 5 ms for 1 (transfer 0x109d880)
* Trying 216.58.207.164...
* TCP_NODELAY set
* Expire in 149991 ms for 3 (transfer 0x109d880)
* Expire in 200 ms for 4 (transfer 0x109d880)
* Connected to www.google.com (216.58.207.164) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
# curl -vk https://www.google.com
* Expire in 0 ms for 6 (transfer 0x133a880)
* Expire in 1 ms for 1 (transfer 0x133a880)
[.. skipping 46 more or less identical lines ..]
* Expire in 4 ms for 1 (transfer 0x133a880)
* Trying 216.58.207.164...
* TCP_NODELAY set
* Expire in 149993 ms for 3 (transfer 0x133a880)
* Expire in 200 ms for 4 (transfer 0x133a880)
* Connected to www.google.com (216.58.207.164) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=Mountain View; O=Google LLC;
CN=www.google.com
* start date: Apr 7 09:49:21 2020 GMT
* expire date: Jun 30 09:49:21 2020 GMT
* issuer: C=US; O=Google Trust Services; CN=GTS CA 1O1
* SSL certificate verify result: unable to get local issuer
certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x133a880)
> GET / HTTP/2
> Host: www.google.com
> User-Agent: curl/7.64.0
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< date: Mon, 04 May 2020 17:57:40 GMT
< expires: -1
< cache-control: private, max-age=0
< content-type: text/html; charset=ISO-8859-1
< p3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< server: gws
< x-xss-protection: 0
< x-frame-options: SAMEORIGIN
< set-cookie: 1P_JAR=2020-05-04-17; expires=Wed, 03-Jun-2020 17:57:40
GMT; path=/; domain=.google.com; Secure
< set-cookie: NID=203=NJeeaDepuErdSOKYdHIR6vtnByU05gHO2DzxoRS3puHM4AsMlNZ5J2aksbNJrJQxfuGuBx_OaG3uyPuuF5tRqJEa4mGmreZ2F9ilyqksUryBh5z7N5y1_QDbDzCvkme1XonAIo_V7rw99ejIfqk8U1nL_tOw5OUSrBZffdLHchA;
expires=Tue, 03-Nov-2020 17:57:40 GMT; path=/; domain=.google.com;
HttpOnly
< alt-svc: h3-Q050=":443"; ma=2592000,h3-Q049=":443";
ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443";
ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000;
v="46,43"
< accept-ranges: none
< vary: Accept-Encoding
<
<!doctype html>
[.. rest of HTML document downloaded ..]
> Also the proposal brought on this list of looking at strace. Perhaps
> limit the trace to file operations, like so:
>
> strace -f -e trace=%file -o trace.out <your curl command here>
>
> This would let you see where curl is looking for certs files.
/etc/ssl/certs contains ~129 certificate files (links to the real
files) and matches the CApath from the curl -v output from above.
I tried the strace and below you can see the result.
# strace -f -e trace=%file -o trace.out curl https://www.google.com
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
# cat trace.out
283 execve("/usr/bin/curl", ["curl", "https://www.google.com";],
0x7ec29e48 /* 7 vars */) = 0
283 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
283 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libcurl.so.4",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libz.so.1",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libpthread.so.0",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libc.so.6",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD,
"/usr/lib/arm-linux-gnueabihf/libnghttp2.so.14",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libidn2.so.0",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/librtmp.so.1",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libssh2.so.1",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libpsl.so.5",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libssl.so.1.1",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD,
"/usr/lib/arm-linux-gnueabihf/libcrypto.so.1.1",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD,
"/usr/lib/arm-linux-gnueabihf/libgssapi_krb5.so.2",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libkrb5.so.3",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD,
"/usr/lib/arm-linux-gnueabihf/libk5crypto.so.3",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libcom_err.so.2",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD,
"/usr/lib/arm-linux-gnueabihf/libldap_r-2.4.so.2",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD,
"/usr/lib/arm-linux-gnueabihf/liblber-2.4.so.2",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD,
"/usr/lib/arm-linux-gnueabihf/libunistring.so.2",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libgnutls.so.30",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libhogweed.so.4",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libnettle.so.6",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libgmp.so.10",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libgcrypt.so.20",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libdl.so.2",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD,
"/usr/lib/arm-linux-gnueabihf/libkrb5support.so.0",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libkeyutils.so.1",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libresolv.so.2",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libsasl2.so.2",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libp11-kit.so.0",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libtasn1.so.6",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libgpg-error.so.0",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/usr/lib/arm-linux-gnueabihf/libffi.so.6",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libgcc_s.so.1",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
283 stat64("/etc/gnutls/default-priorities", 0x7ecb9d60) = -1 ENOENT
(No such file or directory)
283 openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY|O_LARGEFILE) = 3
283 access("/etc/gcrypt/fips_enabled", F_OK) = -1 ENOENT (No such
file or directory)
283 openat(AT_FDCWD, "/proc/sys/crypto/fips_enabled", O_RDONLY) = -1
ENOENT (No such file or directory)
283 openat(AT_FDCWD, "/etc/gcrypt/hwf.deny", O_RDONLY) = -1 ENOENT
(No such file or directory)
283 openat(AT_FDCWD, "/proc/self/auxv", O_RDONLY) = 3
283 openat(AT_FDCWD, "/proc/cpuinfo", O_RDONLY) = 3
283 openat(AT_FDCWD, "/root/.curlrc", O_RDONLY|O_LARGEFILE) = -1
ENOENT (No such file or directory)
284 openat(AT_FDCWD, "/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = 3
284 stat64("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=67, ...}) = 0
284 openat(AT_FDCWD, "/etc/host.conf", O_RDONLY|O_CLOEXEC) = 3
284 openat(AT_FDCWD, "/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 3
284 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
284 openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libnss_files.so.2",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
284 openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
284 openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
284 openat(AT_FDCWD, "/lib/arm-linux-gnueabihf/libnss_dns.so.2",
O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 3
284 openat(AT_FDCWD, "/etc/gai.conf", O_RDONLY|O_CLOEXEC) = 3
284 +++ exited with 0 +++
283 openat(AT_FDCWD, "/usr/lib/ssl/openssl.cnf", O_RDONLY|O_LARGEFILE) = 4
283 stat64("/etc/ssl/certs/99bdd351.0", 0x7ecb9180) = -1 ENOENT (No
such file or directory)
283 openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 4
283 stat64("/etc/ssl/certs/4a6481c9.0", 0x7ecb9180) = -1 ENOENT (No
such file or directory)
283 stat64("/etc/ssl/certs/4a6481c9.0", 0x7ecb9180) = -1 ENOENT (No
such file or directory)
283 +++ exited with 60 +++
On my PC, where curl works fine, I can also see that every access to
files like /etc/ssl/certs/4a6481c9.0 fails, too. So I guess that is
not the problem.
But on the PC I can see that curl reads
/etc/ssl/certs/ca-certificates.crt which is doesn't on armhf. But the
file exists on armhf, has a reasonable size of ~200 kB, and the
contents look unsuspicious. I also ran update-ca-certificates and the
file is identical afterwards.
Greetings,
Mark
Reply to: