"invalid: subscription information too old" and "Connection error 596: Broken pipe"

[udotirol]

Hi,

one of our nodes feels unhappy about its subscription state and displays "invalid: subscription information too old" in the node's subscription page:



When I click "Check", a popup opens and displays "Connection error 596: Broken pipe".



Nothing conclusive in the logs, maybe anyone else has an idea what to do about it?

Thanks

Udo

 

[Stoiko Ivanov]

On a hunch - check if the system tries to connect to shop.proxmox.com via ipv6, but cannot (there was a change in the subscription check code quite a while ago, where broken dual-stack setups could not check their subscription anymore)

else to debug this:
* check on the commandline what `pvesubscription get` and `pvesubscription update` say
* see if you can reach shop.proxmox.com with curl (on port 443)

I hope this helps!

 

[udotirol]

thanks.

I can ping & connect to shop.proxmox.com on the node's commandline, but "pvesubscription update" throws an ugly exception:

root@sisyphos:~# RUST_BACKTRACE=full pvesubscription update
thread '<unnamed>' panicked at 'Failed to add native certificate too root store: UnsupportedCriticalExtension', /usr/share/cargo/registry/ureq-2.4.0/src/rtls.rs:73:14
stack backtrace:
   0:     0x7f1773785c3c - <unknown>
   1:     0x7f17737a161e - <unknown>
   2:     0x7f1773766191 - <unknown>
   3:     0x7f1773771fe5 - <unknown>
   4:     0x7f1773771c42 - <unknown>
   5:     0x7f1773772501 - <unknown>
   6:     0x7f1773785f77 - <unknown>
   7:     0x7f1773785d54 - <unknown>
   8:     0x7f17737721b2 - <unknown>
   9:     0x7f17733ded23 - <unknown>
  10:     0x7f17733dee63 - <unknown>
  11:     0x7f17735b4542 - <unknown>
  12:     0x7f177358e2cb - <unknown>
  13:     0x7f1773757d81 - <unknown>
  14:     0x7f17733d4d22 - <unknown>
  15:     0x7f177358d368 - <unknown>
  16:     0x7f1773580752 - <unknown>
  17:     0x7f177358149d - <unknown>
  18:     0x7f177346ce9f - <unknown>
  19:     0x7f1773443c42 - <unknown>
  20:     0x7f17734446de - <unknown>
  21:     0x7f1773443ca6 - <unknown>
  22:     0x55c0243e2157 - Perl_pp_entersub
  23:     0x55c0243d8846 - Perl_runops_standard
  24:     0x55c024346b1c - perl_run
  25:     0x55c024319482 - main
  26:     0x7f17746ded0a - __libc_start_main
  27:     0x55c0243194ca - _start
  28:                0x0 - <unknown>
fatal runtime error: failed to initiate panic, error 5
Aborted

Previously, the node was running PVE 7.3-4, but the error persists even after upgrading to the latest 7.4-3

 

[Stoiko Ivanov]

hmm - again more on a hunch - do you have any local CA-certificates in the system? (/usr/local/share/ca-certificates) - and if yes - could it be that they contain something which is not really well-formed for a certificate?

 

[udotirol]

yes, there are some local CA certificates in that directory. They have been put there in Nov 2019, so that's quite a while

I however noticed that one of the certificates didn't have .crt but .pem as a filename extension. After listing all known certificates from /etc/ssl/certs/ca-certificates.crt I noticed, that this one certificate was missing from the system.

After renaming the .pem to .crt and running update-ca-certificates, the certificate got successfully imported, but that didn't impress pvesubscription update:

$ update-ca-certificates -v --fresh
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
Doing .
link D-TRUST_Root_Class_3_CA_2_EV_2009.pem -> d4dae3dd.0
link Entrust_Root_Certification_Authority_-_G4.pem -> 5e98733a.0
link Amazon_Root_CA_1.pem -> ce5e74ef.0
link Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.pem -> 3bde41ac.0
link Certum_Trusted_Network_CA_2.pem -> 40193066.0
[...]
link ExampleCom_CA_HTTPS_2018.pem -> 7fdfe7ce.0
link ExampleCom_CA_2018.pem -> ffa5834b.0
[...]
link IdenTrust_Public_Sector_Root_CA_1.pem -> 1e08bfd1.0
link Trustwave_Global_Certification_Authority.pem -> f249de83.0
link certSIGN_ROOT_CA.pem -> 8d86cdd1.0
131 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
$
$ RUST_BACKTRACE=full pvesubscription update
thread '<unnamed>' panicked at 'Failed to add native certificate too root store: UnsupportedCriticalExtension', /usr/share/cargo/registry/ureq-2.4.0/src/rtls.rs:73:14
stack backtrace:
   0:     0x7f9c8bcb92e0 - <unknown>
   1:     0x7f9c8bcd671e - <unknown>
   2:     0x7f9c8bca46f5 - <unknown>
   3:     0x7f9c8bca6eb4 - <unknown>
   4:     0x7f9c8bca6b01 - <unknown>
   5:     0x7f9c8bca748a - <unknown>
   6:     0x7f9c8bcb9637 - <unknown>
   7:     0x7f9c8bcb942c - <unknown>
   8:     0x7f9c8bca7182 - <unknown>
   9:     0x7f9c8b900693 - <unknown>
  10:     0x7f9c8b9007c3 - <unknown>
  11:     0x7f9c8badda27 - <unknown>
  12:     0x7f9c8bac857b - <unknown>
  13:     0x7f9c8bc8b5bf - <unknown>
  14:     0x7f9c8b8f5ad2 - <unknown>
  15:     0x7f9c8bac8898 - <unknown>
  16:     0x7f9c8baaedd4 - <unknown>
  17:     0x7f9c8baaf9dd - <unknown>
  18:     0x7f9c8b96b6eb - <unknown>
  19:     0x7f9c8b9674de - <unknown>
  20:     0x7f9c8b967fb7 - <unknown>
  21:     0x7f9c8b967546 - <unknown>
  22:     0x558e06d62157 - Perl_pp_entersub
  23:     0x558e06d58846 - Perl_runops_standard
  24:     0x558e06cc6b1c - perl_run
  25:     0x558e06c99482 - main
  26:     0x7f9c8cc16d0a - __libc_start_main
  27:     0x558e06c994ca - _start
  28:                0x0 - <unknown>
fatal runtime error: failed to initiate panic, error 5
Aborted

 

[udotirol]

Allright, after temporarily removing the local CA certificates from /usr/local/share/ca-certificates and running "update-ca-certificates --fresh" again, pvesubscription update successfully completes.

So the issue must have something to do with those custom CA certificates, but as they have been in use since 2018 on a very huge number of devices and operating systems, I have my doubts that the problem is really the certificates themselves.

The exception reads "UnsupportedCriticalExtension". When investigating the certificates, they do have two defined critical extensions:

$ openssl x509 -noout -text < ExampleCom_CA_2018.crt
[...]
X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
[...]
            X509v3 CRL Distribution Points: critical

                Full Name:
                  URI:https://foo.example.com/CA/foo.CRL
[...]
$
$ openssl x509 -noout -text < ExampleCom_CA_HTTPS_2018.crt
[...]
X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
[...]
$

So my best guess is that some tool proxmox uses to interact with certificates (ureq?) doesn't like one of the "critical" X509 extensions, but I am a little bit lost here as how to proceed.

 

[Stoiko Ivanov]

maybe it's related to this issue: https://github.com/rustls/rustls-native-certs/issues/48

 

[udotirol]

well, I dug a bit deeper as well and it looks like as if the version of the ureq HTTPS client used (as indicated in the exception /usr/share/cargo/registry/ureq-2.4.0/) is outdated. There is a fix for dealing with those "UnsupportedCriticalExtension" since ureq 2.6.0:

https://github.com/algesten/ureq/pull/564