[SOLVED] No valid subscription [ipv6 non-functioning]

[GarrettB]

Is this error expected for non-PMG users? I just got it for the first time, and I saw that there was a PMG related issue, but I am not a PMG user.

It appears there is a timeout occurring so the information is old. Can you explain what the check process is so I can check into further?
`Connection error 596: Connection timed out`

Spoiler: pveversion output

# pveversion --verbose
proxmox-ve: 7.2-1 (running kernel: 5.15.60-1-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-helper: 7.2-12
pve-kernel-5.15: 7.2-11
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.60-1-pve: 5.15.60-1
pve-kernel-5.15.53-1-pve: 5.15.53-1
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-2-pve: 5.13.19-4
pve-kernel-4.15: 5.4-6
pve-kernel-4.15.18-18-pve: 4.15.18-44
pve-kernel-4.15.17-1-pve: 4.15.17-9
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-10
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.6-1
proxmox-backup-file-restore: 2.2.6-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-2
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-3
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-3
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.5-pve1

 

[t.lamprecht]

Hi,

this is slightly confusing as you talk about PMG but then show the Proxmox VE output of pveversion. Or di you just mean you found an issue that talks about something like that in the context of the PMG and wonder if it affects PVE too?

It appears there is a timeout occurring so the information is old.

Where did you see that, got any more specific log?

What's the output of pvesubscription get | sed -E 's/^(key: p.{5}).*/\1xxxxxxx/'
(should censor the key already)

 

[GarrettB]

Sorry about the confusion. I had found this thread and poorly referenced it https://forum.proxmox.com/threads/no-valid-subscription.115738/

I see the error message right after I log in. Then, when I go to the Subscription page for the node I see the Status is showing as invalid: subscription information too old.

When I run the Check after maybe 20 seconds it shows Connection error - Timeout. Last night when I ran it there was an additional resonse code in the UI modal as noted in the previous post.

If I run System Report it immediately shows again the original No Valid Subscription error, and when I click OK it successfully runs the System Report.

root@pve:~# pvesubscription get | sed -E 's/^(key: p.{5}).*/\1xxxxxxx/'
checktime: 1663493198
key: pve1c-xxxxxxx
message: subscription information too old
nextduedate: 2023-06-02
productname: Proxmox VE Community Subscription 1 CPU/year
regdate: 2018-06-02 00:00:00
serverid: E10BB00A6EB2ADEFC722F65C0D536ACF
sockets: 1
status: invalid
url: https://www.proxmox.com/proxmox-ve/pricing

When I confirm the key and serverid with my Proxmox subscription account, they match.

Also, when I navigate to the Updates page in the UI. I get the error when clicking Refresh but after clicking OK it seems to run updates on the enterprise repo just fine.

starting apt-get update
Hit:1 http://ftp.debian.org/debian bullseye InRelease
Hit:2 http://security.debian.org/debian-security bullseye-security InRelease
Hit:3 http://ftp.debian.org/debian bullseye-updates InRelease
Hit:4 https://enterprise.proxmox.com/debian/pve bullseye InRelease
Reading package lists...
TASK OK

And in case it matters, the only hardware changes since this round of the subscription is a total replacement of RAM cards that were failing. Otherwise, no other hardware changes.

I found this in syslog:

Oct 12 05:26:50 pve pveupdate[1968126]: update subscription info failed: Error checking subscription: https://shop.proxmox.com/modules/servers/licensing/verify.php: Network Error: timed out reading response

I'm not sure what to make of it:

root@pve:~# nslookup shop.proxmox.com
Server:         1.1.1.1
Address:        1.1.1.1#53

Non-authoritative answer:
Name:   shop.proxmox.com
Address: 79.133.36.249
Name:   shop.proxmox.com
Address: 2a01:7e0:0:424::2

root@pve:~# ping 79.133.36.249
PING 79.133.36.249 (79.133.36.249) 56(84) bytes of data.
64 bytes from 79.133.36.249: icmp_seq=1 ttl=47 time=119 ms
64 bytes from 79.133.36.249: icmp_seq=2 ttl=47 time=118 ms
64 bytes from 79.133.36.249: icmp_seq=3 ttl=47 time=117 ms
64 bytes from 79.133.36.249: icmp_seq=4 ttl=47 time=118 ms
^C
--- 79.133.36.249 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 117.474/117.975/118.509/0.377 ms

 

[t.lamprecht]

Does retrying pvesubscription update --force manually keeps outputting that network connection error?

 

[GarrettB]

Yes, same result and there is a delay before it displays.

And if I curl it I get <status>Invalid</status> right away.

root@pve:~# curl https://shop.proxmox.com/modules/servers/licensing/verify.php
<status>Invalid</status>

 

[t.lamprecht]

Yes, same result and there is a delay before it displays.

So something in the network path between your host and the shop.proxmox.com host is either really slow, or adds delay through other reasons. Is there any firewall appliance or the like in between? Also, what changed on the host and network environment (config/hw/..) in the last two weeks, i.e., after the most recent time it worked successfully.

Other (network) details would be good to know too, like is an HTTP proxy required?

And if I curl it I get <status>Invalid</status> right away.

Well, that is an invalid request as the endpoint doesn't accept GET requests at all, but shows that at least curl can connect via HTTP successfully (as the response comes from the shop, not the http or network layer).

 

[GarrettB]

I'm sorry I am not aware of any other changes. There is a firewall in front of it, yes. No changes have been made. It is not showing any other troubles accessing the internet.

A curl POST call also immediately returns an invalid status response. Interestingly, I see firewall activity if I run the curl command, but with the pvesubscription commands I only see a DNS query to Cloudflare...no subsequent call to the shop.proxmox.com domain.

I'm not seeming to have any trouble with reaching the remote server, so it appears the issue is on the node itself, possibly with a failing process related to this particular script. But unfortunately I see no other log activity.

traceroute to 79.133.36.249 (79.133.36.249), 30 hops max, 60 byte packets
 1  192.168.1.1 (192.168.1.1)  0.315 ms  0.286 ms  0.332 ms
 2  rrcs-96-11-215-101.central.biz.rr.com (96.11.215.101)  1.670 ms  2.212 ms  3.029 ms
 3  142-254-151-137.inf.spectrum.com (142.254.151.137)  13.386 ms  14.099 ms  14.197 ms
 4  lag-63.akrpoh1202h.netops.charter.com (24.164.98.209)  36.739 ms  36.900 ms  37.456 ms
 5  lag-32.pltsohae01r.netops.charter.com (24.33.100.6)  20.895 ms  21.685 ms  21.817 ms
 6  lag-30.rcr01clmkohpe.netops.charter.com (65.29.1.40)  24.475 ms  23.391 ms lag-25.rcr01clmkohpe.netops.charter.com (65.29.1.28)  29.868 ms
 7  lag-25.chctilwc00w-bcr00.netops.charter.com (107.14.17.252)  25.319 ms lag-15.chctilwc00w-bcr00.netops.charter.com (66.109.6.68)  32.552 ms lag-515.chctilwc00w-bcr00.netops.charter.com (71.74.44.32)  32.286 ms
 8  lag-11.chcgildt87w-bcr00.netops.charter.com (66.109.6.20)  33.322 ms lag-31.chcgildt87w-bcr00.netops.charter.com (66.109.10.82)  33.581 ms lag-21.chcgildt87w-bcr00.netops.charter.com (66.109.5.136)  32.805 ms
 9  lag-401.pr2.chi10.netops.charter.com (66.109.0.109)  32.576 ms  31.934 ms  30.645 ms
10  chi-b23-link.ip.twelve99.net (62.115.156.204)  35.969 ms  36.769 ms  36.818 ms
11  * nyk-bb2-link.ip.twelve99.net (62.115.137.58)  46.363 ms  34.808 ms
12  ldn-bb4-link.ip.twelve99.net (62.115.112.245)  108.884 ms ldn-bb1-link.ip.twelve99.net (62.115.113.21)  109.009 ms *
13  prs-bb1-link.ip.twelve99.net (62.115.135.25)  111.717 ms  117.382 ms  116.473 ms
14  ffm-bb2-link.ip.twelve99.net (62.115.114.99)  128.307 ms ffm-bb1-link.ip.twelve99.net (62.115.123.12)  134.959 ms ffm-bb2-link.ip.twelve99.net (62.115.114.99)  127.604 ms
15  * ffm-b11-link.ip.twelve99.net (62.115.124.119)  151.892 ms *
16  telia.edge2.fra3.de.first-colo.net (62.115.161.107)  128.329 ms  121.890 ms  123.829 ms
17  ae1.3904.ce2.fra1.de.first-colo.net (212.224.104.9)  130.783 ms  129.941 ms  140.227 ms
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Also, to see if other command line scripts worked, I tried pveperf and it works fine.

Would this correspond with a particular package I could attempt to reinstall?

 

[Stoiko Ivanov]

Since I recently ran into such a case - you could check your routing table (in that case the issue was that there were non-functional ipv6 routes present, which prevented the successful connection, while ipv4 worked fine - it seems curl does a fallback in that situation, which pvesubscription does not)

* `ip route`
* `ip -6 route`
outputs would help to narrow this down.

If this does not resolve the issue - I would suggest running:
*`strace -f -yytt -s 512 -o /tmp/pvesubscription.trace pvesubscription update --force` - and see where things go wrong (in the tracefile)

I hope this helps!

 

[GarrettB]

That was it. There was a ipv6 address and gateway in the networks settings. The file date was from back in May - it's strange I haven't seen the error before now.

Very helpful hint, thank you! It's showing an active subscription now with prompt response.

I really appreciate the support. Thanks to both of you!

 

[Stoiko Ivanov]

That was it. There was a ipv6 address and gateway in the networks settings. The file date was from back in May - it's strange I haven't seen the error before now.

There was a change in the low-level implementation quite recently - so I assume that the old version worked around that issue.

Thanks for confirming my hunch!