After Update to Kernel PVE 5.15.27-1 no login

S

sysadminfromhell

Member
Feb 3, 2022
65
6
13
28
Hi veryone,

i updated today from kernel 5.15.19-2 to kernel PVE 5.15.27-1, after the update i restarted as usual and now I can´t login anymore and the other node shows this node with "?" only.
The syslog says "proxy detected vanished client connection". Is there any fix for that?

Code: 
proxmox-ve: 7.1-1 (running kernel: 5.15.27-1-pve)
pve-manager: 7.1-10 (running version: 7.1-10/6ddebafe)
pve-kernel-5.15: 7.1-13
pve-kernel-helper: 7.1-13
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.27-1-pve: 5.15.27-1
pve-kernel-5.15.19-2-pve: 5.15.19-3
pve-kernel-5.13.19-6-pve: 5.13.19-14
pve-kernel-5.13.19-2-pve: 5.13.19-4
ceph-fuse: 15.2.15-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve2
libproxmox-acme-perl: 1.4.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.1-6
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.1-3
libpve-guest-common-perl: 4.1-1
libpve-http-server-perl: 4.1-1
libpve-storage-perl: 7.1-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.11-1
lxcfs: 4.0.11-pve1
novnc-pve: 1.3.0-2
proxmox-backup-client: 2.1.5-1
proxmox-backup-file-restore: 2.1.5-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.4-7
pve-cluster: 7.1-3
pve-container: 4.1-4
pve-docs: 7.1-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.3-6
pve-ha-manager: 3.3-3
pve-i18n: 2.6-2
pve-qemu-kvm: 6.1.1-2
pve-xtermjs: 4.16.0-1
qemu-server: 7.1-4
smartmontools: 7.2-1
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.2-pve1

Code: 
Mar 18 09:10:00 Nuc01 kernel: [  149.182996] FS-Cache: Loaded
Mar 18 09:10:00 Nuc01 kernel: [  149.197504] FS-Cache: Netfs 'nfs' registered for caching
Mar 18 09:10:00 Nuc01 kernel: [  149.304538] NFS: Registering the id_resolver key type
Mar 18 09:10:00 Nuc01 kernel: [  149.304548] Key type id_resolver registered
Mar 18 09:10:00 Nuc01 kernel: [  149.304549] Key type id_legacy registered
Mar 18 09:10:52 Nuc01 pmxcfs[4016]: [status] notice: received log
Mar 18 09:11:25 Nuc01 pveproxy[4097]: proxy detected vanished client connection
Mar 18 09:11:26 Nuc01 pveproxy[4097]: proxy detected vanished client connection
Mar 18 09:11:26 Nuc01 pveproxy[4096]: proxy detected vanished client connection
Mar 18 09:11:26 Nuc01 pveproxy[4098]: proxy detected vanished client connection
 
Last edited:
Hello,

May you try to restart the following services?:

Bash: 
systemctl restart pvedaemon
systemctl restart pvestatd
systemctl restart pve-manager
systemctl restart pve-cluster
 
I cant restart pve-manager, but the Node is up and running somehow the GUI now shows this error:
Code: 
Error Connection error 596: tls_process_server_certificate: certificate verify failed

systemctl restart pve-manager
Code: 
systemctl restart pve-manager
Failed to start pve-manager.service: Operation refused, unit pve-guests.service may be requested by dependency only (it is configured to refuse manual start/stop).
 
after a pvecm updatecerts the error disappeared but im back to the old error:

Node is up and green but I cant check the status of the VM/Storage:

Code: 
communication failure (0)

i also cannot access the Syslog Page/Update or any other page related to the host node. Its always Communcation failure.
 
Hi,

Did you update the certificates using pvecm updatecerts --force command?

What is the output of the following commands:

Bash: 
pvecm status
mount | grep /etc/pve
 
Did you update the certificates using pvecm updatecerts --force command?
Click to expand...
yes

Code: 
pvecm status
Cluster information
-------------------
Name:             HomeLab
Config Version:   2
Transport:        knet
Secure auth:      on

Quorum information
------------------
Date:             Fri Mar 18 12:50:14 2022
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          0x00000001
Ring ID:          1.10e
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2
Flags:            Quorate

Membership information
----------------------
    Nodeid      Votes Name
0x00000001          1 172.16.24.100 (local)
0x00000002          1 172.16.24.101

Code: 
mount | grep /etc/pve
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
 
Please try to restart the pveproxy service using: systemctl restart pveproxy if that not help, please provide us with Syslog for pveproxy and pve-cluster services
Bash: 
journalctl -u pveproxy -u pve-cluster  > /tmp/syslog.log
 
Code: 
-- Boot 0cf9554a369546f8a623067acb77f4ff --
Mar 18 12:15:01 Nuc01 systemd[1]: Starting The Proxmox VE cluster filesystem...
Mar 18 12:15:01 Nuc01 pmxcfs[1046]: [quorum] crit: quorum_initialize failed: 2
Mar 18 12:15:01 Nuc01 pmxcfs[1046]: [quorum] crit: can't initialize service
Mar 18 12:15:01 Nuc01 pmxcfs[1046]: [confdb] crit: cmap_initialize failed: 2
Mar 18 12:15:01 Nuc01 pmxcfs[1046]: [confdb] crit: can't initialize service
Mar 18 12:15:01 Nuc01 pmxcfs[1046]: [dcdb] crit: cpg_initialize failed: 2
Mar 18 12:15:01 Nuc01 pmxcfs[1046]: [dcdb] crit: can't initialize service
Mar 18 12:15:01 Nuc01 pmxcfs[1046]: [status] crit: cpg_initialize failed: 2
Mar 18 12:15:01 Nuc01 pmxcfs[1046]: [status] crit: can't initialize service
Mar 18 12:15:02 Nuc01 systemd[1]: Started The Proxmox VE cluster filesystem.
Mar 18 12:15:03 Nuc01 systemd[1]: Starting PVE API Proxy Server...
Mar 18 12:15:04 Nuc01 pveproxy[1110]: starting server
Mar 18 12:15:04 Nuc01 pveproxy[1110]: starting 3 worker(s)
Mar 18 12:15:04 Nuc01 pveproxy[1110]: worker 1111 started
Mar 18 12:15:04 Nuc01 pveproxy[1110]: worker 1112 started
Mar 18 12:15:04 Nuc01 pveproxy[1110]: worker 1113 started
Mar 18 12:15:04 Nuc01 systemd[1]: Started PVE API Proxy Server.
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [status] notice: update cluster info (cluster name  HomeLab, version = 2)
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [status] notice: node has quorum
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [dcdb] notice: members: 1/1046, 2/2008
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [dcdb] notice: starting data syncronisation
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [dcdb] notice: received sync request (epoch 1/1046/00000001)
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [status] notice: members: 1/1046, 2/2008
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [status] notice: starting data syncronisation
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [status] notice: received sync request (epoch 1/1046/00000001)
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [dcdb] notice: received all states
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [dcdb] notice: leader is 1/1046
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [dcdb] notice: synced members: 1/1046, 2/2008
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [dcdb] notice: start sending inode updates
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [dcdb] notice: sent all (0) updates
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [dcdb] notice: all data is up to date
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [status] notice: received all states
Mar 18 12:15:07 Nuc01 pmxcfs[1046]: [status] notice: all data is up to date
Mar 18 12:15:15 Nuc01 pmxcfs[1046]: [status] notice: received log
Mar 18 12:15:23 Nuc01 pmxcfs[1046]: [status] notice: received log
Mar 18 12:17:49 Nuc01 pveproxy[1113]: proxy detected vanished client connection
Mar 18 12:17:52 Nuc01 pveproxy[1113]: proxy detected vanished client connection
Mar 18 12:17:52 Nuc01 pveproxy[1111]: proxy detected vanished client connection
Mar 18 12:17:52 Nuc01 pveproxy[1112]: proxy detected vanished client connection
Mar 18 12:17:54 Nuc01 pveproxy[1111]: proxy detected vanished client connection
Mar 18 12:17:55 Nuc01 pveproxy[1111]: proxy detected vanished client connection
Mar 18 12:18:20 Nuc01 pveproxy[1111]: proxy detected vanished client connection
Mar 18 12:18:27 Nuc01 pmxcfs[1046]: [status] notice: received log
Mar 18 12:33:28 Nuc01 pmxcfs[1046]: [status] notice: received log
Mar 18 12:48:41 Nuc01 pmxcfs[1046]: [status] notice: received log
Mar 18 13:03:41 Nuc01 pmxcfs[1046]: [status] notice: received log
Mar 18 13:12:01 Nuc01 pveproxy[1111]: proxy detected vanished client connection
Mar 18 13:13:32 Nuc01 pveproxy[1112]: proxy detected vanished client connection
Mar 18 13:13:49 Nuc01 pveproxy[1113]: proxy detected vanished client connection
Mar 18 13:15:01 Nuc01 pmxcfs[1046]: [dcdb] notice: data verification successful
Mar 18 13:15:24 Nuc01 pveproxy[1112]: proxy detected vanished client connection
Mar 18 13:16:05 Nuc01 pveproxy[1113]: Clearing outdated entries from certificate cache
Mar 18 13:16:05 Nuc01 pveproxy[1111]: Clearing outdated entries from certificate cache
Mar 18 13:17:05 Nuc01 pveproxy[1113]: proxy detected vanished client connection
Mar 18 13:17:05 Nuc01 pveproxy[1113]: proxy detected vanished client connection
Mar 18 13:17:07 Nuc01 pveproxy[1112]: proxy detected vanished client connection
Mar 18 13:17:08 Nuc01 pveproxy[1113]: proxy detected vanished client connection
Mar 18 13:17:08 Nuc01 pveproxy[1111]: proxy detected vanished client connection
Mar 18 13:18:42 Nuc01 pmxcfs[1046]: [status] notice: received log
Mar 18 13:34:41 Nuc01 pmxcfs[1046]: [status] notice: received log
Mar 18 13:42:13 Nuc01 systemd[1]: Stopping PVE API Proxy Server...
Mar 18 13:42:14 Nuc01 pveproxy[1110]: received signal TERM
Mar 18 13:42:14 Nuc01 pveproxy[1110]: server closing
Mar 18 13:42:14 Nuc01 pveproxy[1113]: worker exit
Mar 18 13:42:14 Nuc01 pveproxy[1112]: worker exit
Mar 18 13:42:14 Nuc01 pveproxy[1110]: worker 1111 finished
Mar 18 13:42:14 Nuc01 pveproxy[1110]: worker 1113 finished
Mar 18 13:42:14 Nuc01 pveproxy[1110]: worker 1112 finished
Mar 18 13:42:14 Nuc01 pveproxy[1110]: server stopped
Mar 18 13:42:14 Nuc01 pveproxy[11385]: got inotify poll request in wrong process - disabling inotify
Mar 18 13:42:15 Nuc01 pveproxy[11385]: worker exit
Mar 18 13:42:15 Nuc01 systemd[1]: pveproxy.service: Succeeded.
Mar 18 13:42:15 Nuc01 systemd[1]: Stopped PVE API Proxy Server.
Mar 18 13:42:15 Nuc01 systemd[1]: pveproxy.service: Consumed 10.863s CPU time.
Mar 18 13:42:15 Nuc01 systemd[1]: Starting PVE API Proxy Server...
Mar 18 13:42:16 Nuc01 pveproxy[11390]: starting server
Mar 18 13:42:16 Nuc01 pveproxy[11390]: starting 3 worker(s)
Mar 18 13:42:16 Nuc01 pveproxy[11390]: worker 11391 started
Mar 18 13:42:16 Nuc01 pveproxy[11390]: worker 11392 started
Mar 18 13:42:16 Nuc01 pveproxy[11390]: worker 11393 started
Mar 18 13:42:16 Nuc01 systemd[1]: Started PVE API Proxy Server.
Mar 18 13:42:57 Nuc01 pveproxy[11391]: proxy detected vanished client connection
Mar 18 13:43:23 Nuc01 pveproxy[11391]: proxy detected vanished client connection
 
I see now when the problem appears:
If I reboot the Node everything is fine UNTIL i Click on on of the storages markes with a ?, then everythings starts loading and i get a communcation error.
Where I can see more information about it ?
Its a NAS mounted via iSCSI, no problems so far with it on the other node.
 
sysadminfromhell said:
If I reboot the Node everything is fine UNTIL i Click on on of the storages markes with a ?, then everythings starts loading and i get a communcation error.
Click to expand...
Did you see any other error message in the Syslog when you click on storage?

Have you checked from the hosts file on both nodes /etc/hosts?

can you update the cert again on one node let's say Nuc01 and restart the pveproxy service?
 
Moayad said:
Did you see any other error message in the Syslog when you click on storage?

Have you checked from the hosts file on both nodes /etc/hosts?

can you update the cert again on one node let's say Nuc01 and restart the pveproxy service?
Click to expand...
1. No, its just loading until I get communcation failure

2. Yes, it looks totally normal and fine for me:

Code: 
cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
172.16.24.100 Nuc01.fritz.box Nuc01

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

cat /etc/hosts
127.0.0.1       localhost.localdomain localhost
172.16.24.101   PiMox01.fritz.box PiMox01

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Code: 
pvecm updatecerts --force
(re)generate node files
generate new node certificate
merge authorized SSH keys and known hosts
root@Nuc01:/etc/pve# systemctl restart pveproxy.service

==> Same Issue
 
i restarted the NAS to ensure everything is OK on this site.
Syslog just shows:
Code: 
Mar 18 15:16:51 Nuc01 pmxcfs[1819]: [status] notice: received log
Mar 18 15:17:01 Nuc01 CRON[7027]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Mar 18 15:17:39 Nuc01 pveproxy[2793]: proxy detected vanished client connection
Mar 18 15:18:26 Nuc01 pveproxy[2793]: proxy detected vanished client connection
Mar 18 15:20:26 Nuc01 pveproxy[2794]: proxy detected vanished client connection

The GUI stays on:
Connection failure (0)
later then:
connection timeout (595)
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom