KVM and container wont start after upgrade and rebooting

[nttec]

I'm running

Kernel Version Linux 5.4.114-1-pve #1 SMP PVE 5.4.114-1 (Sun, 09 May 2021 17:13:05 +0200)

Kernel Version Linux 5.4.114-1-pve #1 SMP PVE 5.4.114-1 (Sun, 09 May 2021 17:13:05 +0200)

all went well after the upgrade yesterday, but run some issue that we needed to run this code

ceph config set mon auth_allow_insecure_global_id_reclaim false

and now after 24hrs of doing that, we cant start a kvm and we can't login to the gui.

We did manage to login again to the gui after we reboot the machines.

but our issue now is that all KVM and container is not running and wont event start.

Any idea where should we look at?

ohour pool has a "?" on it andwe are not sure what that is.



EDIT:


we turn this to true

ceph config set mon auth_allow_insecure_global_id_reclaim true

and everything start again. But we are back with the same warning before

clients are using insecure global_id reclaim

mons are allowing insecure global_id reclaim

 

[fiona]

Hi,
sounds like the clients are still using the insecure global_id reclaim for some reason. Rebooting the nodes should be more than enough for the KVM/container/PVE processes to pick up the new client library version, not sure why it didn't seem to work for you. See also this thread for more information.

What is the output of pveversion -v? Are the PVE Ceph repositories configured correctly? Is the cluster external?

You should use

ceph config set mon auth_allow_insecure_global_id_reclaim false

only once clients are using the upgraded version, i.e. after the warning for the clients disappears.

 

[nttec]

here's the result for pverversion -v

proxmox-ve: 6.4-1 (running kernel: 5.4.114-1-pve)
pve-manager: 6.4-6 (running version: 6.4-6/be2fa32c)
pve-kernel-5.4: 6.4-2
pve-kernel-helper: 6.4-2
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.114-1-pve: 5.4.114-1
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.78-2-pve: 5.4.78-2
pve-kernel-5.4.65-1-pve: 5.4.65-1
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-5.4.41-1-pve: 5.4.41-1
pve-kernel-4.15: 5.4-12
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-4.15.18-24-pve: 4.15.18-52
pve-kernel-4.15.18-21-pve: 4.15.18-48
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.2-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.1.0
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.4-1
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.4-3
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.2-2
libpve-storage-perl: 6.4-1
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.1.6-2
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.5-5
pve-cluster: 6.4-1
pve-container: 3.3-5
pve-docs: 6.4-2
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-3
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-6
pve-xtermjs: 4.7.0-3
qemu-server: 6.4-2
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1

Are the PVE Ceph repositories configured correctly?

-this is what we have on our pve repository.

deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise

Is the cluster external?

- yes cluster are external

 

[fiona]

ceph-fuse: 12.2.11+dfsg1-2.1+b1

Are the PVE Ceph repositories configured correctly?
-this is what we have on our pve repository.
deb https://enterprise.proxmox.com/debian/pve buster pve-enterprise

While this is the enterprise repository, our Ceph repositories are separate. The client version from Debian buster most likely don't include the fixes (they are still from Ceph 12 aka Luminous).

Is the cluster external?

- yes cluster are external

Please see the post I already linked to. It mentions what to do in that case.