proxmox 6.3 breaks LXC & systemd-timesyncd loops on boot

bleomycin

Renowned Member
Mar 20, 2011
49
7
73
Recently upgraded from proxmox 6.2 to 6.3 and this is when the trouble started. I eventually gave up troubleshooting and just completed a fresh install from the 6.3 iso followed by a quick apt dist-upgrade, which is when the exact same issues have happened again.

When my 5-drive raidz disks are attached to the system on boot systemd-timesyncd.service fails to start and endlessly loops preventing the system from booting. If i remove the drives the system boots fine and I can plug them in and mount them afterwards, however now anything related to LXC is now broken. Unable to start or create CT's.

If i remove the 5-drive raidz from the system and reboot CT's inexplicably start working again and everything is fine. No idea how to even start debugging this. Thanks for any help!

Code:
proxmox-ve: 6.3-1 (running kernel: 5.11.7-1-pve)
pve-manager: 6.3-6 (running version: 6.3-6/2184247e)
pve-kernel-5.11: 7.0-0+3~bpo10
pve-kernel-5.4: 6.3-8
pve-kernel-helper: 6.3-8
pve-kernel-5.11.7-1-pve: 5.11.7-1~bpo10
pve-kernel-5.4.106-1-pve: 5.4.106-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.1.0-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: residual config
ifupdown2: 3.0.0-1+pve3
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.20-pve1
libproxmox-acme-perl: 1.0.8
libproxmox-backup-qemu0: 1.0.3-1
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.1-3
libpve-common-perl: 6.3-5
libpve-guest-common-perl: 3.1-5
libpve-http-server-perl: 3.1-1
libpve-storage-perl: 6.3-7
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.6-2
lxcfs: 4.0.6-pve1
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.12-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.4-9
pve-cluster: 6.2-1
pve-container: 3.3-4
pve-docs: 6.3-1
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.2-2
pve-ha-manager: 3.1-1
pve-i18n: 2.3-1
pve-qemu-kvm: 5.2.0-5
pve-xtermjs: 4.7.0-3
qemu-server: 6.3-10
smartmontools: 7.2-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 2.0.4-pve1
Code:
pct config 110
arch: amd64
cores: 1
hostname: hoobs
memory: 2500
net0: name=eth0,bridge=vmbr0,hwaddr=1A:06:11:A0:D7:6A,ip=dhcp,type=veth
onboot: 1
ostype: ubuntu
parent: backup1
rootfs: local-zfs:subvol-110-disk-0,size=10G
swap: 256
unprivileged: 1


Code:
Full output pastebin: https://pastebin.com/6k44ks0t
lxc-start -n 110 -F -l DEBUG -o /tmp/lxc-110.log
lxc-start: 110: start.c: lxc_spawn: 1673 Operation not permitted - Failed to clone a new set of namespaces
lxc-start: 110: start.c: __lxc_start: 1999 Failed to spawn container "110"
lxc-start: 110: conf.c: userns_exec_1: 4104 Failed to clone process in new user namespace
lxc-start: 110: tools/lxc_start.c: main: 308 The container failed to start
 
Last edited:
Could you try booting the latest 5.4 kernel? 5.11 has not been officially released yet (probably with 7.0).
 
  • Like
Reactions: bleomycin
Could you try booting the latest 5.4 kernel? 5.11 has not been officially released yet (probably with 7.0).
Yes sorry that is what I did first, I only installed 5.11 as a last ditch effort to see if any difference was made.
 
Could you try booting the latest 5.4 kernel? 5.11 has not been officially released yet (probably with 7.0).
Is there anything else you recommend I can try to troubleshoot? I can roll back to a 6.2 installation and never upgrade but knowing this is likely to re-occur the second I upgrade again has me worried.
 
* Please share the logs (`journalctl -b`output after booting) of the boot that is not working due to the timesyncd loop (haven't seen something like this yet)
* please share your storage.cfg and zpool status output
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!