Containers do not start after upgrade{SOLVED}

[guletz]

Hi,

I have several nodes in a cluster with some CTs on them.
Before update(using 6.x), today all CTs(debian, centos) are able to run. When I make updates on 2 nodes, the containers do not start. But if I move any of my CTs, on any other PMX host without updates that you seen in the picture, then they start without any problem!

Thx. a lot!

Good luck/Bafta

 

[Moayad]

Hi,

Please post debugging for LXC container and output of pveversion -v as well

lxc-start -n ID -F -l DEBUG -o /tmp/lxc-ID.log

 

[guletz]

Thx. a lot for reply! Problem solved. Long story:

My cluster was running PMX 5.x. I make upgrade to ver. 6.x! On 2 nodes of 7 it was a problem with one network interface(the same model on both nodes) who was renamed by the upgrade installer. So one of my bridge was not working. The strange fact is this containers do not use this faulty bridge(they use a operational bridge, different from faulty bridge ), Today I see the problem, and after I add the new name of this NIC, and then all my CTs are up and running.


Thz. a lot for your time @Moayad, really apreciated!

Good luck / Bafta!

 

[guletz]

Please post debugging for LXC container and output of pveversion -v as well

But I was in mistake, even now my all bridges are Ok, the errror is happend again after the server reboot:

lxc-start -n 2020109 -F -l DEBUG -o /tmp/lxc-2020109.log lxc-start: 2020109: conf.c: mount_autodev: 1074 Permission denied - Failed to create "/dev" directory lxc-start: 2020109: conf.c: lxc_setup: 3311 Failed to mount "/dev" lxc-start: 2020109: start.c: do_start: 1231 Failed to setup container "2020109" lxc-start: 2020109: sync.c: __sync_wait: 41 An error occurred in another process (expected sequence number 5) lxc-start: 2020109: start.c: __lxc_start: 1957 Failed to spawn container "2020109" lxc-start: 2020109: tools/lxc_start.c: main: 308 The container failed to start lxc-start: 2020109: tools/lxc_start.c: main: 314 Additional information can be obtained by setting the --logfile and --logpriority options .......

pveversion -v
proxmox-ve: 6.2-1 (running kernel: 5.4.44-2-pve)
pve-manager: 6.2-10 (running version: 6.2-10/a20769ed)
pve-kernel-5.4: 6.2-4
pve-kernel-helper: 6.2-4
pve-kernel-5.4.44-2-pve: 5.4.44-2
pve-kernel-4.15: 5.4-19
pve-kernel-4.15.18-30-pve: 4.15.18-58
pve-kernel-4.15.18-12-pve: 4.15.18-36
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.4
libpve-access-control: 6.1-2
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.1-5
libpve-guest-common-perl: 3.1-1
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.2-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.2-9
pve-cluster: 6.1-8
pve-container: 3.1-12
pve-docs: 6.2-5
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-2
pve-firmware: 3.1-1
pve-ha-manager: 3.0-9
pve-i18n: 2.1-3
pve-qemu-kvm: 5.0.0-11
pve-xtermjs: 4.3.0-1
qemu-server: 6.2-11
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.4-pve1

 

[Moayad]

Hi,

Is the HA running?

 

[guletz]

Hi again,

After some time spent for the problem, I have discovery that the zfs pool mount point of this storage was not mounted after boot! Insted it was created only the normal folders(like /rpool/data/subvol-201812190-disk-1) inside my boot PMX drive(LVM) and not in the zfs pool space! And normally my PMX boot drive was almost full.
I destroy my zfs pool, remove all my data from the /rpool(-> LVM root), re-create zfs pool(from zfs history), and after several test(reboot node, move /migrate CTs, check usage space on LVM and rpool, stop/start CTs) I think the problem is solve now!

Thx. for your time @Moayad

Good luck/Bafta

 

[Moayad]

Hi,

Thank you so much for sharing your solution

 

[guletz]

Thank you so much for sharing your solution

With pleasure! I guess the systemd process have start to mount too late zfs at boot time and the PMX storage have start to populate this zfs mount points. I will watch this aspects for the next weeks.


Good luck /Bafta!