Software: proxmox-ve 7.0-2 Fresh install
Storage: CEPH
Hardware: 6 Node cluster all running same version of proxmox-ve
Feature: Containers
Problem: Trying to migrate a CT from one host to another will fail IF the CT has never been run on the node before.
Error Example.
In my case the CT will migrate to the new host however it will not start. Any attempt to change or manage the CT will produce an error like the following.
Fix:
On the host on the host where the CT will not start check if a directory /var/lib/lxc/CT-ID is present. If not make the directory with the CT-ID then from the webGUI restart the CT.
For example CT-ID: 111 is gray and won't start on host pve2 after being migrated from host pve1.
root@pve2:~# mkdir /var/lib/lxc/111
In the WebGUI Right click on the CT and click Start, wait a few seconds and CT will online and can now be manage.
Can you reproduce the error?
Yes. Delete /var/lib/lxc/CT-ID on a host where the CT WAS running, then migrate back to that host.
Points of interest:
* This was a fresh install of Proxmox 7. No Upgrade was used.
* CTs and VMs were backups from Proxmox 6.4
* VMs do not have this problem
* I have enable KRDB on the CEPH Storage used by the CTs.
* If I make a new CT in Proxmox 7 using centos-8-default_20201210_amd64.tar.xz image, I still get the same error when I try to migrate, and the same fix works.
* If I make /var/lib/lxc/CT-ID before I migrate a CT, or if /var/lib/lxc/CT-ID is present on the host being migrated to, no problem.
Anyone seen this before?
root@hpe001:~# pveversion --verbose
proxmox-ve: 7.0-2 (running kernel: 5.11.22-1-pve)
pve-manager: 7.0-8 (running version: 7.0-8/b1dbf562)
pve-kernel-5.11: 7.0-3
pve-kernel-helper: 7.0-3
pve-kernel-5.11.22-1-pve: 5.11.22-2
ceph: 16.2.5-pve1
ceph-fuse: 16.2.5-pve1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.0.0-1+pve5
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.1.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-4
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-2
lxcfs: 4.0.8-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.1-1
proxmox-backup-file-restore: 2.0.1-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.2-4
pve-cluster: 7.0-3
pve-container: 4.0-5
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.2-4
pve-ha-manager: 3.3-1
pve-i18n: 2.4-1
pve-qemu-kvm: 6.0.0-2
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-7
smartmontools: 7.2-1
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.4-pve1
Storage: CEPH
Hardware: 6 Node cluster all running same version of proxmox-ve
Feature: Containers
Problem: Trying to migrate a CT from one host to another will fail IF the CT has never been run on the node before.
Error Example.
Code:
2021-07-20 14:08:46 shutdown CT 107
2021-07-20 14:08:51 starting migration of CT 107 to node 'hpe004' (10.4.4.4)
2021-07-20 14:08:51 volume 'ceph-lxc:vm-107-disk-1' is on shared storage 'ceph-lxc'
2021-07-20 14:08:51 start final cleanup
2021-07-20 14:08:52 start container on target node
2021-07-20 14:08:52 # /usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=hpe004' root@10.4.4.4 pct start 107
2021-07-20 14:08:53 unable to open file '/var/lib/lxc/107/rules.seccomp.tmp.388292' - No such file or directory
2021-07-20 14:08:53 ERROR: command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=hpe004' root@10.4.4.4 pct start 107' failed: exit code 255
2021-07-20 14:08:53 ERROR: migration finished with problems (duration 00:00:07)
TASK ERROR: migration problemsIn my case the CT will migrate to the new host however it will not start. Any attempt to change or manage the CT will produce an error like the following.
Code:
unable to open file '/var/lib/lxc/CT-ID/rules.seccomp.tmp.354433' - No such file or directory (500)Fix:
On the host on the host where the CT will not start check if a directory /var/lib/lxc/CT-ID is present. If not make the directory with the CT-ID then from the webGUI restart the CT.
For example CT-ID: 111 is gray and won't start on host pve2 after being migrated from host pve1.
root@pve2:~# mkdir /var/lib/lxc/111
In the WebGUI Right click on the CT and click Start, wait a few seconds and CT will online and can now be manage.
Can you reproduce the error?
Yes. Delete /var/lib/lxc/CT-ID on a host where the CT WAS running, then migrate back to that host.
Points of interest:
* This was a fresh install of Proxmox 7. No Upgrade was used.
* CTs and VMs were backups from Proxmox 6.4
* VMs do not have this problem
* I have enable KRDB on the CEPH Storage used by the CTs.
* If I make a new CT in Proxmox 7 using centos-8-default_20201210_amd64.tar.xz image, I still get the same error when I try to migrate, and the same fix works.
* If I make /var/lib/lxc/CT-ID before I migrate a CT, or if /var/lib/lxc/CT-ID is present on the host being migrated to, no problem.
Anyone seen this before?
root@hpe001:~# pveversion --verbose
proxmox-ve: 7.0-2 (running kernel: 5.11.22-1-pve)
pve-manager: 7.0-8 (running version: 7.0-8/b1dbf562)
pve-kernel-5.11: 7.0-3
pve-kernel-helper: 7.0-3
pve-kernel-5.11.22-1-pve: 5.11.22-2
ceph: 16.2.5-pve1
ceph-fuse: 16.2.5-pve1
corosync: 3.1.2-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.0.0-1+pve5
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.21-pve1
libproxmox-acme-perl: 1.1.1
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-4
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-2
libpve-storage-perl: 7.0-7
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-2
lxcfs: 4.0.8-pve1
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.1-1
proxmox-backup-file-restore: 2.0.1-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.2-4
pve-cluster: 7.0-3
pve-container: 4.0-5
pve-docs: 7.0-5
pve-edk2-firmware: 3.20200531-1
pve-firewall: 4.2-2
pve-firmware: 3.2-4
pve-ha-manager: 3.3-1
pve-i18n: 2.4-1
pve-qemu-kvm: 6.0.0-2
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-7
smartmontools: 7.2-1
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.4-pve1
Last edited:
