Proxmox Virtual Environment 8.2.2 - LXC High Availibilty after upgrade PVE v7 to v8

Chris · Jul 16, 2024

I did a test on one production server with the correct setup for CEPH (no hardware RAID).
Upgraded from PVE 7.4-18 to PVE 8.2.4.

We also have Debian 12 LXC's and with HA migration to the node with the latest version of PVE, it doesn't want to start.
I tested with a Debian LXC with id 102:

Bash:

task started by HA resource agent
run_buffer: 571 Script exited with status 255
lxc_init: 845 Failed to run lxc.hook.pre-start for container "101"
__lxc_start: 2034 Failed to initialize container "101"
TASK ERROR: startup for container '101' failed

Journalctl -xe mentions a ext4 error:

Bash:

pve1# journalctl -xe
Jul 16 10:42:11 pve1 kernel: loop0: detected capacity change from 0 to 20971520
Jul 16 10:42:11 pve1 kernel: EXT4-fs error (device loop0): ext4_get_journal_inode:5779: inode #8: comm mount: iget: checksum invalid
Jul 16 10:42:11 pve1 kernel: EXT4-fs (loop0): no journal found
Jul 16 10:42:11 pve1 kernel: I/O error, dev loop0, sector 20971392 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Jul 16 10:42:11 pve1 pvedaemon[2135]: unable to get PID for CT 101 (not running?)
Jul 16 10:42:11 pve1 pve-ha-lrm[32154]: startup for container '101' failed
Jul 16 10:42:11 pve1 pve-ha-lrm[32152]: <root@pam> end task UPID:pve1:00007D9A:00015395:66963261:vzstart:101:root@pam: startup for container '101' failed
Jul 16 10:42:11 pve1 pve-ha-lrm[32152]: unable to start service ct:101
Jul 16 10:42:11 pve1 pve-ha-lrm[4461]: restart policy: retry number 2 for service 'ct:101'
Jul 16 10:42:13 pve1 systemd[1]: pve-container@101.service: Main process exited, code=exited, status=1/FAILURE

The configuration of the LCX:

Bash:

# pct config 101 --current
arch: amd64
cores: 1
description: # Vaultwarden LXC general%0A## Network config%0A| NIC | IPv4 | MAC | %0A| ---%3A--- | ---%3A--- | ---%3A--- |%0A| net0 | 192.168.16.101 | E2%3A61%3ADC%3A07%3A1F%3A8F |%0A
features: keyctl=1,nesting=1
hostname: vaultwarden.internal.robotronic.be
memory: 512
nameserver: 192.168.16.238
net0: name=eth0,bridge=vmbr0,gw=192.168.23.254,hwaddr=E2:61:DC:07:1F:8F,ip=192.168.16.101/21,type=veth
onboot: 1
ostype: debian
rootfs: vm-lxc-storage:101/vm-101-disk-0.raw,size=10G
searchdomain: internal.robotronic.be
swap: 512
tags: debian12;webserver
unprivileged: 1

I ran a file system check on the LXC disk:

Bash:

pve1# pct fsck 101


fsck from util-linux 2.38.1
/mnt/pve/cephfs_cluster/vm-lxc-storage/images/101/vm-101-disk-0.raw: Superblock has an invalid journal (inode 8).
CLEARED.
*** journal has been deleted ***

/mnt/pve/cephfs_cluster/vm-lxc-storage/images/101/vm-101-disk-0.raw: Resize inode not valid.

/mnt/pve/cephfs_cluster/vm-lxc-storage/images/101/vm-101-disk-0.raw: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
command 'fsck -a -l /mnt/pve/cephfs_cluster/vm-lxc-storage/images/101/vm-101-disk-0.raw' failed: exit code 4

Then I ran it manually on the lxc disk (a lot of inode issues):

Bash:

pve1# fsck -l /mnt/pve/cephfs_cluster/vm-lxc-storage/images/101/vm-101-disk-0.raw
Inode 146317 ref count is 2, should be 1.  Fix? yes

Unattached inode 146319
Connect to /lost+found? yes


/mnt/pve/cephfs_cluster/vm-lxc-storage/images/101/vm-101-disk-0.raw: ***** FILE SYSTEM WAS MODIFIED *****
/mnt/pve/cephfs_cluster/vm-lxc-storage/images/101/vm-101-disk-0.raw: 31726/655360 files (0.4% non-contiguous), 608611/2621440 blocks

I tried to startup the LXC in debug mode:

Bash:

#  pct start 101 --debug
mount_autodev: 1028 Permission denied - Failed to create "/dev" directory
lxc_setup: 3898 Failed to mount "/dev"
do_start: 1273 Failed to setup container "101"
sync_wait: 34 An error occurred in another process (expected sequence number 3)
__lxc_start: 2114 Failed to spawn container "101"
INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "101", config section "lxc"
DEBUG    utils - ../src/lxc/utils.c:run_buffer:560 - Script exec /usr/share/lxc/hooks/lxc-pve-prestart-hook 101 lxc pre-start produced output: /etc/os-release file not found and autodetection failed, falling back to 'unmanaged'
WARNING: /etc not present in CT, is the rootfs mounted?
got unexpected ostype (unmanaged != debian)

INFO     cgfsng - ../src/lxc/cgroups/cgfsng.c:unpriv_systemd_create_scope:1498 - Running privileged, not using a systemd unit
DEBUG    seccomp - ../src/lxc/seccomp.c:parse_config_v2:664 - Host native arch is [3221225534]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "[all]"
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "kexec_load errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[246:kexec_load] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "open_by_handle_at errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[304:open_by_handle_at] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "init_module errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[175:init_module] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "finit_module errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[313:finit_module] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "delete_module errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[176:delete_module] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "ioctl errno 1 [1,0x9400,SCMP_CMP_MASKED_EQ,0xff00]"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:555 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[16:ioctl] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:555 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:555 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:1036 - Merging compat seccomp contexts into main context
INFO     start - ../src/lxc/start.c:lxc_init:882 - Container "101" is initialized
INFO     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_create:1669 - The monitor process uses "lxc.monitor/101" as cgroup
DEBUG    storage - ../src/lxc/storage/storage.c:storage_query:231 - Detected rootfs type "dir"
DEBUG    storage - ../src/lxc/storage/storage.c:storage_query:231 - Detected rootfs type "dir"
INFO     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_payload_create:1777 - The container process uses "lxc/101/ns" as inner and "lxc/101" as limit cgroup
INFO     start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWUSER
INFO     start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWNS
INFO     start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWPID
INFO     start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWUTS
INFO     start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWIPC
INFO     start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWCGROUP
DEBUG    start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved user namespace via fd 17 and stashed path as user:/proc/37044/fd/17
DEBUG    start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved mnt namespace via fd 18 and stashed path as mnt:/proc/37044/fd/18
DEBUG    start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved pid namespace via fd 19 and stashed path as pid:/proc/37044/fd/19
DEBUG    start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved uts namespace via fd 20 and stashed path as uts:/proc/37044/fd/20
DEBUG    start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved ipc namespace via fd 21 and stashed path as ipc:/proc/37044/fd/21
DEBUG    start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved cgroup namespace via fd 22 and stashed path as cgroup:/proc/37044/fd/22
DEBUG    idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newuidmap" does have the setuid bit set
DEBUG    idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newgidmap" does have the setuid bit set
DEBUG    idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:178 - Functional newuidmap and newgidmap binary found
INFO     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits:3528 - Limits for the unified cgroup hierarchy have been setup
DEBUG    idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newuidmap" does have the setuid bit set
DEBUG    idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newgidmap" does have the setuid bit set
INFO     idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:176 - Caller maps host root. Writing mapping directly
NOTICE   utils - ../src/lxc/utils.c:lxc_drop_groups:1572 - Dropped supplimentary groups
INFO     start - ../src/lxc/start.c:do_start:1105 - Unshared CLONE_NEWNET
NOTICE   utils - ../src/lxc/utils.c:lxc_drop_groups:1572 - Dropped supplimentary groups
NOTICE   utils - ../src/lxc/utils.c:lxc_switch_uid_gid:1548 - Switched to gid 0
NOTICE   utils - ../src/lxc/utils.c:lxc_switch_uid_gid:1557 - Switched to uid 0
DEBUG    start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved net namespace via fd 5 and stashed path as net:/proc/37044/fd/5
INFO     utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "101", config section "net"
DEBUG    network - ../src/lxc/network.c:netdev_configure_server_veth:876 - Instantiated veth tunnel "veth101i0 <--> vethgd6BVY"
DEBUG    conf - ../src/lxc/conf.c:lxc_mount_rootfs:1240 - Mounted rootfs "/var/lib/lxc/101/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)"
INFO     conf - ../src/lxc/conf.c:setup_utsname:679 - Set hostname to "vaultwarden.internal.robotronic.be"
DEBUG    network - ../src/lxc/network.c:setup_hw_addr:3863 - Mac address "E2:61:DC:07:1F:8F" on "eth0" has been setup
DEBUG    network - ../src/lxc/network.c:lxc_network_setup_in_child_namespaces_common:4004 - Network device "eth0" has been setup
INFO     network - ../src/lxc/network.c:lxc_setup_network_in_child_namespaces:4061 - Finished setting up network devices with caller assigned names
INFO     conf - ../src/lxc/conf.c:mount_autodev:1023 - Preparing "/dev"
ERROR    conf - ../src/lxc/conf.c:mount_autodev:1028 - Permission denied - Failed to create "/dev" directory
INFO     conf - ../src/lxc/conf.c:mount_autodev:1084 - Prepared "/dev"
ERROR    conf - ../src/lxc/conf.c:lxc_setup:3898 - Failed to mount "/dev"
ERROR    start - ../src/lxc/start.c:do_start:1273 - Failed to setup container "101"
ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
DEBUG    network - ../src/lxc/network.c:lxc_delete_network:4217 - Deleted network devices
ERROR    start - ../src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "101"
WARN     start - ../src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 16 for process 37080
startup for container '101' failed

When migrating the container back to another PVE node with version of 7.4-18 I now have the following error:

Bash:

task started by HA resource agent
mount_autodev: 1225 Permission denied - Failed to create "/dev" directory
lxc_setup: 4395 Failed to mount "/dev"
do_start: 1272 Failed to setup container "101"
sync_wait: 34 An error occurred in another process (expected sequence number 3)
__lxc_start: 2107 Failed to spawn container "101"
TASK ERROR: startup for container '101' failed

Any idea's?
I want to update the whole production cluster only if I'm 100% sure that the containers can run on the latest PVE version...

Thank you for further investigating this!

Please post the full output of pveversion -v for the node upgraded to PVE8 in code tags.

Also, try to restore the container from backup to a different node (running PVE7) and only migrate it to the PVE8 node when that is booted into an older kernel version, does the issue persist in that case? Further, try to use the latest 6.8.8 kernel on the PVE8 host and also check if the container filesystem seems corrupted.

Also, if possible migrate the container from the CephFS backed storage to a different shared storage and see if the issue persists (again, with a container restored from backup onto a working PVE7 node).

Jera92 · Jul 18, 2024

Chris said:
Please post the full output of pveversion -v for the node upgraded to PVE8 in code tags.

Bash:

pve1# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.8-2-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-14
proxmox-kernel-6.8: 6.8.8-2
proxmox-kernel-6.8.8-2-pve-signed: 6.8.8-2
pve-kernel-5.15.158-1-pve: 5.15.158-1
pve-kernel-5.15.152-1-pve: 5.15.152-1
pve-kernel-5.15.131-1-pve: 5.15.131-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph: 17.2.7-pve3
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
openvswitch-switch: 3.1.0-2+deb12u1
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.12-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.0-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

Chris said:
Also, try to restore the container from backup to a different node (running PVE7) and only migrate it to the PVE8 node when that is booted into an older kernel version, does the issue persist in that case? Further, try to use the latest 6.8.8 kernel on the PVE8 host and also check if the container filesystem seems corrupted.

Also, if possible migrate the container from the CephFS backed storage to a different shared storage and see if the issue persists (again, with a container restored from backup onto a working PVE7 node).

I migrated LXC101 to PVE1 when booted into kernel mode 5.15.158-1-pve (as shown on the attached screenshot).
Then the container worked as it should, no ext4 errors.
I also tried with LXC106 also a Debian 12 container, and worked perfectly when booted into kernel mode 5.15.158-1-pve.

Is it kernel related? What can I do right now?..
Thanks for your help, I really appreciate it!

Chris · Jul 18, 2024

Jera92 said:

Bash:

pve1# pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.8-2-pve)
pve-manager: 8.2.4 (running version: 8.2.4/faa83925c9641325)
proxmox-kernel-helper: 8.1.0
pve-kernel-5.15: 7.4-14
proxmox-kernel-6.8: 6.8.8-2
proxmox-kernel-6.8.8-2-pve-signed: 6.8.8-2
pve-kernel-5.15.158-1-pve: 5.15.158-1
pve-kernel-5.15.152-1-pve: 5.15.152-1
pve-kernel-5.15.131-1-pve: 5.15.131-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph: 17.2.7-pve3
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.1
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.7
libpve-cluster-perl: 8.0.7
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.3
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.9
libpve-storage-perl: 8.2.3
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
openvswitch-switch: 3.1.0-2+deb12u1
proxmox-backup-client: 3.2.7-1
proxmox-backup-file-restore: 3.2.7-1
proxmox-firewall: 0.4.2
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.2.3
pve-cluster: 8.0.7
pve-container: 5.1.12
pve-docs: 8.2.2
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.1
pve-firewall: 5.0.7
pve-firmware: 3.12-1
pve-ha-manager: 4.0.5
pve-i18n: 3.2.2
pve-qemu-kvm: 9.0.0-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.4-pve1

I migrated LXC101 to PVE1 when booted into kernel mode 5.15.158-1-pve (as shown on the attached screenshot).
Then the container worked as it should, no ext4 errors.
I also tried with LXC106 also a Debian 12 container, and worked perfectly when booted into kernel mode 5.15.158-1-pve.

Is it kernel related? What can I do right now?..
Thanks for your help, I really appreciate it!

I would suggest to move the container from the CephFS backed storage to a RBD backed storage [0] and check if you can use the latest kernel. I would suggest to use this over CephFS for containers.
Also, you could check if the issue persists when using the Fuse client instead of the kernel cephfs client to see if the issue persists [1], if that is not the case it would indeed point to the in-kernel ceph client.

Please, also share you current storage configuration cat /etc/pve/storage.cfg

[0] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#ceph_rados_block_devices
[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#storage_cephfs

Edit: Do I understand correctly that you setup a dedicated directory based storage on top of the CephFS backed storage? That as well I would recommend against.

Jera92 · Jul 18, 2024

Chris said:
I would suggest to move the container from the CephFS backed storage to a RBD backed storage [0] and check if you can use the latest kernel. I would suggest to use this over CephFS for containers.
Also, you could check if the issue persists when using the Fuse client instead of the kernel cephfs client to see if the issue persists [1], if that is not the case it would indeed point to the in-kernel ceph client.

Owkay, I can try this in our sandbox environment.
Can I create RBD storage on top of an existing ceph cluster on the PVE nodes itself? I'm unexpierenced with RBD...
I found this command to create it on a pve node:
pvesm add rbd <storage-name> –pool <replicated-pool> –data-pool <ec-pool>
And via this documentation, I guess I can add the storage to a node?
https://knowledgebase.45drives.com/kb/kb450244-configuring-external-ceph-rbd-storage-with-proxmox/

Chris said:
Please, also share you current storage configuration cat /etc/pve/storage.cfg

The storage configuration:

Bash:

pve1# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,vztmpl,backup

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

cephfs: cephfs_cluster
        path /mnt/pve/cephfs_cluster
        content vztmpl,backup,iso
        fs-name cephfs_cluster

dir: vm-lxc-storage
        path /mnt/pve/cephfs_cluster/vm-lxc-storage
        content rootdir,images
        prune-backups keep-all=1
        shared 1

dir: ISO
        path /mnt/pve/cephfs_cluster/iso
        content iso
        prune-backups keep-all=1
        shared 1

cifs: vm-backup-qnap
        path /mnt/pve/vm-backup-qnap
        server 192.168.16.229
        share PVE-backup
        content backup
        prune-backups keep-all=1
        username Sandbox

cifs: synology-backup-daily
        path /mnt/pve/synology-backup-daily
        server synology.internal.robotronic.be
        share Backup
        content backup
        prune-backups keep-daily=7,keep-monthly=12,keep-weekly=4,keep-yearly=5
        subdir /10_PVE_Backup/DailyBackup
        username PVEBackup

cifs: synology-backup-weekly
        path /mnt/pve/synology-backup-weekly
        server synology.internal.robotronic.be
        share Backup
        content backup
        prune-backups keep-daily=7,keep-monthly=12,keep-weekly=4,keep-yearly=5
        subdir /10_PVE_Backup/WeeklyBackup
        username PVEBackup

Chris · Jul 18, 2024

Jera92 said:
dir: vm-lxc-storage path /mnt/pve/cephfs_cluster/vm-lxc-storage

Yes, this is definitely recommended against. Please use a RBD backed storage instead of creating a directory based storage on top of the CephFS storage.

You can do so by navigating to Datacenter > Storage > Add > RBD and select the ceph pool there. Further, make sure to select also Container for the allowed content types. Once the storage is set up, move the container filesystems to this storage.

Jera92 · Jul 18, 2024

Chris said:
Yes, this is definitely recommended against. Please use a RBD backed storage instead of creating a directory based storage on top of the CephFS storage.

You can do so by navigating to Datacenter > Storage > Add > RBD and select the ceph pool there. Further, make sure to select also Container for the allowed content types. Once the storage is set up, move the container filesystems to this storage.

Thank you for your fast reply!

Just to be 100% sure, should I create the RBD storage with the option "Use Proxmox VE managed hyper-converged ceph pool" on or off?
What's the difference between these?
I attached the options show on the screenshots.

Chris · Jul 18, 2024

Jera92 said:
Thank you for your fast reply!

Just to be 100% sure, should I create the RBD storage with the option "Use Proxmox VE managed hyper-converged ceph pool" on or off?
What's the difference between these?
I attached the options show on the screenshots.

Yes, you want to have the checkbox enabled since this Ceph cluster is managed by your PVE, right?

Jera92 · Jul 18, 2024

Chris said:
Yes, you want to have the checkbox enabled since this Ceph cluster is managed by your PVE, right?

Thanks, yes, that's true

Well, I created it the storage with the following commands:

Bash:

pve1-sandbox# cp /etc/ceph/ceph.conf /etc/pve/priv/ceph/pve_rbd.conf
pve1-sandbox# cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/pve_rbd.keyring
pve1-sandbox# pvesm add rbd rbd_lxc -pool pve_rbd_pool -data-pool pve_rbd_data_pool

But the PVE GUI shows it's inactive/unavailable as on the screenshot.
Have I done something wrong?..

Bash:

pve1-sandbox# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content vztmpl,backup,iso

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

cephfs: cephfs
        path /mnt/pve/cephfs
        content vztmpl,backup,iso
        fs-name cephfs

dir: vm-lxc-storage
        path /mnt/pve/cephfs/vm-lxc-storage
        content rootdir,images
        prune-backups keep-all=1
        shared 1

dir: ISO
        path /mnt/pve/cephfs/ISO
        content iso
        prune-backups keep-all=1
        shared 1

cifs: vm-backup-qnap
        path /mnt/pve/vm-backup-qnap
        server 192.168.16.229
        share PVE-backup
        content backup
        prune-backups keep-daily=7,keep-monthly=12,keep-weekly=4,keep-yearly=5
        username Sandbox

rbd: rbd_lxc
        content rootdir
        data-pool pve_rbd_data_pool
        krbd 1
        pool pve_rbd_pool

Chris · Jul 18, 2024

Jera92 said:

Thanks, yes, that's true

Well, I created it the storage with the following commands:

Bash:

pve1-sandbox# cp /etc/ceph/ceph.conf /etc/pve/priv/ceph/pve_rbd.conf
pve1-sandbox# cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/pve_rbd.keyring
pve1-sandbox# pvesm add rbd rbd_lxc -pool pve_rbd_pool -data-pool pve_rbd_data_pool

But the PVE GUI shows it's inactive/unavailable as on the screenshot.
Have I done something wrong?..

Bash:

pve1-sandbox# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content vztmpl,backup,iso

lvmthin: local-lvm
        thinpool data
        vgname pve
        content rootdir,images

cephfs: cephfs
        path /mnt/pve/cephfs
        content vztmpl,backup,iso
        fs-name cephfs

dir: vm-lxc-storage
        path /mnt/pve/cephfs/vm-lxc-storage
        content rootdir,images
        prune-backups keep-all=1
        shared 1

dir: ISO
        path /mnt/pve/cephfs/ISO
        content iso
        prune-backups keep-all=1
        shared 1

cifs: vm-backup-qnap
        path /mnt/pve/vm-backup-qnap
        server 192.168.16.229
        share PVE-backup
        content backup
        prune-backups keep-daily=7,keep-monthly=12,keep-weekly=4,keep-yearly=5
        username Sandbox

rbd: rbd_lxc
        content rootdir
        data-pool pve_rbd_data_pool
        krbd 1
        pool pve_rbd_pool

Try without the optional krdb flag.

What is the status of you ceph services? Check pveceph status

Jera92 · Jul 18, 2024

Chris said:
Try without the optional krdb flag.

What is the status of you ceph services? Check pveceph status

I disabled the krdb flag.

Bash:

pve1-sandbox#  pveceph status
  cluster:
    id:     966f472b-71aa-455b-ab51-4bf1617bf92e
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum pve1-sandbox,pve2-sandbox,pve3-sandbox (age 2w)
    mgr: pve3-sandbox(active, since 2w), standbys: pve2-sandbox, pve1-sandbox
    mds: 1/1 daemons up, 2 standby
    osd: 14 osds: 14 up (since 2w), 14 in (since 4M)

  data:
    volumes: 1/1 healthy
    pools:   3 pools, 65 pgs
    objects: 79.13k objects, 300 GiB
    usage:   871 GiB used, 25 TiB / 25 TiB avail
    pgs:     65 active+clean

  io:
    client:   2.3 KiB/s rd, 66 KiB/s wr, 0 op/s rd, 8 op/s wr

Chris · Jul 18, 2024

Jera92 said:
pve1-sandbox# cp /etc/ceph/ceph.conf /etc/pve/priv/ceph/pve_rbd.conf pve1-sandbox# cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/pve_rbd.keyring

Ahm, why did you copy these files? This should not be required. Does the setup of the RBD storage via the WebUI fail?

Jera92 · Jul 18, 2024

Chris said:
Ahm, why did you copy these files? This should not be required. Does the setup of the RBD storage via the WebUI fail?

Well, I tried with the command and then the rbd_lxc storage shows up in the GUI.

Bash:

pve1-sandbox# pvesm add rbd rbd_lxc -pool pve_rbd_pool -data-pool pve_rbd_data_pool

When I create the RBD storage by navigating to Datacenter > Storage > Add > RBD, I need to select a Pool but can't create one over there, so I'm unable te complete it via the GUI...
Am I missing a step?..

Chris · Jul 18, 2024

Jera92 said:
Well, I tried with the command and then the rbd_lxc storage shows up in the GUI.

Bash:

pve1-sandbox# pvesm add rbd rbd_lxc -pool pve_rbd_pool -data-pool pve_rbd_data_pool

When I create the RBD storage by navigating to Datacenter > Storage > Add > RBD, I need to select a Pool but can't create one over there, so I'm unable te complete it via the GUI...
Am I missing a step?..

Well, you have to have a pool first... This can be created via <Node> > Ceph > Pools. There you can also directly add it as storage.

Jera92 · Jul 18, 2024

Chris said:
Well, you have to have a pool first... This can be created via <Node> > Ceph > Pools. There you can also directly add it as storage.

Thanks, I got it now, I'm sorry for this mistake, should have known it..

Well, I made the RBD storage on the sandbox cluster before creating it on the production cluster, restored backup of an LXC on the RBD storage booted and worked. Did a live migration and this also works on the latest PVE version.

So, for LXC's shared storage, always use RBD?
And for VM's cephFS shared storage?

I appreciate your help and thank you very much!

Chris · Jul 19, 2024

Jera92 said:
So, for LXC's shared storage, always use RBD?
And for VM's cephFS shared storage?

No, you should use Rados block devices for CTs AND VMs. Their are better suited for the task. As you found out already, the Proxmox VE does not even let you store such content types on the CephFS backed storage without you nesting the storages (which as already stated, you should not do).

So I do recommend to migrate all of your CTs and VMs currently located on the directory storage nested on top of the CephFS storage onto the RDB storage. This should also be more performant.

Jera92 · Jul 24, 2024

Chris said:
No, you should use Rados block devices for CTs AND VMs. Their are better suited for the task. As you found out already, the Proxmox VE does not even let you store such content types on the CephFS backed storage without you nesting the storages (which as already stated, you should not do).

So I do recommend to migrate all of your CTs and VMs currently located on the directory storage nested on top of the CephFS storage onto the RDB storage. This should also be more performant.

Alright, I get it now.
I migrated all LXC's succesfully to RBD storage and will alwaus use RBD backed storage for VM & LXC's right now.

But I now have an issue concerning disk migration, with the Move disk to storage option on a vm under VMID > Hardware > Hard Disk and then selecting Disk Action > Move Storage.
When booting the VM from the moved VM disk on RBD backed storage, grub fails. Booting in a system rescue Linux distro shows me the the root and home partition have an unknown filesystem, previously it was an EXT4 filesystem, as shown on the screenshot:

I'm unable to mount it with the following commands:

Bash:

# mkdir /mnt/sda3
# mount -t ext4 /dev/sda3 /mnt/sda3

How can I solve the VM disk migration process?

Search

Search

Proxmox Virtual Environment 8.2.2 - LXC High Availibilty after upgrade PVE v7 to v8

Chris

Proxmox Staff Member

Jera92

Active Member

Attachments

Chris

Proxmox Staff Member

Jera92

Active Member

Chris

Proxmox Staff Member

Jera92

Active Member

Attachments

Chris

Proxmox Staff Member

Jera92

Active Member

Attachments

Chris

Proxmox Staff Member

Jera92

Active Member

Chris

Proxmox Staff Member

Jera92

Active Member

Chris

Proxmox Staff Member

Jera92

Active Member

Chris

Proxmox Staff Member

Jera92

Active Member