[SOLVED] HELP! My LXC Containers Won't Start After System Reinstall!

L

linaste

New Member
Aug 20, 2023
21
2
3
I've made backups for all of my containers, then I reinstall the whole system, after that I restored and tried to boot them. Then an error occurs:
Bash: 
run_buffer: 322 Script exited with status 1
lxc_setup: 4437 Failed to run mount hooks
do_start: 1272 Failed to setup container "104"
sync_wait: 34 An error occurred in another process (expected sequence number 4)
__lxc_start: 2107 Failed to spawn container "104"
TASK ERROR: startup for container '104' failed
Please help solve this problem!

ps. During reinstall, I changed my file system from ext4 to btrfs, could this be a problem?
 
linaste said:
I've made backups for all of my containers, then I reinstall the whole system, after that I restored and tried to boot them. Then an error occurs:
Bash: 
run_buffer: 322 Script exited with status 1
lxc_setup: 4437 Failed to run mount hooks
do_start: 1272 Failed to setup container "104"
sync_wait: 34 An error occurred in another process (expected sequence number 4)
__lxc_start: 2107 Failed to spawn container "104"
TASK ERROR: startup for container '104' failed
Please help solve this problem!
Click to expand...
Did you reinstall the same PVE version or a newer one? Keep in mind that for example support for old LXCs, that don'T support cgroup2, dropped with PVE7.
linaste said:
ps. During reinstall, I changed my file system from ext4 to btrfs, could this be a problem?
Click to expand...
Shouldn't make a difference.
 
Last edited:
Dunuin said:
Dud you reinstall the same PVE version or a newer one? Keep in mind that for example support for old LXCs, that don'T support cgroup2, dropped with PVE7.
Click to expand...
Thank for your reply. I reinstalled it from 7 to 8. Could you please tell me where should I check for cgroup2?
 
Its described in the known issues of PVE 7.0:
  • Container:
    • cgroupv2 support by the container’s OS is needed to run in a pure cgroupv2 environment. Containers running systemd version 231 or newer support cgroupv2 [1], as do containers that do not use systemd as init system in the first place (e.g., Alpine Linux or Devuan).
    CentOS 7 and Ubuntu 16.10 are two prominent examples for Linux distributions releases, which have a systemd version that is too old to run in a cgroupv2 environment, for details and possible fixes see: https://pve.proxmox.com/pve-docs/chapter-pct.html#pct_cgroup_compat
Click to expand...
But shouldn`t be a problem when it already worked on PVE7 unless you needed to do the workaround back then and nw forgot to do it too.
 
Last edited:
Dunuin said:
But shouldn`t be a problem when it already worked on PVE7 unless you needed to do the workaround back then and nw forgot to do it too.
Click to expand...
Is there anything like logs or configs I can provide that could make it easier to find out the problem?
 
You can start the LXC in debug mode for some more error reporting: pct start YourVMID --debug 1

And have a look at the nodes syslog in the webUI at the time of the start.

You get the LXCs config with: pct config YourVMID
 
Dunuin said:
You can start the LXC in debug mode for some more error reporting: pct start YourVMID --debug 1

And have a look at the nodes syslog in the webUI at the time of the start.

You get the LXCs config with: pct config YourVMID
Click to expand...
pct start 114 --debug 1:
Bash: 
run_buffer: 322 Script exited with status 1
lxc_setup: 4437 Failed to run mount hooks
do_start: 1272 Failed to setup container "114"
sync_wait: 34 An error occurred in another process (expected sequence number 4)
__lxc_start: 2107 Failed to spawn container "114"
3221225534]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:524 - Set seccomp rule to reject force umounts
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "[all]"
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "kexec_load errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[246:kexec_load] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "open_by_handle_at errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[304:open_by_handle_at] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "init_module errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[175:init_module] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "finit_module errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[313:finit_module] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:807 - Processing "delete_module errno 1"
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding native rule for syscall[176:delete_module] action[327681:errno] arch[0]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741827]
INFO     seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:564 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741886]
INFO     seccomp - ../src/lxc/seccomp.c:parse_config_v2:1017 - Merging compat seccomp contexts into main context
INFO     start - ../src/lxc/start.c:lxc_init:881 - Container "114" is initialized
INFO     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_create:1391 - The monitor process uses "lxc.monitor/114" as cgroup
DEBUG    storage - ../src/lxc/storage/storage.c:storage_query:231 - Detected rootfs type "dir"
INFO     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_payload_create:1499 - The container process uses "lxc/114/ns" as inner and "lxc/114" as limit cgroup
INFO     start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWNS
INFO     start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWPID
INFO     start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWUTS
INFO     start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWIPC
INFO     start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWNET
INFO     start - ../src/lxc/start.c:lxc_spawn:1762 - Cloned CLONE_NEWCGROUP
DEBUG    start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved mnt namespace via fd 18 and stashed path as mnt:/proc/119114/fd/18
DEBUG    start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved pid namespace via fd 19 and stashed path as pid:/proc/119114/fd/19
DEBUG    start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved uts namespace via fd 20 and stashed path as uts:/proc/119114/fd/20
DEBUG    start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved ipc namespace via fd 21 and stashed path as ipc:/proc/119114/fd/21
DEBUG    start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved net namespace via fd 22 and stashed path as net:/proc/119114/fd/22
DEBUG    start - ../src/lxc/start.c:lxc_try_preserve_namespace:139 - Preserved cgroup namespace via fd 23 and stashed path as cgroup:/proc/119114/fd/23
WARN     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits_legacy:3155 - Invalid argument - Ignoring legacy cgroup limits on pure cgroup2 system
INFO     cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits:3251 - Limits for the unified cgroup hierarchy have been setup
INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxc/lxcnetaddbr" for container "114", config section "net"
DEBUG    network - ../src/lxc/network.c:netdev_configure_server_veth:852 - Instantiated veth tunnel "veth114i0 <--> vethqVwMye"
DEBUG    conf - ../src/lxc/conf.c:lxc_mount_rootfs:1437 - Mounted rootfs "/var/lib/lxc/114/rootfs" onto "/usr/lib/x86_64-linux-gnu/lxc/rootfs" with options "(null)"
INFO     conf - ../src/lxc/conf.c:setup_utsname:876 - Set hostname to "example"
DEBUG    network - ../src/lxc/network.c:setup_hw_addr:3821 - Mac address "46:B5:00:C2:AA:BB" on "eth0" has been setup
DEBUG    network - ../src/lxc/network.c:lxc_network_setup_in_child_namespaces_common:3962 - Network device "eth0" has been setup
INFO     network - ../src/lxc/network.c:lxc_setup_network_in_child_namespaces:4019 - Finished setting up network devices with caller assigned names
INFO     conf - ../src/lxc/conf.c:mount_autodev:1220 - Preparing "/dev"
INFO     conf - ../src/lxc/conf.c:mount_autodev:1281 - Prepared "/dev"
DEBUG    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:736 - Invalid argument - Tried to ensure procfs is unmounted
DEBUG    conf - ../src/lxc/conf.c:lxc_mount_auto_mounts:759 - Invalid argument - Tried to ensure sysfs is unmounted
DEBUG    conf - ../src/lxc/conf.c:mount_entry:2445 - Remounting "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" to respect bind or remount options
DEBUG    conf - ../src/lxc/conf.c:mount_entry:2464 - Flags for "/sys/fs/fuse/connections" were 4110, required extra flags are 14
DEBUG    conf - ../src/lxc/conf.c:mount_entry:2508 - Mounted "/sys/fs/fuse/connections" on "/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections" with filesystem type "none"
DEBUG    cgfsng - ../src/lxc/cgroups/cgfsng.c:__cgroupfs_mount:1909 - Mounted cgroup filesystem cgroup2 onto 20((null))
INFO     conf - ../src/lxc/conf.c:run_script_argv:338 - Executing script "/usr/share/lxcfs/lxc.mount.hook" for container "114", config section "lxc"
DEBUG    conf - ../src/lxc/conf.c:run_buffer:311 - Script exec /usr/share/lxcfs/lxc.mount.hook 114 lxc mount produced output: missing /var/lib/lxcfs/proc/ - lxcfs not running?

ERROR    conf - ../src/lxc/conf.c:run_buffer:322 - Script exited with status 1
ERROR    conf - ../src/lxc/conf.c:lxc_setup:4437 - Failed to run mount hooks
ERROR    start - ../src/lxc/start.c:do_start:1272 - Failed to setup container "114"
ERROR    sync - ../src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 4)
DEBUG    network - ../src/lxc/network.c:lxc_delete_network:4173 - Deleted network devices
ERROR    start - ../src/lxc/start.c:__lxc_start:2107 - Failed to spawn container "114"
WARN     start - ../src/lxc/start.c:lxc_abort:1036 - No such process - Failed to send SIGKILL via pidfd 17 for process 119133

TASK ERROR: startup for container '114' failed

syslog:
Bash: 
Aug 21 08:40:17 pve kernel: vmbr0: port 4(fwpr114p0) entered blocking state
Aug 21 08:40:17 pve kernel: vmbr0: port 4(fwpr114p0) entered disabled state
Aug 21 08:40:17 pve kernel: device fwpr114p0 entered promiscuous mode
Aug 21 08:40:17 pve kernel: vmbr0: port 4(fwpr114p0) entered blocking state
Aug 21 08:40:17 pve kernel: vmbr0: port 4(fwpr114p0) entered forwarding state
Aug 21 08:40:17 pve kernel: fwbr114i0: port 1(fwln114i0) entered blocking state
Aug 21 08:40:17 pve kernel: fwbr114i0: port 1(fwln114i0) entered disabled state
Aug 21 08:40:17 pve kernel: device fwln114i0 entered promiscuous mode
Aug 21 08:40:17 pve kernel: fwbr114i0: port 1(fwln114i0) entered blocking state
Aug 21 08:40:17 pve kernel: fwbr114i0: port 1(fwln114i0) entered forwarding state
Aug 21 08:40:17 pve kernel: fwbr114i0: port 2(veth114i0) entered blocking state
Aug 21 08:40:17 pve kernel: fwbr114i0: port 2(veth114i0) entered disabled state
Aug 21 08:40:17 pve kernel: device veth114i0 entered promiscuous mode
Aug 21 08:40:17 pve kernel: eth0: renamed from vethqVwMye
Aug 21 08:40:17 pve pct[119110]: startup for container '114' failed
Aug 21 08:40:17 pve pct[119109]: <root@pam> end task UPID:pve:0001D146:0027E916:64E2B26F:vzstart:114:root@pam: startup for container '114' failed
Aug 21 08:40:17 pve kernel: fwbr114i0: port 2(veth114i0) entered disabled state
Aug 21 08:40:17 pve kernel: device veth114i0 left promiscuous mode
Aug 21 08:40:17 pve kernel: fwbr114i0: port 2(veth114i0) entered disabled state
Aug 21 08:40:18 pve kernel: EXT4-fs (loop0): unmounting filesystem 98c44243-2a3e-4afb-8d96-0ad9974cb01c.
Aug 21 08:40:18 pve kernel: fwbr114i0: port 1(fwln114i0) entered disabled state
Aug 21 08:40:18 pve kernel: vmbr0: port 4(fwpr114p0) entered disabled state
Aug 21 08:40:18 pve kernel: device fwln114i0 left promiscuous mode
Aug 21 08:40:18 pve kernel: fwbr114i0: port 1(fwln114i0) entered disabled state
Aug 21 08:40:18 pve kernel: device fwpr114p0 left promiscuous mode
Aug 21 08:40:18 pve kernel: vmbr0: port 4(fwpr114p0) entered disabled state
Aug 21 08:40:18 pve systemd[1]: pve-container-debug@114.service: Main process exited, code=exited, status=1/FAILURE
Aug 21 08:40:18 pve systemd[1]: pve-container-debug@114.service: Failed with result 'exit-code'.

pct config 114:
Bash: 
arch: amd64
cores: 4
hostname: example
memory: 2048
mp0: /mnt/tm_ssd,mp=/home/example/tm_ssd
mp1: /mnt/Share,mp=/home/example/Share
nameserver: 192.168.1.254
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.254,hwaddr=46:B5:B2:C2:3F:5D,ip=192.168.1.114/24,type=veth
onboot: 1
ostype: debian
rootfs: local-nvme:114/vm-114-disk-0.raw,size=10G
swap: 512
lxc.apparmor.profile: unconfined
 
Last edited:
UPDATE:
I tried to create a brand new LXC container, and got the same error. So maybe the old backups are not to blame.
 
Dunuin said:
Keep in mind that for example support for old LXCs, that don'T support cgroup2, dropped with PVE7.
Click to expand...
There did have a warning log before the error occurred: WARN cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits_legacy:3155 - Invalid argument - Ignoring legacy cgroup limits on pure cgroup2 system

So maybe it have something to do with cgroup2?
 
Hi,
please post the output of systemctl status lxcfs.service.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back