Hi
After carefully reading and upgrading all my nodes, I saw that I was missing a step with removing systemd-boot package, as I use grub-efi and it could cause problems.
After removing it I cannot start any LXC/VM at all in my whole cluster.
here a debug log from a starting lxc:
Also checked this thread: https://forum.proxmox.com/threads/s...failed-failed-to-create-network-device.100745
The issue happens across all my nodes except the ones I did not already run the command `apt remove systemd-boot` as described in docs: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9#Systemd-boot meta-package changes the bootloader configuration automatically and should be uninstalled
On a node where the lxc wont start, where I removed systemd-boot:
On a working node it looks like this(systemd-boot still installed):
Im not 100% certain this comes from removing systemd-boot, but as far as I remember after the initial migration all containers seemed to start fine, and the two nodes that still work were the ones I did not yet remove systemd-boot. All nodes do however still reboot fine, so systemd-boot is most likely not needed to boot.
After carefully reading and upgrading all my nodes, I saw that I was missing a step with removing systemd-boot package, as I use grub-efi and it could cause problems.
After removing it I cannot start any LXC/VM at all in my whole cluster.
here a debug log from a starting lxc:
Code:
root@lenovo4:~# pct start 100 --debug
netdev_configure_server_veth: 669 Operation not supported - Failed to create veth pair "veth100i0" and "vethJDzaGo"
lxc_create_network_priv: 3466 Operation not supported - Failed to create network device
lxc_spawn: 1847 Failed to create the network
__lxc_start: 2114 Failed to spawn container "100"
0 20250810182811.140 INFO utils - ../src/lxc/utils.c:run_script_argv:587 - Executing script "/usr/share/lxc/hooks/lxc-pve-prestart-hook" for container "100", config section "lxc"
INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:unpriv_systemd_create_scope:1508 - Running privileged, not using a systemd unit
DEBUG seccomp - ../src/lxc/seccomp.c:parse_config_v2:664 - Host native arch is [3221225534]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "reject_force_umount # comment this to allow umount -f; not recommended"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:532 - Set seccomp rule to reject force umounts
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "[all]"
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "kexec_load errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[246:kexec_load] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741827]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[246:kexec_load] action[327681:errno] arch[1073741886]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "open_by_handle_at errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[304:open_by_handle_at] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741827]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[304:open_by_handle_at] action[327681:errno] arch[1073741886]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "init_module errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[175:init_module] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741827]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[175:init_module] action[327681:errno] arch[1073741886]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "finit_module errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[313:finit_module] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741827]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[313:finit_module] action[327681:errno] arch[1073741886]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "delete_module errno 1"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[176:delete_module] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741827]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[176:delete_module] action[327681:errno] arch[1073741886]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:815 - Processing "ioctl errno 1 [1,0x9400,SCMP_CMP_MASKED_EQ,0xff00]"
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:555 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding native rule for syscall[16:ioctl] action[327681:errno] arch[0]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:555 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741827]
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:555 - arg_cmp[0]: SCMP_CMP(1, 7, 65280, 37888)
INFO seccomp - ../src/lxc/seccomp.c:do_resolve_add_rule:572 - Adding compat rule for syscall[16:ioctl] action[327681:errno] arch[1073741886]
INFO seccomp - ../src/lxc/seccomp.c:parse_config_v2:1036 - Merging compat seccomp contexts into main context
INFO start - ../src/lxc/start.c:lxc_init:882 - Container "100" is initialized
INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_monitor_create:1679 - The monitor process uses "lxc.monitor/100" as cgroup
DEBUG storage - ../src/lxc/storage/storage.c:storage_query:231 - Detected rootfs type "dir"
DEBUG storage - ../src/lxc/storage/storage.c:storage_query:231 - Detected rootfs type "dir"
INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_payload_create:1787 - The container process uses "lxc/100/ns" as inner and "lxc/100" as limit cgroup
INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWUSER
INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWNS
INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWPID
INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWUTS
INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWIPC
INFO start - ../src/lxc/start.c:lxc_spawn:1769 - Cloned CLONE_NEWCGROUP
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved user namespace via fd 17 and stashed path as user:/proc/2566/fd/17
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved mnt namespace via fd 18 and stashed path as mnt:/proc/2566/fd/18
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved pid namespace via fd 19 and stashed path as pid:/proc/2566/fd/19
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved uts namespace via fd 20 and stashed path as uts:/proc/2566/fd/20
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved ipc namespace via fd 21 and stashed path as ipc:/proc/2566/fd/21
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved cgroup namespace via fd 22 and stashed path as cgroup:/proc/2566/fd/22
DEBUG idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newuidmap" does have the setuid bit set
DEBUG idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newgidmap" does have the setuid bit set
DEBUG idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:178 - Functional newuidmap and newgidmap binary found
INFO cgfsng - ../src/lxc/cgroups/cgfsng.c:cgfsng_setup_limits:3538 - Limits for the unified cgroup hierarchy have been setup
DEBUG idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newuidmap" does have the setuid bit set
DEBUG idmap_utils - ../src/lxc/idmap_utils.c:idmaptool_on_path_and_privileged:93 - The binary "/usr/bin/newgidmap" does have the setuid bit set
INFO idmap_utils - ../src/lxc/idmap_utils.c:lxc_map_ids:176 - Caller maps host root. Writing mapping directly
NOTICE utils - ../src/lxc/utils.c:lxc_drop_groups:1477 - Dropped supplimentary groups
INFO start - ../src/lxc/start.c:do_start:1105 - Unshared CLONE_NEWNET
NOTICE utils - ../src/lxc/utils.c:lxc_drop_groups:1477 - Dropped supplimentary groups
NOTICE utils - ../src/lxc/utils.c:lxc_switch_uid_gid:1453 - Switched to gid 0
NOTICE utils - ../src/lxc/utils.c:lxc_switch_uid_gid:1462 - Switched to uid 0
DEBUG start - ../src/lxc/start.c:lxc_try_preserve_namespace:140 - Preserved net namespace via fd 5 and stashed path as net:/proc/2566/fd/5
ERROR network - ../src/lxc/network.c:netdev_configure_server_veth:669 - Operation not supported - Failed to create veth pair "veth100i0" and "vethJDzaGo"
ERROR network - ../src/lxc/network.c:lxc_create_network_priv:3466 - Operation not supported - Failed to create network device
ERROR start - ../src/lxc/start.c:lxc_spawn:1847 - Failed to create the network
DEBUG network - ../src/lxc/network.c:lxc_delete_network:4220 - Deleted network devices
ERROR start - ../src/lxc/start.c:__lxc_start:2114 - Failed to spawn container "100"
WARN start - ../src/lxc/start.c:lxc_abort:1037 - No such process - Failed to send SIGKILL via pidfd 16 for process 2584
startup for container '100' failed
Code:
root@lenovo4:~# pveversion
pve-manager/9.0.3/025864202ebb6109 (running kernel: 6.14.8-2-pve)Also checked this thread: https://forum.proxmox.com/threads/s...failed-failed-to-create-network-device.100745
The issue happens across all my nodes except the ones I did not already run the command `apt remove systemd-boot` as described in docs: https://pve.proxmox.com/wiki/Upgrade_from_8_to_9#Systemd-boot meta-package changes the bootloader configuration automatically and should be uninstalled
On a node where the lxc wont start, where I removed systemd-boot:
Code:
root@lenovo4:~# lsmod | grep veth
root@lenovo4:~# modprobe veth
modprobe: FATAL: Module veth not found in directory /lib/modules/6.14.8-2-pve
root@lenovo4:~# cat /etc/modules
# /etc/modules is obsolete and has been replaced by /etc/modules-load.d/.
# Please see modules-load.d(5) and modprobe.d(5) for details.
#
# Updating this file still works, but it is undocumented and unsupported.
root@lenovo4:~# cat /etc/modules-load.d/
modules.conf qemu-server.conf
root@lenovo4:~# cat /etc/modules-load.d/modules.conf
# /etc/modules is obsolete and has been replaced by /etc/modules-load.d/.
# Please see modules-load.d(5) and modprobe.d(5) for details.
#
# Updating this file still works, but it is undocumented and unsupported.
root@lenovo4:~# cat /etc/modules-load.d/qemu-server.conf
vhost_net
root@lenovo4:~# uname -r
6.14.8-2-pveOn a working node it looks like this(systemd-boot still installed):
Code:
root@lenovo1:~# modprobe veth
root@lenovo1:~# lsmod | grep veth
veth 40960 0
root@lenovo1:~# cat /etc/modules
# /etc/modules is obsolete and has been replaced by /etc/modules-load.d/.
# Please see modules-load.d(5) and modprobe.d(5) for details.
#
# Updating this file still works, but it is undocumented and unsupported.
root@lenovo1:~# cat /etc/modules-load.d/
modules.conf qemu-server.conf
root@lenovo1:~# cat /etc/modules-load.d/modules.conf
# /etc/modules is obsolete and has been replaced by /etc/modules-load.d/.
# Please see modules-load.d(5) and modprobe.d(5) for details.
#
# Updating this file still works, but it is undocumented and unsupported.
root@lenovo1:~# cat /etc/modules-load.d/qemu-server.conf
vhost_net
root@lenovo1:~# uname -r
6.14.8-2-pveIm not 100% certain this comes from removing systemd-boot, but as far as I remember after the initial migration all containers seemed to start fine, and the two nodes that still work were the ones I did not yet remove systemd-boot. All nodes do however still reboot fine, so systemd-boot is most likely not needed to boot.
Last edited:
