Debian container boot failures after update to v7.0

[rdpok]

Hello. New to Proxmox and LXC in general, so this is probably just some stupid user issue but after updating from v6 to v7 I've found that with any container I create, there are strange errors getting their services up. So far tried debian-10-turnkey-zoneminder_16.1-1_amd64 and the default debian-11-standard_11.0-1_amd64.

Below are some logs related to latter one, as I suspect this default template should work without major hiccups? For me the boot takes 5 minutes (Raise network interfaces timeout, only black Console during this) and there are multiple services failing. The container is created with default settings; Unprivileged=Yes, nesting=1.

 ~ pveversion -v
proxmox-ve: 7.0-2 (running kernel: 5.11.22-5-pve)
pve-manager: 7.0-13 (running version: 7.0-13/7aa7e488)
pve-kernel-helper: 7.1-2
pve-kernel-5.11: 7.0-8
pve-kernel-5.4: 6.4-7
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.4.143-1-pve: 5.4.143-1
pve-kernel-5.4.140-1-pve: 5.4.140-1
pve-kernel-5.4.73-1-pve: 5.4.73-1
ceph-fuse: 14.2.21-1
corosync: 3.1.5-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: 0.8.36+pve1
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.22-pve1
libproxmox-acme-perl: 1.4.0
libproxmox-backup-qemu0: 1.2.0-1
libpve-access-control: 7.0-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.0-10
libpve-guest-common-perl: 4.0-2
libpve-http-server-perl: 4.0-3
libpve-storage-perl: 7.0-12
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 4.0.9-4
lxcfs: 4.0.8-pve2
novnc-pve: 1.2.0-3
proxmox-backup-client: 2.0.11-1
proxmox-backup-file-restore: 2.0.11-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.3-6
pve-cluster: 7.0-3
pve-container: 4.1-1
pve-docs: 7.0-5
pve-edk2-firmware: 3.20210831-1
pve-firewall: 4.2-4
pve-firmware: 3.3-2
pve-ha-manager: 3.3-1
pve-i18n: 2.5-1
pve-qemu-kvm: 6.0.0-4
pve-xtermjs: 4.12.0-1
qemu-server: 7.0-16
smartmontools: 7.2-pve2
spiceterm: 3.2-2
vncterm: 1.7-1
zfsutils-linux: 2.0.6-pve1~bpo10+1

CT template and creation:

 ~ pveam list local                                           
NAME                                                         SIZE 
local:vztmpl/debian-11-standard_11.0-1_amd64.tar.gz          232.15MB

 *** CT creation in web UI with default options ***
extracting archive '/var/lib/vz/template/cache/debian-11-standard_11.0-1_amd64.tar.gz'
Total bytes read: 684820480 (654MiB, 81MiB/s)
Detected container architecture: amd64
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
done: SHA256:... root@debian
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
done: SHA256:Wz5/... root@debian
Creating SSH host key 'ssh_host_dsa_key' - this may take some time ...
done: SHA256:/... root@debian
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
done: SHA256:... root@debian
TASK OK

Config:

 ~ cat /etc/pve/lxc/102.conf
arch: amd64
cores: 4
features: nesting=1
hostname: debian
memory: 2080
net0: name=eth0,bridge=vmbr0,firewall=1,hwaddr=BA:D9:87:12:F3:8F,ip=dhcp,ip6=dhcp,type=veth
ostype: debian
rootfs: hdd1t:subvol-102-disk-0,size=64G
swap: 2080
unprivileged: 1

Boot messages (3 failing services)

~ lxc-start -n 102 -F -l DEBUG -o /tmp/lxc-102.log
systemd 247.3-6 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +ZSTD +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=unified)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Debian GNU/Linux 11 (bullseye)!

Set hostname to <debian>.
Queued start job for default target Graphical Interface.
[  OK  ] Created slice system-container\x2dgetty.slice.
[  OK  ] Created slice system-modprobe.slice.
[  OK  ] Created slice system-postfix.slice.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Reached target Paths.
[  OK  ] Reached target Remote Encrypted Volumes.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Reached target Slices.
[  OK  ] Reached target Swap.
[  OK  ] Listening on Syslog Socket.
[  OK  ] Listening on initctl Compatibility Named Pipe.
systemd-journald-audit.socket: Failed to create listening socket (audit 1): Operation not permitted
systemd-journald-audit.socket: Failed to listen on sockets: Operation not permitted
systemd-journald-audit.socket: Failed with result 'resources'.
[FAILED] Failed to listen on Journal Audit Socket.
See 'systemctl status systemd-journald-audit.socket' for details.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Listening on Journal Socket.
[  OK  ] Listening on Network Service Netlink Socket.
         Mounting POSIX Message Queue File System...
         Starting Wait for network to be configured by ifupdown...
         Starting Load Kernel Module configfs...
         Starting Load Kernel Module drm...
         Starting Load Kernel Module fuse...
         Starting nftables...
systemd-journald.service: Attaching egress BPF program to cgroup /sys/fs/cgroup/system.slice/systemd-journald.service failed: Invalid argument
         Starting Journal Service...
         Starting Load Kernel Modules...
         Starting Remount Root and Kernel File Systems...
         Starting Helper to synchronize boot up for ifupdown...
[  OK  ] Mounted POSIX Message Queue File System.
modprobe@configfs.service: Succeeded.
[  OK  ] Finished Load Kernel Module configfs.
modprobe@drm.service: Succeeded.
[  OK  ] Finished Load Kernel Module drm.
modprobe@fuse.service: Succeeded.
[  OK  ] Finished Load Kernel Module fuse.
         Mounting Kernel Configuration File System...
[  OK  ] Finished Remount Root and Kernel File Systems.
         Starting Create System Users...
sys-kernel-config.mount: Mount process exited, code=exited, status=32/n/a
sys-kernel-config.mount: Failed with result 'exit-code'.
[FAILED] Failed to mount Kernel Configuration File System.
See 'systemctl status sys-kernel-config.mount' for details.
[  OK  ] Finished Helper to synchronize boot up for ifupdown.
[  OK  ] Finished nftables.
[  OK  ] Reached target Network (Pre).
[  OK  ] Finished Load Kernel Modules.
         Starting Apply Kernel Variables...
[  OK  ] Finished Create System Users.
         Starting Create Static Device Nodes in /dev...
[  OK  ] Finished Apply Kernel Variables.
[  OK  ] Finished Create Static Device Nodes in /dev.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Raise network interfaces...
         Starting Network Service...
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Network Service.
         Starting Wait for Network to be Configured...
[  OK  ] Finished Flush Journal to Persistent Storage.
         Starting Create Volatile Files and Directories...
[  OK  ] Finished Wait for Network to be Configured.
[  OK  ] Finished Create Volatile Files and Directories.
         Starting Network Name Resolution...
[  OK  ] Reached target System Time Set.
[  OK  ] Reached target System Time Synchronized.
         Starting Update UTMP about System Boot/Shutdown...
[  OK  ] Finished Update UTMP about System Boot/Shutdown.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily apt download activities.
[  OK  ] Started Daily apt upgrade and clean activities.
[  OK  ] Started Periodic ext4 Online Metadata Check for All Filesystems.
[  OK  ] Started Daily rotation of log files.
[  OK  ] Started Daily man-db regeneration.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Listening on OpenBSD Secure Shell server socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
[  OK  ] Started Regular background program processing daemon.
[  OK  ] Started D-Bus System Message Bus.
         Starting Remove Stale Online ext4 Metadata Check Snapshots...
         Starting System Logging Service...
         Starting User Login Management...
[  OK  ] Started System Logging Service.
[  OK  ] Finished Remove Stale Online ext4 Metadata Check Snapshots.
[  OK  ] Started User Login Management.
[  OK  ] Started Network Name Resolution.
[  OK  ] Reached target Host and Network Name Lookups.
[  OK  ] Finished Wait for network to be configured by ifupdown.
[**    ] A start job is running for Raise network interfaces (23s / 5min)  ***BOOT STUCK HERE FOR THE 5 MIN ***
[FAILED] Failed to start Raise network interfaces.
See 'systemctl status networking.service' for details.
[  OK  ] Reached target Network.
[  OK  ] Reached target Network is Online.
         Starting Postfix Mail Transport Agent (instance -)...
         Starting Permit User Sessions...
[  OK  ] Finished Permit User Sessions.
[  OK  ] Started Console Getty.
[  OK  ] Started Container Getty on /dev/tty1.
[  OK  ] Started Container Getty on /dev/tty2.
[  OK  ] Reached target Login Prompts.
[  OK  ] Started Postfix Mail Transport Agent (instance -).
         Starting Postfix Mail Transport Agent...
[  OK  ] Finished Postfix Mail Transport Agent.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Finished Update UTMP about System Runlevel Changes.

Debian GNU/Linux 11 debian console

debian login:

LXC start debug log attached (due to post size limit).


Any ideas where to troubleshoot? In case this might be more problematic issue, a full reinstall of PVE is still on the table, as so far I've been only getting to know the system and testing stuff, mostly Win10 & macOS VMs. This is pretty much the first major hurdle, and looks like one I'm not able to pass

 

[rdpok]

Some further details of the failing services from within the container:

root@debian:~# systemctl       
  UNIT                                 LOAD   ACTIVE SUB       DESCRIPTION
  -.mount                              loaded active mounted   Root Mount
  dev-.lxc-proc.mount                  loaded active mounted   /dev/.lxc/proc
  dev-.lxc-sys.mount                   loaded active mounted   /dev/.lxc/sys
  dev-full.mount                       loaded active mounted   /dev/full
  dev-mqueue.mount                     loaded active mounted   POSIX Message Queue File System
  dev-null.mount                       loaded active mounted   /dev/null
  dev-ptmx.mount                       loaded active mounted   /dev/ptmx
  dev-random.mount                     loaded active mounted   /dev/random
  dev-tty.mount                        loaded active mounted   /dev/tty
  dev-tty1.mount                       loaded active mounted   /dev/tty1
  dev-tty2.mount                       loaded active mounted   /dev/tty2
  dev-urandom.mount                    loaded active mounted   /dev/urandom
  dev-zero.mount                       loaded active mounted   /dev/zero
  proc-cpuinfo.mount                   loaded active mounted   /proc/cpuinfo
  proc-diskstats.mount                 loaded active mounted   /proc/diskstats
  proc-loadavg.mount                   loaded active mounted   /proc/loadavg
  proc-meminfo.mount                   loaded active mounted   /proc/meminfo
  proc-stat.mount                      loaded active mounted   /proc/stat
  proc-swaps.mount                     loaded active mounted   /proc/swaps
  proc-sys-net.mount                   loaded active mounted   /proc/sys/net
  proc-sysrq\x2dtrigger.mount          loaded active mounted   /proc/sysrq-trigger
  proc-uptime.mount                    loaded active mounted   /proc/uptime
  run-user-0.mount                     loaded active mounted   /run/user/0
  sys-devices-system-cpu-online.mount  loaded active mounted   /sys/devices/system/cpu/online
  sys-devices-virtual-net.mount        loaded active mounted   /sys/devices/virtual/net
  sys-fs-fuse-connections.mount        loaded active mounted   FUSE Control File System
* sys-kernel-config.mount              loaded failed failed    Kernel Configuration File System                       
  systemd-ask-password-console.path    loaded active waiting   Dispatch Password Requests to Console Directory Watch
  systemd-ask-password-wall.path       loaded active waiting   Forward Password Requests to Wall Directory Watch     
  init.scope                           loaded active running   System and Service Manager
  session-c1.scope                     loaded active running   Session c1 of user root                               
  console-getty.service                loaded active running   Console Getty
  container-getty@1.service            loaded active running   Container Getty on /dev/tty1
  container-getty@2.service            loaded active running   Container Getty on /dev/tty2
  cron.service                         loaded active running   Regular background program processing daemon
  dbus.service                         loaded active running   D-Bus System Message Bus
  ifupdown-pre.service                 loaded active exited    Helper to synchronize boot up for ifupdown
  ifupdown-wait-online.service         loaded active exited    Wait for network to be configured by ifupdown
* networking.service                   loaded failed failed    Raise network interfaces
  nftables.service                     loaded active exited    nftables
  postfix.service                      loaded active exited    Postfix Mail Transport Agent
  postfix@-.service                    loaded active running   Postfix Mail Transport Agent (instance -)
  rsyslog.service                      loaded active running   System Logging Service
  systemd-journal-flush.service        loaded active exited    Flush Journal to Persistent Storage
  systemd-journald.service             loaded active running   Journal Service
  systemd-logind.service               loaded active running   User Login Management
  systemd-modules-load.service         loaded active exited    Load Kernel Modules
  systemd-networkd-wait-online.service loaded active exited    Wait for Network to be Configured
  systemd-networkd.service             loaded active running   Network Service
  systemd-remount-fs.service           loaded active exited    Remount Root and Kernel File Systems
  systemd-resolved.service             loaded active running   Network Name Resolution
  systemd-sysctl.service               loaded active exited    Apply Kernel Variables
  systemd-sysusers.service             loaded active exited    Create System Users
  systemd-tmpfiles-setup-dev.service   loaded active exited    Create Static Device Nodes in /dev
  systemd-tmpfiles-setup.service       loaded active exited    Create Volatile Files and Directories
  systemd-update-utmp.service          loaded active exited    Update UTMP about System Boot/Shutdown
  systemd-user-sessions.service        loaded active exited    Permit User Sessions
  user-runtime-dir@0.service           loaded active exited    User Runtime Directory /run/user/0
  user@0.service                       loaded active running   User Manager for UID 0                                 
  -.slice                              loaded active active    Root Slice
  system-container\x2dgetty.slice      loaded active active    system-container\x2dgetty.slice
  system-modprobe.slice                loaded active active    system-modprobe.slice
  system-postfix.slice                 loaded active active    system-postfix.slice
  system.slice                         loaded active active    System Slice
  user-0.slice                         loaded active active    User Slice of UID 0
  user.slice                           loaded active active    User and Session Slice                                 
  dbus.socket                          loaded active running   D-Bus System Message Bus Socket
  ssh.socket                           loaded active listening OpenBSD Secure Shell server socket
  syslog.socket                        loaded active running   Syslog Socket
  systemd-initctl.socket               loaded active listening initctl Compatibility Named Pipe
* systemd-journald-audit.socket        loaded failed failed    Journal Audit Socket
  systemd-journald-dev-log.socket      loaded active running   Journal Socket (/dev/log)
  systemd-journald.socket              loaded active running   Journal Socket
  systemd-networkd.socket              loaded active running   Network Service Netlink Socket                         
  basic.target                         loaded active active    Basic System
  cryptsetup.target                    loaded active active    Local Encrypted Volumes
  getty.target                         loaded active active    Login Prompts
  graphical.target                     loaded active active    Graphical Interface
  local-fs-pre.target                  loaded active active    Local File Systems (Pre)
  local-fs.target                      loaded active active    Local File Systems
  multi-user.target                    loaded active active    Multi-User System
  network-online.target                loaded active active    Network is Online
  network-pre.target                   loaded active active    Network (Pre)
  network.target                       loaded active active    Network
  nss-lookup.target                    loaded active active    Host and Network Name Lookups
  paths.target                         loaded active active    Paths
  remote-cryptsetup.target             loaded active active    Remote Encrypted Volumes
  remote-fs.target                     loaded active active    Remote File Systems
  slices.target                        loaded active active    Slices
  sockets.target                       loaded active active    Sockets
  swap.target                          loaded active active    Swap
  sysinit.target                       loaded active active    System Initialization
  time-set.target                      loaded active active    System Time Set
  time-sync.target                     loaded active active    System Time Synchronized
  timers.target                        loaded active active    Timers                                                 
  apt-daily-upgrade.timer              loaded active waiting   Daily apt upgrade and clean activities
  apt-daily.timer                      loaded active waiting   Daily apt download activities
  e2scrub_all.timer                    loaded active waiting   Periodic ext4 Online Metadata Check for All Filesystems
  logrotate.timer                      loaded active waiting   Daily rotation of log files
  man-db.timer                         loaded active waiting   Daily man-db regeneration
  systemd-tmpfiles-clean.timer         loaded active waiting   Daily Cleanup of Temporary Directories

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.
101 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.




root@debian:~# systemctl status networking.service
* networking.service - Raise network interfaces
     Loaded: loaded (/lib/systemd/system/networking.service; enabled; vendor preset: enabled)
     Active: failed (Result: timeout) since Mon 2021-11-01 16:18:41 UTC; 44s ago
       Docs: man:interfaces(5)
    Process: 67 ExecStart=/sbin/ifup -a --read-environment (code=exited, status=1/FAILURE)
   Main PID: 67 (code=exited, status=1/FAILURE)
        CPU: 144ms

Nov 01 16:17:58 debian ifup[179]: XMT:  | X-- Request renew in  +3600
Nov 01 16:17:58 debian ifup[179]: XMT:  | X-- Request rebind in +5400
Nov 01 16:17:58 debian ifup[179]: XMT: Solicit on eth0, interval 109810ms.
Nov 01 16:17:58 debian dhclient[179]: XMT: Solicit on eth0, interval 109810ms.
Nov 01 16:18:41 debian systemd[1]: networking.service: start operation timed out. Terminating.
Nov 01 16:18:41 debian systemd[1]: networking.service: Main process exited, code=exited, status=1/F>
Nov 01 16:18:41 debian ifup[67]: Got signal Terminated, terminating...
Nov 01 16:18:41 debian ifup[67]: ifup: failed to bring up eth0
Nov 01 16:18:41 debian systemd[1]: networking.service: Failed with result 'timeout'.
Nov 01 16:18:41 debian systemd[1]: Failed to start Raise network interfaces.



root@debian:~# systemctl status systemd-journald-audit.socket 
* systemd-journald-audit.socket - Journal Audit Socket
     Loaded: loaded (/lib/systemd/system/systemd-journald-audit.socket; static)
     Active: failed (Result: resources)
   Triggers: * systemd-journald.service
       Docs: man:systemd-journald.service(8)
             man:journald.conf(5)
     Listen: audit 1 (Netlink)


root@debian:~# systemctl status sys-kernel-config.mount
* sys-kernel-config.mount - Kernel Configuration File System
     Loaded: loaded (/lib/systemd/system/sys-kernel-config.mount; static)
     Active: failed (Result: exit-code) since Thu 2021-11-04 18:12:57 UTC; 16min ago
      Where: /sys/kernel/config
       What: configfs
       Docs: https://www.kernel.org/doc/Documentation/filesystems/configfs/configfs.txt
             https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
        CPU: 2ms

Nov 04 18:12:57 debian mount[61]: mount: /sys/kernel/config: permission denied.
Warning: journal has been rotated since unit was started, output may be incomplete.

root@debian:~# systemctl stop sys-kernel-config.mount
root@debian:~# systemctl start sys-kernel-config.mount
Job failed. See "journalctl -xe" for details.

root@debian:~# journalctl -xe
Nov 04 18:37:41 debian systemd[1]: Starting Load Kernel Module configfs...
-- Subject: A start job for unit modprobe@configfs.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit modprobe@configfs.service has begun execution.
--
-- The job identifier is 389.
Nov 04 18:37:41 debian systemd[1]: modprobe@configfs.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit modprobe@configfs.service has successfully entered the 'dead' state.
Nov 04 18:37:41 debian systemd[1]: Finished Load Kernel Module configfs.
-- Subject: A start job for unit modprobe@configfs.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit modprobe@configfs.service has finished successfully.
--
-- The job identifier is 389.
Nov 04 18:37:41 debian systemd[1]: Mounting Kernel Configuration File System...
-- Subject: A start job for unit sys-kernel-config.mount has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit sys-kernel-config.mount has begun execution.
--
-- The job identifier is 387.
Nov 04 18:37:41 debian mount[675]: mount: /sys/kernel/config: permission denied.
Nov 04 18:37:41 debian systemd[1]: sys-kernel-config.mount: Mount process exited, code=exited, status=32/n/a
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An n/a= process belonging to unit sys-kernel-config.mount has exited.
--
-- The process' exit code is 'exited' and its exit status is 32.
Nov 04 18:37:41 debian systemd[1]: sys-kernel-config.mount: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit sys-kernel-config.mount has entered the 'failed' state with result 'exit-code'.
Nov 04 18:37:41 debian systemd[1]: Failed to mount Kernel Configuration File System.
-- Subject: A start job for unit sys-kernel-config.mount has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit sys-kernel-config.mount has finished with a failure.
--
-- The job identifier is 387 and the job result is failed.

 

[fiona]

Hi,
I suspect the problematic setting is using DHCP for IPv6. Is your DHCP server configured for that?

 

[rdpok]

So this was indeed a stupid user issue as I thought, thanks a lot for the tip!

Disabling DHCP for IPv6 made the network service start without issues. The sys-kernel-config.mount and systemd-journald-audit.socket do still fail but I suppose that's not critical. At least I have not noticed that affecting anything.

 

[hvisage]

The sys-kernel-config.mount and systemd-journald-audit.socket do still fail but I suppose that's not critical. At least I have not noticed that affecting anything.

Yeah, other than a simple check for "degraded" then lits up my LXC dashboards like christmas trees