[SOLVED] LXC won't start

[ferdl9999]

Hello,

since yesterday I have a problem with my lxc containers. First I noticed the following behavior:

When attaching a screen inside the container (with screen -r ...) the following message appeared: WriteMessage: Too much references: can't splice (German: WriteMessage: Zu viele Referenzen: can't splice).

I first tried shutting down the container and booting it up again. But the start of the container failed with the following message:

Job for pve-container@225.service failed because the control process exited with error code.
See "systemctl status pve-container@225.service" and "journalctl -xe" for details.
TASK ERROR: command 'systemctl start pve-container@225' failed: exit code 1

Then I searched a bit on the internet. I tried booting up the server with the following command: lxc-start -n 225 --foreground --logfile test.log --logpriority=2
The server was booting up fine. The content of the test.log file is the following:

lxc-start 225 20201123084310.952 ERROR    conf - conf.c:lxc_setup_boot_id:3527 - Permission denied - Failed to mount /dev/.lxc-boot-id to /proc/sys/kernel/random/boot_id

After shutting down the server again, I tried booting it up just via the following command: lxc-start -n 225

The problem is occuring on all of the lxc containers on this node. It is not a problem with this specific container.

Here is the output of pveversion -v:

proxmox-ve: 6.1-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.1-7 (running version: 6.1-7/13e58d5e)
pve-kernel-helper: 6.1-6
pve-kernel-5.3: 6.1-5
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.3-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.15-pve1
libpve-access-control: 6.0-6
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.0-13
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-4
libpve-storage-perl: 6.1-5
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-3
pve-cluster: 6.1-4
pve-container: 3.0-21
pve-docs: 6.1-6
pve-edk2-firmware: 2.20200229-1
pve-firewall: 4.0-10
pve-firmware: 3.0-6
pve-ha-manager: 3.0-8
pve-i18n: 2.0-4
pve-qemu-kvm: 4.1.1-3
pve-xtermjs: 4.3.0-1
qemu-server: 6.1-6
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.3-pve1

When I run apt update and apt upgrade and apt dist-upgrade, there are no new packages available.

By the way: After starting the container with the command lxc-start -n <ID>, the problem with attaching a screen still persists. Maybe this is related to the startup problem?

On this node are a lot of VMs running, so a reboot of the node wouldn't be optimal.

Thank you for help time and your help!
Best Regards

 

[juliokele]

"When I run apt update and apt upgrade and apt dist-upgrade, there are no new packages available."
your system is not up to date, proxmox-ve: 6.1-2 vs. proxmox-ve: 6.2-2 (11.23.2020)

please check your Package Repositorys, for example Proxmox VE No-Subscription Repository:
https://pve.proxmox.com/wiki/Package_Repositories#sysadmin_no_subscription_repo

 

[ferdl9999]

Thank you very much. I have replaced the contents of /etc/apt/sources.list with the contents for the PVE No-Subscription repository and upgraded my system.

After that, I tried again starting up an LXC server, and the error message have changed. The error message is now:

pct start 225

lxc_can_use_pidfd: 1855 Invalid argument - Kernel does not support waiting on processes through pidfds
lxc_send_ttys_to_parent: 1001 Too many references: cannot splice - Failed to send 2 ttys to parent
lxc_create_ttys: 1021 Failed to send ttys to parent
do_start: 1224 Failed to setup container "225"
__sync_wait: 41 An error occurred in another process (expected sequence number 5)
__lxc_start: 1950 Failed to spawn container "225"
startup for container '225' failed

What does Too many references: cannot splice - Failed to send 2 ttys to parent mean?

 

[juliokele]

please post your pve version again (pveversion -v)

 

[ferdl9999]

Sure, here is the output:

proxmox-ve: 6.2-2 (running kernel: 5.3.18-2-pve)
pve-manager: 6.2-15 (running version: 6.2-15/48bd51b6)
pve-kernel-5.4: 6.3-1
pve-kernel-helper: 6.3-1
pve-kernel-5.3: 6.1-6
pve-kernel-5.4.73-1-pve: 5.4.73-1
pve-kernel-5.3.18-3-pve: 5.3.18-3
pve-kernel-5.3.18-2-pve: 5.3.18-2
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.4-pve1
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
ksm-control-daemon: 1.3-1
libjs-extjs: 6.0.1-10
libknet1: 1.16-pve1
libproxmox-acme-perl: 1.0.5
libpve-access-control: 6.1-3
libpve-apiclient-perl: 3.0-3
libpve-common-perl: 6.2-4
libpve-guest-common-perl: 3.1-3
libpve-http-server-perl: 3.0-6
libpve-storage-perl: 6.2-10
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve4
lxc-pve: 4.0.3-1
lxcfs: 4.0.3-pve3
novnc-pve: 1.1.0-1
proxmox-backup-client: 1.0.1-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.3-10
pve-cluster: 6.2-1
pve-container: 3.2-3
pve-docs: 6.2-6
pve-edk2-firmware: 2.20200531-1
pve-firewall: 4.1-3
pve-firmware: 3.1-3
pve-ha-manager: 3.1-1
pve-i18n: 2.2-2
pve-qemu-kvm: 5.1.0-6
pve-xtermjs: 4.7.0-2
qemu-server: 6.2-20
smartmontools: 7.1-pve2
spiceterm: 3.1-1
vncterm: 1.6-2
zfsutils-linux: 0.8.5-pve1

 

[juliokele]

please restart your server, the newest kernel is not yet running...

 

[ferdl9999]

Thank you, it's working fine again. A simple upgrade did the trick. But I don't knot what caused this error, it happened from one moment to another.