Server goes on strike after reinstallation after starting an lxc or vm [pveproxy[896 897]: detected empty handle]

[dimla]

Hello everyone,
I have two s740 Futro that I previously ran in a Proxmox cluster with ZFS. But since there were always problems with reboot, I wanted to let the two run as separate nodes.

So now to my problem, previously they ran without any problems. But not anymore after reinstalling pve 7.4 and 8.0.

The BIOS is currently v1.13.0 VT-d is active, 16GB RAM has already been replaced by 4GB as a test, m.2 480GB has been replaced by a 240GB, no other drives are connected.
As a test, I also controlled the server separately with just one client in an isolated network, so to speak.

However, everything comes out the same.

As soon as I start an LXC container or a VM, the web interface loads very slowly or fails, there are connection interruptions via ssh, and access works very slowly.

After starting an empty and new LXC container on Ubuntu 22.04, the load goes up to 1.00 0.95 0.62, the following appears in the log under Proxmox.

Nov 16 13:35:11 pve1 pvedaemon[1985]: starting CT 100: UPID:pve1:000007C1:0000B550:65560C7F:vzstart:100:root@pam:
Nov 16 13:35:11 pve1 pvedaemon[887]: <root@pam> starting task UPID:pve1:000007C1:0000B550:65560C7F:vzstart:100:root@pam:
Nov 16 13:35:11 pve1 systemd[1]: Created slice system-pve\x2dcontainer.slice - PVE LXC Container Slice.
Nov 16 13:35:11 pve1 systemd[1]: Started pve-container@100.service - PVE LXC Container: 100.
Nov 16 13:35:14 pve1 kernel: EXT4-fs (dm-6): mounted filesystem 4b41e9a3-e218-41ba-b88d-78e451dde4ad with ordered data mode. Quota mode: none.
Nov 16 13:35:15 pve1 audit[2015]: AVC apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=2015 comm="apparmor_parser"
Nov 16 13:35:15 pve1 kernel: kauditd_printk_skb: 13 callbacks suppressed
Nov 16 13:35:15 pve1 kernel: audit: type=1400 audit(1700138115.273:25): apparmor="STATUS" operation="profile_load" profile="/usr/bin/lxc-start" name="lxc-100_</var/lib/lxc>" pid=2015 comm="apparmor_parser"
Nov 16 13:35:16 pve1 kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Nov 16 13:35:16 pve1 kernel: vmbr0: port 2(fwpr100p0) entered disabled state
Nov 16 13:35:16 pve1 kernel: device fwpr100p0 entered promiscuous mode
Nov 16 13:35:16 pve1 kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Nov 16 13:35:16 pve1 kernel: vmbr0: port 2(fwpr100p0) entered forwarding state
Nov 16 13:35:16 pve1 kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Nov 16 13:35:16 pve1 kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Nov 16 13:35:16 pve1 kernel: device fwln100i0 entered promiscuous mode
Nov 16 13:35:16 pve1 kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Nov 16 13:35:16 pve1 kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state
Nov 16 13:35:16 pve1 kernel: fwbr100i0: port 2(veth100i0) entered blocking state
Nov 16 13:35:16 pve1 kernel: fwbr100i0: port 2(veth100i0) entered disabled state
Nov 16 13:35:16 pve1 kernel: device veth100i0 entered promiscuous mode
Nov 16 13:35:17 pve1 kernel: eth0: renamed from vethjqeXas
Nov 16 13:35:17 pve1 pvedaemon[887]: <root@pam> end task UPID:pve1:000007C1:0000B550:65560C7F:vzstart:100:root@pam: OK
Nov 16 13:35:18 pve1 audit[2166]: AVC apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="nvidia_modprobe" pid=2166 comm="apparmor_parser"
Nov 16 13:35:18 pve1 kernel: audit: type=1400 audit(1700138118.549:26): apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="nvidia_modprobe" pid=2166 comm="apparmor_parser"
Nov 16 13:35:18 pve1 kernel: audit: type=1400 audit(1700138118.549:27): apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="nvidia_modprobe//kmod" pid=2166 comm="apparmor_parser"
Nov 16 13:35:18 pve1 audit[2166]: AVC apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="nvidia_modprobe//kmod" pid=2166 comm="apparmor_parser"
Nov 16 13:35:18 pve1 audit[2168]: AVC apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="/usr/bin/man" pid=2168 comm="apparmor_parser"
Nov 16 13:35:18 pve1 audit[2168]: AVC apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="man_filter" pid=2168 comm="apparmor_parser"
Nov 16 13:35:18 pve1 audit[2168]: AVC apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="man_groff" pid=2168 comm="apparmor_parser"
Nov 16 13:35:18 pve1 kernel: audit: type=1400 audit(1700138118.557:28): apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="/usr/bin/man" pid=2168 comm="apparmor_parser"
Nov 16 13:35:18 pve1 kernel: audit: type=1400 audit(1700138118.557:29): apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="man_filter" pid=2168 comm="apparmor_parser"
Nov 16 13:35:18 pve1 kernel: audit: type=1400 audit(1700138118.557:30): apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="man_groff" pid=2168 comm="apparmor_parser"
Nov 16 13:35:18 pve1 audit[2167]: AVC apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=2167 comm="apparmor_parser"
Nov 16 13:35:18 pve1 audit[2167]: AVC apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=2167 comm="apparmor_parser"
Nov 16 13:35:18 pve1 audit[2167]: AVC apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=2167 comm="apparmor_parser"
Nov 16 13:35:18 pve1 audit[2167]: AVC apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="/{,usr/}sbin/dhclient" pid=2167 comm="apparmor_parser"
Nov 16 13:35:18 pve1 audit[2170]: AVC apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="tcpdump" pid=2170 comm="apparmor_parser"
Nov 16 13:35:18 pve1 audit[2165]: AVC apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="lsb_release" pid=2165 comm="apparmor_parser"
Nov 16 13:35:18 pve1 kernel: audit: type=1400 audit(1700138118.565:31): apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=2167 comm="apparmor_parser"
Nov 16 13:35:18 pve1 kernel: audit: type=1400 audit(1700138118.565:32): apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=2167 comm="apparmor_parser"
Nov 16 13:35:18 pve1 kernel: audit: type=1400 audit(1700138118.565:33): apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=2167 comm="apparmor_parser"
Nov 16 13:35:18 pve1 kernel: audit: type=1400 audit(1700138118.565:34): apparmor="STATUS" operation="profile_load" label="lxc-100_</var/lib/lxc>//&:lxc-100_<-var-lib-lxc>:unconfined" name="/{,usr/}sbin/dhclient" pid=2167 comm="apparmor_parser"
Nov 16 13:35:18 pve1 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Nov 16 13:35:18 pve1 kernel: fwbr100i0: port 2(veth100i0) entered blocking state
Nov 16 13:35:18 pve1 kernel: fwbr100i0: port 2(veth100i0) entered forwarding state
Nov 16 13:35:19 pve1 pvestatd[885]: modified cpu set for lxc/100: 0
...
Nov 16 13:47:07 pve1 pveproxy[897]: detected empty handle
Nov 16 13:47:10 pve1 pveproxy[896]: detected empty handle
Nov 16 13:47:11 pve1 kernel: perf: interrupt took too long (3962 > 3957), lowering kernel.perf_event_max_sample_rate to 50250
Nov 16 13:47:12 pve1 pveproxy[897]: detected empty handle
Nov 16 13:47:15 pve1 pveproxy[896]: detected empty handle
Nov 16 13:47:17 pve1 pveproxy[897]: detected empty handle
Nov 16 13:47:20 pve1 pveproxy[896]: detected empty handle

The really strange thing is that I have two s740s and the same problem occurs on both.

 

[dimla]

Between run the LXC




5 min after reboot

 

[Lucaaa]

same

 

[Lucaaa]

Issue found:
root@pom3nvme:~# journalctl -b --priority=4 -f Dec 13 00:58:27 pom3nvme pveproxy[469409]: detected empty handle Dec 13 00:58:27 pom3nvme pveproxy[469409]: detected empty handle Dec 13 00:58:29 pom3nvme pveproxy[469409]: detected empty handle Dec 13 00:58:32 pom3nvme pveproxy[469411]: detected empty handle Dec 13 00:58:32 pom3nvme pveproxy[469409]: detected empty handle Dec 13 00:58:33 pom3nvme pveproxy[469411]: detected empty handle Dec 13 00:58:34 pom3nvme pveproxy[469409]: detected empty handle Dec 13 00:58:37 pom3nvme pveproxy[469409]: detected empty handle Dec 13 00:58:39 pom3nvme pveproxy[469409]: detected empty handle Dec 13 00:58:42 pom3nvme pveproxy[469409]: detected empty handle Dec 13 00:58:44 pom3nvme pveproxy[469409]: detected empty handle ^C

 

[Lucaaa]

What's the trouble?

 

[dperezu76]

what is it the issue?