[SOLVED] kernel: landlock - Error on VM boot - assistance required

D

DerekG

Well-Known Member
Mar 30, 2021
141
53
48
45
Hi all,

I have Proxmox 9.1.1 which was fully operational until I made 2 changes at the same time and am unsure why a single VM (out of 9 VM's/CT's) fails to complete the boot process.

The 1st change was hardware replacement, and as it's deep inside the NAS box, I really don't want to go back and fit the old parts at this stage.

2nd I had Kernel Linux 6.14.11-4-pve (it might have been the 6.8 kernel) pinned automatically after the upgrade from PVE 8.4, a couple of days ago I unpinned and upgraded to kernel 6.17.2-1-pve but this is the 1st reboot after that upgrade.

Now I get the following error (around 30 times) during the boot process:

Nov 26 22:37:17 pve-5 pveproxy[2214]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 26 22:37:17 pve-5 pveproxy[2214]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 26 22:37:17 pve-5 pveproxy[2214]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 26 22:37:18 pve-5 kernel: landlock: Disabled but requested by user space. You should enable Landlock at boot time: https://docs.kernel.org/userspace-api/landlock.html#boot-time-configuration

The VM in question is booted according to Proxmox, I can ping the device OK, but I am unable to access the console or the web interface to see what is going on.

I did try looking up the document detailed but my Linux is not good enough to understand what I am reading.

Note: I have returned to the 6.14.11 kernel, but the error remains.


Update on the above:

I have just pinned kernel Linux 6.8.12-14-pve and the VM in question is fully functional again.

So the error is introduced with 6.14 & persist with 6.17.

I would still like to resolve that matter if anyone can help.

Thank you.
 
Last edited:
Hmm I noticed these exact errors, googling brings up this thread only, in my case my server failed connection to my Proxmox Backup Server and then threw these errors like so:

Code: 
Nov 28 05:41:34 Node_C pvestatd[2257]: primary-backups: error fetching datastores - 500 read timeout
Nov 28 05:41:34 Node_C pvestatd[2257]: status update time (9.088 seconds)
Nov 28 05:41:42 Node_C pvestatd[2257]: primary-backups: error fetching datastores - 500 Can't connect to 192.168.1.77:8007
Nov 28 05:41:42 Node_C pvestatd[2257]: status update time (7.522 seconds)
Nov 28 05:41:53 Node_C pvestatd[2257]: primary-backups: error fetching datastores - 500 Can't connect to 192.168.1.77:8007
Nov 28 05:41:53 Node_C pvestatd[2257]: status update time (7.507 seconds)
Nov 28 05:42:02 Node_C pvestatd[2257]: primary-backups: error fetching datastores - 500 Can't connect to 192.168.1.77:8007
Nov 28 05:42:02 Node_C pvestatd[2257]: status update time (7.519 seconds)
Nov 28 05:42:13 Node_C pvestatd[2257]: primary-backups: error fetching datastores - 500 Can't connect to 192.168.1.77:8007
Nov 28 05:42:13 Node_C pvestatd[2257]: status update time (7.526 seconds)
Nov 28 05:42:22 Node_C pvestatd[2257]: primary-backups: error fetching datastores - 500 Can't connect to 192.168.1.77:8007
Nov 28 05:42:22 Node_C pvestatd[2257]: status update time (7.517 seconds)
Nov 28 05:42:33 Node_C pvestatd[2257]: primary-backups: error fetching datastores - 500 Can't connect to 192.168.1.77:8007
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 28 05:50:03 Node_C pveproxy[2163275]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.

Once the backup server recovered these errors disappeared so I'm guessing it's failing to pull metrics from the specific item in question.

It's not like your VM is dead it just can't be viewed from the console, might be worth looking into this @fweber as rolling back Kernels is not a real great solution.
 
complexplaster27 said:
rolling back Kernels is not a real great solution
Click to expand...

Thanks for your reply complexplaster27,

The device is available for me to furnish any diagnostics information which might be of use, just let me know what's required.

Best regards

DerekG
 
Hi!

DerekG said:
Nov 26 22:37:17 pve-5 pveproxy[2214]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 26 22:37:17 pve-5 pveproxy[2214]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 26 22:37:17 pve-5 pveproxy[2214]: Use of uninitialized value $value in addition (+) at /usr/share/perl5/PVE/PullMetric.pm line 72.
Nov 26 22:37:18 pve-5 kernel: landlock: Disabled but requested by user space. You should enable Landlock at boot time: https://docs.kernel.org/userspace-api/landlock.html#boot-time-configuration
Click to expand...
Have you installed any additional software, which actively uses the landlock kernel module? This warning is only reported, if one of the landlock syscalls (landlock_create_ruleset, landlock_add_rule, landlock_restrict_self) are trapped directly and AFAIK we don't use them directly.

I'll look into removing the uninitialized value warnings though.
 
+ What is the VM config of the VM that fails to start (e.g. cat /etc/pve/qemu-server/<vmid>.conf)?
 
dakralex said:
Have you installed any additional software, which actively uses the landlock kernel module? This warning is only reported, if one of the landlock syscalls (landlock_create_ruleset, landlock_add_rule, landlock_restrict_self) are trapped directly and AFAIK we don't use them directly.
Click to expand...
The node in question is AMD based, it does host a Windows 11 VM with the AMD iGPU passthrough, but that VM starts without issue.

The VM which fails to boot is a TrueNAS VM, with NVMe's & SATA interface passthrough, details of the config:

Code: 
root@pve-5:~# cat /etc/pve/qemu-server/410.conf
#TrueNAS with 3 pools
#
#tank - 4x NVMe (RaidZ1) 12TB
#
#SSD - 2x SSD (Mirror) 1TB
#
#Rust - 4x HDD (RaidZ1) 12TB
#
#SMART disk checks from CLI. Check /dev/id with #lsblk - check%3A  #smartctl -a /dev/nvme0n1
#
#Check NVMe link speed
#
#lspci -vvvv|grep 'LnkSta%3A' | tail -n 5
agent: 1
balloon: 0
bios: ovmf
boot: order=scsi0;net0;ide2
cores: 4
cpu: host,flags=+aes
efidisk0: local-lvm:vm-410-disk-0,efitype=4m,size=528K
hostpci0: 0000:68:00.0,pcie=1
hostpci1: 0000:6b:00.0,pcie=1
hostpci2: 0000:6c:00.0,pcie=1
hostpci3: 0000:6d:00.0,pcie=1
hostpci4: 0000:67:00.0,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 32768
meta: creation-qemu=9.2.0,ctime=1743866294
name: truenas-pve-4
net0: virtio=BC:24:11:1D:6E:F9,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-410-disk-1,discard=on,iothread=1,size=32G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=13ea9c05-c9e8-44e5-a6ff-e08a65a7a8ae
sockets: 1
startup: order=1
tablet: 0
tags: 192.168.10.11
vmgenid: 52fa7a4e-d626-4a86-95c1-712e2a81e02d
root@pve-5:~#

I hope that helps with the diagnostics.

DerekG
 
I cannot see anything obviously wrong about the config. Can you post the VM start log and the syslog around the VM startup?
 
dakralex said:
I cannot see anything obviously wrong about the config. Can you post the VM start log and the syslog around the VM startup?
Click to expand...
Sorry for the delay, I need to make some downtime available to get the information requested as the TrueNAS running on that node is critical to multiple applications in my lab.
 
Sorry it's taken so long, it's not easy to take the TrueNAS down for diagnosis here.

I think there has been an update to kernel 6.17.2-2 since my last update. And this is the log from VM boot:

Code: 
Dec 14 17:12:49 pve-5 pvestatd[2344]: status update time (9.508 seconds)
Dec 14 17:13:00 pve-5 pvedaemon[2376]: <root@pam> starting task UPID:pve-5:0000CBD4:000615EA:693EF01C:qmstart:410:root@pam:
Dec 14 17:13:00 pve-5 pvedaemon[52180]: start VM 410: UPID:pve-5:0000CBD4:000615EA:693EF01C:qmstart:410:root@pam:
Dec 14 17:13:00 pve-5 pvestatd[2344]: storage 'proxmox' is not online
Dec 14 17:13:03 pve-5 kernel: vfio-pci 0000:68:00.0: resetting
Dec 14 17:13:03 pve-5 kernel: vfio-pci 0000:68:00.0: reset done
Dec 14 17:13:03 pve-5 kernel: vfio-pci 0000:6b:00.0: resetting
Dec 14 17:13:03 pve-5 kernel: vfio-pci 0000:6b:00.0: reset done
Dec 14 17:13:03 pve-5 kernel: vfio-pci 0000:6c:00.0: resetting
Dec 14 17:13:03 pve-5 kernel: vfio-pci 0000:6c:00.0: reset done
Dec 14 17:13:03 pve-5 kernel: vfio-pci 0000:6d:00.0: resetting
Dec 14 17:13:03 pve-5 kernel: vfio-pci 0000:6d:00.0: reset done
Dec 14 17:13:03 pve-5 kernel: vfio-pci 0000:67:00.0: resetting
Dec 14 17:13:03 pve-5 kernel: vfio-pci 0000:67:00.0: reset done
Dec 14 17:13:03 pve-5 systemd[1]: 410.scope: Deactivated successfully.
Dec 14 17:13:03 pve-5 systemd[1]: Stopped 410.scope.
Dec 14 17:13:03 pve-5 systemd[1]: 410.scope: Consumed 13min 15.150s CPU time, 32.1G memory peak.
Dec 14 17:13:03 pve-5 pvestatd[2344]: status update time (13.088 seconds)
Dec 14 17:13:04 pve-5 kernel: vmbr0: port 14(tap410i0) entered disabled state
Dec 14 17:13:04 pve-5 kernel: tap410i0 (unregistering): left allmulticast mode
Dec 14 17:13:04 pve-5 kernel: vmbr0: port 14(tap410i0) entered disabled state
Dec 14 17:13:04 pve-5 qmeventd[52321]: Starting cleanup for 410
Dec 14 17:13:04 pve-5 qmeventd[52321]: trying to acquire lock...
Dec 14 17:13:04 pve-5 systemd[1]: Started 410.scope.
Dec 14 17:13:05 pve-5 kernel: tap410i0: entered promiscuous mode
Dec 14 17:13:05 pve-5 kernel: vmbr0: port 14(tap410i0) entered blocking state
Dec 14 17:13:05 pve-5 kernel: vmbr0: port 14(tap410i0) entered disabled state
Dec 14 17:13:05 pve-5 kernel: tap410i0: entered allmulticast mode
Dec 14 17:13:05 pve-5 kernel: vmbr0: port 14(tap410i0) entered blocking state
Dec 14 17:13:05 pve-5 kernel: vmbr0: port 14(tap410i0) entered forwarding state
Dec 14 17:13:06 pve-5 kernel: vfio-pci 0000:68:00.0: resetting
Dec 14 17:13:06 pve-5 kernel: vfio-pci 0000:68:00.0: reset done
Dec 14 17:13:06 pve-5 kernel: vfio-pci 0000:6b:00.0: resetting
Dec 14 17:13:06 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 6a5
Dec 14 17:13:06 pve-5 kernel: vfio-pci 0000:6b:00.0: reset done
Dec 14 17:13:06 pve-5 kernel: vfio-pci 0000:6c:00.0: resetting
Dec 14 17:13:06 pve-5 kernel: vfio-pci 0000:6c:00.0: reset done
Dec 14 17:13:06 pve-5 kernel: vfio-pci 0000:6d:00.0: resetting
Dec 14 17:13:07 pve-5 kernel: vfio-pci 0000:6d:00.0: reset done
Dec 14 17:13:07 pve-5 kernel: vfio-pci 0000:67:00.0: resetting
Dec 14 17:13:07 pve-5 kernel: vfio-pci 0000:67:00.0: reset done
Dec 14 17:13:07 pve-5 kernel: vfio-pci 0000:67:00.0: resetting
Dec 14 17:13:07 pve-5 pvestatd[2344]: storage 'proxmox' is not online
Dec 14 17:13:07 pve-5 kernel: vfio-pci 0000:67:00.0: reset done
Dec 14 17:13:07 pve-5 kernel: vfio-pci 0000:6d:00.0: resetting
Dec 14 17:13:07 pve-5 kernel: vfio-pci 0000:6d:00.0: reset done
Dec 14 17:13:07 pve-5 kernel: vfio-pci 0000:6c:00.0: resetting
Dec 14 17:13:07 pve-5 kernel: vfio-pci 0000:6c:00.0: reset done
Dec 14 17:13:07 pve-5 kernel: vfio-pci 0000:6b:00.0: resetting
Dec 14 17:13:07 pve-5 kernel: vfio-pci 0000:6b:00.0: reset done
Dec 14 17:13:07 pve-5 kernel: vfio-pci 0000:68:00.0: resetting
Dec 14 17:13:07 pve-5 kernel: vfio-pci 0000:68:00.0: reset done
Dec 14 17:13:07 pve-5 pvedaemon[52180]: VM 410 started with PID 52338.
Dec 14 17:13:07 pve-5 qmeventd[52321]:  OK
Dec 14 17:13:07 pve-5 qmeventd[52321]: vm still running
Dec 14 17:13:07 pve-5 pvedaemon[2376]: <root@pam> end task UPID:pve-5:0000CBD4:000615EA:693EF01C:qmstart:410:root@pam: OK
Dec 14 17:13:12 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 6c9
Dec 14 17:13:12 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 6d1
Dec 14 17:13:20 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 6f5
Dec 14 17:13:21 pve-5 corosync[2317]:   [KNET  ] link: host: 4 link: 0 is down
Dec 14 17:13:21 pve-5 corosync[2317]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Dec 14 17:13:21 pve-5 corosync[2317]:   [KNET  ] host: host: 4 has no active links
Dec 14 17:13:22 pve-5 corosync[2317]:   [KNET  ] link: Resetting MTU for link 0 because host 4 joined
Dec 14 17:13:22 pve-5 corosync[2317]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Dec 14 17:13:22 pve-5 corosync[2317]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Dec 14 17:13:22 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 6f9
Dec 14 17:13:24 pve-5 pvestatd[2344]: storage 'proxmox' is not online
Dec 14 17:13:26 pve-5 pvestatd[2344]: status update time (13.744 seconds)
Dec 14 17:13:28 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 70b 70c 70e
Dec 14 17:13:29 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 71d
Dec 14 17:13:39 pve-5 pvestatd[2344]: storage 'proxmox' is not online
Dec 14 17:13:40 pve-5 pvestatd[2344]: status update time (13.595 seconds)
Dec 14 17:13:44 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 748
Dec 14 17:13:44 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 748
Dec 14 17:13:44 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 748
Dec 14 17:13:44 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 748
Dec 14 17:13:44 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 757
Dec 14 17:13:48 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 75f
Dec 14 17:13:52 pve-5 pvestatd[2344]: storage 'proxmox' is not online
Dec 14 17:13:53 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 765
Dec 14 17:13:53 pve-5 pvestatd[2344]: status update time (13.075 seconds)
Dec 14 17:13:59 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 776
Dec 14 17:14:00 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 780
Dec 14 17:14:01 pve-5 pvestatd[2344]: storage 'proxmox' is not online
Dec 14 17:14:08 pve-5 pvestatd[2344]: status update time (14.578 seconds)
Dec 14 17:14:08 pve-5 corosync[2317]:   [KNET  ] link: host: 5 link: 0 is down
Dec 14 17:14:08 pve-5 corosync[2317]:   [KNET  ] link: host: 6 link: 0 is down
Dec 14 17:14:08 pve-5 corosync[2317]:   [KNET  ] link: host: 4 link: 0 is down
Dec 14 17:14:08 pve-5 corosync[2317]:   [KNET  ] host: host: 5 (passive) best link: 0 (pri: 1)
Dec 14 17:14:08 pve-5 corosync[2317]:   [KNET  ] host: host: 5 has no active links
Dec 14 17:14:08 pve-5 corosync[2317]:   [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1)
Dec 14 17:14:08 pve-5 corosync[2317]:   [KNET  ] host: host: 6 has no active links
Dec 14 17:14:08 pve-5 corosync[2317]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Dec 14 17:14:08 pve-5 corosync[2317]:   [KNET  ] host: host: 4 has no active links
Dec 14 17:14:08 pve-5 corosync[2317]:   [KNET  ] link: Resetting MTU for link 0 because host 6 joined
Dec 14 17:14:08 pve-5 corosync[2317]:   [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1)
Dec 14 17:14:08 pve-5 corosync[2317]:   [KNET  ] link: Resetting MTU for link 0 because host 5 joined
Dec 14 17:14:08 pve-5 corosync[2317]:   [KNET  ] host: host: 5 (passive) best link: 0 (pri: 1)
Dec 14 17:14:08 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 7af
Dec 14 17:14:09 pve-5 corosync[2317]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Dec 14 17:14:10 pve-5 corosync[2317]:   [KNET  ] link: Resetting MTU for link 0 because host 4 joined
Dec 14 17:14:10 pve-5 corosync[2317]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Dec 14 17:14:11 pve-5 corosync[2317]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Dec 14 17:14:12 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 7bb
Dec 14 17:14:12 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 7be
Dec 14 17:14:12 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 7c2
Dec 14 17:14:12 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 7c2
Dec 14 17:14:12 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 7c2
Dec 14 17:14:14 pve-5 pvestatd[2344]: storage 'proxmox' is not online
Dec 14 17:14:18 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 7d6
Dec 14 17:14:22 pve-5 corosync[2317]:   [KNET  ] link: host: 6 link: 0 is down
Dec 14 17:14:22 pve-5 corosync[2317]:   [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1)
Dec 14 17:14:22 pve-5 corosync[2317]:   [KNET  ] host: host: 6 has no active links
Dec 14 17:14:22 pve-5 pvestatd[2344]: status update time (14.109 seconds)
Dec 14 17:14:22 pve-5 corosync[2317]:   [KNET  ] link: Resetting MTU for link 0 because host 6 joined
Dec 14 17:14:22 pve-5 corosync[2317]:   [KNET  ] host: host: 6 (passive) best link: 0 (pri: 1)
Dec 14 17:14:22 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 7e1 7e2
Dec 14 17:14:23 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 7e2
Dec 14 17:14:23 pve-5 corosync[2317]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Dec 14 17:14:26 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 7ee
Dec 14 17:14:29 pve-5 pvestatd[2344]: storage 'proxmox' is not online
Dec 14 17:14:34 pve-5 pvestatd[2344]: status update time (12.157 seconds)
Dec 14 17:14:35 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 808
Dec 14 17:14:35 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 808 80d
Dec 14 17:14:35 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 808
Dec 14 17:14:35 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 808
Dec 14 17:14:45 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 830
Dec 14 17:14:45 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 830
Dec 14 17:14:53 pve-5 pvestatd[2344]: storage 'proxmox' is not online
Dec 14 17:14:53 pve-5 pvestatd[2344]: status update time (19.622 seconds)
Dec 14 17:15:01 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 856
Dec 14 17:15:01 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 85a
Dec 14 17:15:01 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 85a
Dec 14 17:15:03 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 876
Dec 14 17:15:03 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 876
Dec 14 17:15:05 pve-5 pvestatd[2344]: storage 'proxmox' is not online
Dec 14 17:15:07 pve-5 corosync[2317]:   [KNET  ] link: host: 4 link: 0 is down
Dec 14 17:15:07 pve-5 corosync[2317]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Dec 14 17:15:07 pve-5 corosync[2317]:   [KNET  ] host: host: 4 has no active links
Dec 14 17:15:07 pve-5 corosync[2317]:   [KNET  ] link: Resetting MTU for link 0 because host 4 joined
Dec 14 17:15:07 pve-5 corosync[2317]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Dec 14 17:15:08 pve-5 corosync[2317]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Dec 14 17:15:09 pve-5 corosync[2317]:   [KNET  ] link: host: 5 link: 0 is down
Dec 14 17:15:09 pve-5 corosync[2317]:   [KNET  ] host: host: 5 (passive) best link: 0 (pri: 1)
Dec 14 17:15:09 pve-5 corosync[2317]:   [KNET  ] host: host: 5 has no active links
Dec 14 17:15:09 pve-5 corosync[2317]:   [KNET  ] link: Resetting MTU for link 0 because host 5 joined
Dec 14 17:15:09 pve-5 corosync[2317]:   [KNET  ] host: host: 5 (passive) best link: 0 (pri: 1)
Dec 14 17:15:10 pve-5 corosync[2317]:   [KNET  ] pmtud: Global data MTU changed to: 1397
Dec 14 17:15:11 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 896
Dec 14 17:15:12 pve-5 pvestatd[2344]: status update time (18.455 seconds)
Dec 14 17:15:21 pve-5 pvestatd[2344]: storage 'proxmox' is not online
Dec 14 17:15:24 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 8c7
Dec 14 17:15:25 pve-5 pvestatd[2344]: status update time (13.317 seconds)
Dec 14 17:15:26 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 8c9
Dec 14 17:15:32 pve-5 corosync[2317]:   [KNET  ] link: host: 2 link: 0 is down
Dec 14 17:15:32 pve-5 corosync[2317]:   [KNET  ] link: host: 1 link: 0 is down
Dec 14 17:15:32 pve-5 corosync[2317]:   [KNET  ] link: host: 4 link: 0 is down
Dec 14 17:15:32 pve-5 corosync[2317]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Dec 14 17:15:32 pve-5 corosync[2317]:   [KNET  ] host: host: 2 has no active links
Dec 14 17:15:32 pve-5 corosync[2317]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Dec 14 17:15:32 pve-5 corosync[2317]:   [KNET  ] host: host: 1 has no active links
Dec 14 17:15:32 pve-5 corosync[2317]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Dec 14 17:15:32 pve-5 corosync[2317]:   [KNET  ] host: host: 4 has no active links
Dec 14 17:15:34 pve-5 corosync[2317]:   [KNET  ] link: Resetting MTU for link 0 because host 1 joined
Dec 14 17:15:34 pve-5 corosync[2317]:   [KNET  ] host: host: 1 (passive) best link: 0 (pri: 1)
Dec 14 17:15:34 pve-5 corosync[2317]:   [KNET  ] link: Resetting MTU for link 0 because host 4 joined
Dec 14 17:15:34 pve-5 corosync[2317]:   [KNET  ] link: Resetting MTU for link 0 because host 2 joined
Dec 14 17:15:34 pve-5 corosync[2317]:   [KNET  ] host: host: 4 (passive) best link: 0 (pri: 1)
Dec 14 17:15:34 pve-5 corosync[2317]:   [KNET  ] host: host: 2 (passive) best link: 0 (pri: 1)
Dec 14 17:15:34 pve-5 corosync[2317]:   [TOTEM ] Retransmit List: 4
Dec 14 17:15:34 pve-5 corosync[2317]:   [QUORUM] Sync members[7]: 1 2 3 4 5 6 7
Dec 14 17:15:34 pve-5 corosync[2317]:   [TOTEM ] A new membership (1.ba93) was formed. Members
Dec 14 17:15:34 pve-5 pvestatd[2344]: storage 'proxmox' is not online
Dec 14 17:15:35 pve-5 corosync[2317]:   [QUORUM] Members[7]: 1 2 3 4 5 6 7
Dec 14 17:15:35 pve-5 corosync[2317]:   [MAIN  ] Completed service synchronization, ready to provide service.
 
Was there any console output for the qmstart task for the VM 410? Is there anything shown at boot on the machine? Is the boot disk (scsi0) still intact (e.g., no fs corruption, system files are readable)?

Else I could not see anything off from the syslog itself between start and end of the qmstart task.
 
Last edited:
dakralex said:
Was there any console output for the qmstart task for the VM 410? Is there anything shown at boot on the machine? Is the boot disk (scsi0) still intact (e.g., no fs corruption, system files are readable)?
Click to expand...
The system is booting into TrueNAS, I can hear the sequence of the drives as the zfs initializes them.

The problem appears to me as no network access to the TrueNAS VM, not via the web, the console or the network shares. The VM is using the same virtio NIC driver as the other guests. The only difference is that the TN is the only guest with an internal bridge network. Having said that I have another TrueNAS (on an Intel host) and that doesn't exhibit the same issue.

However there are multiple VM/CT's also connected on the same bridge network and those are having no issues.
 
Just a brief update on this TrueNAS VM issue.

Today I decide to build a fresh TrueNAS VM with the host on kernel 6.17.4-1 and I am unable to get the VM to boot for installation, and with the passthrough devices attached the system would not boot. I pinned the issue down to the ASM1166 SATA Controller, as soon as that device is installed the guest goes into that state. I have now upgraded the firmware on that controller but it didn't make any difference.

It's now late so I'll have to revert back to this issue another day.
 
uzumo said:
Is this separate from the following issue?
Click to expand...

Issue resolved.

As per the post, I had to both upgrade the firmware on the SATA controller AND set Rombar to off (where it defaults to on) in order for the kernel 6.14 or later to allow the VM to fully boot.

I should have recognised this fault as I have had the same on another TrueNAS install with a separate ASM PCI card (only the fault didn't show up the same on that Intel based system). In that case the firmware was updated but Rombar was set to on, and yet the VM still functions OK.
 
  • Like
Reactions: uzumo
I'm not sure if it was a change in the operating specifications or a bug, but I'm glad it's fixed.

If it's a working configuration, I don't think it has any adverse effects.
 
You must log in or register to reply here.
Top Bottom