I see the same
vblank wait timed out trace with our nested PVE labs. A nomodeset seems to eliminate these (or serial console would probably too).
Last edited:
Opt-in Linux 7.0 Kernel for Proxmox VE 9 available on test and no-subscription
- Thread starter t.lamprecht
- Start date
Hi. Apparently the same thing happened to me. Went from 9.1.6 to 9.1.9 today and got kernel panic :
Code:
unable to mount root fs on unknown-block(0,0)Rebooted back to 6.17.13-7-pve and all seem ok.
journalctl does not seem to show much. But maybe I'm running the wrong command.
@fabian : Is there any way I can help before removing the 7.0.x kernel ?
Test of iperf3-ver-tailscale between two LXCs on the same machine show great performance:
Code:
09:42 user@samba:~ > iperf3 -c ts.ip.same.machine1 -t 5
Connecting to host ts.ip.same.machine1, port 5201
[ 5] local ts.ip.same.machine2 port 59294 connected to ts.ip.same.machine1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.38 GBytes 11.9 Gbits/sec 0 4.16 MBytes
[ 5] 1.00-2.00 sec 1.40 GBytes 12.0 Gbits/sec 0 4.16 MBytes
[ 5] 2.00-3.00 sec 1.39 GBytes 11.9 Gbits/sec 0 4.16 MBytes
[ 5] 3.00-4.00 sec 1.39 GBytes 12.0 Gbits/sec 0 4.16 MBytes
[ 5] 4.00-5.00 sec 1.38 GBytes 11.8 Gbits/sec 0 4.16 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-5.00 sec 6.94 GBytes 11.9 Gbits/sec 0 sender
[ 5] 0.00-5.00 sec 6.94 GBytes 11.9 Gbits/sec receiver
iperf Done.
09:43 user@samba:~ > iperf3 -c ts.ip.same.machine1 -t 5 -R
Connecting to host ts.ip.same.machine1, port 5201
Reverse mode, remote host ts.ip.same.machine1 is sending
[ 5] local ts.ip.same.machine2 port 59310 connected to ts.ip.same.machine1 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.39 GBytes 11.9 Gbits/sec
[ 5] 1.00-2.00 sec 1.40 GBytes 12.0 Gbits/sec
[ 5] 2.00-3.00 sec 1.41 GBytes 12.2 Gbits/sec
[ 5] 3.00-4.00 sec 1.42 GBytes 12.2 Gbits/sec
[ 5] 4.00-5.00 sec 1.41 GBytes 12.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-5.00 sec 7.03 GBytes 12.1 Gbits/sec 0 sender
[ 5] 0.00-5.00 sec 7.03 GBytes 12.1 Gbits/sec receiver
iperf Done
.However, the issue persists between two different machines on the same LAN (or beyond) - hugely asymmetric throughput:
(n.b. 10GbE to the main switch, but the receiving end here has only a USB 2.5 GbE dongle)
Code:
09:49 user@samba:~ > iperf3 -c ts.ip.same.lan1 -t 5
Connecting to host ts.ip.same.lan1, port 5201
[ 5] local ts.ip.same.lan2 port 41570 connected to ts.ip.same.lan1 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 512 KBytes 4.19 Mbits/sec 43 2.40 KBytes
[ 5] 1.00-2.00 sec 768 KBytes 6.29 Mbits/sec 38 2.40 KBytes
[ 5] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec 24 2.40 KBytes
[ 5] 3.00-4.00 sec 384 KBytes 3.15 Mbits/sec 41 1.20 KBytes
[ 5] 4.00-5.00 sec 384 KBytes 3.14 Mbits/sec 32 1.20 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-5.00 sec 2.12 MBytes 3.56 Mbits/sec 178 sender
[ 5] 0.00-5.00 sec 2.12 MBytes 3.56 Mbits/sec receiver
iperf Done.
09:54 user@samba:~ > iperf3 -c ts.ip.same.lan1 -t 5 -R
Connecting to host ts.ip.same.lan1, port 5201
Reverse mode, remote host ts.ip.same.lan1 is sending
[ 5] local ts.ip.same.lan2 port 46490 connected to ts.ip.same.lan1 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 117 MBytes 984 Mbits/sec
[ 5] 1.00-2.00 sec 137 MBytes 1.15 Gbits/sec
[ 5] 2.00-3.00 sec 123 MBytes 1.03 Gbits/sec
[ 5] 3.00-4.00 sec 156 MBytes 1.31 Gbits/sec
[ 5] 4.00-5.00 sec 151 MBytes 1.27 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-5.00 sec 688 MBytes 1.15 Gbits/sec 1 sender
[ 5] 0.00-5.00 sec 685 MBytes 1.15 Gbits/sec receiver
iperf Done.This leads me to conclude:
TCP over WireGuard/Tailscale works correctly as shown here - at 11.9 Gbps between LXCs on the same host (Linux pve-homeserver25 7.0.2-2-pve), but collapses to ~4 Mbps when packets egress through the physical Mellanox ConnectX-4 Lx NIC. The regression is therefore isolated to the mlx5 transmit path handling of WireGuard UDP packets in kernel 7.x.
Not sure how to take diagnosis any further and would love to get some Proxmox input on this issue...
Code:
02:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Subsystem: Mellanox Technologies Stand-up ConnectX-4 Lx EN, 25GbE dual-port SFP28, PCIe3.0 x8, MCX4121A-ACAT
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
02:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
Subsystem: Mellanox Technologies Stand-up ConnectX-4 Lx EN, 25GbE dual-port SFP28, PCIe3.0 x8, MCX4121A-ACAT
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core
Querying Mellanox devices firmware ...
Device #1:
----------
Device Type: ConnectX4LX
Part Number: MCX4121A-ACA_Ax
Description: ConnectX-4 Lx EN network interface card; 25GbE dual-port SFP28; PCIe3.0 x8; ROHS R6
PSID: MT_2420110034
PCI Device Name: /dev/mst/mt4117_pciconf0
Base MAC: 0c42a12d0cd2
Versions: Current Available
FW 14.32.1912 14.32.1010
PXE 3.6.0502 3.6.0502
UEFI 14.25.0017 14.25.0017
Status: Up to date...of course worth remembering others, with completely different NICs, are also seeing this issue (I know of at least 4 reports between here and this reddit thread: https://www.reddit.com/r/Proxmox/comments/1t7xs12/proxmox_tailscale_lxc_regression_wkernel_7/ )
I am yet to see any non Proxmox reports that sound similar though.
Last edited:
I'm not sure your conclusion is totally correct - since on the incoming
iperf3 -R you don't show that regression, even though it is to be assumed that those packets are also coming in on the Mellanox ConnectX-4 Lx NIC.As a test - you could try using an iperf3 test from the host itself or another LXC/VM (non user@samba) to eliminate other variables (such as samba etc.).
Good luck.
Already done, the LXC is seemingly irrelevant - the samba one is Debian13 with nothing but samba on it, but the issue also occurs with a Ubuntu 24.04 LXC with Jellyfin on it. I don't have (and don't really want) Tailscale installed on the PVE host itself so can't really test from there. (I have of course also tested iperf3 without Tailscale, and I then do get the full expected speeds).
The conclusion was an issue with the transmit path, not the receive, as the results demonstrate I believe.
The conclusion was an issue with the transmit path, not the receive, as the results demonstrate I believe.
yes, please open a new thread (feel free to mention @ me) and include a journal of the working kernel and details about your setup and hardware, in particular where your rootfs is stored
Is this opt in? When I updated to PVE 9.1, This kernel was installed by default.
I also no 6.17 mentioned in the thread, I have 6.8 which was already there on proxmox 8, and this 7.0 was installed during the apt dist-upgrade process.
I also no 6.17 mentioned in the thread, I have 6.8 which was already there on proxmox 8, and this 7.0 was installed during the apt dist-upgrade process.
Last edited:
After upgrade to latest Proxmox 9 and its Linux kernel 7.0.x , Windows 11 VM will not boot and will crash with automatic startup repair failing.
When I pin kernel to 6.17.13-8-pve via `proxmox-boot-tool kernel pin 6.17.13-8-pve --next-boot` , it works perfectly fine.
There's no GPU or PCIe passthrough that I am doing.
I am using Intel N150 host and using host type for the VM.
When I pin kernel to 6.17.13-8-pve via `proxmox-boot-tool kernel pin 6.17.13-8-pve --next-boot` , it works perfectly fine.
There's no GPU or PCIe passthrough that I am doing.
I am using Intel N150 host and using host type for the VM.
Last edited:
See the "Known issues", whether it applies to your case (Windows 11 might also be affected?):
Interesting update - especially the decision to align the default kernel with the upcoming Ubuntu 26.04 base. The jump to the 7.0 kernel should bring noticeable improvements for newer AMD EPYC and Intel Xeon platforms, along with better NVMe and networking support.
Across all my 6-7 Zen3/4/5 Servers especially, either zfs 2.4.2 or since kernel 7.0.2-2/4...
The perfomance increased a lot.
I mean i can feel it, so its more as 10%.
Even on old intel Servers i can feel some performance increase, not as huge as on Zen4 but there is some too.
And the new novnc/terminal console is a huge improvement either!!!
Thanks a lot!
This is definitively the biggest or only performance increase i see since any kernel or update since years.
However its the first time i have an issue either with some weird PBS Storage timeouts or not available, thats why i camed across the forum to read if someone else is affected.
But so far i think its probably an issue on my side.
So for now i would say, its amazing. Its one of the best updates, maybe the best since years.
EDIT: My issue is SMB related to Hetzner CIFS-Storage beeing unreliable, so it has nothing todo with Proxmox. Everything is perfect like always.
Cheers
The perfomance increased a lot.
I mean i can feel it, so its more as 10%.
Even on old intel Servers i can feel some performance increase, not as huge as on Zen4 but there is some too.
And the new novnc/terminal console is a huge improvement either!!!
Thanks a lot!
This is definitively the biggest or only performance increase i see since any kernel or update since years.
However its the first time i have an issue either with some weird PBS Storage timeouts or not available, thats why i camed across the forum to read if someone else is affected.
But so far i think its probably an issue on my side.
So for now i would say, its amazing. Its one of the best updates, maybe the best since years.
EDIT: My issue is SMB related to Hetzner CIFS-Storage beeing unreliable, so it has nothing todo with Proxmox. Everything is perfect like always.
Cheers
Last edited:
It seems workaround will be merged in proxmox soon: https://lore.proxmox.com/all/20260515155810.229819-1-f.ebner@proxmox.com/T/See the "Known issues", whether it applies to your case (Windows 11 might also be affected?):
May I inquire how the additional attack vector of RUST in the kernel is being triaged?
Is it of no concern on the one hand or actively being stripped out on the other?
Or anything in between of course.
Kernel 7.0 is unusable for me.
As in previous post (still waiting for a moderator) 7.0.2-4 also has issues with SR-IOV of Intel X540-AT4.
Going back to 6.17
Code:
May 17 20:02:05 px1 pveproxy[3689]: worker 406420 started
May 17 20:02:30 px1 QEMU[3776]: kvm: vfio_container_dma_map(0x5bc5a7745dd0, 0xe0cd0000, 0x1000, 0x717b85a6e000) = -28 (No space left on device)
May 17 20:02:30 px1 QEMU[3776]: kvm: vfio_container_dma_map(0x5bc5a7745dd0, 0xe0ccf000, 0x1000, 0x717a4960b000) = -28 (No space left on device)
May 17 20:02:30 px1 QEMU[3776]: kvm: vfio_container_dma_map(0x5bc5a7745dd0, 0xe0cce000, 0x1000, 0x717a4960a000) = -28 (No space left on device)
May 17 20:02:30 px1 kernel: DMAR: DRHD: handling fault status reg 2
May 17 20:02:30 px1 kernel: DMAR: [DMA Read NO_PASID] Request device [04:10.0] fault addr 0xe0ca5000 [fault reason 0x06] PTE Read access is not set
May 17 20:02:30 px1 kernel: DMAR: DRHD: handling fault status reg 102
May 17 20:02:30 px1 kernel: DMAR: [DMA Read NO_PASID] Request device [04:10.0] fault addr 0xe0ca3000 [fault reason 0x06] PTE Read access is not set
May 17 20:02:30 px1 kernel: DMAR: DRHD: handling fault status reg 202
May 17 20:02:30 px1 kernel: DMAR: [DMA Read NO_PASID] Request device [04:10.0] fault addr 0xe0ca1000 [fault reason 0x06] PTE Read access is not set
May 17 20:02:30 px1 kernel: DMAR: DRHD: handling fault status reg 302
May 17 20:02:30 px1 QEMU[3776]: kvm: vfio_container_dma_map(0x5bc5a7745dd0, 0xe0ccd000, 0x1000, 0x717b87523000) = -28 (No space left on device)
...
May 17 20:02:33 px1 QEMU[3776]: kvm: vfio_container_dma_map(0x5bc5a7745dd0, 0xe0c70000, 0x1000, 0x717a1d328000) = -28 (No space left on device)
May 17 20:02:34 px1 kernel: vfio-pci 0000:04:10.0: timed out waiting for pending transaction; performing function level reset anyway
May 17 20:02:34 px1 kernel: ixgbe 0000:04:00.0 enp4s0f0: 7 Spoofed packets detected
May 17 20:02:46 px1 pvestatd[3650]: storage 'NAS_VM_Backups' is not online
May 17 20:02:47 px1 pvestatd[3650]: status update time (10.498 seconds)
May 17 20:02:57 px1 QEMU[3776]: kvm: vfio_container_dma_map(0x5bc5a7745dd0, 0xe0c6f000, 0x1000, 0x717a1d32a000) = -28 (No space left on device)Going back to 6.17
Last edited: