Opt-in Linux 6.14 Kernel for Proxmox VE 8 available on test & no-subscription

gfngfn256 · Sunday at 19:59

jauling said:
does your N100

Don't have one. Maybe ask t.lamprecht who seems to have one.

zenowl77 · Sunday at 22:23

GreenDamTan said:
new branch for 17.5 and 16.9
https://gitlab.com/GreenDamTan/vgpu-proxmox/-/tree/17.5_16.9_kernel_6.12_and_6.14?ref_type=heads
View attachment 84566

this works perfectly thank you. great work as always.

for reference i applied the patch to the driver .run file, installed it, installed 6.14 and rebooted. the modules built fine and all is well.

also if you do not mind me asking, i noticed you have a pascal on 17.5? is there a trick to that? i replaced my vgpuConfig.xml file, which works fine to make vGPU function, but linux VMs say driver mismatch on any driver version (and they do not work) and windows will still only use 535.... just wondering if i am doing something wrong or missing something (it seems like i am. i could never figure out this or getting LXCs to use the gpu right with the kvm driver haha. )

jauling · Sunday at 22:36

t.lamprecht said:
DDR5 has built in basic EEC and Intel N100 had an issue where memory error events were not correctly handled, this has been fixed with a commit that landed in Linux 6.13, and thus it can indeed happen that 6.14 will show these messages while older kernel do not.

This does not necessarily mean that the HW must be broken, the reporting of it could be broken too – some vendors sell Intel N100 based systems with memory installed already, that memory is often some noname cheap stuff, might even work fine but could have other issues. And, there really could be a problem with your HW, e.g. the memory DIMMs or the motherboard or the CPU and if that's the case then disabling the EDAC module is only silencing the messenger and symptoms, not really fixing anything.

FWIW, I got an Intel N100 based system running PVE, and it does not show these errors when booting our 6.14 kernel, I bought memory separately and chose a reputable vendor, so at least it's not a general issue to Intel N100 based systems.

Not sure if there is a difference between POLLED and INTERRUPT when the igen6_edac is loaded, but I noticed that was a difference between 6.8 and 6.14 kernels.

t.lamprecht · 2025-04-07T07:57:17+0200

jauling said:
Not sure if there is a difference between POLLED and INTERRUPT when the igen6_edac is loaded, but I noticed that was a difference between 6.8 and 6.14 kernels.

Yeah, polling is exactly what the patch I linked implements, from the commit message:

Some PCs with Intel N100 (with PCI device 8086:461c, DID_ADL_N_SKU4) experienced issues with error interrupts not working, [...].
Add polling mode support for these machines to ensure that memory error events are handled.

Again, I'd not be surprised if it's an issue in the firmware of your board, those are often done rather sloppy for those cheaper motherboards. Adding the polling mode seems to be a workaround for something like that too.

UltraCoder · 2025-04-07T17:11:26+0200

Switched to 6.14 kernel. Found an issue with Realtek 8125 NIC: iperf3 reports 2.35 Gbits/s in one direction, but only 2.24 Gbits/s in reverse direction.

Code:

# iperf3 -c router.home
Connecting to host router.home, port 5201
[  5] local 192.168.64.2 port 51652 connected to 192.168.64.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   282 MBytes  2.37 Gbits/sec    0    749 KBytes       
[  5]   1.00-2.00   sec   281 MBytes  2.35 Gbits/sec   44    837 KBytes       
[  5]   2.00-3.00   sec   281 MBytes  2.35 Gbits/sec    0    837 KBytes       
[  5]   3.00-4.00   sec   281 MBytes  2.36 Gbits/sec    0    837 KBytes       
[  5]   4.00-5.00   sec   281 MBytes  2.35 Gbits/sec    0    837 KBytes       
[  5]   5.00-6.00   sec   280 MBytes  2.35 Gbits/sec    0    837 KBytes       
[  5]   6.00-7.00   sec   281 MBytes  2.35 Gbits/sec    0    837 KBytes       
[  5]   7.00-8.00   sec   281 MBytes  2.36 Gbits/sec    0    837 KBytes       
[  5]   8.00-9.00   sec   281 MBytes  2.36 Gbits/sec    0    837 KBytes       
[  5]   9.00-10.00  sec   280 MBytes  2.35 Gbits/sec    0    837 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.74 GBytes  2.36 Gbits/sec   44             sender
[  5]   0.00-10.00  sec  2.00 GBytes  1.72 Gbits/sec                  receiver

iperf Done.
# iperf3 -c router.home -R
Connecting to host router.home, port 5201
Reverse mode, remote host router.home is sending
[  5] local 192.168.64.2 port 39748 connected to 192.168.64.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   268 MBytes  2.24 Gbits/sec                 
[  5]   1.00-2.00   sec   267 MBytes  2.24 Gbits/sec                 
[  5]   2.00-3.00   sec   267 MBytes  2.24 Gbits/sec                 
[  5]   3.00-4.00   sec   267 MBytes  2.24 Gbits/sec                 
[  5]   4.00-5.00   sec   267 MBytes  2.24 Gbits/sec                 
[  5]   5.00-6.00   sec   267 MBytes  2.24 Gbits/sec                 
[  5]   6.00-7.00   sec   267 MBytes  2.24 Gbits/sec                 
[  5]   7.00-8.00   sec   267 MBytes  2.24 Gbits/sec                 
[  5]   8.00-9.00   sec   267 MBytes  2.24 Gbits/sec                 
[  5]   9.00-10.00  sec   267 MBytes  2.24 Gbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.00 GBytes  1.72 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.61 GBytes  2.24 Gbits/sec                  receiver

iperf Done.

gfngfn256 · 2025-04-07T17:18:56+0200

UltraCoder said:
iperf3 reports 2.35 Gbits/s in one direction, but only 2.24 Gbits/s in reverse direction.

A margin of 4.68%! Not sure if that is indicative of anything.

UltraCoder · 2025-04-07T17:38:51+0200

gfngfn256 said:
A margin of 4.68%! Not sure if that is indicative of anything.

On 6.11 kernel bitrate is symmetric:

Code:

# iperf3 -c router.home
Connecting to host router.home, port 5201
[  5] local 192.168.64.2 port 42146 connected to 192.168.64.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   282 MBytes  2.36 Gbits/sec   88    730 KBytes     
[  5]   1.00-2.00   sec   280 MBytes  2.35 Gbits/sec    0    747 KBytes     
[  5]   2.00-3.00   sec   281 MBytes  2.35 Gbits/sec    0    747 KBytes     
[  5]   3.00-4.00   sec   281 MBytes  2.36 Gbits/sec   18    748 KBytes     
[  5]   4.00-5.00   sec   280 MBytes  2.35 Gbits/sec    0    748 KBytes     
[  5]   5.00-6.00   sec   281 MBytes  2.36 Gbits/sec    0    748 KBytes     
[  5]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec    0    768 KBytes     
[  5]   7.00-8.00   sec   281 MBytes  2.35 Gbits/sec    0    793 KBytes     
[  5]   8.00-9.00   sec   281 MBytes  2.36 Gbits/sec    0    793 KBytes     
[  5]   9.00-10.00  sec   280 MBytes  2.35 Gbits/sec    0    874 KBytes     
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.74 GBytes  2.35 Gbits/sec  106             sender
[  5]   0.00-10.00  sec  2.00 GBytes  1.72 Gbits/sec                  receiver

iperf Done.
# iperf3 -c router.home -R
Connecting to host router.home, port 5201
Reverse mode, remote host router.home is sending
[  5] local 192.168.64.2 port 39790 connected to 192.168.64.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   280 MBytes  2.35 Gbits/sec               
[  5]   1.00-2.00   sec   281 MBytes  2.35 Gbits/sec               
[  5]   2.00-3.00   sec   280 MBytes  2.35 Gbits/sec               
[  5]   3.00-4.00   sec   280 MBytes  2.35 Gbits/sec               
[  5]   4.00-5.00   sec   281 MBytes  2.35 Gbits/sec               
[  5]   5.00-6.00   sec   281 MBytes  2.35 Gbits/sec               
[  5]   6.00-7.00   sec   280 MBytes  2.35 Gbits/sec               
[  5]   7.00-8.00   sec   280 MBytes  2.35 Gbits/sec               
[  5]   8.00-9.00   sec   281 MBytes  2.35 Gbits/sec               
[  5]   9.00-10.00  sec   280 MBytes  2.35 Gbits/sec               
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.00 GBytes  1.72 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.74 GBytes  2.35 Gbits/sec                  receiver

iperf Done.

Update: Propably, this has nothing to do with R8125. Previously I tested throughput between my PC and VM (VM sending via Virtio device). But throughput between PC and Proxmox host itself is ok:

Code:

# iperf3 -c proxmox.home -R
Connecting to host proxmox.home, port 5201
Reverse mode, remote host proxmox.home is sending
[  5] local 192.168.64.2 port 57700 connected to 192.168.64.100 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   281 MBytes  2.35 Gbits/sec                
[  5]   1.00-2.00   sec   281 MBytes  2.35 Gbits/sec                
[  5]   2.00-3.00   sec   281 MBytes  2.35 Gbits/sec                
[  5]   3.00-4.00   sec   281 MBytes  2.35 Gbits/sec                
[  5]   4.00-5.00   sec   280 MBytes  2.35 Gbits/sec                
[  5]   5.00-6.00   sec   280 MBytes  2.35 Gbits/sec                
[  5]   6.00-7.00   sec   281 MBytes  2.35 Gbits/sec                
[  5]   7.00-8.00   sec   281 MBytes  2.35 Gbits/sec                
[  5]   8.00-9.00   sec   280 MBytes  2.35 Gbits/sec                
[  5]   9.00-10.00  sec   280 MBytes  2.35 Gbits/sec                
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.00 GBytes  1.72 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  2.74 GBytes  2.35 Gbits/sec                  receiver

iperf Done.

gfngfn256 · 2025-04-07T18:40:43+0200

UltraCoder said:
On 6.11 kernel bitrate is symmetric

On kernel Linux 6.8.12-9-pve (2025-03-16T19:18Z this would be "unsymmetric" in your books (- not mine!):

Code:

root@WRKSPC-DEBIAN:~# iperf3 -c 192.168.1.205 -R
Connecting to host 192.168.1.205, port 5201
Reverse mode, remote host 192.168.1.205 is sending
[  5] local 192.168.1.204 port 52986 connected to 192.168.1.205 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  9.03 GBytes  77.6 Gbits/sec
[  5]   1.00-2.00   sec  9.21 GBytes  79.2 Gbits/sec
[  5]   2.00-3.00   sec  9.52 GBytes  81.7 Gbits/sec
[  5]   3.00-4.00   sec  9.54 GBytes  81.9 Gbits/sec
[  5]   4.00-5.00   sec  9.49 GBytes  81.5 Gbits/sec
[  5]   5.00-6.00   sec  9.51 GBytes  81.7 Gbits/sec
[  5]   6.00-7.00   sec  9.54 GBytes  82.0 Gbits/sec
[  5]   7.00-8.00   sec  9.54 GBytes  81.9 Gbits/sec
[  5]   8.00-9.00   sec  9.12 GBytes  78.3 Gbits/sec
[  5]   9.00-10.00  sec  9.23 GBytes  79.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  93.7 GBytes  80.5 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  93.7 GBytes  80.5 Gbits/sec                  receiver

iperf Done.
root@WRKSPC-DEBIAN:~# iperf3 -c 192.168.1.205
Connecting to host 192.168.1.205, port 5201
[  5] local 192.168.1.204 port 47862 connected to 192.168.1.205 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  9.60 GBytes  82.5 Gbits/sec    0    462 KBytes
[  5]   1.00-2.00   sec  9.59 GBytes  82.4 Gbits/sec    0    462 KBytes
[  5]   2.00-3.00   sec  9.59 GBytes  82.4 Gbits/sec    0    462 KBytes
[  5]   3.00-4.00   sec  9.59 GBytes  82.4 Gbits/sec    0    462 KBytes
[  5]   4.00-5.00   sec  9.57 GBytes  82.2 Gbits/sec    0    650 KBytes
[  5]   5.00-6.00   sec  9.59 GBytes  82.4 Gbits/sec    0    650 KBytes
[  5]   6.00-7.00   sec  9.55 GBytes  82.1 Gbits/sec    0    650 KBytes
[  5]   7.00-8.00   sec  9.59 GBytes  82.4 Gbits/sec    0    683 KBytes
[  5]   8.00-9.00   sec  9.57 GBytes  82.2 Gbits/sec    0    683 KBytes
[  5]   9.00-10.00  sec  9.58 GBytes  82.3 Gbits/sec    0    683 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  95.8 GBytes  82.3 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  95.8 GBytes  82.3 Gbits/sec                  receiver

iperf Done.

So a deviation of (a whopping!) 2.19% - nothing to write home about! I notice a lot of Retr on your end (on both kernels) - probably NW activity etc.

UltraCoder · 2025-04-07T19:29:11+0200

gfngfn256 said:
So a deviation of (a whopping!) 2.19% - nothing to write home about! I notice a lot of Retr on your end (on both kernels) - probably NW activity etc.

1. My network is practically idle, but difference in throughput is about 110 Mbits/s.
2. My results are repeatable. I'm measuring 2.35 Gbit/s in both directions on kernel 6.11, reboot into 6.14 and measuring 2.35 Gbits/s in one direction and 2.24 Gbits/s in the other.
3. This has nothing to do with retranslations, because:
a) in submitted results retranslations shown only for forward direction, which still has 2.35-2.36 Gbits/s, no matter if there were retranslations;
b) I ran the same test from the opposite side (VM as a client, PC as a server, VM is sending), and saw zero retranslations, and still only 2.24-2.25 Gbits/s.
4. I think we should not compare 2.5G network with 100G one in terms of deviation. With 100G network even CPU or PCI-E might become bottleneck.

gfngfn256 · 2025-04-07T19:37:26+0200

UltraCoder said:
retranslations

AFAIK Retr stands for Retransmitted, as in Retransmitted packets.

UltraCoder said:
4. I think we should not compare 2.5G network with 100G one in terms of deviation. With 100G network even CPU or PCI-E might become bottleneck.

My tests were done CT to CT on same Proxmox host. Maybe try that in your tests & compare results.

MikeMcr · 2025-04-07T21:27:37+0200

jauling said:

6.14.0-1-pve is reporting some kind of ECC errors (I'm not running ECC memory with N100 lol). My dmesg is completely filled with these errors:

Code:

# dmesg | grep igen6
[    3.775370] caller igen6_probe+0x1bc/0x8e0 [igen6_edac] mapping multiple BARs
[    3.775412] EDAC MC0: Giving out device to module igen6_edac controller Intel_client_SoC MC#0: DEV 0000:00:00.0 (POLLED)
[    3.775430] EDAC igen6 MC0: HANDLING IBECC MEMORY ERROR
[    3.775431] EDAC igen6 MC0: ADDR 0x7fffffffe0
[    3.775469] EDAC igen6: v2.5.1
[    4.824379] EDAC igen6 MC0: HANDLING IBECC MEMORY ERROR
[    4.824382] EDAC igen6 MC0: ADDR 0x7fffffffe0
[    5.852327] EDAC igen6 MC0: HANDLING IBECC MEMORY ERROR
[    5.852335] EDAC igen6 MC0: ADDR 0x7fffffffe0
[    6.872532] EDAC igen6 MC0: HANDLING IBECC MEMORY ERROR
[    6.872536] EDAC igen6 MC0: ADDR 0x7fffffffe0
[    7.896351] EDAC igen6 MC0: HANDLING IBECC MEMORY ERROR
[    7.896355] EDAC igen6 MC0: ADDR 0x7fffffffe0
[    8.920317] EDAC igen6 MC0: HANDLING IBECC MEMORY ERROR
[    8.920320] EDAC igen6 MC0: ADDR 0x7fffffffe0

I've blacklist the igen6_edac module, but still with this 6.14 kernel my idle CPU frequency is still higher than with 6.8, so I've reverted.

Not sure about anyone else, but when running 6.8.12-9-pve, the igen6_edac module also loads, but the only dmesg entries are these:

Code:

# dmesg | grep igen6
[    3.455293] caller igen6_probe+0x193/0x8b0 [igen6_edac] mapping multiple BARs
[    3.455337] EDAC MC0: Giving out device to module igen6_edac controller Intel_client_SoC MC#0: DEV 0000:00:00.0 (INTERRUPT)
[    3.455389] EDAC igen6 MC0: HANDLING IBECC MEMORY ERROR
[    3.455391] EDAC igen6 MC0: ADDR 0x7fffffffe0
[    3.455431] EDAC igen6: v2.5.1

Same problem with a GMKtec N150. That constant flood of messages can't be good for the SSD so I've reverted back to 6.11.

gfngfn256 · 2025-04-07T21:43:41+0200

Seems like this "EDAC igen6 MC0: HANDLING IBECC MEMORY ERROR" flooding on kernel >6.13 is reported here & here. It would appear to be an Intel N-series CPU problem.

t.lamprecht · 2025-04-07T22:47:41+0200

gfngfn256 said:
Seems like this "EDAC igen6 MC0: HANDLING IBECC MEMORY ERROR" flooding on kernel >6.13 is reported here & here. It would appear to be an Intel N-series CPU problem.

Thanks for the pointer, so seems really like an issue with some specific N CPUs/motherboard combos that can luckily be worked around in the kernel.
I checked and saw that the POC patch was sent as an actual one and the maintainer accepted it. So I cherry-picked that for our next 6.14 build (few weeks at max), disable edac until then as workaround if you run an affected system and still really want to run 6.14.

UltraCoder · 2025-04-07T23:51:43+0200

gfngfn256 said:
AFAIK Retr stands for Retransmitted, as in Retransmitted packets.

Oh, of course, I meant retransmissions.

Sorry, English is not my native language.

gfngfn256 said:
My tests were done CT to CT on same Proxmox host. Maybe try that in your tests & compare results.

OK, I performed additional tests and wrote results in the table:

It looks to me, that tests inside Proxmox are not representative, and the difference is within the margin of error. The bottleneck here is, propably, CPU (AMD Ryzen 1700 in my case), so I don't really care.
But tests with PC are not good. 2.5G (2.35G with MTU 1500) should not be a problem ever. But we can clearly see, that upgrading kernel results in ~4-5% loss in throughput. Importantly, the test between PVE and PC shows no losses, so maybe this has something to do with changes in linux bridge?

GreenDamTan · 2025-04-08T05:16:45+0200

zenowl77 said:
this works perfectly thank you. great work as always.

for reference i applied the patch to the driver .run file, installed it, installed 6.14 and rebooted. the modules built fine and all is well.

also if you do not mind me asking, i noticed you have a pascal on 17.5? is there a trick to that? i replaced my vgpuConfig.xml file, which works fine to make vGPU function, but linux VMs say driver mismatch on any driver version (and they do not work) and windows will still only use 535.... just wondering if i am doing something wrong or missing something (it seems like i am. i could never figure out this or getting LXCs to use the gpu right with the kvm driver haha. )

https://github.com/GreenDamTan/vgpu_unlock-rs/commit/a651cf35c0097426c369c2783c749b2ed28bce39
try to use A5500 device id in pascal card

Search

Search

Opt-in Linux 6.14 Kernel for Proxmox VE 8 available on test & no-subscription

gfngfn256

Famous Member

zenowl77

Member

jauling

New Member

t.lamprecht

Proxmox Staff Member

UltraCoder

New Member

gfngfn256

Famous Member

UltraCoder

New Member

gfngfn256

Famous Member

UltraCoder

New Member

gfngfn256

Famous Member

MikeMcr

New Member

gfngfn256

Famous Member

t.lamprecht

Proxmox Staff Member

UltraCoder

New Member

GreenDamTan

New Member

We value your privacy