4.15 based test kernel for PVE 5.x available

0xFelix · Jun 5, 2018

The 4.15.17-3 test kernel with the intel out of tree drivers fixes the problem I have with my HP DL160 G6. The network is coming up normally again after booting.

micro · Jun 9, 2018

I was unable to boot one of my nodes after upgrading it yesterday to PVE 5.2 and rebooting with the new 4.15 (kernel pve-kernel-4.15.17-2-pve / 4.15.17-10). The booting process was stuck with slowly spitting out some messages:

watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1].

and then Call trace.....

which I can attach if you are interested in, but it is very ugly because of some remote IMM2 console artifacts.

I guess the problem was that I'm having some Intel gigabit cards too?

95:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
95:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)

After two more unsuccessful attempts to boot with 4.15 I changed the default grub boot order and booted successfully with kernel 4.13

Looks like PVE team already have decided that the problem is somewhere in the kernel tree (ixgbe driver) and decided to revert back to the separate (external driver) in a new build of 4.15 in the test repository.

I can't afford to experiment with test repository kernels (except something critical happens), so for now I'm keeping the 4.13. I prefer to wait some time when this build arrives to the enterprise repository I give 4.15 a chance again.

t.lamprecht · Jun 11, 2018

We now have a PVE 4.15 kernel with out-of-tree Intel Ethernet drivers available on our community repository. If you have the 'pve-kernel-4.15' meta package installed the update should get in automatically with an

Code:

apt update
apt full-upgrade

The respective package which should get installed is named: pve-kernel-4.15.17-3-pve in version 4.15.17-12

This is a newer version as the manual downloadable kernel from my previous post, as you may have noticed.
It is the same in regards to OOT intel drivers but also includes a backport fix for SUN NICs in combination with Open vSwitch.
So if you installed the manual version and do not use SUN NICs (or OVS) then you may skip this specific version.

djzort · Jun 14, 2018

is a debug package available to help analyzing crashes?
i.e. pve-kernel-4.15.17-2-pve-dbg ?

resoli · Jun 14, 2018

Reposting here https://forum.proxmox.com/threads/i...onnections-over-jumbo-frames-anomalies.44555/

I noticed this thread just now, sorry for the noise ...

========
I have recently routinely updated my 3 node pve 5.2 cluster setup, installing new pve-kernel-4.15.17-3-pve .

I started to have problems in syncronizing drbd resources over a link with jumbo frames enabled:

Jun 14 08:59:26 pve1 kernel: [40906.042440] drbd vm-102-disk-1/0 drbd103 pve3: Began resync as SyncSource (will sync 3476 KB [869 bits set]).
Jun 14 08:59:39 pve1 kernel: [40918.313936] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 6
Jun 14 08:59:45 pve1 kernel: [40924.458069] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 5
Jun 14 08:59:51 pve1 kernel: [40930.602191] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 4
Jun 14 08:59:57 pve1 kernel: [40936.746355] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 3

My configuration uses drbd9 in a dedicated network mesh configuration described here:

https://lists.gt.net/drbd/users/28251#28251

In brief, I put two interfaces in a "drbdbr" bridge on each host, blocking forwarding with ebtables rules.

Each interface has jumbo frames (mtu=9000) enabled.

I suspect the problem is in the update (out of tree) igb driver,

Intel(R) Gigabit Ethernet Linux Driver - version 5.3.5.18

because in the forum I saw there was problems with jumbo frames in the past.

I reverted back to the previous "pve-kernel-4.15.17-2-pve" kernel with in-tree igb driver:

igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
to restore correct functionality. Unfortunately removing latest kernel has as side effect the removal of proxmox-ve and pve-kernel-4.15 packages.

Any hint?

Thanks,
rob

Alwin · Jun 14, 2018

@resoli, please don't double post, this makes it just confusing for people to help you.
https://forum.proxmox.com/threads/i...onnections-over-jumbo-frames-anomalies.44555/

resoli · Jun 14, 2018

@Alwin Ehm, I would like to note respectfully that this is not double posting. I re-posted here because I thought was less confusing, not more. I have stated clearly (I guess) in the former thread (as well) that my issue is more appropriate here.

t.lamprecht · Jun 14, 2018

Your message here is totally fine, you already excused for the extra noise, Alwin may just have overlooked it.

I'll try to investigate, initial IGB tests looked just fine here, I'll retry high MTU tests.

I've to admit that I really do not like the intel ethernet driver situation at the moment, even if it got already better...

resoli · Jun 14, 2018

Hello Thomas, nice to meet you here ... I think also that @Alwin overlooked it, after all that post seems duplicated a lot

.

Regarding Intel drivers, I had absolutely nothing to blame igb for till now; 4.15.17-2 (with in-tree igb driver) works flawlessly, and I have the neat feeling that this kernel (thanks to better bridging support?) gave a boost in drbd9 performance (remember that my situation is quite particular with two interfaces in a no-forward bridge for implementing mesh).

I will keep 4.15 series for the moment, will be glad to help in tracking down this ...

bye,
rob

t.lamprecht · Jun 15, 2018

Hi Roberto, likewise!

Ah, not the driver code itself directly, it's just that the in vs out of tree model, which the continue to develop simultaneously, makes life not easier for us and PVE admins...
The have often subtle bug differences and the out-of-tree ones are released in an unperiodic manner. Intel people say that the intree one has more features and will be the way to go, in the long run at least.
But it seems we got bitten more by the intree than the outtree one from subtle bugs, so the switch back again...

resoli said:
4.15.17-2 (with in-tree igb driver) works flawlessly,

Hmm, which kernel did you run previously to the 4.15 ones? Because 4.13 and 4.4 (from PVE 4) had all the out of tree driver, like now 4.15.17-3 has again.

resoli said:
o restore correct functionality. Unfortunately removing latest kernel has as side effect the removal of proxmox-ve and pve-kernel-4.15 packages.

You should be able to install an older version of pve-kernel-4.15 meta package to address this

resoli · Jun 15, 2018

t.lamprecht said:
Hmm, which kernel did you run previously to the 4.15 ones? Because 4.13 and 4.4 (from PVE 4) had all the out of tree driver, like now 4.15.17-3 has again.

I am under pve-no-subscription, and dist-upgrading quite often, so i passed form 4.13 and then through 4.15.17-1 4.15.17-2 and 4.15.17-3 . Only latest give me unresolvable syncronization problems under drbd9 with "mtu 9000" in place.

t.lamprecht said:
You should be able to install an older version of pve-kernel-4.15 meta package to address this

Aaah right!

Anyway at the moment I kept 4.15.17-3 installed and managed to default booting to 4.15.17-2 .

bye,
rob

resoli · Jun 15, 2018

As a further note, I must say that the syncronization problem is very subtle. Most resources syncronize immediately, but two or three are steadily refusing: syncronization starts, goes till 100%, but never ends; in the syslog of the source node I see (quoting from my previous message):

resoli said:
Jun 14 08:59:26 pve1 kernel: [40906.042440] drbd vm-102-disk-1/0 drbd103 pve3: Began resync as SyncSource (will sync 3476 KB [869 bits set]).
Jun 14 08:59:39 pve1 kernel: [40918.313936] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 6
Jun 14 08:59:45 pve1 kernel: [40924.458069] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 5
Jun 14 08:59:51 pve1 kernel: [40930.602191] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 4
Jun 14 08:59:57 pve1 kernel: [40936.746355] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 3

rob

resoli · Jun 15, 2018

Another note: In the least powerful node (used only as quorum and storage redundancy) I had to add a fourth interface, which uses "jme" driver. This one is in bridge with an "igb" one to form the "drbdbr" interface. Will try to change this and put two igb in the bridge and see if something changes.

rob

resoli · Jun 15, 2018

A question: Under BSD the man page of igb suggests some tunables, and a couple are jumbo frame related:

http://manpages.ubuntu.com/manpages/trusty/man4/if_igb.4freebsd.html

kern.ipc.nmbclusters
The maximum number of mbuf clusters allowed. If the system has
more than one igb card or jumbo frames are enabled, this value
will need to be increased.

kern.ipc.nmbjumbo9k
The maximum number of mbuf 9k jumbo clusters allowed. Increasing
this to allow for at least 8192 extra clusters per interface can
allow for an mtu of 8192.

Are there similar tunables for linux os?

rob

t.lamprecht · Jun 15, 2018

resoli said:
I am under pve-no-subscription, and dist-upgrading quite often, so i passed form 4.13 and then through 4.15.17-1 4.15.17-2 and 4.15.17-3 . Only latest give me unresolvable syncronization problems under drbd9 with "mtu 9000" in place.

Can confirm problems with the current 4.15 out of tree igb driver and MTUs which aren't 1500 (the default).
I wasn't able to find the source today, I'll continue to investigate next week...

resoli said:
kern.ipc.nmbclusters
kern.ipc.nmbjumbo9k

Are there similar tunables for linux os?

No, not AFAIK.

resoli · Jun 15, 2018

t.lamprecht said:
Can confirm problems with the current 4.15 out of tree igb driver and MTUs which aren't 1500 (the default).
I wasn't able to find the source today, I'll continue to investigate next week...

Great.

t.lamprecht said:
No, not AFAIK.

Ok, many thanks, have a nice we.

rob

Antonio Blanco · Jun 18, 2018

t.lamprecht said:
We now have a PVE 4.15 kernel with out-of-tree Intel Ethernet drivers available on our community repository. If you have the 'pve-kernel-4.15' meta package installed the update should get in automatically with an

Code:

apt update apt full-upgrade

The respective package which should get installed is named: pve-kernel-4.15.17-3-pve in version 4.15.17-12

This is a newer version as the manual downloadable kernel from my previous post, as you may have noticed.
It is the same in regards to OOT intel drivers but also includes a backport fix for SUN NICs in combination with Open vSwitch.
So if you installed the manual version and do not use SUN NICs (or OVS) then you may skip this specific version.

Thank you for the effort but kernel 4.15.17-12 didn't work for me, still the same errors and instability with intel drivers as before
the issue doesn't present after applying the ethtool command solved in the same way as I explained before

t.lamprecht · Jun 19, 2018

resoli said:
I am under pve-no-subscription, and dist-upgrading quite often, so i passed form 4.13 and then through 4.15.17-1 4.15.17-2 and 4.15.17-3 . Only latest give me unresolvable syncronization problems under drbd9 with "mtu 9000" in place.

I found an issue which fixes my reproducer here, I reported it upstream with a patch and build a new kernel with the fix included, it was just uploaded to pvetest, you could manually download it from:

http://download.proxmox.com/debian/...pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb

Code:

# md5sum pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb
93ee36afcf147ea4ab9ef0d04ca7e412  pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb
# sha512sum pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb
f8766a267b66614abe35e99ebb68bff6dd3fa01b494ac1b27782029fa6c2ca59c0a75ddd6e95eb649a05ae04000fe9293d491914bac57b3301b6910160fa3c0a  pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb

Antonio Blanco said:
Thank you for the effort but kernel 4.15.17-12 didn't work for me, still the same errors and instability with intel drivers as before
the issue doesn't present after applying the ethtool command solved in the same way as I explained before

Hmm, no good...
Could you please post the output of:

Code:

ethtool -e eth0 length 256
ethtool -i eth0

(where 'eth0' should be replaced with a problematic e1000e NIC link name)

taf_321 · Jun 19, 2018

Hello!

t.lamprecht said:
I found an issue which fixes my reproducer here, I reported it upstream with a patch and build a new kernel with the fix included, it was just uploaded to pvetest, you could manually download it from:

With version pve-kernel-4.15.17-3-pve_4.15.17-13, my problem https://forum.proxmox.com/threads/jumbo-frame-pve-kernel-4-15-17-3-pve-not-work.44654/ was solved.

Antonio Blanco · Jun 19, 2018

t.lamprecht said:
I found an issue which fixes my reproducer here, I reported it upstream with a patch and build a new kernel with the fix included, it was just uploaded to pvetest, you could manually download it from:

http://download.proxmox.com/debian/...pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb

Code:

# md5sum pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb 93ee36afcf147ea4ab9ef0d04ca7e412 pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb # sha512sum pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb f8766a267b66614abe35e99ebb68bff6dd3fa01b494ac1b27782029fa6c2ca59c0a75ddd6e95eb649a05ae04000fe9293d491914bac57b3301b6910160fa3c0a pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb

Hmm, no good...
Could you please post the output of:

Code:

ethtool -e eth0 length 256 ethtool -i eth0

(where 'eth0' should be replaced with a problematic e1000e NIC link name)

Sure, here:

Code:

 ethtool -e eth0 length 256
Offset        Values
------        ------
0x0000:        90 1b 0e a3 c4 5f 01 08 ff ff 84 00 8e 00 00 80
0x0010:        ff ff ff ff c3 10 1f 12 34 17 b7 15 00 00 00 00
0x0020:        00 00 00 00 00 80 05 a7 2c 30 00 16 00 00 00 0c
0x0030:        f4 18 42 0a 43 08 13 01 b7 15 ad ba b7 15 b8 15
0x0040:        ad ba b7 15 ad ba b7 15 00 00 80 80 00 4e 86 08
0x0050:        00 00 00 00 07 00 00 20 20 00 00 00 00 0e 00 00
0x0060:        00 01 00 40 0a 01 07 40 ff ff ff ff ff ff ff ff
0x0070:        ff ff ff ff ff ff ff ff ff ff 00 02 ff ff ef f7
0x0080:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0090:        00 00 00 00 00 00 ff ff ff ff ff ff ff ff ff ff
0x00a0:        94 b0 00 08 0a 00 04 90 b0 47 40 24 c2 c1 21 fb
0x00b0:        80 60 1f 00 00 48 10 00 40 60 1f 00 04 d1 11 00
0x00c0:        03 0a 12 00 00 00 1f 00 04 b4 30 00 1c 00 31 00
0x00d0:        06 b4 30 00 09 00 31 00 07 b4 30 00 10 00 31 00
0x00e0:        0a b4 30 00 18 00 31 00 0c b4 30 00 18 00 31 00
0x00f0:        0d b4 30 00 18 00 31 00 01 fd 30 00 2c 9c 31 00

Code:

ethtool -e eth0
Offset        Values
------        ------
0x0000:        90 1b 0e a3 c4 5f 01 08 ff ff 84 00 8e 00 00 80
0x0010:        ff ff ff ff c3 10 1f 12 34 17 b7 15 00 00 00 00
0x0020:        00 00 00 00 00 80 05 a7 2c 30 00 16 00 00 00 0c
0x0030:        f4 18 42 0a 43 08 13 01 b7 15 ad ba b7 15 b8 15
0x0040:        ad ba b7 15 ad ba b7 15 00 00 80 80 00 4e 86 08
0x0050:        00 00 00 00 07 00 00 20 20 00 00 00 00 0e 00 00
0x0060:        00 01 00 40 0a 01 07 40 ff ff ff ff ff ff ff ff
0x0070:        ff ff ff ff ff ff ff ff ff ff 00 02 ff ff ef f7
0x0080:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0090:        00 00 00 00 00 00 ff ff ff ff ff ff ff ff ff ff
0x00a0:        94 b0 00 08 0a 00 04 90 b0 47 40 24 c2 c1 21 fb
0x00b0:        80 60 1f 00 00 48 10 00 40 60 1f 00 04 d1 11 00
0x00c0:        03 0a 12 00 00 00 1f 00 04 b4 30 00 1c 00 31 00
0x00d0:        06 b4 30 00 09 00 31 00 07 b4 30 00 10 00 31 00
0x00e0:        0a b4 30 00 18 00 31 00 0c b4 30 00 18 00 31 00
0x00f0:        0d b4 30 00 18 00 31 00 01 fd 30 00 2c 9c 31 00
0x0100:        4c b6 30 00 05 00 31 00 ff ff ff ff ff ff ff ff
0x0110:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0120:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0130:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0140:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0150:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0160:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0170:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0180:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0190:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x01a0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x01b0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x01c0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x01d0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x01e0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x01f0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0200:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0210:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0220:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0230:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0240:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0250:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0260:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0270:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0280:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0290:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x02a0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x02b0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x02c0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x02d0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x02e0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x02f0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0300:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0310:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0320:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0330:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0340:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0350:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0360:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0370:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0380:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0390:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x03a0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x03b0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x03c0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x03d0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x03e0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x03f0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0400:        69 53 84 03 01 00 00 00 00 00 00 00 00 00 00 00
0x0410:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0420:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0430:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0440:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0450:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0460:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0470:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0480:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0490:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04a0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04b0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04c0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04d0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04e0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04f0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0500:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0510:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0520:        00 00 00 00 00 00 00 00 03 3d 00 00 00 00 00 00
0x0530:        00 00 00 00 00 00 00 00 00 00 00 00 bc 0c 00 00
0x0540:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0550:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0560:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0570:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0580:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0590:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05a0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05b0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05c0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05d0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05e0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05f0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0600:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0610:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0620:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0630:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0640:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0650:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0660:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0670:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0680:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0690:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x06a0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x06b0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x06c0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x06d0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x06e0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x06f0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0700:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0710:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0720:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0730:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0740:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0750:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0760:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0770:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0780:        00 00 00 00 ff ff ff ff ff ff ff ff ff ff ff ff
0x0790:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07a0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07b0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07c0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07d0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07e0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07f0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0800:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0810:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0820:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0830:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0840:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0850:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0860:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0870:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0880:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0890:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08a0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08b0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08c0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08d0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08e0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08f0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0900:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0910:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0920:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0930:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0940:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0950:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0960:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0970:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0980:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0990:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09a0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09b0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09c0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09d0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09e0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09f0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a00:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a10:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a20:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a30:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a40:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a50:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a60:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a70:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a80:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a90:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0aa0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ab0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ac0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ad0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ae0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0af0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b00:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b10:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b20:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b30:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b40:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b50:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b60:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b70:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b80:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b90:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ba0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0bb0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0bc0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0bd0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0be0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0bf0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c00:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c10:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c20:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c30:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c40:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c50:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c60:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c70:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c80:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c90:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ca0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0cb0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0cc0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0cd0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ce0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0cf0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d00:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d10:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d20:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d30:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d40:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d50:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d60:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d70:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d80:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d90:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0da0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0db0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0dc0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0dd0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0de0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0df0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e00:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e10:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e20:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e30:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e40:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e50:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e60:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e70:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e80:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e90:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ea0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0eb0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ec0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ed0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ee0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ef0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f00:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f10:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f20:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f30:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f40:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f50:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f60:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f70:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f80:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f90:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fa0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fb0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fc0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fd0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fe0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ff0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

And another server:

Code:

ethtool -e enp0s31f6: length 256
Offset        Values
------        ------
0x0000:        90 1b 0e bb 34 93 01 08 ff ff 84 00 8e 00 00 80
0x0010:        ff ff ff ff c3 10 1f 12 34 17 b7 15 00 00 00 00
0x0020:        00 00 00 00 00 80 05 a7 2c 30 00 16 00 00 00 0c
0x0030:        f4 18 42 0a 43 08 13 01 b7 15 ad ba b7 15 b8 15
0x0040:        ad ba b7 15 ad ba b7 15 00 00 80 80 00 4e 86 08
0x0050:        00 00 00 00 07 00 00 20 20 00 00 00 00 0e 00 00
0x0060:        00 01 00 40 04 01 07 40 ff ff ff ff ff ff ff ff
0x0070:        ff ff ff ff ff ff ff ff ff ff 00 02 ff ff 85 ac
0x0080:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0090:        00 00 00 00 00 00 ff ff ff ff ff ff ff ff ff ff
0x00a0:        94 b0 00 08 0a 00 04 90 b0 47 40 24 c2 c1 21 fb
0x00b0:        80 60 1f 00 00 48 10 00 40 60 1f 00 04 d1 11 00
0x00c0:        03 0a 12 00 00 00 1f 00 04 b4 30 00 1c 00 31 00
0x00d0:        06 b4 30 00 09 00 31 00 07 b4 30 00 10 00 31 00
0x00e0:        0a b4 30 00 18 00 31 00 0c b4 30 00 18 00 31 00
0x00f0:        0d b4 30 00 18 00 31 00 01 fd 30 00 2c 9c 31 00

4.15 based test kernel for PVE 5.x available

Member

Renowned Member

Proxmox Staff Member

Member

Renowned Member

Proxmox Retired Staff

Renowned Member

Proxmox Staff Member

Renowned Member

Proxmox Staff Member

Renowned Member

Renowned Member

Renowned Member

Renowned Member

Proxmox Staff Member

Renowned Member

Member

Proxmox Staff Member

New Member

Member