4.15 based test kernel for PVE 5.x available

The 4.15.17-3 test kernel with the intel out of tree drivers fixes the problem I have with my HP DL160 G6. The network is coming up normally again after booting.
 
  • Like
Reactions: micro and tjh
I was unable to boot one of my nodes after upgrading it yesterday to PVE 5.2 and rebooting with the new 4.15 (kernel pve-kernel-4.15.17-2-pve / 4.15.17-10). The booting process was stuck with slowly spitting out some messages:

watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1].
and then Call trace.....

which I can attach if you are interested in, but it is very ugly because of some remote IMM2 console artifacts.

I guess the problem was that I'm having some Intel gigabit cards too?


95:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
95:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)


After two more unsuccessful attempts to boot with 4.15 I changed the default grub boot order and booted successfully with kernel 4.13

Looks like PVE team already have decided that the problem is somewhere in the kernel tree (ixgbe driver) and decided to revert back to the separate (external driver) in a new build of 4.15 in the test repository.

I can't afford to experiment with test repository kernels (except something critical happens), so for now I'm keeping the 4.13. I prefer to wait some time when this build arrives to the enterprise repository I give 4.15 a chance again.
 
We now have a PVE 4.15 kernel with out-of-tree Intel Ethernet drivers available on our community repository. If you have the 'pve-kernel-4.15' meta package installed the update should get in automatically with an
Code:
apt update
apt full-upgrade

The respective package which should get installed is named: pve-kernel-4.15.17-3-pve in version 4.15.17-12

This is a newer version as the manual downloadable kernel from my previous post, as you may have noticed.
It is the same in regards to OOT intel drivers but also includes a backport fix for SUN NICs in combination with Open vSwitch.
So if you installed the manual version and do not use SUN NICs (or OVS) then you may skip this specific version.
 
is a debug package available to help analyzing crashes?
i.e. pve-kernel-4.15.17-2-pve-dbg ?
 
Reposting here https://forum.proxmox.com/threads/i...onnections-over-jumbo-frames-anomalies.44555/

I noticed this thread just now, sorry for the noise ...

========
I have recently routinely updated my 3 node pve 5.2 cluster setup, installing new pve-kernel-4.15.17-3-pve .

I started to have problems in syncronizing drbd resources over a link with jumbo frames enabled:


Jun 14 08:59:26 pve1 kernel: [40906.042440] drbd vm-102-disk-1/0 drbd103 pve3: Began resync as SyncSource (will sync 3476 KB [869 bits set]).
Jun 14 08:59:39 pve1 kernel: [40918.313936] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 6
Jun 14 08:59:45 pve1 kernel: [40924.458069] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 5
Jun 14 08:59:51 pve1 kernel: [40930.602191] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 4
Jun 14 08:59:57 pve1 kernel: [40936.746355] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 3


My configuration uses drbd9 in a dedicated network mesh configuration described here:

https://lists.gt.net/drbd/users/28251#28251

In brief, I put two interfaces in a "drbdbr" bridge on each host, blocking forwarding with ebtables rules.

Each interface has jumbo frames (mtu=9000) enabled.

I suspect the problem is in the update (out of tree) igb driver,

Intel(R) Gigabit Ethernet Linux Driver - version 5.3.5.18

because in the forum I saw there was problems with jumbo frames in the past.

I reverted back to the previous "pve-kernel-4.15.17-2-pve" kernel with in-tree igb driver:

igb: Intel(R) Gigabit Ethernet Network Driver - version 5.4.0-k
to restore correct functionality. Unfortunately removing latest kernel has as side effect the removal of proxmox-ve and pve-kernel-4.15 packages.

Any hint?

Thanks,
rob
 
@Alwin Ehm, I would like to note respectfully that this is not double posting. I re-posted here because I thought was less confusing, not more. I have stated clearly (I guess) in the former thread (as well) that my issue is more appropriate here.
 
Your message here is totally fine, you already excused for the extra noise, Alwin may just have overlooked it.

I'll try to investigate, initial IGB tests looked just fine here, I'll retry high MTU tests.

I've to admit that I really do not like the intel ethernet driver situation at the moment, even if it got already better...
 
Hello Thomas, nice to meet you here ... I think also that @Alwin overlooked it, after all that post seems duplicated a lot :) .

Regarding Intel drivers, I had absolutely nothing to blame igb for till now; 4.15.17-2 (with in-tree igb driver) works flawlessly, and I have the neat feeling that this kernel (thanks to better bridging support?) gave a boost in drbd9 performance (remember that my situation is quite particular with two interfaces in a no-forward bridge for implementing mesh).

I will keep 4.15 series for the moment, will be glad to help in tracking down this ...

bye,
rob
 
Hi Roberto, likewise!

Ah, not the driver code itself directly, it's just that the in vs out of tree model, which the continue to develop simultaneously, makes life not easier for us and PVE admins...
The have often subtle bug differences and the out-of-tree ones are released in an unperiodic manner. Intel people say that the intree one has more features and will be the way to go, in the long run at least.
But it seems we got bitten more by the intree than the outtree one from subtle bugs, so the switch back again...

4.15.17-2 (with in-tree igb driver) works flawlessly,
Hmm, which kernel did you run previously to the 4.15 ones? Because 4.13 and 4.4 (from PVE 4) had all the out of tree driver, like now 4.15.17-3 has again.

o restore correct functionality. Unfortunately removing latest kernel has as side effect the removal of proxmox-ve and pve-kernel-4.15 packages.

You should be able to install an older version of pve-kernel-4.15 meta package to address this
 
Hmm, which kernel did you run previously to the 4.15 ones? Because 4.13 and 4.4 (from PVE 4) had all the out of tree driver, like now 4.15.17-3 has again.

I am under pve-no-subscription, and dist-upgrading quite often, so i passed form 4.13 and then through 4.15.17-1 4.15.17-2 and 4.15.17-3 . Only latest give me unresolvable syncronization problems under drbd9 with "mtu 9000" in place.

You should be able to install an older version of pve-kernel-4.15 meta package to address this

Aaah right!

Anyway at the moment I kept 4.15.17-3 installed and managed to default booting to 4.15.17-2 .

bye,
rob
 
As a further note, I must say that the syncronization problem is very subtle. Most resources syncronize immediately, but two or three are steadily refusing: syncronization starts, goes till 100%, but never ends; in the syslog of the source node I see (quoting from my previous message):

Jun 14 08:59:26 pve1 kernel: [40906.042440] drbd vm-102-disk-1/0 drbd103 pve3: Began resync as SyncSource (will sync 3476 KB [869 bits set]).
Jun 14 08:59:39 pve1 kernel: [40918.313936] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 6
Jun 14 08:59:45 pve1 kernel: [40924.458069] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 5
Jun 14 08:59:51 pve1 kernel: [40930.602191] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 4
Jun 14 08:59:57 pve1 kernel: [40936.746355] drbd vm-102-disk-1 pve3: [drbd_s_vm-102-d/3075] sending time expired, ko = 3

rob
 
Another note: In the least powerful node (used only as quorum and storage redundancy) I had to add a fourth interface, which uses "jme" driver. This one is in bridge with an "igb" one to form the "drbdbr" interface. Will try to change this and put two igb in the bridge and see if something changes.

rob
 
A question: Under BSD the man page of igb suggests some tunables, and a couple are jumbo frame related:

http://manpages.ubuntu.com/manpages/trusty/man4/if_igb.4freebsd.html
kern.ipc.nmbclusters
The maximum number of mbuf clusters allowed. If the system has
more than one igb card or jumbo frames are enabled, this value
will need to be increased.

kern.ipc.nmbjumbo9k
The maximum number of mbuf 9k jumbo clusters allowed. Increasing
this to allow for at least 8192 extra clusters per interface can
allow for an mtu of 8192.
Are there similar tunables for linux os?

rob
 
I am under pve-no-subscription, and dist-upgrading quite often, so i passed form 4.13 and then through 4.15.17-1 4.15.17-2 and 4.15.17-3 . Only latest give me unresolvable syncronization problems under drbd9 with "mtu 9000" in place.

Can confirm problems with the current 4.15 out of tree igb driver and MTUs which aren't 1500 (the default).
I wasn't able to find the source today, I'll continue to investigate next week...


kern.ipc.nmbclusters
kern.ipc.nmbjumbo9k


Are there similar tunables for linux os?

No, not AFAIK.
 
We now have a PVE 4.15 kernel with out-of-tree Intel Ethernet drivers available on our community repository. If you have the 'pve-kernel-4.15' meta package installed the update should get in automatically with an
Code:
apt update
apt full-upgrade

The respective package which should get installed is named: pve-kernel-4.15.17-3-pve in version 4.15.17-12

This is a newer version as the manual downloadable kernel from my previous post, as you may have noticed.
It is the same in regards to OOT intel drivers but also includes a backport fix for SUN NICs in combination with Open vSwitch.
So if you installed the manual version and do not use SUN NICs (or OVS) then you may skip this specific version.

Thank you for the effort but kernel 4.15.17-12 didn't work for me, still the same errors and instability with intel drivers as before
the issue doesn't present after applying the ethtool command solved in the same way as I explained before
 
I am under pve-no-subscription, and dist-upgrading quite often, so i passed form 4.13 and then through 4.15.17-1 4.15.17-2 and 4.15.17-3 . Only latest give me unresolvable syncronization problems under drbd9 with "mtu 9000" in place.

I found an issue which fixes my reproducer here, I reported it upstream with a patch and build a new kernel with the fix included, it was just uploaded to pvetest, you could manually download it from:

http://download.proxmox.com/debian/...pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb
Code:
# md5sum pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb
93ee36afcf147ea4ab9ef0d04ca7e412  pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb
# sha512sum pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb
f8766a267b66614abe35e99ebb68bff6dd3fa01b494ac1b27782029fa6c2ca59c0a75ddd6e95eb649a05ae04000fe9293d491914bac57b3301b6910160fa3c0a  pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb

Thank you for the effort but kernel 4.15.17-12 didn't work for me, still the same errors and instability with intel drivers as before
the issue doesn't present after applying the ethtool command solved in the same way as I explained before

Hmm, no good...
Could you please post the output of:
Code:
ethtool -e eth0 length 256
ethtool -i eth0
(where 'eth0' should be replaced with a problematic e1000e NIC link name)
 
I found an issue which fixes my reproducer here, I reported it upstream with a patch and build a new kernel with the fix included, it was just uploaded to pvetest, you could manually download it from:

http://download.proxmox.com/debian/...pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb
Code:
# md5sum pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb
93ee36afcf147ea4ab9ef0d04ca7e412  pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb
# sha512sum pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb
f8766a267b66614abe35e99ebb68bff6dd3fa01b494ac1b27782029fa6c2ca59c0a75ddd6e95eb649a05ae04000fe9293d491914bac57b3301b6910160fa3c0a  pve-kernel-4.15.17-3-pve_4.15.17-13_amd64.deb

Hmm, no good...
Could you please post the output of:
Code:
ethtool -e eth0 length 256
ethtool -i eth0
(where 'eth0' should be replaced with a problematic e1000e NIC link name)

Sure, here:
Code:
 ethtool -e eth0 length 256
Offset        Values
------        ------
0x0000:        90 1b 0e a3 c4 5f 01 08 ff ff 84 00 8e 00 00 80
0x0010:        ff ff ff ff c3 10 1f 12 34 17 b7 15 00 00 00 00
0x0020:        00 00 00 00 00 80 05 a7 2c 30 00 16 00 00 00 0c
0x0030:        f4 18 42 0a 43 08 13 01 b7 15 ad ba b7 15 b8 15
0x0040:        ad ba b7 15 ad ba b7 15 00 00 80 80 00 4e 86 08
0x0050:        00 00 00 00 07 00 00 20 20 00 00 00 00 0e 00 00
0x0060:        00 01 00 40 0a 01 07 40 ff ff ff ff ff ff ff ff
0x0070:        ff ff ff ff ff ff ff ff ff ff 00 02 ff ff ef f7
0x0080:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0090:        00 00 00 00 00 00 ff ff ff ff ff ff ff ff ff ff
0x00a0:        94 b0 00 08 0a 00 04 90 b0 47 40 24 c2 c1 21 fb
0x00b0:        80 60 1f 00 00 48 10 00 40 60 1f 00 04 d1 11 00
0x00c0:        03 0a 12 00 00 00 1f 00 04 b4 30 00 1c 00 31 00
0x00d0:        06 b4 30 00 09 00 31 00 07 b4 30 00 10 00 31 00
0x00e0:        0a b4 30 00 18 00 31 00 0c b4 30 00 18 00 31 00
0x00f0:        0d b4 30 00 18 00 31 00 01 fd 30 00 2c 9c 31 00
Code:
ethtool -e eth0
Offset        Values
------        ------
0x0000:        90 1b 0e a3 c4 5f 01 08 ff ff 84 00 8e 00 00 80
0x0010:        ff ff ff ff c3 10 1f 12 34 17 b7 15 00 00 00 00
0x0020:        00 00 00 00 00 80 05 a7 2c 30 00 16 00 00 00 0c
0x0030:        f4 18 42 0a 43 08 13 01 b7 15 ad ba b7 15 b8 15
0x0040:        ad ba b7 15 ad ba b7 15 00 00 80 80 00 4e 86 08
0x0050:        00 00 00 00 07 00 00 20 20 00 00 00 00 0e 00 00
0x0060:        00 01 00 40 0a 01 07 40 ff ff ff ff ff ff ff ff
0x0070:        ff ff ff ff ff ff ff ff ff ff 00 02 ff ff ef f7
0x0080:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0090:        00 00 00 00 00 00 ff ff ff ff ff ff ff ff ff ff
0x00a0:        94 b0 00 08 0a 00 04 90 b0 47 40 24 c2 c1 21 fb
0x00b0:        80 60 1f 00 00 48 10 00 40 60 1f 00 04 d1 11 00
0x00c0:        03 0a 12 00 00 00 1f 00 04 b4 30 00 1c 00 31 00
0x00d0:        06 b4 30 00 09 00 31 00 07 b4 30 00 10 00 31 00
0x00e0:        0a b4 30 00 18 00 31 00 0c b4 30 00 18 00 31 00
0x00f0:        0d b4 30 00 18 00 31 00 01 fd 30 00 2c 9c 31 00
0x0100:        4c b6 30 00 05 00 31 00 ff ff ff ff ff ff ff ff
0x0110:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0120:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0130:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0140:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0150:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0160:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0170:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0180:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0190:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x01a0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x01b0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x01c0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x01d0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x01e0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x01f0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0200:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0210:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0220:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0230:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0240:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0250:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0260:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0270:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0280:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0290:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x02a0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x02b0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x02c0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x02d0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x02e0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x02f0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0300:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0310:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0320:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0330:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0340:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0350:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0360:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0370:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0380:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0390:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x03a0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x03b0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x03c0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x03d0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x03e0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x03f0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0400:        69 53 84 03 01 00 00 00 00 00 00 00 00 00 00 00
0x0410:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0420:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0430:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0440:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0450:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0460:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0470:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0480:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0490:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04a0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04b0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04c0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04d0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04e0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x04f0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0500:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0510:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0520:        00 00 00 00 00 00 00 00 03 3d 00 00 00 00 00 00
0x0530:        00 00 00 00 00 00 00 00 00 00 00 00 bc 0c 00 00
0x0540:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0550:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0560:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0570:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0580:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0590:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05a0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05b0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05c0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05d0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05e0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x05f0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0600:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0610:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0620:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0630:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0640:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0650:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0660:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0670:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0680:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0690:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x06a0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x06b0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x06c0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x06d0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x06e0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x06f0:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0700:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0710:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0720:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0730:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0740:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0750:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0760:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0770:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0780:        00 00 00 00 ff ff ff ff ff ff ff ff ff ff ff ff
0x0790:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07a0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07b0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07c0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07d0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07e0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x07f0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0800:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0810:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0820:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0830:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0840:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0850:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0860:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0870:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0880:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0890:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08a0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08b0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08c0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08d0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08e0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x08f0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0900:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0910:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0920:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0930:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0940:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0950:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0960:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0970:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0980:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0990:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09a0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09b0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09c0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09d0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09e0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x09f0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a00:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a10:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a20:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a30:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a40:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a50:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a60:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a70:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a80:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0a90:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0aa0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ab0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ac0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ad0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ae0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0af0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b00:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b10:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b20:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b30:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b40:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b50:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b60:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b70:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b80:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0b90:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ba0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0bb0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0bc0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0bd0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0be0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0bf0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c00:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c10:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c20:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c30:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c40:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c50:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c60:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c70:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c80:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0c90:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ca0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0cb0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0cc0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0cd0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ce0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0cf0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d00:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d10:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d20:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d30:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d40:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d50:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d60:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d70:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d80:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0d90:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0da0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0db0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0dc0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0dd0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0de0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0df0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e00:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e10:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e20:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e30:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e40:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e50:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e60:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e70:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e80:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0e90:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ea0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0eb0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ec0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ed0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ee0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ef0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f00:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f10:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f20:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f30:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f40:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f50:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f60:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f70:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f80:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0f90:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fa0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fb0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fc0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fd0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0fe0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
0x0ff0:        ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

And another server:
Code:
ethtool -e enp0s31f6: length 256
Offset        Values
------        ------
0x0000:        90 1b 0e bb 34 93 01 08 ff ff 84 00 8e 00 00 80
0x0010:        ff ff ff ff c3 10 1f 12 34 17 b7 15 00 00 00 00
0x0020:        00 00 00 00 00 80 05 a7 2c 30 00 16 00 00 00 0c
0x0030:        f4 18 42 0a 43 08 13 01 b7 15 ad ba b7 15 b8 15
0x0040:        ad ba b7 15 ad ba b7 15 00 00 80 80 00 4e 86 08
0x0050:        00 00 00 00 07 00 00 20 20 00 00 00 00 0e 00 00
0x0060:        00 01 00 40 04 01 07 40 ff ff ff ff ff ff ff ff
0x0070:        ff ff ff ff ff ff ff ff ff ff 00 02 ff ff 85 ac
0x0080:        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0090:        00 00 00 00 00 00 ff ff ff ff ff ff ff ff ff ff
0x00a0:        94 b0 00 08 0a 00 04 90 b0 47 40 24 c2 c1 21 fb
0x00b0:        80 60 1f 00 00 48 10 00 40 60 1f 00 04 d1 11 00
0x00c0:        03 0a 12 00 00 00 1f 00 04 b4 30 00 1c 00 31 00
0x00d0:        06 b4 30 00 09 00 31 00 07 b4 30 00 10 00 31 00
0x00e0:        0a b4 30 00 18 00 31 00 0c b4 30 00 18 00 31 00
0x00f0:        0d b4 30 00 18 00 31 00 01 fd 30 00 2c 9c 31 00
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!