"PME: Spurious native interrupt!" Kernel Meldungen

smasty · Jan 6, 2020

Hallo zusammen,

nachdem ich nun den ersten PVE-Knoten von 6.0.9 auf 6.1.5 (und damit Kernel 5.3.13) angehoben habe, wird mein Syslog mit folgenden Meldungen kontinuierlich befüllt:

[...]
Jan 6 10:00:01 kernel: [ 8076.840412] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:04 kernel: [ 8078.958690] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:06 kernel: [ 8080.971962] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:06 kernel: [ 8080.988809] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:09 kernel: [ 8084.103786] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:09 kernel: [ 8084.103963] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:11 kernel: [ 8086.457771] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:16 kernel: [ 8091.345566] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:16 kernel: [ 8091.353984] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:26 kernel: [ 8101.770559] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:27 kernel: [ 8102.724566] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:27 kernel: [ 8102.869279] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:28 kernel: [ 8103.431608] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:31 kernel: [ 8106.841754] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:31 kernel: [ 8106.856727] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:33 kernel: [ 8108.399521] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:38 kernel: [ 8113.399608] pcieport 0000:00:03.0: PME: Spurious native interrupt!
Jan 6 10:00:39 kernel: [ 8114.350334] pcieport 0000:00:03.0: PME: Spurious native interrupt!
[...]

Wie man an den zeitlichen Abständen sieht, kommen da schon viele Log Messages zusammen...

Es geht dabei wohl um folgendes PCI-Device:

# lspci -vv -t|grep 03.0
| +-03.0-[83]--
+-03.0-[04-05]--+-00.0 Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection

Die Meldungen erscheinen allerdings erst im Log, nachdem ich wieder eine VM zurückmigriert habe bzw. dort eine VM läuft.

Ich vermute, dass sich das mit dem/den nächsten Kernel Updates erledigen wird, hat hier jemand ähnliches im Log?

rdfeij · Jan 8, 2020

Same problem here. Since update yesterday from version 5.4.x to 6.1-5
We have the same Intel 10Gbit interfaces.
We had over 64GB of logging in kern.log filling up OS disk

rdfeij · Jan 9, 2020

rdfeij said:
Same problem here. Since update yesterday from version 5.4.x to 6.1-5
We have the same Intel 10Gbit interfaces.
We had over 64GB of logging in kern.log filling up OS disk

This morning our 3th node crashed again (out of diskspace)
After deleting:
/var/log/syslog
/var/log/messages
/var/log/kern.log
and a reboot it is working again.

The spurious interrupt records are filling up logging very fast.
For now solved by creating a cronjob emptying these files every hour.

rdfeij · Feb 6, 2020

@smasty Have you a solution yet?

smasty · Feb 7, 2020

rdfeij said:
@smasty Have you a solution yet?

I have created a filter for rsyslog and wait for a fix.

spyfly · Apr 14, 2020

I am experiencing the same issue with the same network card, as seen below.

Code:

lspci -vv -t|grep 03.2
             +-03.2-[07]--+-00.0  Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection

My syslog looks like this as well:

Code:

[...]
Apr 14 20:43:02 *** kernel: [2413144.982743] pcieport 0000:00:03.2: PME: Spurious native interrupt!
Apr 14 20:43:07 *** kernel: [2413150.297445] pcieport 0000:00:03.2: PME: Spurious native interrupt!
Apr 14 20:43:07 *** kernel: [2413150.505500] pcieport 0000:00:03.2: PME: Spurious native interrupt!
Apr 14 20:43:18 *** kernel: [2413161.499213] pcieport 0000:00:03.2: PME: Spurious native interrupt!
[...]

csutcliff · Jul 15, 2020

Hi,

I have the same issue atm with the 82599ES driver on proxmox. I've found that if you disable interrupt coalescing on the affected cards then the errors stop happening.

sudo ethtool -C <iface> rx-usecs 0

NTS · Jul 16, 2020

Hi
We have the same issue:

Bash:

root@proxmox1:/etc/rsyslog.d# lspci
...
04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
...

dmesg:

Bash:

[Thu Jul 16 15:45:32 2020] pcieport 0000:00:02.2: PME: Spurious native interrupt!

We run Virtual Environment 6.2-10 since today, and still got this messages
any ideas, or does we have run the ethtool command like above?

@csutcliff - did you got any issues after this command ?

csutcliff · Jul 16, 2020

NTS said:
Hi
We have the same issue:

Bash:

root@proxmox1:/etc/rsyslog.d# lspci ... 04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) ...

dmesg:

Bash:

[Thu Jul 16 15:45:32 2020] pcieport 0000:00:02.2: PME: Spurious native interrupt!

We run Virtual Environment 6.2-10 since today, and still got this messages
any ideas, or does we have run the ethtool command like above?

@csutcliff - did you got any issues after this command ?

I haven't had any issue since turning off interrupt coalescing. The expected side effect of doing this is slight cpu usage increase as there will be more interrupts. In my environment this is immeasurably small.

rdfeij · Jul 16, 2020

NTS said:
Hi
We have the same issue:

Bash:

root@proxmox1:/etc/rsyslog.d# lspci ... 04:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 04:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) ...

dmesg:

Bash:

[Thu Jul 16 15:45:32 2020] pcieport 0000:00:02.2: PME: Spurious native interrupt!

We run Virtual Environment 6.2-10 since today, and still got this messages
any ideas, or does we have run the ethtool command like above?

@csutcliff - did you got any issues after this command ?

Run the command above. Keep in mind the interface will reinitialize and disconnect for a few seconds.
It worked for our enviroment. CPU raise not noticable.

NTS · Aug 11, 2020

rdfeij said:
Run the command above. Keep in mind the interface will reinitialize and disconnect for a few seconds.
It worked for our enviroment. CPU raise not noticable.

Hi Rdfeji
Did you ran the command below on the physical or bond interface?

Bash:

sudo ethtool -C <iface> rx-usecs 0

spyfly · Aug 11, 2020

NTS said:
Hi Rdfeji
Did you ran the command below on the physical or bond interface?

Bash:

sudo ethtool -C <iface> rx-usecs 0

I ran it on the physical ones.

NTS · Aug 11, 2020

spyfly said:
I ran it on the physical ones.

Ok fine! and its reboot persistant?

spyfly · Nov 8, 2020

NTS said:
Ok fine! and its reboot persistant?

My workaround is to just run a @reboot crontab with the following cmdline:

Bash:

@reboot /sbin/ethtool -C ens7f0 rx-usecs 0 && /sbin/ethtool -C ens7f1 rx-usecs 0

Don't forget to replace ens7f0 and ens7f1 with your network interfaces

NTS · Nov 9, 2020

fine, @reboot is straight forward.
i did a script as follows - but untested yet!

Bash:

root@proxmox1:/etc/network/if-pre-up.d# cat ethtool-rx-usecs
#!/bin/sh
ETHTOOL=/usr/sbin/ethtool
$ETHTOOL -C eno49 rx-usecs 0
$ETHTOOL -C eno50 rx-usecs 0

And of course, change interfaces as needed

rdfeij · Feb 22, 2021

Funny, i'm still investigating. Last weekend i updated all nodes.
I still have the mentioned temporary fix active (run with Cron @reboot)and after a reboot i saw "some but not many" spurious logs again.
Trying to manual disable interrupts it said "nothing to change" so Cron is working.
After enabling interupts again spurious messages were gone?????
Having to investigate this further and still awaiting fix????

Search

Search

"PME: Spurious native interrupt!" Kernel Meldungen

smasty

Member

rdfeij

Member

rdfeij

Member

rdfeij

Member

smasty

Member

spyfly

Active Member

csutcliff

Member

NTS

Member

csutcliff

Member

rdfeij

Member

NTS

Member

spyfly

Active Member

NTS

Member

spyfly

Active Member

NTS

Member

rdfeij

Member