Page fault on VM

[Isnubi]

Hello !

Since weeks, I got the same error appears on multiple virtual machines on my Proxmox server.
The error appear on the same hosts, but randomly (once a day, fourth a hour, ...).
All the hosts affected are Debian 11 systems, but not all my hosts are affected.
Mostly, I got a Docker platform and my Centreon server that goes down with this error.

Here's one of the error taken from syslog.

Jul 25 10:37:45 docker-debian11 kernel: [ 6152.813206] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.813428] BUG: unable to handle page fault for address: ffffa0d76fda7580
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.813584] #PF: supervisor instruction fetch in kernel mode
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.813730] #PF: error_code(0x0011) - permissions violation
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.813873] PGD 3a401067 P4D 3a401067 PUD 3a405067 PMD 107766063 PTE 800000042fda7063
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.814063] Oops: 0011 [#1] SMP PTI
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.814207] CPU: 3 PID: 416 Comm: dockerd Not tainted 5.10.0-23-amd64 #1 Debian 5.10.179-1
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.814380] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.814596] RIP: 0010:0xffffa0d76fda7580
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.814726] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 da 6f d7 a0 ff ff 01 00 00 00 00 00 00 00 <c0> fc 9d c2 4f b4 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.815067] RSP: 0018:ffffb44fc0367df8 EFLAGS: 00010212
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.815207] RAX: ffffa0d76fda7580 RBX: ffffa0d443c0d420 RCX: ffffa0d4437ae618
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.815362] RDX: ffffb44fc0367ea0 RSI: ffffb44fc0367e10 RDI: ffffa0d443c0d3c0
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.815515] RBP: ffffa0d443c0d410 R08: ffffa0d443c18600 R09: ffffa0d443c0d3c0
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.815668] R10: ffffa0d443c0d410 R11: 0000000000000000 R12: ffffb44fc0367e10
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.815819] R13: 0000000000000000 R14: ffffb44fc0367ea0 R15: ffffa0d443c0d3c0
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.815972] FS:  00007f9324ef9700(0000) GS:ffffa0d76fd80000(0000) knlGS:0000000000000000
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.816088] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.816152] CR2: ffffa0d76fda7580 CR3: 000000010759a000 CR4: 00000000000006e0
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.816225] Call Trace:
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.816302]  ? ep_scan_ready_list.constprop.0+0xab/0x1e0
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.816371]  ? do_epoll_wait+0x247/0x670
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.816423]  ? ep_read_events_proc+0xe0/0xe0
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.816497]  ? __sys_recvmsg+0x89/0xa0
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.816664]  ? ep_unregister_pollwait.constprop.0+0xa0/0xa0
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.816808]  ? __x64_sys_epoll_pwait+0x49/0xb0
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.816946]  ? do_syscall_64+0x33/0x80
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.817079]  ? entry_SYSCALL_64_after_hwframe+0x61/0xc6
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.817232] Modules linked in: veth xt_nat xt_tcpudp xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo nft_counter xt_addrtype nft_compat nf_tables libcrc32c nfnetlink br_netfilter bridge stp llc overlay bochs_drm drm_vram_helper drm_ttm_helper ttm drm_kms_helper evdev joydev sg cec virtio_balloon virtio_console serio_raw button qemu_fw_cfg pcspkr drm fuse configfs ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic hid_generic usbhid hid sd_mod t10_pi crc_t10dif crct10dif_generic crct10dif_common virtio_net net_failover failover virtio_scsi ata_generic uhci_hcd ehci_hcd ata_piix libata usbcore usb_common scsi_mod psmouse virtio_pci virtio_ring virtio i2c_piix4 floppy
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.818582] CR2: ffffa0d76fda7580
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.818709] ---[ end trace 6f455c7d08b8e89c ]---
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.818852] RIP: 0010:0xffffa0d76fda7580
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.818981] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 da 6f d7 a0 ff ff 01 00 00 00 00 00 00 00 <c0> fc 9d c2 4f b4 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.819283] RSP: 0018:ffffb44fc0367df8 EFLAGS: 00010212
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.819380] RAX: ffffa0d76fda7580 RBX: ffffa0d443c0d420 RCX: ffffa0d4437ae618
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.819470] RDX: ffffb44fc0367ea0 RSI: ffffb44fc0367e10 RDI: ffffa0d443c0d3c0
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.819559] RBP: ffffa0d443c0d410 R08: ffffa0d443c18600 R09: ffffa0d443c0d3c0
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.819666] R10: ffffa0d443c0d410 R11: 0000000000000000 R12: ffffb44fc0367e10
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.819784] R13: 0000000000000000 R14: ffffb44fc0367ea0 R15: ffffa0d443c0d3c0
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.819873] FS:  00007f9324ef9700(0000) GS:ffffa0d76fd80000(0000) knlGS:0000000000000000
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.819965] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 25 10:37:45 docker-debian11 kernel: [ 6152.822369] CR2: ffffa0d76fda7580 CR3: 000000010759a000 CR4: 00000000000006e0

And here's the pveversion -v of my Proxmox server

proxmox-ve: 7.4-1 (running kernel: 5.15.108-1-pve)
pve-manager: 7.4-16 (running version: 7.4-16/0f39f621)
pve-kernel-5.15: 7.4-4
pve-kernel-5.15.108-1-pve: 5.15.108-1
pve-kernel-5.15.107-2-pve: 5.15.107-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.2-3
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-3
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
proxmox-backup-client: 2.4.2-1
proxmox-backup-file-restore: 2.4.2-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.7.3
pve-cluster: 7.3-3
pve-container: 4.4-6
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+1
pve-firewall: 4.3-4
pve-firmware: 3.6-5
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.0-8
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-4
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.11-pve1

If I check the KVM of my server, I also see that error printed in the terminal.

[370602.340154] DMAR: [INTER-REMAP] Request device [02:00.0] fault index 0x2f [fault reason 0x26] Blocked an interrupt request due to source-id verification failure

Did anyone have an idea to resolve that?

 

[gurubert]

bad memory?

 

[Isnubi]

bad memory?

I don't have any memory alert on my server
Can a memtest86 can help to see if it came from the physical memory ?

 

[gurubert]

Can a memtest86 can help to see if it came from the physical memory ?

Yes, this test will run for several hours.