QEMU/KVM process crashed with a segmentation fault on CPU 7 | Backup Failure.

[SamerXtn]

Hello

i have a backup of 2disk sum 2.3 TB everynight start 00:00 the backup is taking around 5 hours to be done, it was going normally. last night i got backup Failure i went to check i found the VM down, checking the log i found

root@proxmox:~# journalctl --since "2025-02-04 03:00:00" --until "2025-02-04 06:00:00" | grep -i "error\|warning\|fail\|oom\|critical"
Feb 04 03:45:24 proxmox pvestatd[2491]: VM 111 qmp command failed - VM 111 qmp command 'query-proxmox-support' failed - unable to connect to VM 111 qmp socket - timeout after 51 retries
Feb 04 05:03:22 proxmox kernel: kvm[4507]: segfault at 4 ip 000065551de7e597 sp 000077d45d0d0f08 error 6 in qemu-system-x86_64[65551dd82000+6a4000] likely on CPU 7 (core 3, socket 0)
Feb 04 05:03:23 proxmox pvescheduler[1487087]: VM 111 qmp command failed - VM 111 not running
Feb 04 05:03:23 proxmox pvescheduler[1487087]: VM 111 qmp command failed - VM 111 not running
Feb 04 05:03:23 proxmox pvescheduler[1487087]: VM 111 qmp command failed - VM 111 not running
Feb 04 05:03:29 proxmox pvescheduler[1487087]: ERROR: Backup of VM 111 failed - VM 111 not running
Feb 04 05:03:29 proxmox pvescheduler[1487087]: INFO: Backup job finished with errors
Feb 04 05:03:29 proxmox pvescheduler[1487087]: job errors
root@proxmox:~#

root@proxmox:~# dmesg -T | grep "2025-02-04"
root@proxmox:~# ls -la /var/crash/
ls: cannot access '/var/crash/': No such file or directory
root@proxmox:~# journalctl -k --since "2025-02-04 03:00:00" --until "2025-02-04 06:00:00"
Feb 04 05:02:32 proxmox kernel: usb 2-3.1: reset SuperSpeed USB device number 3 using xhci_hcd
Feb 04 05:03:22 proxmox kernel: kvm[4507]: segfault at 4 ip 000065551de7e597 sp 000077d45d0d0f08 error 6 in qemu-system-x86_64[65551dd82000+6a4000] likely o>
Feb 04 05:03:22 proxmox kernel: Code: 8d 3d ad ee 60 00 e8 f8 40 f0 ff 0f 1f 84 00 00 00 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 8b 87 b8 0>
Feb 04 05:03:22 proxmox kernel: fwbr111i0: port 2(tap111i0) entered disabled state
Feb 04 05:03:22 proxmox kernel: tap111i0 (unregistering): left allmulticast mode
Feb 04 05:03:22 proxmox kernel: fwbr111i0: port 2(tap111i0) entered disabled state
Feb 04 05:03:23 proxmox kernel: fwbr111i0: port 1(fwln111i0) entered disabled state
Feb 04 05:03:23 proxmox kernel: vmbr0: port 3(fwpr111p0) entered disabled state
Feb 04 05:03:23 proxmox kernel: fwln111i0 (unregistering): left allmulticast mode
Feb 04 05:03:23 proxmox kernel: fwln111i0 (unregistering): left promiscuous mode
Feb 04 05:03:23 proxmox kernel: fwbr111i0: port 1(fwln111i0) entered disabled state
Feb 04 05:03:23 proxmox kernel: fwpr111p0 (unregistering): left allmulticast mode
Feb 04 05:03:23 proxmox kernel: fwpr111p0 (unregistering): left promiscuous mode
Feb 04 05:03:23 proxmox kernel: vmbr0: port 3(fwpr111p0) entered disabled state
lines 1-14/14 (END)

as you can see Critical Event :

Feb 04 05:03:22 proxmox kernel: kvm[4507]: segfault at 4 ip 000065551de7e597 sp 000077d45d0d0f08 error 6 in qemu-system-x86_64[65551dd82000+6a4000] likely on CPU 7 (core 3, socket 0)

This caused the VM's network interfaces to shut down:

• tap111i0 disabled
• fwbr111i0 ports disabled
• vmbr0 port 3 disabled

and The backup job then failed because it couldn't find the running VM

Vm Config:

/etc/pve/qemu-server/111.conf
agent: enabled=1
bios: ovmf
boot: order=sata0;ide0
cores: 6
cpu: x86-64-v2-AES
efidisk0: local-lvm:vm-111-disk-2,efitype=4m,pre-enrolled-keys=1,size=4M
ide0: local:iso/Virtio-Win-0.1.262_Drivers_08-2024.iso,media=cdrom,size=708140K
ide2: none,media=cdrom
memory: 32768
meta: creation-qemu=9.0.2,ctime=1734169526
name: WindowsServer
net0: virtio=BC:24:11:04:7D:F3,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
sata0: local-lvm:vm-111-disk-0,size=487808M
sata1: local-lvm:vm-111-disk-1,size=1907200M
scsihw: virtio-scsi-single
smbios1: uuid=adea1593-fafd-4281-ac54-322f7eed920d
sockets: 1
usb0: host=2-3.1
vmgenid: c4cd6be6-36bd-4315-bb97-610eb0e6706d

 

[fba]

Hello,
there is no common reason for qemu-system-x86_64 to segfault. Probably some of your hardware is getting faulty and should be replaced. Start with checking the memory with programs like memtest86.

 

[fabian]

this can either be a hardware issue (CPU/ram/mainboard/disk ) or a bug in qemu.. could you also post "pveversion -v"? how long was the VM running before it crashed?