QM running (internal-error)

[kriznik]

Hello,
after years of running on proxmox, I've experienced from yesterday issue with one of my VM's which stopped responding.
It showed "running (internal-error) on the UI

whole pvestatsd needed to be restarted in order to see statuses of the remaining VM's (but that might be side effect).
when performing backup to the Proxmox Backup server Host completely frozen.
I've disabled backup and host is now running even when that given VM is in internal error.

no clue what happened. Found this in the syslog, can somebody please help me decode what that means?
Host has microcode installed and been running fine for long time.

Sep 29 04:15:42 inception QEMU[1564]: error: kvm run failed Invalid argument

Thanks a lot

Sep 29 03:46:54 inception smartd[670]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 68 to 69
Sep 29 04:13:25 inception kernel: perf: interrupt took too long (3985 > 3950), lowering kernel.perf_event_max_sample_rate to 50000
Sep 29 04:15:42 inception kernel: get_mmio_spte: reserved bits set on MMU-present spte, addr 0x469c4d24, hierarchy:
Sep 29 04:15:42 inception kernel: ------ spte = 0x108387907 level = 4, rsvd bits = 0xfff80000000f8
Sep 29 04:15:42 inception kernel: ------ spte = 0x109e3c907 level = 3, rsvd bits = 0xfff8000000078
Sep 29 04:15:42 inception kernel: ------ spte = 0x3807f3907 level = 2, rsvd bits = 0xfff8000000078
Sep 29 04:15:42 inception kernel: ------ spte = 0x6000101d6d7eb77 level = 1, rsvd bits = 0xfff8000000000
Sep 29 04:15:42 inception kernel: ------------[ cut here ]------------
Sep 29 04:15:42 inception kernel: WARNING: CPU: 0 PID: 1626 at arch/x86/kvm/mmu/mmu.c:4099 kvm_mmu_page_fault+0x7f3/0xb40 [kvm]
Sep 29 04:15:42 inception kernel: Modules linked in: wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_comment nft_compat 8021q garp mrp rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache netfs veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc binfmt_misc nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common iwlmvm x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_hdmi mac80211 kvm_intel libarc4 mei_pxp mei_hdcp i915 iwlwifi snd_hda_codec_realtek drm_buddy snd_hda_codec_generic ttm ledtrig_audio kvm drm_display_helper snd_hda_intel irqbypass snd_intel_dspcfg snd_intel_sdw_acpi crct10dif_pclmul btusb polyval_generic cec ghash_clmulni_intel btrtl cryptd rc_core btbcm snd_hda_codec sha512_ssse3 btintel btmtk rapl snd_hda_core
Sep 29 04:15:42 inception kernel:  mei_me drm_kms_helper cfg80211 bluetooth i2c_algo_bit spi_nor pcspkr snd_hwdep intel_cstate at24 syscopyarea sysfillrect ecdh_generic mtd snd_pcm mei ecc snd_timer sysimgblt snd soundcore acpi_pad mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap efi_pstore drm dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c xhci_pci spi_intel_platform xhci_pci_renesas spi_intel r8169 ehci_pci crc32_pclmul xhci_hcd ahci ehci_hcd i2c_i801 realtek i2c_smbus libahci lpc_ich video wmi
Sep 29 04:15:42 inception kernel: CPU: 0 PID: 1626 Comm: CPU 1/KVM Tainted: P           O       6.2.16-14-pve #1
Sep 29 04:15:42 inception kernel: Hardware name: GIGABYTE GB-BXCE-3205/MQLPCAP-00, BIOS F4 12/08/2015
Sep 29 04:15:42 inception kernel: RIP: 0010:kvm_mmu_page_fault+0x7f3/0xb40 [kvm]
Sep 29 04:15:42 inception kernel: Code: 9b 48 01 c8 49 8b 8c c7 b0 00 00 00 49 83 fc 05 0f 87 b7 02 00 00 4c 89 c6 48 c7 c7 e8 3d 5e c1 e8 02 6d 7d f7 45 39 f5 7e a3 <0f> 0b b8 ea ff ff ff e9 2f fa ff ff 0f 0b 4c 89 e7 44 89 9d 30 ff
Sep 29 04:15:42 inception kernel: RSP: 0018:ffffbda8c313bb70 EFLAGS: 00010202
Sep 29 04:15:42 inception kernel: RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000000
Sep 29 04:15:42 inception kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Sep 29 04:15:42 inception kernel: RBP: ffffbda8c313bc60 R08: 0000000000000000 R09: 0000000000000000
Sep 29 04:15:42 inception kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
Sep 29 04:15:42 inception kernel: R13: 0000000000000001 R14: 0000000000000000 R15: ffffa090cc954a90
Sep 29 04:15:42 inception kernel: FS:  00007f3456bbb6c0(0000) GS:ffffa093cee00000(0000) knlGS:ffff968f3bd00000
Sep 29 04:15:42 inception kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 29 04:15:42 inception kernel: CR2: 00007fc66f842000 CR3: 000000010b4b8005 CR4: 00000000000726f0
Sep 29 04:15:42 inception kernel: Call Trace:
Sep 29 04:15:42 inception kernel:  <TASK>
Sep 29 04:15:42 inception kernel:  ? show_regs+0x6d/0x80
Sep 29 04:15:42 inception kernel:  ? __warn+0x89/0x160
Sep 29 04:15:42 inception kernel:  ? kvm_mmu_page_fault+0x7f3/0xb40 [kvm]
Sep 29 04:15:42 inception kernel:  ? report_bug+0x17e/0x1b0
Sep 29 04:15:42 inception kernel:  ? handle_bug+0x46/0x90
Sep 29 04:15:42 inception kernel:  ? exc_invalid_op+0x18/0x80
Sep 29 04:15:42 inception kernel:  ? asm_exc_invalid_op+0x1b/0x20
Sep 29 04:15:42 inception kernel:  ? kvm_mmu_page_fault+0x7f3/0xb40 [kvm]
Sep 29 04:15:42 inception kernel:  ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
Sep 29 04:15:42 inception kernel:  handle_ept_misconfig+0x4f/0x1d0 [kvm_intel]
Sep 29 04:15:42 inception kernel:  vmx_handle_exit+0x90d/0xa40 [kvm_intel]
Sep 29 04:15:42 inception kernel:  kvm_arch_vcpu_ioctl_run+0xe02/0x1750 [kvm]
Sep 29 04:15:42 inception kernel:  ? vmx_vcpu_put+0x134/0x220 [kvm_intel]
Sep 29 04:15:42 inception kernel:  ? restore_fpregs_from_fpstate+0x47/0xf0
Sep 29 04:15:42 inception kernel:  ? kvm_arch_vcpu_put+0x1a6/0x200 [kvm]
Sep 29 04:15:42 inception kernel:  kvm_vcpu_ioctl+0x297/0x7c0 [kvm]
Sep 29 04:15:42 inception kernel:  ? __fget_light+0xa5/0x120
Sep 29 04:15:42 inception kernel:  __x64_sys_ioctl+0xa0/0xe0
Sep 29 04:15:42 inception kernel:  do_syscall_64+0x5b/0x90
Sep 29 04:15:42 inception kernel:  ? fire_user_return_notifiers+0x3a/0x80
Sep 29 04:15:42 inception kernel:  ? exit_to_user_mode_prepare+0x39/0x190
Sep 29 04:15:42 inception kernel:  ? syscall_exit_to_user_mode+0x37/0x60
Sep 29 04:15:42 inception kernel:  ? do_syscall_64+0x67/0x90
Sep 29 04:15:42 inception kernel:  ? do_syscall_64+0x67/0x90
Sep 29 04:15:42 inception kernel:  ? syscall_exit_to_user_mode+0x37/0x60
Sep 29 04:15:42 inception kernel:  ? do_syscall_64+0x67/0x90
Sep 29 04:15:42 inception kernel:  ? do_syscall_64+0x67/0x90
Sep 29 04:15:42 inception kernel:  entry_SYSCALL_64_after_hwframe+0x73/0xdd
Sep 29 04:15:42 inception kernel: RIP: 0033:0x7f345fa77b3b
Sep 29 04:15:42 inception kernel: Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
Sep 29 04:15:42 inception kernel: RSP: 002b:00007f3456bb5fb0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Sep 29 04:15:42 inception kernel: RAX: ffffffffffffffda RBX: 00005611284fbc60 RCX: 00007f345fa77b3b
Sep 29 04:15:42 inception kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000001d
Sep 29 04:15:42 inception kernel: RBP: 000000000000ae80 R08: 000056112696ce00 R09: 0000000000000000
Sep 29 04:15:42 inception kernel: R10: 0000000000000002 R11: 0000000000000246 R12: 0000000000000000
Sep 29 04:15:42 inception kernel: R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000000
Sep 29 04:15:42 inception kernel:  </TASK>
Sep 29 04:15:42 inception kernel: ---[ end trace 0000000000000000 ]---
Sep 29 04:15:42 inception QEMU[1564]: error: kvm run failed Invalid argument
Sep 29 04:15:42 inception QEMU[1564]: RAX=0000000000265b0a RBX=0000000000003ae0 RCX=0000000000000017 RDX=0000000000004016
Sep 29 04:15:42 inception QEMU[1564]: RSI=00000000c9bc4d18 RDI=00007fc69ee43c40 RBP=00007fc6698fe098 RSP=00007fc6698fdac0
Sep 29 04:15:42 inception QEMU[1564]: R8 =000000000000012c R9 =00000000c905e548 R10=0000000100551738 R11=000000002004ddcd
Sep 29 04:15:42 inception QEMU[1564]: R12=0000000000000000 R13=0000000000000001 R14=00007fc6698fd670 R15=00007fc68002ad80
Sep 29 04:15:42 inception QEMU[1564]: RIP=00007fc6bb4bb026 RFL=00010202 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
Sep 29 04:15:42 inception QEMU[1564]: ES =0000 0000000000000000 ffffffff 00c00000
Sep 29 04:15:42 inception QEMU[1564]: CS =0033 0000000000000000 ffffffff 00a0fb00 DPL=3 CS64 [-RA]
Sep 29 04:15:42 inception QEMU[1564]: SS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
Sep 29 04:15:42 inception QEMU[1564]: DS =0000 0000000000000000 ffffffff 00c00000
Sep 29 04:15:42 inception QEMU[1564]: FS =0000 00007fc6698ff6c0 ffffffff 00c00000
Sep 29 04:15:42 inception QEMU[1564]: GS =0000 0000000000000000 ffffffff 00c00000
Sep 29 04:15:42 inception QEMU[1564]: LDT=0000 0000000000000000 ffffffff 00c00000
Sep 29 04:15:42 inception QEMU[1564]: TR =0040 fffffe000003e000 00004087 00008b00 DPL=0 TSS64-busy
Sep 29 04:15:42 inception QEMU[1564]: GDT=     fffffe000003c000 0000007f
Sep 29 04:15:42 inception QEMU[1564]: IDT=     fffffe0000000000 00000fff
Sep 29 04:15:42 inception QEMU[1564]: CR0=80050033 CR2=00007fc66f842000 CR3=0000000102f6d000 CR4=000006e0
Sep 29 04:15:42 inception QEMU[1564]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
Sep 29 04:15:42 inception QEMU[1564]: DR6=00000000ffff0ff0 DR7=0000000000000400
Sep 29 04:15:42 inception QEMU[1564]: EFER=0000000000000d01
Sep 29 04:15:42 inception QEMU[1564]: Code=9f f4 00 00 00 81 e3 fe ff 1f 00 83 fb 00 0f 84 26 e1 00 00 <8b> 76 0c 48 3b 06 48 8b fe 48 bb 40 3c e4 9e c6 7f 00 00 8b 7f 08 48 c1 e7 03 48 3b bb 48
Sep 29 04:16:54 inception smartd[670]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 69 to 68
Sep 29 04:17:01 inception CRON[30258]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 29 04:17:01 inception CRON[30259]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)

 

[fiona]

Hi,
please share the output of pveversion -v and the VM configuration qm config <ID> --current. If the issue occurs again, you might want to try booting an older kernel to see if that makes a difference.

 

[kriznik]

Thanks for the reply
here it is.
(btw frozen again completely, with backup disabled, so my initial thoughts was wrong, and host just frozen again)

root@inception:~# pveversion -v                                                                                                                                                                         
proxmox-ve: 8.0.2 (running kernel: 6.2.16-14-pve)                                                                                                                                                       
pve-manager: 8.0.4 (running version: 8.0.4/d258a813cfa6b390)                                                                                                                                             
pve-kernel-6.2: 8.0.5                                                                                                                                                                                   
proxmox-kernel-helper: 8.0.3                                                                                                                                                                             
proxmox-kernel-6.2.16-14-pve: 6.2.16-14                                                                                                                                                                 
proxmox-kernel-6.2: 6.2.16-14                                                                                                                                                                           
ceph-fuse: 17.2.6-pve1+3                                                                                                                                                                                 
corosync: 3.1.7-pve3                                                                                                                                                                                     
criu: 3.17.1-2                                                                                                                                                                                           
glusterfs-client: 10.3-5                                                                                                                                                                                 
ifupdown2: 3.2.0-1+pmx4                                                                                                                                                                                 
ksm-control-daemon: 1.4-1                                                                                                                                                                               
libjs-extjs: 7.0.0-4                                                                                                                                                                                     
libknet1: 1.25-pve1                                                                                                                                                                                     
libproxmox-acme-perl: 1.4.6                                                                                                                                                                             
libproxmox-backup-qemu0: 1.4.0                                                                                                                                                                           
libproxmox-rs-perl: 0.3.1                                                                                                                                                                               
libpve-access-control: 8.0.5                                                                                                                                                                             
libpve-apiclient-perl: 3.3.1                                                                                                                                                                             
libpve-common-perl: 8.0.8                                                                                                                                                                               
libpve-guest-common-perl: 5.0.4                                                                                                                                                                         
libpve-http-server-perl: 5.0.4                                                                                                                                                                           
libpve-rs-perl: 0.8.5                                                                                                                                                                                   
libpve-storage-perl: 8.0.2                                                                                                                                                                               
libspice-server1: 0.15.1-1                                                                                                                                                                               
lvm2: 2.03.16-2                                                                                                                                                                                         
lxc-pve: 5.0.2-4                                                                                                                                                                                         
lxcfs: 5.0.3-pve3                                                                                                                                                                                       
novnc-pve: 1.4.0-2                                                                                                                                                                                       
proxmox-backup-client: 3.0.2-1                                                                                                                                                                           
proxmox-backup-file-restore: 3.0.2-1                                                                                                                                                                     
proxmox-kernel-helper: 8.0.3                                                                                                                                                                             
proxmox-mail-forward: 0.2.0                                                                                                                                                                             
proxmox-mini-journalreader: 1.4.0                                                                                                                                                                       
proxmox-widget-toolkit: 4.0.6                                                                                                                                                                           
pve-cluster: 8.0.3                                                                                                                                                                                       
pve-container: 5.0.4                                                                                                                                                                                     
pve-docs: 8.0.4                                                                                                                                                                                         
pve-edk2-firmware: 3.20230228-4                                                                                                                                                                         
pve-firewall: 5.0.3                                                                                                                                                                                     
pve-firmware: 3.8-2                                                                                                                                                                                     
pve-ha-manager: 4.0.2                                                                                                                                                                                   
pve-i18n: 3.0.7                                                                                                                                                                                         
pve-qemu-kvm: 8.0.2-6                                                                                                                                                                                   
pve-xtermjs: 4.16.0-3                                                                                                                                                                                   
qemu-server: 8.0.7                                                                                                                                                                                       
smartmontools: 7.3-pve1                                                                                                                                                                                 
spiceterm: 3.3.0                                                                                                                                                                                         
swtpm: 0.8.0+pve1                                                                                                                                                                                       
vncterm: 1.8.0                                                                                                                                                                                           
zfsutils-linux: 2.1.12-pve1                                                                                                                                                                             
root@inception:~#

root@inception:~# qm config 602 --current                                                                                                                                                               
agent: 1                                                                                                                                                                                                 
boot: order=scsi0;ide2                                                                                                                                                                                   
cores: 2                                                                                                                                                                                                 
description: # debian 12%0A# openhab 4%0A# 10.10.1.20                                                                                                                                                   
ide2: none,media=cdrom                                                                                                                                                                                   
memory: 4096                                                                                                                                                                                             
meta: creation-qemu=7.1.0,ctime=1670535142                                                                                                                                                               
name: openhab                                                                                                                                                                                           
net0: virtio=FE:A8:AC:D7:E7:34,bridge=vmbr0,firewall=1                                                                                                                                                   
numa: 0                                                                                                                                                                                                 
onboot: 1                                                                                                                                                                                               
ostype: l26                                                                                                                                                                                             
scsi0: local-ssd:vm-602-disk-0,iothread=1,size=8G                                                                                                                                                       
scsihw: virtio-scsi-single                                                                                                                                                                               
smbios1: uuid=0d85b9d8-5cc7-46a1-be35-539ce095db35                                                                                                                                                       
sockets: 1                                                                                                                                                                                               
startup: order=3                                                                                                                                                                                         
tags:                                                                                                                                                                                                   
vmgenid: d110a7b5-06ca-4610-922f-cbf9d11a1a2a                                                                                                                                                           
root@inception:~#

 

[kriznik]

this is actual frozen issue syslog:

This time it looks bit different in the syslog
(i had to do hardrestart, as ssh is not accesible when host is frozen)

Might be issue with kernel? how to safely install older one back? (asking for friend )

Sep 29 09:20:10 inception kernel: usercopy: Kernel memory overwrite attempt detected to process stack (offset 18446744073709551288, size 16)!
Sep 29 09:20:10 inception kernel: ------------[ cut here ]------------
Sep 29 09:20:10 inception kernel: kernel BUG at mm/usercopy.c:102!
Sep 29 09:20:10 inception kernel: invalid opcode: 0000 [#1] PREEMPT SMP PTI
Sep 29 09:20:10 inception kernel: CPU: 0 PID: 90901 Comm: lxc-info Tainted: P        W  O       6.2.16-14-pve #1
Sep 29 09:20:10 inception kernel: Hardware name: GIGABYTE GB-BXCE-3205/MQLPCAP-00, BIOS F4 12/08/2015
Sep 29 09:20:10 inception kernel: RIP: 0010:usercopy_abort+0x6c/0x80
Sep 29 09:20:10 inception kernel: Code: 52 ba 51 48 c7 c2 f7 3e 57 ba 41 52 48 c7 c7 c8 ef 53 ba 48 0f 45 d6 48 c7 c6 65 24 52 ba 48 89 c1 49 0f 45 f3 e8 34 08 d2 ff <0f> 0b 49 c7 c1 e0 4d 56 ba 4d 89 ca 4d 89 c8 eb a8 0f 1f 00 90 90
Sep 29 09:20:10 inception kernel: RSP: 0018:ffffbda8cc313bc0 EFLAGS: 00010246
Sep 29 09:20:10 inception kernel: RAX: 000000000000006b RBX: ffffbda8cc313d30 RCX: 0000000000000000
Sep 29 09:20:10 inception kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Sep 29 09:20:10 inception kernel: RBP: ffffbda8cc313bd8 R08: 0000000000000000 R09: 0000000000000000
Sep 29 09:20:10 inception kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000010
Sep 29 09:20:10 inception kernel: R13: 0000000000000000 R14: ffffbda8cc313d40 R15: 0000000000000010
Sep 29 09:20:10 inception kernel: FS:  00007f498211a7c0(0000) GS:ffffa093cee00000(0000) knlGS:0000000000000000
Sep 29 09:20:10 inception kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 29 09:20:10 inception kernel: CR2: 000055ec0185f4b0 CR3: 00000003590ea006 CR4: 00000000000726f0
Sep 29 09:20:10 inception kernel: Call Trace:
Sep 29 09:20:10 inception kernel:  <TASK>
Sep 29 09:20:10 inception kernel:  ? show_regs+0x6d/0x80
Sep 29 09:20:10 inception kernel:  ? die+0x37/0xa0
Sep 29 09:20:10 inception kernel:  ? do_trap+0xd4/0xf0
Sep 29 09:20:10 inception kernel:  ? do_error_trap+0x71/0xb0
Sep 29 09:20:10 inception kernel:  ? usercopy_abort+0x6c/0x80
Sep 29 09:20:10 inception kernel:  ? exc_invalid_op+0x52/0x80
Sep 29 09:20:10 inception kernel:  ? usercopy_abort+0x6c/0x80
Sep 29 09:20:10 inception kernel:  ? asm_exc_invalid_op+0x1b/0x20
Sep 29 09:20:10 inception kernel:  ? usercopy_abort+0x6c/0x80
Sep 29 09:20:10 inception kernel:  __check_object_size+0xfc/0x300
Sep 29 09:20:10 inception kernel:  ? ep_poll_callback+0x23f/0x290
Sep 29 09:20:10 inception kernel:  iovec_from_user.part.0+0x53/0x1a0
Sep 29 09:20:10 inception kernel:  __import_iovec+0x4b/0x190
Sep 29 09:20:10 inception kernel:  import_iovec+0x1b/0x30
Sep 29 09:20:10 inception kernel:  copy_msghdr_from_user+0x80/0xd0
Sep 29 09:20:10 inception kernel: WARNING: kernel stack frame pointer at 0000000049f30804 in lxc-info:90901 has bad value 0000000093f4bc66
Sep 29 09:20:10 inception kernel: unwind stack type:1 next_sp:0000000000000000 mask:0x2 graph_idx:0
---- edited too much text
Sep 29 09:20:10 inception kernel:  ? ___sys_sendmsg+0x7f/0xf0
Sep 29 09:20:10 inception kernel:  ? __sys_sendmsg+0x89/0xf0
Sep 29 09:20:10 inception kernel:  ? __x64_sys_sendmsg+0x1d/0x30
Sep 29 09:20:10 inception kernel:  ? do_syscall_64+0x5b/0x90
Sep 29 09:20:10 inception kernel:  ? entry_SYSCALL_64_after_hwframe+0x73/0xdd
Sep 29 09:20:10 inception kernel:  </TASK>
Sep 29 09:20:10 inception kernel: Modules linked in: wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_comment nft_compat 8021q garp mrp rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache netfs veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc binfmt_misc nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common iwlmvm x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_hdmi mac80211 kvm_intel libarc4 mei_pxp mei_hdcp i915 iwlwifi snd_hda_codec_realtek drm_buddy snd_hda_codec_generic ttm ledtrig_audio kvm drm_display_helper snd_hda_intel irqbypass snd_intel_dspcfg snd_intel_sdw_acpi crct10dif_pclmul btusb polyval_generic cec ghash_clmulni_intel btrtl cryptd rc_core btbcm snd_hda_codec sha512_ssse3 btintel btmtk rapl snd_hda_core
Sep 29 09:20:10 inception kernel:  mei_me drm_kms_helper cfg80211 bluetooth i2c_algo_bit spi_nor pcspkr snd_hwdep intel_cstate at24 syscopyarea sysfillrect ecdh_generic mtd snd_pcm mei ecc snd_timer sysimgblt snd soundcore acpi_pad mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap efi_pstore drm dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c xhci_pci spi_intel_platform xhci_pci_renesas spi_intel r8169 ehci_pci crc32_pclmul xhci_hcd ahci ehci_hcd i2c_i801 realtek i2c_smbus libahci lpc_ich video wmi
Sep 29 09:20:10 inception kernel: ---[ end trace 0000000000000000 ]---
Sep 29 09:20:10 inception kernel: RIP: 0010:usercopy_abort+0x6c/0x80
Sep 29 09:20:10 inception kernel: Code: 52 ba 51 48 c7 c2 f7 3e 57 ba 41 52 48 c7 c7 c8 ef 53 ba 48 0f 45 d6 48 c7 c6 65 24 52 ba 48 89 c1 49 0f 45 f3 e8 34 08 d2 ff <0f> 0b 49 c7 c1 e0 4d 56 ba 4d 89 ca 4d 89 c8 eb a8 0f 1f 00 90 90
Sep 29 09:20:10 inception kernel: RSP: 0018:ffffbda8cc313bc0 EFLAGS: 00010246
Sep 29 09:20:10 inception kernel: RAX: 000000000000006b RBX: ffffbda8cc313d30 RCX: 0000000000000000
Sep 29 09:20:10 inception kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Sep 29 09:20:10 inception kernel: RBP: ffffbda8cc313bd8 R08: 0000000000000000 R09: 0000000000000000
Sep 29 09:20:10 inception kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000010
Sep 29 09:20:10 inception kernel: R13: 0000000000000000 R14: ffffbda8cc313d40 R15: 0000000000000010
Sep 29 09:20:10 inception kernel: FS:  00007f498211a7c0(0000) GS:ffffa093cee00000(0000) knlGS:0000000000000000
Sep 29 09:20:10 inception kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 29 09:20:10 inception kernel: CR2: 000055ec0185f4b0 CR3: 00000003590ea006 CR4: 00000000000726f0
Sep 29 09:20:10 inception pvestatd[54384]: command 'lxc-info -n 102 -p' failed: got signal 11
Sep 29 09:46:35 inception pvedaemon[65264]: <root@pam> successful auth for user 'root@pam'
Sep 29 09:55:41 inception kernel: BUG: unable to handle page fault for address: ffffa190cca083d8
Sep 29 09:55:41 inception kernel: #PF: supervisor read access in kernel mode
Sep 29 09:55:41 inception kernel: #PF: error_code(0x0000) - not-present page
Sep 29 09:55:41 inception kernel: PGD 0 P4D 0
Sep 29 09:55:41 inception kernel: Oops: 0000 [#2] PREEMPT SMP PTI
Sep 29 09:55:41 inception kernel: CPU: 0 PID: 101971 Comm: pvestatd Tainted: P      D W  O       6.2.16-14-pve #1
Sep 29 09:55:41 inception kernel: Hardware name: GIGABYTE GB-BXCE-3205/MQLPCAP-00, BIOS F4 12/08/2015
Sep 29 09:55:41 inception kernel: RIP: 0010:unlink_file_vma+0x17/0x70
Sep 29 09:55:41 inception kernel: Code: 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 48 8b 47 70 48 85 c0 74 52 55 48 89 e5 41 55 41 54 53 <4c> 8b a0 d8 00 00 00 48 89 fb 4d 8d 6c 24 58 4c 89 ef e8 22 ec cc
Sep 29 09:55:41 inception kernel: RSP: 0018:ffffbda8c3fb7c20 EFLAGS: 00010286
Sep 29 09:55:41 inception kernel: RAX: ffffa190cca08300 RBX: ffffa090c75353f0 RCX: 0000000000000000
Sep 29 09:55:41 inception kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffa090c75353f0
Sep 29 09:55:41 inception kernel: RBP: ffffbda8c3fb7c38 R08: 0000000000000000 R09: 0000000000000000
Sep 29 09:55:41 inception kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffffffffff
Sep 29 09:55:41 inception kernel: R13: 00007fdc9eaa3000 R14: ffffa090c75353f0 R15: ffffa090c7535888
Sep 29 09:55:41 inception kernel: FS:  0000000000000000(0000) GS:ffffa093cee00000(0000) knlGS:0000000000000000
Sep 29 09:55:41 inception kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 29 09:55:41 inception kernel: CR2: ffffa190cca083d8 CR3: 000000021ee10005 CR4: 00000000000726f0
Sep 29 09:55:41 inception kernel: Call Trace:
Sep 29 09:55:41 inception kernel:  <TASK>
Sep 29 09:55:41 inception kernel:  ? show_regs+0x6d/0x80
Sep 29 09:55:41 inception kernel:  ? __die+0x24/0x80
Sep 29 09:55:41 inception kernel:  ? page_fault_oops+0x176/0x500
Sep 29 09:55:41 inception kernel:  ? unlink_file_vma+0x17/0x70
Sep 29 09:55:41 inception kernel:  ? kernelmode_fixup_or_oops+0xb2/0x140
Sep 29 09:55:41 inception kernel:  ? __bad_area_nosemaphore+0x1a5/0x2c0
Sep 29 09:55:41 inception kernel:  ? bad_area_nosemaphore+0x16/0x30
Sep 29 09:55:41 inception kernel:  ? do_kern_addr_fault+0x7b/0xa0
Sep 29 09:55:41 inception kernel:  ? exc_page_fault+0x10a/0x1b0
Sep 29 09:55:41 inception kernel:  ? asm_exc_page_fault+0x27/0x30
Sep 29 09:55:41 inception kernel:  ? unlink_file_vma+0x17/0x70
Sep 29 09:55:41 inception kernel:  free_pgtables+0x105/0x160
Sep 29 09:55:41 inception kernel:  exit_mmap+0x142/0x310
Sep 29 09:55:41 inception kernel:  __mmput+0x41/0x140
Sep 29 09:55:41 inception kernel:  mmput+0x2e/0x40
Sep 29 09:55:41 inception kernel:  do_exit+0x2ef/0xac0
Sep 29 09:55:41 inception kernel:  do_group_exit+0x35/0x90
Sep 29 09:55:41 inception kernel:  __x64_sys_exit_group+0x18/0x20
Sep 29 09:55:41 inception kernel:  do_syscall_64+0x5b/0x90
Sep 29 09:55:41 inception kernel:  ? exit_to_user_mode_prepare+0x39/0x190
Sep 29 09:55:41 inception kernel:  ? irqentry_exit_to_user_mode+0x17/0x20
Sep 29 09:55:41 inception kernel:  ? irqentry_exit+0x43/0x50
Sep 29 09:55:41 inception kernel:  ? exc_page_fault+0x91/0x1b0
Sep 29 09:55:41 inception kernel:  entry_SYSCALL_64_after_hwframe+0x73/0xdd
Sep 29 09:55:41 inception kernel: RIP: 0033:0x7fdca2560249
Sep 29 09:55:41 inception kernel: Code: Unable to access opcode bytes at 0x7fdca256021f.
Sep 29 09:55:41 inception kernel: RSP: 002b:00007ffcf4eeffc8 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7
Sep 29 09:55:41 inception kernel: RAX: ffffffffffffffda RBX: 000055712f24f2a0 RCX: 00007fdca2560249
Sep 29 09:55:41 inception kernel: RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
Sep 29 09:55:41 inception kernel: RBP: 0000000000000001 R08: ffffffffffffff78 R09: 0000000000000000
Sep 29 09:55:41 inception kernel: R10: 00007fdca249e1e0 R11: 0000000000000202 R12: 0000000000000006
Sep 29 09:55:41 inception kernel: R13: 0000000000000000 R14: 0000557134013d78 R15: 000055712f7a8e58
Sep 29 09:55:41 inception kernel:  </TASK>
Sep 29 09:55:41 inception kernel: Modules linked in: 
-- edited too much text ---
Sep 29 09:55:41 inception kernel: CR2: ffffa190cca083d8
Sep 29 09:55:41 inception kernel: ---[ end trace 0000000000000000 ]---
Sep 29 09:55:41 inception kernel: RIP: 0010:usercopy_abort+0x6c/0x80
Sep 29 09:55:41 inception kernel: Code: 52 ba 51 48 c7 c2 f7 3e 57 ba 41 52 48 c7 c7 c8 ef 53 ba 48 0f 45 d6 48 c7 c6 65 24 52 ba 48 89 c1 49 0f 45 f3 e8 34 08 d2 ff <0f> 0b 49 c7 c1 e0 4d 56 ba 4d 89 ca 4d 89 c8 eb a8 0f 1f 00 90 90
Sep 29 09:55:41 inception kernel: RSP: 0018:ffffbda8cc313bc0 EFLAGS: 00010246
Sep 29 09:55:41 inception kernel: RAX: 000000000000006b RBX: ffffbda8cc313d30 RCX: 0000000000000000
Sep 29 09:55:41 inception kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Sep 29 09:55:41 inception kernel: RBP: ffffbda8cc313bd8 R08: 0000000000000000 R09: 0000000000000000
Sep 29 09:55:41 inception kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000010
Sep 29 09:55:41 inception kernel: R13: 0000000000000000 R14: ffffbda8cc313d40 R15: 0000000000000010
Sep 29 09:55:41 inception kernel: FS:  0000000000000000(0000) GS:ffffa093cee00000(0000) knlGS:0000000000000000
Sep 29 09:55:41 inception kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 29 09:55:41 inception kernel: CR2: ffffa190cca083d8 CR3: 000000021ee10005 CR4: 00000000000726f0
Sep 29 09:55:41 inception kernel: note: pvestatd[101971] exited with irqs disabled
Sep 29 09:55:41 inception kernel: Fixing recursive fault but reboot is needed!
Sep 29 09:55:41 inception kernel: BUG: scheduling while atomic: pvestatd/101971/0x00000000
Sep 29 09:55:41 inception kernel: Modules linked in: 
-- edited too much text ---
Sep 29 09:55:41 inception kernel: CPU: 0 PID: 101971 Comm: pvestatd Tainted: P      D W  O       6.2.16-14-pve #1
Sep 29 09:55:41 inception kernel: Hardware name: GIGABYTE GB-BXCE-3205/MQLPCAP-00, BIOS F4 12/08/2015
Sep 29 09:55:41 inception kernel: Call Trace:
Sep 29 09:55:41 inception kernel:  <TASK>
Sep 29 09:55:41 inception kernel:  dump_stack_lvl+0x48/0x70
Sep 29 09:55:41 inception kernel:  dump_stack+0x10/0x20
Sep 29 09:55:41 inception kernel:  __schedule_bug+0x64/0x80
Sep 29 09:55:41 inception kernel:  __schedule+0xf54/0x1510
Sep 29 09:55:41 inception kernel:  ? vprintk_default+0x1d/0x30
Sep 29 09:55:41 inception kernel:  ? _printk+0x60/0x90
Sep 29 09:55:41 inception kernel:  do_task_dead+0x44/0x50
Sep 29 09:55:41 inception kernel:  make_task_dead+0x15a/0x180
Sep 29 09:55:41 inception kernel:  rewind_stack_and_make_dead+0x17/0x20
Sep 29 09:55:41 inception kernel: RIP: 0033:0x7fdca2560249
Sep 29 09:55:41 inception kernel: Code: Unable to access opcode bytes at 0x7fdca256021f.
Sep 29 09:55:41 inception kernel: RSP: 002b:00007ffcf4eeffc8 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7
Sep 29 09:55:41 inception kernel: RAX: ffffffffffffffda RBX: 000055712f24f2a0 RCX: 00007fdca2560249
Sep 29 09:55:41 inception kernel: RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
Sep 29 09:55:41 inception kernel: RBP: 0000000000000001 R08: ffffffffffffff78 R09: 0000000000000000
Sep 29 09:55:41 inception kernel: R10: 00007fdca249e1e0 R11: 0000000000000202 R12: 0000000000000006
Sep 29 09:55:41 inception kernel: R13: 0000000000000000 R14: 0000557134013d78 R15: 000055712f7a8e58
Sep 29 09:55:41 inception kernel:  </TASK>
-- Reboot --

 

[fiona]

Might be issue with kernel? how to safely install older one back? (asking for friend )

By default, an older kernel will stay installed, so you could just select it during boot. Otherwise, use apt install proxmox-kernel-<version> first.

Sep 29 09:20:10 inception kernel: usercopy: Kernel memory overwrite attempt detected to process stack (offset 18446744073709551288, size 16)!
Sep 29 09:20:10 inception kernel: ------------[ cut here ]------------
Sep 29 09:20:10 inception kernel: kernel BUG at mm/usercopy.c:102!
Sep 29 09:20:10 inception kernel: invalid opcode: 0000 [#1] PREEMPT SMP PTI
Sep 29 09:20:10 inception kernel: CPU: 0 PID: 90901 Comm: lxc-info Tainted: P        W  O       6.2.16-14-pve #1
Sep 29 09:20:10 inception kernel: Hardware name: GIGABYTE GB-BXCE-3205/MQLPCAP-00, BIOS F4 12/08/2015
Sep 29 09:20:10 inception kernel: RIP: 0010:usercopy_abort+0x6c/0x80
Sep 29 09:20:10 inception kernel: Code: 52 ba 51 48 c7 c2 f7 3e 57 ba 41 52 48 c7 c7 c8 ef 53 ba 48 0f 45 d6 48 c7 c6 65 24 52 ba 48 89 c1 49 0f 45 f3 e8 34 08 d2 ff <0f> 0b 49 c7 c1 e0 4d 56 ba 4d 89 ca 4d 89 c8 eb a8 0f 1f 00 90 90
Sep 29 09:20:10 inception kernel: RSP: 0018:ffffbda8cc313bc0 EFLAGS: 00010246
Sep 29 09:20:10 inception kernel: RAX: 000000000000006b RBX: ffffbda8cc313d30 RCX: 0000000000000000
Sep 29 09:20:10 inception kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Sep 29 09:20:10 inception kernel: RBP: ffffbda8cc313bd8 R08: 0000000000000000 R09: 0000000000000000
Sep 29 09:20:10 inception kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000010
Sep 29 09:20:10 inception kernel: R13: 0000000000000000 R14: ffffbda8cc313d40 R15: 0000000000000010
Sep 29 09:20:10 inception kernel: FS:  00007f498211a7c0(0000) GS:ffffa093cee00000(0000) knlGS:0000000000000000
Sep 29 09:20:10 inception kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 29 09:20:10 inception kernel: CR2: 000055ec0185f4b0 CR3: 00000003590ea006 CR4: 00000000000726f0
Sep 29 09:20:10 inception kernel: Call Trace:
Sep 29 09:20:10 inception kernel:  <TASK>
Sep 29 09:20:10 inception kernel:  ? show_regs+0x6d/0x80
Sep 29 09:20:10 inception kernel:  ? die+0x37/0xa0
Sep 29 09:20:10 inception kernel:  ? do_trap+0xd4/0xf0
Sep 29 09:20:10 inception kernel:  ? do_error_trap+0x71/0xb0
Sep 29 09:20:10 inception kernel:  ? usercopy_abort+0x6c/0x80
Sep 29 09:20:10 inception kernel:  ? exc_invalid_op+0x52/0x80
Sep 29 09:20:10 inception kernel:  ? usercopy_abort+0x6c/0x80
Sep 29 09:20:10 inception kernel:  ? asm_exc_invalid_op+0x1b/0x20
Sep 29 09:20:10 inception kernel:  ? usercopy_abort+0x6c/0x80
Sep 29 09:20:10 inception kernel:  __check_object_size+0xfc/0x300
Sep 29 09:20:10 inception kernel:  ? ep_poll_callback+0x23f/0x290
Sep 29 09:20:10 inception kernel:  iovec_from_user.part.0+0x53/0x1a0
Sep 29 09:20:10 inception kernel:  __import_iovec+0x4b/0x190
Sep 29 09:20:10 inception kernel:  import_iovec+0x1b/0x30
Sep 29 09:20:10 inception kernel:  copy_msghdr_from_user+0x80/0xd0
Sep 29 09:20:10 inception kernel: WARNING: kernel stack frame pointer at 0000000049f30804 in lxc-info:90901 has bad value 0000000093f4bc66
Sep 29 09:20:10 inception kernel: unwind stack type:1 next_sp:0000000000000000 mask:0x2 graph_idx:0
---- edited too much text
Sep 29 09:20:10 inception kernel:  ? ___sys_sendmsg+0x7f/0xf0
Sep 29 09:20:10 inception kernel:  ? __sys_sendmsg+0x89/0xf0
Sep 29 09:20:10 inception kernel:  ? __x64_sys_sendmsg+0x1d/0x30
Sep 29 09:20:10 inception kernel:  ? do_syscall_64+0x5b/0x90
Sep 29 09:20:10 inception kernel:  ? entry_SYSCALL_64_after_hwframe+0x73/0xdd
Sep 29 09:20:10 inception kernel:  </TASK>
Sep 29 09:20:10 inception kernel: Modules linked in: wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_comment nft_compat 8021q garp mrp rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache netfs veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc binfmt_misc nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common iwlmvm x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_hdmi mac80211 kvm_intel libarc4 mei_pxp mei_hdcp i915 iwlwifi snd_hda_codec_realtek drm_buddy snd_hda_codec_generic ttm ledtrig_audio kvm drm_display_helper snd_hda_intel irqbypass snd_intel_dspcfg snd_intel_sdw_acpi crct10dif_pclmul btusb polyval_generic cec ghash_clmulni_intel btrtl cryptd rc_core btbcm snd_hda_codec sha512_ssse3 btintel btmtk rapl snd_hda_core
Sep 29 09:20:10 inception kernel:  mei_me drm_kms_helper cfg80211 bluetooth i2c_algo_bit spi_nor pcspkr snd_hwdep intel_cstate at24 syscopyarea sysfillrect ecdh_generic mtd snd_pcm mei ecc snd_timer sysimgblt snd soundcore acpi_pad mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap efi_pstore drm dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c xhci_pci spi_intel_platform xhci_pci_renesas spi_intel r8169 ehci_pci crc32_pclmul xhci_hcd ahci ehci_hcd i2c_i801 realtek i2c_smbus libahci lpc_ich video wmi
Sep 29 09:20:10 inception kernel: ---[ end trace 0000000000000000 ]---
Sep 29 09:20:10 inception kernel: RIP: 0010:usercopy_abort+0x6c/0x80
Sep 29 09:20:10 inception kernel: Code: 52 ba 51 48 c7 c2 f7 3e 57 ba 41 52 48 c7 c7 c8 ef 53 ba 48 0f 45 d6 48 c7 c6 65 24 52 ba 48 89 c1 49 0f 45 f3 e8 34 08 d2 ff <0f> 0b 49 c7 c1 e0 4d 56 ba 4d 89 ca 4d 89 c8 eb a8 0f 1f 00 90 90
Sep 29 09:20:10 inception kernel: RSP: 0018:ffffbda8cc313bc0 EFLAGS: 00010246
Sep 29 09:20:10 inception kernel: RAX: 000000000000006b RBX: ffffbda8cc313d30 RCX: 0000000000000000
Sep 29 09:20:10 inception kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Sep 29 09:20:10 inception kernel: RBP: ffffbda8cc313bd8 R08: 0000000000000000 R09: 0000000000000000
Sep 29 09:20:10 inception kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000010
Sep 29 09:20:10 inception kernel: R13: 0000000000000000 R14: ffffbda8cc313d40 R15: 0000000000000010
Sep 29 09:20:10 inception kernel: FS:  00007f498211a7c0(0000) GS:ffffa093cee00000(0000) knlGS:0000000000000000
Sep 29 09:20:10 inception kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 29 09:20:10 inception kernel: CR2: 000055ec0185f4b0 CR3: 00000003590ea006 CR4: 00000000000726f0
Sep 29 09:20:10 inception pvestatd[54384]: command 'lxc-info -n 102 -p' failed: got signal 11
Sep 29 09:46:35 inception pvedaemon[65264]: <root@pam> successful auth for user 'root@pam'
Sep 29 09:55:41 inception kernel: BUG: unable to handle page fault for address: ffffa190cca083d8
Sep 29 09:55:41 inception kernel: #PF: supervisor read access in kernel mode
Sep 29 09:55:41 inception kernel: #PF: error_code(0x0000) - not-present page
Sep 29 09:55:41 inception kernel: PGD 0 P4D 0
Sep 29 09:55:41 inception kernel: Oops: 0000 [#2] PREEMPT SMP PTI
Sep 29 09:55:41 inception kernel: CPU: 0 PID: 101971 Comm: pvestatd Tainted: P      D W  O       6.2.16-14-pve #1
Sep 29 09:55:41 inception kernel: Hardware name: GIGABYTE GB-BXCE-3205/MQLPCAP-00, BIOS F4 12/08/2015
Sep 29 09:55:41 inception kernel: RIP: 0010:unlink_file_vma+0x17/0x70
Sep 29 09:55:41 inception kernel: Code: 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 48 8b 47 70 48 85 c0 74 52 55 48 89 e5 41 55 41 54 53 <4c> 8b a0 d8 00 00 00 48 89 fb 4d 8d 6c 24 58 4c 89 ef e8 22 ec cc
Sep 29 09:55:41 inception kernel: RSP: 0018:ffffbda8c3fb7c20 EFLAGS: 00010286
Sep 29 09:55:41 inception kernel: RAX: ffffa190cca08300 RBX: ffffa090c75353f0 RCX: 0000000000000000
Sep 29 09:55:41 inception kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffa090c75353f0
Sep 29 09:55:41 inception kernel: RBP: ffffbda8c3fb7c38 R08: 0000000000000000 R09: 0000000000000000
Sep 29 09:55:41 inception kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffffffffff
Sep 29 09:55:41 inception kernel: R13: 00007fdc9eaa3000 R14: ffffa090c75353f0 R15: ffffa090c7535888
Sep 29 09:55:41 inception kernel: FS:  0000000000000000(0000) GS:ffffa093cee00000(0000) knlGS:0000000000000000
Sep 29 09:55:41 inception kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 29 09:55:41 inception kernel: CR2: ffffa190cca083d8 CR3: 000000021ee10005 CR4: 00000000000726f0
Sep 29 09:55:41 inception kernel: Call Trace:
Sep 29 09:55:41 inception kernel:  <TASK>
Sep 29 09:55:41 inception kernel:  ? show_regs+0x6d/0x80
Sep 29 09:55:41 inception kernel:  ? __die+0x24/0x80
Sep 29 09:55:41 inception kernel:  ? page_fault_oops+0x176/0x500
Sep 29 09:55:41 inception kernel:  ? unlink_file_vma+0x17/0x70
Sep 29 09:55:41 inception kernel:  ? kernelmode_fixup_or_oops+0xb2/0x140
Sep 29 09:55:41 inception kernel:  ? __bad_area_nosemaphore+0x1a5/0x2c0
Sep 29 09:55:41 inception kernel:  ? bad_area_nosemaphore+0x16/0x30
Sep 29 09:55:41 inception kernel:  ? do_kern_addr_fault+0x7b/0xa0
Sep 29 09:55:41 inception kernel:  ? exc_page_fault+0x10a/0x1b0
Sep 29 09:55:41 inception kernel:  ? asm_exc_page_fault+0x27/0x30
Sep 29 09:55:41 inception kernel:  ? unlink_file_vma+0x17/0x70
Sep 29 09:55:41 inception kernel:  free_pgtables+0x105/0x160
Sep 29 09:55:41 inception kernel:  exit_mmap+0x142/0x310
Sep 29 09:55:41 inception kernel:  __mmput+0x41/0x140
Sep 29 09:55:41 inception kernel:  mmput+0x2e/0x40
Sep 29 09:55:41 inception kernel:  do_exit+0x2ef/0xac0
Sep 29 09:55:41 inception kernel:  do_group_exit+0x35/0x90
Sep 29 09:55:41 inception kernel:  __x64_sys_exit_group+0x18/0x20
Sep 29 09:55:41 inception kernel:  do_syscall_64+0x5b/0x90
Sep 29 09:55:41 inception kernel:  ? exit_to_user_mode_prepare+0x39/0x190
Sep 29 09:55:41 inception kernel:  ? irqentry_exit_to_user_mode+0x17/0x20
Sep 29 09:55:41 inception kernel:  ? irqentry_exit+0x43/0x50
Sep 29 09:55:41 inception kernel:  ? exc_page_fault+0x91/0x1b0
Sep 29 09:55:41 inception kernel:  entry_SYSCALL_64_after_hwframe+0x73/0xdd
Sep 29 09:55:41 inception kernel: RIP: 0033:0x7fdca2560249
Sep 29 09:55:41 inception kernel: Code: Unable to access opcode bytes at 0x7fdca256021f.
Sep 29 09:55:41 inception kernel: RSP: 002b:00007ffcf4eeffc8 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7
Sep 29 09:55:41 inception kernel: RAX: ffffffffffffffda RBX: 000055712f24f2a0 RCX: 00007fdca2560249
Sep 29 09:55:41 inception kernel: RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
Sep 29 09:55:41 inception kernel: RBP: 0000000000000001 R08: ffffffffffffff78 R09: 0000000000000000
Sep 29 09:55:41 inception kernel: R10: 00007fdca249e1e0 R11: 0000000000000202 R12: 0000000000000006
Sep 29 09:55:41 inception kernel: R13: 0000000000000000 R14: 0000557134013d78 R15: 000055712f7a8e58
Sep 29 09:55:41 inception kernel:  </TASK>
Sep 29 09:55:41 inception kernel: Modules linked in:
-- edited too much text ---
Sep 29 09:55:41 inception kernel: CR2: ffffa190cca083d8
Sep 29 09:55:41 inception kernel: ---[ end trace 0000000000000000 ]---
Sep 29 09:55:41 inception kernel: RIP: 0010:usercopy_abort+0x6c/0x80
Sep 29 09:55:41 inception kernel: Code: 52 ba 51 48 c7 c2 f7 3e 57 ba 41 52 48 c7 c7 c8 ef 53 ba 48 0f 45 d6 48 c7 c6 65 24 52 ba 48 89 c1 49 0f 45 f3 e8 34 08 d2 ff <0f> 0b 49 c7 c1 e0 4d 56 ba 4d 89 ca 4d 89 c8 eb a8 0f 1f 00 90 90
Sep 29 09:55:41 inception kernel: RSP: 0018:ffffbda8cc313bc0 EFLAGS: 00010246
Sep 29 09:55:41 inception kernel: RAX: 000000000000006b RBX: ffffbda8cc313d30 RCX: 0000000000000000
Sep 29 09:55:41 inception kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Sep 29 09:55:41 inception kernel: RBP: ffffbda8cc313bd8 R08: 0000000000000000 R09: 0000000000000000
Sep 29 09:55:41 inception kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000010
Sep 29 09:55:41 inception kernel: R13: 0000000000000000 R14: ffffbda8cc313d40 R15: 0000000000000010
Sep 29 09:55:41 inception kernel: FS:  0000000000000000(0000) GS:ffffa093cee00000(0000) knlGS:0000000000000000
Sep 29 09:55:41 inception kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 29 09:55:41 inception kernel: CR2: ffffa190cca083d8 CR3: 000000021ee10005 CR4: 00000000000726f0
Sep 29 09:55:41 inception kernel: note: pvestatd[101971] exited with irqs disabled
Sep 29 09:55:41 inception kernel: Fixing recursive fault but reboot is needed!
Sep 29 09:55:41 inception kernel: BUG: scheduling while atomic: pvestatd/101971/0x00000000
Sep 29 09:55:41 inception kernel: Modules linked in:
-- edited too much text ---
Sep 29 09:55:41 inception kernel: CPU: 0 PID: 101971 Comm: pvestatd Tainted: P      D W  O       6.2.16-14-pve #1
Sep 29 09:55:41 inception kernel: Hardware name: GIGABYTE GB-BXCE-3205/MQLPCAP-00, BIOS F4 12/08/2015
Sep 29 09:55:41 inception kernel: Call Trace:
Sep 29 09:55:41 inception kernel:  <TASK>
Sep 29 09:55:41 inception kernel:  dump_stack_lvl+0x48/0x70
Sep 29 09:55:41 inception kernel:  dump_stack+0x10/0x20
Sep 29 09:55:41 inception kernel:  __schedule_bug+0x64/0x80
Sep 29 09:55:41 inception kernel:  __schedule+0xf54/0x1510
Sep 29 09:55:41 inception kernel:  ? vprintk_default+0x1d/0x30
Sep 29 09:55:41 inception kernel:  ? _printk+0x60/0x90
Sep 29 09:55:41 inception kernel:  do_task_dead+0x44/0x50
Sep 29 09:55:41 inception kernel:  make_task_dead+0x15a/0x180
Sep 29 09:55:41 inception kernel:  rewind_stack_and_make_dead+0x17/0x20
Sep 29 09:55:41 inception kernel: RIP: 0033:0x7fdca2560249
Sep 29 09:55:41 inception kernel: Code: Unable to access opcode bytes at 0x7fdca256021f.
Sep 29 09:55:41 inception kernel: RSP: 002b:00007ffcf4eeffc8 EFLAGS: 00000202 ORIG_RAX: 00000000000000e7
Sep 29 09:55:41 inception kernel: RAX: ffffffffffffffda RBX: 000055712f24f2a0 RCX: 00007fdca2560249
Sep 29 09:55:41 inception kernel: RDX: 000000000000003c RSI: 00000000000000e7 RDI: 0000000000000000
Sep 29 09:55:41 inception kernel: RBP: 0000000000000001 R08: ffffffffffffff78 R09: 0000000000000000
Sep 29 09:55:41 inception kernel: R10: 00007fdca249e1e0 R11: 0000000000000202 R12: 0000000000000006
Sep 29 09:55:41 inception kernel: R13: 0000000000000000 R14: 0000557134013d78 R15: 000055712f7a8e58
Sep 29 09:55:41 inception kernel:  </TASK>
-- Reboot --

Hmm, those messages are quite different then the first one you posted. Might point to hardware issues. I suggest you check your memory with e.g. memtest86+ which can be started from the Proxmox VE installer ISO.

 

[kriznik]

I have exact same hw with same proxmox, so I've migrated everyhting and that problematic instance will now run without any VM's to narrow down the issue.
Also I have cluster running, so I have options.
I did memcheck and diskcheck couple of times, all results been OK, which does not telling anything it cannot be wrong indeed

I've seen similar iissue recently in this forum, which hasn't been solved yet ... so who knows
https://forum.proxmox.com/threads/internal-error-on-vm.128839/

 

[fiona]

I have exact same hw with same proxmox, so I've migrated everyhting and that problematic instance will now run without any VM's to narrow down the issue.
Also I have cluster running, so I have options.
I did memcheck and diskcheck couple of times, all results been OK, which does not telling anything it cannot be wrong indeed

What host CPU do you have? Are the latest CPU microcode and BIOS updates installed?

I've seen similar iissue recently in this forum, which hasn't been solved yet ... so who knows
https://forum.proxmox.com/threads/internal-error-on-vm.128839/

That one is a bit of a different error though:
KVM internal error. Suberror: 3
instead of your
error: kvm run failed Invalid argument
and the host didn't freeze.

 

[kriznik]

Yes I do have microcode installed (latest).
Bit of the update: when running without any VM's it's not freezing of course. Did another memtest, and everything went OK.

I'm bit clueless for now actually

 

[fiona]

You can still test with an older kernel. It could be a regression there.

 

[kriznik]

so bit of update, node itself (without any VM's) running 2days without an issue. So I guess I need somehow debug if some exact VM is causing issue, or any.
Is there some sort of debug which I can use ? as in syslog there is nothing more and on VM there is nothing really.

 

[fiona]

so bit of update, node itself (without any VM's) running 2days without an issue. So I guess I need somehow debug if some exact VM is causing issue, or any.

This error in the system logs does include the PID (i.e. 1564) of the QEMU process:

Sep 29 04:15:42 inception QEMU[1564]: error: kvm run failed Invalid argument

You can use qm list or to see which VM ID it belongs to next time the issue happens.