Hello,
after years of running on proxmox, I've experienced from yesterday issue with one of my VM's which stopped responding.
It showed "running (internal-error) on the UI
whole pvestatsd needed to be restarted in order to see statuses of the remaining VM's (but that might be side effect).
when performing backup to the Proxmox Backup server Host completely frozen.
I've disabled backup and host is now running even when that given VM is in internal error.
no clue what happened. Found this in the syslog, can somebody please help me decode what that means?
Host has microcode installed and been running fine for long time.
Sep 29 04:15:42 inception QEMU[1564]: error: kvm run failed Invalid argument
Thanks a lot
after years of running on proxmox, I've experienced from yesterday issue with one of my VM's which stopped responding.
It showed "running (internal-error) on the UI
whole pvestatsd needed to be restarted in order to see statuses of the remaining VM's (but that might be side effect).
when performing backup to the Proxmox Backup server Host completely frozen.
I've disabled backup and host is now running even when that given VM is in internal error.
no clue what happened. Found this in the syslog, can somebody please help me decode what that means?
Host has microcode installed and been running fine for long time.
Sep 29 04:15:42 inception QEMU[1564]: error: kvm run failed Invalid argument
Thanks a lot
Code:
Sep 29 03:46:54 inception smartd[670]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 68 to 69
Sep 29 04:13:25 inception kernel: perf: interrupt took too long (3985 > 3950), lowering kernel.perf_event_max_sample_rate to 50000
Sep 29 04:15:42 inception kernel: get_mmio_spte: reserved bits set on MMU-present spte, addr 0x469c4d24, hierarchy:
Sep 29 04:15:42 inception kernel: ------ spte = 0x108387907 level = 4, rsvd bits = 0xfff80000000f8
Sep 29 04:15:42 inception kernel: ------ spte = 0x109e3c907 level = 3, rsvd bits = 0xfff8000000078
Sep 29 04:15:42 inception kernel: ------ spte = 0x3807f3907 level = 2, rsvd bits = 0xfff8000000078
Sep 29 04:15:42 inception kernel: ------ spte = 0x6000101d6d7eb77 level = 1, rsvd bits = 0xfff8000000000
Sep 29 04:15:42 inception kernel: ------------[ cut here ]------------
Sep 29 04:15:42 inception kernel: WARNING: CPU: 0 PID: 1626 at arch/x86/kvm/mmu/mmu.c:4099 kvm_mmu_page_fault+0x7f3/0xb40 [kvm]
Sep 29 04:15:42 inception kernel: Modules linked in: wireguard curve25519_x86_64 libchacha20poly1305 chacha_x86_64 poly1305_x86_64 libcurve25519_generic libchacha ip6_udp_tunnel udp_tunnel nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_comment nft_compat 8021q garp mrp rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache netfs veth ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter bpfilter nf_tables bonding tls softdog sunrpc binfmt_misc nfnetlink_log nfnetlink intel_rapl_msr intel_rapl_common iwlmvm x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_hdmi mac80211 kvm_intel libarc4 mei_pxp mei_hdcp i915 iwlwifi snd_hda_codec_realtek drm_buddy snd_hda_codec_generic ttm ledtrig_audio kvm drm_display_helper snd_hda_intel irqbypass snd_intel_dspcfg snd_intel_sdw_acpi crct10dif_pclmul btusb polyval_generic cec ghash_clmulni_intel btrtl cryptd rc_core btbcm snd_hda_codec sha512_ssse3 btintel btmtk rapl snd_hda_core
Sep 29 04:15:42 inception kernel: mei_me drm_kms_helper cfg80211 bluetooth i2c_algo_bit spi_nor pcspkr snd_hwdep intel_cstate at24 syscopyarea sysfillrect ecdh_generic mtd snd_pcm mei ecc snd_timer sysimgblt snd soundcore acpi_pad mac_hid zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) vhost_net vhost vhost_iotlb tap efi_pstore drm dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq simplefb dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c xhci_pci spi_intel_platform xhci_pci_renesas spi_intel r8169 ehci_pci crc32_pclmul xhci_hcd ahci ehci_hcd i2c_i801 realtek i2c_smbus libahci lpc_ich video wmi
Sep 29 04:15:42 inception kernel: CPU: 0 PID: 1626 Comm: CPU 1/KVM Tainted: P O 6.2.16-14-pve #1
Sep 29 04:15:42 inception kernel: Hardware name: GIGABYTE GB-BXCE-3205/MQLPCAP-00, BIOS F4 12/08/2015
Sep 29 04:15:42 inception kernel: RIP: 0010:kvm_mmu_page_fault+0x7f3/0xb40 [kvm]
Sep 29 04:15:42 inception kernel: Code: 9b 48 01 c8 49 8b 8c c7 b0 00 00 00 49 83 fc 05 0f 87 b7 02 00 00 4c 89 c6 48 c7 c7 e8 3d 5e c1 e8 02 6d 7d f7 45 39 f5 7e a3 <0f> 0b b8 ea ff ff ff e9 2f fa ff ff 0f 0b 4c 89 e7 44 89 9d 30 ff
Sep 29 04:15:42 inception kernel: RSP: 0018:ffffbda8c313bb70 EFLAGS: 00010202
Sep 29 04:15:42 inception kernel: RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000000
Sep 29 04:15:42 inception kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Sep 29 04:15:42 inception kernel: RBP: ffffbda8c313bc60 R08: 0000000000000000 R09: 0000000000000000
Sep 29 04:15:42 inception kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
Sep 29 04:15:42 inception kernel: R13: 0000000000000001 R14: 0000000000000000 R15: ffffa090cc954a90
Sep 29 04:15:42 inception kernel: FS: 00007f3456bbb6c0(0000) GS:ffffa093cee00000(0000) knlGS:ffff968f3bd00000
Sep 29 04:15:42 inception kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 29 04:15:42 inception kernel: CR2: 00007fc66f842000 CR3: 000000010b4b8005 CR4: 00000000000726f0
Sep 29 04:15:42 inception kernel: Call Trace:
Sep 29 04:15:42 inception kernel: <TASK>
Sep 29 04:15:42 inception kernel: ? show_regs+0x6d/0x80
Sep 29 04:15:42 inception kernel: ? __warn+0x89/0x160
Sep 29 04:15:42 inception kernel: ? kvm_mmu_page_fault+0x7f3/0xb40 [kvm]
Sep 29 04:15:42 inception kernel: ? report_bug+0x17e/0x1b0
Sep 29 04:15:42 inception kernel: ? handle_bug+0x46/0x90
Sep 29 04:15:42 inception kernel: ? exc_invalid_op+0x18/0x80
Sep 29 04:15:42 inception kernel: ? asm_exc_invalid_op+0x1b/0x20
Sep 29 04:15:42 inception kernel: ? kvm_mmu_page_fault+0x7f3/0xb40 [kvm]
Sep 29 04:15:42 inception kernel: ? __kvm_io_bus_write+0x2d/0xc0 [kvm]
Sep 29 04:15:42 inception kernel: handle_ept_misconfig+0x4f/0x1d0 [kvm_intel]
Sep 29 04:15:42 inception kernel: vmx_handle_exit+0x90d/0xa40 [kvm_intel]
Sep 29 04:15:42 inception kernel: kvm_arch_vcpu_ioctl_run+0xe02/0x1750 [kvm]
Sep 29 04:15:42 inception kernel: ? vmx_vcpu_put+0x134/0x220 [kvm_intel]
Sep 29 04:15:42 inception kernel: ? restore_fpregs_from_fpstate+0x47/0xf0
Sep 29 04:15:42 inception kernel: ? kvm_arch_vcpu_put+0x1a6/0x200 [kvm]
Sep 29 04:15:42 inception kernel: kvm_vcpu_ioctl+0x297/0x7c0 [kvm]
Sep 29 04:15:42 inception kernel: ? __fget_light+0xa5/0x120
Sep 29 04:15:42 inception kernel: __x64_sys_ioctl+0xa0/0xe0
Sep 29 04:15:42 inception kernel: do_syscall_64+0x5b/0x90
Sep 29 04:15:42 inception kernel: ? fire_user_return_notifiers+0x3a/0x80
Sep 29 04:15:42 inception kernel: ? exit_to_user_mode_prepare+0x39/0x190
Sep 29 04:15:42 inception kernel: ? syscall_exit_to_user_mode+0x37/0x60
Sep 29 04:15:42 inception kernel: ? do_syscall_64+0x67/0x90
Sep 29 04:15:42 inception kernel: ? do_syscall_64+0x67/0x90
Sep 29 04:15:42 inception kernel: ? syscall_exit_to_user_mode+0x37/0x60
Sep 29 04:15:42 inception kernel: ? do_syscall_64+0x67/0x90
Sep 29 04:15:42 inception kernel: ? do_syscall_64+0x67/0x90
Sep 29 04:15:42 inception kernel: entry_SYSCALL_64_after_hwframe+0x73/0xdd
Sep 29 04:15:42 inception kernel: RIP: 0033:0x7f345fa77b3b
Sep 29 04:15:42 inception kernel: Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
Sep 29 04:15:42 inception kernel: RSP: 002b:00007f3456bb5fb0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
Sep 29 04:15:42 inception kernel: RAX: ffffffffffffffda RBX: 00005611284fbc60 RCX: 00007f345fa77b3b
Sep 29 04:15:42 inception kernel: RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 000000000000001d
Sep 29 04:15:42 inception kernel: RBP: 000000000000ae80 R08: 000056112696ce00 R09: 0000000000000000
Sep 29 04:15:42 inception kernel: R10: 0000000000000002 R11: 0000000000000246 R12: 0000000000000000
Sep 29 04:15:42 inception kernel: R13: 0000000000000001 R14: 0000000000000001 R15: 0000000000000000
Sep 29 04:15:42 inception kernel: </TASK>
Sep 29 04:15:42 inception kernel: ---[ end trace 0000000000000000 ]---
Sep 29 04:15:42 inception QEMU[1564]: error: kvm run failed Invalid argument
Sep 29 04:15:42 inception QEMU[1564]: RAX=0000000000265b0a RBX=0000000000003ae0 RCX=0000000000000017 RDX=0000000000004016
Sep 29 04:15:42 inception QEMU[1564]: RSI=00000000c9bc4d18 RDI=00007fc69ee43c40 RBP=00007fc6698fe098 RSP=00007fc6698fdac0
Sep 29 04:15:42 inception QEMU[1564]: R8 =000000000000012c R9 =00000000c905e548 R10=0000000100551738 R11=000000002004ddcd
Sep 29 04:15:42 inception QEMU[1564]: R12=0000000000000000 R13=0000000000000001 R14=00007fc6698fd670 R15=00007fc68002ad80
Sep 29 04:15:42 inception QEMU[1564]: RIP=00007fc6bb4bb026 RFL=00010202 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
Sep 29 04:15:42 inception QEMU[1564]: ES =0000 0000000000000000 ffffffff 00c00000
Sep 29 04:15:42 inception QEMU[1564]: CS =0033 0000000000000000 ffffffff 00a0fb00 DPL=3 CS64 [-RA]
Sep 29 04:15:42 inception QEMU[1564]: SS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA]
Sep 29 04:15:42 inception QEMU[1564]: DS =0000 0000000000000000 ffffffff 00c00000
Sep 29 04:15:42 inception QEMU[1564]: FS =0000 00007fc6698ff6c0 ffffffff 00c00000
Sep 29 04:15:42 inception QEMU[1564]: GS =0000 0000000000000000 ffffffff 00c00000
Sep 29 04:15:42 inception QEMU[1564]: LDT=0000 0000000000000000 ffffffff 00c00000
Sep 29 04:15:42 inception QEMU[1564]: TR =0040 fffffe000003e000 00004087 00008b00 DPL=0 TSS64-busy
Sep 29 04:15:42 inception QEMU[1564]: GDT= fffffe000003c000 0000007f
Sep 29 04:15:42 inception QEMU[1564]: IDT= fffffe0000000000 00000fff
Sep 29 04:15:42 inception QEMU[1564]: CR0=80050033 CR2=00007fc66f842000 CR3=0000000102f6d000 CR4=000006e0
Sep 29 04:15:42 inception QEMU[1564]: DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
Sep 29 04:15:42 inception QEMU[1564]: DR6=00000000ffff0ff0 DR7=0000000000000400
Sep 29 04:15:42 inception QEMU[1564]: EFER=0000000000000d01
Sep 29 04:15:42 inception QEMU[1564]: Code=9f f4 00 00 00 81 e3 fe ff 1f 00 83 fb 00 0f 84 26 e1 00 00 <8b> 76 0c 48 3b 06 48 8b fe 48 bb 40 3c e4 9e c6 7f 00 00 8b 7f 08 48 c1 e7 03 48 3b bb 48
Sep 29 04:16:54 inception smartd[670]: Device: /dev/sdb [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 69 to 68
Sep 29 04:17:01 inception CRON[30258]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Sep 29 04:17:01 inception CRON[30259]: (root) CMD (cd / && run-parts --report /etc/cron.hourly)