Web interface become and unresponsive after random time being online

[nirungu]

The Web interface become unresponsive and unusable. all VMs apart from MacOS VM (204) still seem to work, interface just has question marks on evething the tasks doess seem to update when I login via SSH and type "reboot" it says Bulk Shutdown VMs and Containers - unable to read tail (got 0 bytes)
Eventually managed to get to reboot by ssh into it and using the the command systemctl --force --force reboot
This happens randomly it can be weeks sometimes

 

[bbgeek17]

Hi @nirungu , welcome to the forum.

The PVE GUI is a separate service from PVE virtualization layer (QEMU). The GUI relies on a number of services, in particular it needs "pvestatd" to return good information.
"pvestatd" could be "confused" due to various environmental issues: full root filesystem, bad network configuration (host/IP), cluster services issues, hardware failures, etc.

You need to examine your system's "journal" (journalctl -n 500), available space, cluster state (if any), etc.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[nirungu]

I found the following
Jul 23 14:48:14 proxmox kernel: ------------[ cut here ]------------
Jul 23 14:48:14 proxmox kernel: kernel BUG at mm/ksm.c:1036!
Jul 23 14:48:14 proxmox kernel: invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
Jul 23 14:48:14 proxmox kernel: CPU: 10 PID: 219 Comm: ksmd Tainted: P U O 6.8.12-11-pve #1
Jul 23 14:48:14 proxmox kernel: Hardware name: Dell Inc. Precision 3660/0PRR48, BIOS 2.12.0 01/30/2024
Jul 23 14:48:14 proxmox kernel: RIP: 0010:remove_rmap_item_from_tree+0x1c2/0x1d0
Jul 23 14:48:14 proxmox kernel: Code: 28 48 8d 34 d0 e8 de 76 c7 00 48 8b 43 18 e9 8e fe ff ff e8 b0 d2 fa ff e9 4d ff ff ff 4c 89 e7 e8 33 32 f4 ff e9 0c ff ff ff <0f> 0b 4c 8d 60 ff e9 f3 fe ff ff 0f 1f 00 90 90 90 90 90 90 90 90
Jul 23 14:48:14 proxmox kernel: RSP: 0018:ffffb9a40097fdf0 EFLAGS: 00010202
Jul 23 14:48:14 proxmox kernel: RAX: 000071f01fe3d000 RBX: ffff9d2ee5a2cfc0 RCX: 0000000000000000
Jul 23 14:48:14 proxmox kernel: RDX: 00000000e01c3002 RSI: 0000000000000004 RDI: ffff9d2ee5a2cfc0
Jul 23 14:48:14 proxmox kernel: RBP: ffffb9a40097fe08 R08: 0000000000000000 R09: 0000000000000000
Jul 23 14:48:14 proxmox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
Jul 23 14:48:14 proxmox kernel: R13: ffff9d2ee5a2cf40 R14: ffff9d2c5c690900 R15: ffffe0ac7af80f40
Jul 23 14:48:14 proxmox kernel: FS: 0000000000000000(0000) GS:ffff9d3fcf100000(0000) knlGS:0000000000000000
Jul 23 14:48:14 proxmox kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 23 14:48:14 proxmox kernel: CR2: 000000dcb3b1825a CR3: 00000001a3af4006 CR4: 0000000000f72ef0
Jul 23 14:48:14 proxmox kernel: PKRU: 55555554
Jul 23 14:48:14 proxmox kernel: Call Trace:
Jul 23 14:48:14 proxmox kernel: <TASK>
Jul 23 14:48:14 proxmox kernel: ? show_regs+0x6d/0x80
Jul 23 14:48:14 proxmox kernel: ? die+0x37/0xa0
Jul 23 14:48:14 proxmox kernel: ? do_trap+0xd4/0xf0
Jul 23 14:48:14 proxmox kernel: ? do_error_trap+0x71/0xb0
Jul 23 14:48:14 proxmox kernel: ? remove_rmap_item_from_tree+0x1c2/0x1d0
Jul 23 14:48:14 proxmox kernel: ? exc_invalid_op+0x52/0x80
Jul 23 14:48:14 proxmox kernel: ? remove_rmap_item_from_tree+0x1c2/0x1d0
Jul 23 14:48:14 proxmox kernel: ? asm_exc_invalid_op+0x1b/0x20
Jul 23 14:48:14 proxmox kernel: ? remove_rmap_item_from_tree+0x1c2/0x1d0
Jul 23 14:48:14 proxmox kernel: ksm_scan_thread+0x806/0x22c0
Jul 23 14:48:14 proxmox kernel: ? __pfx_ksm_scan_thread+0x10/0x10
Jul 23 14:48:14 proxmox kernel: kthread+0xef/0x120
Jul 23 14:48:14 proxmox kernel: ? __pfx_kthread+0x10/0x10
Jul 23 14:48:14 proxmox kernel: ret_from_fork+0x44/0x70
Jul 23 14:48:14 proxmox kernel: ? __pfx_kthread+0x10/0x10
Jul 23 14:48:14 proxmox kernel: ret_from_fork_asm+0x1b/0x30
Jul 23 14:48:14 proxmox kernel: </TASK>
Jul 23 14:48:14 proxmox kernel: Modules linked in: tcp_diag inet_diag nls_utf8 cifs cifs_arc4 nls_ucs2_utils rdma_cm iw_cm ib_cm ib_core cifs_md4 cfg80211 veth rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace netfs ebtable_filter ebtables ip_set ip6table_raw iptable_ra>
Jul 23 14:48:14 proxmox kernel: processor_thermal_device_pci i915 soundwire_bus sha1_ssse3 processor_thermal_device snd_soc_core aesni_intel processor_thermal_wt_hint processor_thermal_rfim snd_compress drm_buddy ac97_bus intel_rapl_msr processor_thermal_rapl crypto_si>
Jul 23 14:48:14 proxmox kernel: blake2b_generic xor raid6_pq dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c hid_generic usbkbd usbmouse usbhid hid ixgbe xhci_pci xfrm_algo nvme intel_lpss_pci r8169 rtsx_pci_sdmmc i2c_i801 xhci_pci_renesas ahci spi_int>
Jul 23 14:48:14 proxmox kernel: ---[ end trace 0000000000000000 ]---
Jul 23 14:48:14 proxmox kernel: RIP: 0010:remove_rmap_item_from_tree+0x1c2/0x1d0
Jul 23 14:48:14 proxmox kernel: Code: 28 48 8d 34 d0 e8 de 76 c7 00 48 8b 43 18 e9 8e fe ff ff e8 b0 d2 fa ff e9 4d ff ff ff 4c 89 e7 e8 33 32 f4 ff e9 0c ff ff ff <0f> 0b 4c 8d 60 ff e9 f3 fe ff ff 0f 1f 00 90 90 90 90 90 90 90 90
Jul 23 14:48:14 proxmox kernel: RSP: 0018:ffffb9a40097fdf0 EFLAGS: 00010202
Jul 23 14:48:14 proxmox kernel: RAX: 000071f01fe3d000 RBX: ffff9d2ee5a2cfc0 RCX: 0000000000000000
Jul 23 14:48:14 proxmox kernel: RDX: 00000000e01c3002 RSI: 0000000000000004 RDI: ffff9d2ee5a2cfc0
Jul 23 14:48:14 proxmox kernel: RBP: ffffb9a40097fe08 R08: 0000000000000000 R09: 0000000000000000
Jul 23 14:48:14 proxmox kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
Jul 23 14:48:14 proxmox kernel: R13: ffff9d2ee5a2cf40 R14: ffff9d2c5c690900 R15: ffffe0ac7af80f40
Jul 23 14:48:14 proxmox kernel: FS: 0000000000000000(0000) GS:ffff9d3fcf100000(0000) knlGS:0000000000000000
Jul 23 14:48:14 proxmox kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Jul 23 14:48:14 proxmox kernel: CR2: 000000dcb3b1825a CR3: 00000001a3af4001 CR4: 0000000000f72ef0
Jul 23 14:48:14 proxmox kernel: PKRU: 55555554
Jul 23 14:55:01 proxmox CRON[937594]: pam_unix(cron:session): session opened for

Then there was these later which explains the no resposive interface

Jul 24 13:53:40 proxmox pveproxy[529146]: proxy detected vanished client connection
Jul 24 13:53:45 proxmox pveproxy[529146]: proxy detected vanished client connection
Jul 24 13:53:45 proxmox pveproxy[529147]: proxy detected vanished client connection
Jul 24 13:54:07 proxmox pveproxy[529146]: proxy detected vanished client connection
Jul 24 13:54:09 proxmox pveproxy[529146]: proxy detected vanished client connection
Jul 24 13:54:10 proxmox pveproxy[529145]: proxy detected vanished client connection
Jul 24 13:54:15 proxmox pveproxy[529146]: proxy detected vanished client connection
Jul 24 13:54:15 proxmox pveproxy[529146]: proxy detected vanished client connection
Jul 24 13:54:25 proxmox pveproxy[529147]: proxy detected vanished client connection
Jul 24 13:54:26 proxmox pveproxy[529145]: proxy detected vanished client connection
Jul 24 13:54:26 proxmox pveproxy[529147]: proxy detected vanished client connection
Jul 24 13:54:37 proxmox pveproxy[529146]: proxy detected vanished client connection
Jul 24 13:55:01 proxmox CRON[1401958]: pam_unix(cron:session): session opened for user