Web interface partially not responding - question marks on all nodes / BUG: kernel NULL pointer dereference

[moxmox]

My instance of proxmox seems to be working fine (all vms'v conatiners are running fine.

However when going to the the web interface It just shows question marks against everything. Its happened twice in the last week so not sure if it related to a recent update?

rebooting fixed it last time but want to fix without rebooting

stopping the pvestatd service seems to fail

Mar 15 15:32:13 pve systemd[1]: Stopping PVE Status Daemon...
Mar 15 15:32:14 pve pvestatd[8340]: received signal TERM
Mar 15 15:32:14 pve pvestatd[8340]: server closing
Mar 15 15:32:14 pve pvestatd[8340]: server stopped
Mar 15 15:33:45 pve systemd[1]: pvestatd.service: State 'stop-sigterm' timed out. Killing.
Mar 15 15:33:45 pve systemd[1]: pvestatd.service: Killing process 28142 (udevadm) with signal SIGKILL.
Mar 15 15:35:15 pve systemd[1]: pvestatd.service: Processes still around after SIGKILL. Ignoring.

and it just hangs trying to start it, any ideas?

 

[moxmox]

trying to start it again fails with this

Mar 15 15:45:47 pve systemd[1]: pvestatd.service: Processes still around after final SIGKILL. Entering failed mode.
Mar 15 15:45:47 pve systemd[1]: pvestatd.service: Failed with result 'timeout'.
Mar 15 15:45:47 pve systemd[1]: Failed to start PVE Status Daemon.

 

[moxmox]

hmm guess its probably caused by this

[549694.775318] BUG: kernel NULL pointer dereference, address: 0000000000000014
[549694.775471] #PF: supervisor read access in kernel mode
[549694.776082] #PF: error_code(0x0000) - not-present page
[549694.776832] PGD 0 P4D 0
[549694.777561] Oops: 0000 [#1] SMP PTI
[549694.778293] CPU: 7 PID: 23045 Comm: kworker/7:0 Tainted: P        W  O                                      5.3.13-3-pve #1
[549694.779011] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X99                                 WS, BIOS P3.50 04/06/2018
[549694.779740] Workqueue: events key_garbage_collector
[549694.780460] RIP: 0010:keyring_gc_check_iterator+0x30/0x40
[549694.781166] Code: 48 83 e7 fc b8 01 00 00 00 48 89 e5 f6 87 80 00 00 00 21 7                                5 19 48 8b 57 58 48 39 16 7c 05 48 85 d2 7f 0b 48 8b 87 a0 00 00 00 <0f> b6 40 1                                4 5d c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55
[549694.782618] RSP: 0018:ffff9a6f8aff7db8 EFLAGS: 00010246
[549694.783335] RAX: 0000000000000000 RBX: ffff8c78f54e7600 RCX: ffff9a6f8aff7e2                                0
[549694.784035] RDX: 0000000000000000 RSI: ffff9a6f8aff7e20 RDI: ffff8c7eb720990                                0
[549694.784727] RBP: ffff9a6f8aff7db8 R08: 0000000000000010 R09: 000000000000000                                0
[549694.785414] R10: 000000000000000f R11: 0000000000000000 R12: ffff8c78f54e769                                0
[549694.786087] R13: ffffffffa8e27dd0 R14: ffff8c7eb7209d00 R15: ffff8c78f54e767                                8
[549694.786752] FS:  0000000000000000(0000) GS:ffff8c835f9c0000(0000) knlGS:0000                                000000000000
[549694.787410] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[549694.788059] CR2: 0000000000000014 CR3: 0000001f32d14002 CR4: 00000000001626e                                0
[549694.788709] Call Trace:
[549694.789347]  assoc_array_subtree_iterate+0x5c/0x100
[549694.789967]  assoc_array_iterate+0x19/0x20
[549694.790579]  keyring_gc+0x43/0x80
[549694.791176]  key_garbage_collector+0x35a/0x400
[549694.791762]  process_one_work+0x20f/0x3d0
[549694.792342]  worker_thread+0x34/0x400
[549694.792907]  kthread+0x120/0x140
[549694.793454]  ? process_one_work+0x3d0/0x3d0
[549694.793992]  ? __kthread_parkme+0x70/0x70
[549694.794521]  ret_from_fork+0x35/0x40
[549694.795039] Modules linked in: nfsv4 w1_smem ds2490 wire tcp_diag inet_diag                                 rpcsec_gss_krb5 nf_conntrack_netlink xt_nat xt_MASQUERADE xfrm_user xfrm_algo ip                                table_nat nf_nat nfsv3 nfs veth ebtable_filter ebtables ip6table_raw ip6t_REJECT                                 nf_reject_ipv6 ip6table_filter ip6_tables iptable_raw xt_mac ipt_REJECT nf_reje                                ct_ipv4 xt_set xt_physdev xt_addrtype xt_multiport xt_conntrack nf_conntrack nf_                                defrag_ipv6 nf_defrag_ipv4 xt_comment xt_tcpudp xt_mark ip_set_hash_net ip_set m                                d4 cmac nls_utf8 cifs libarc4 fscache binfmt_misc iptable_filter bpfilter bondin                                g softdog nfnetlink_log nfnetlink snd_hda_codec_hdmi intel_rapl_msr intel_rapl_c                                ommon sb_edac x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm irqbypass crct                                10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_codec_realtek snd_hda_code                                c_generic ledtrig_audio nouveau snd_hda_intel aesni_intel snd_hda_codec video ae                                s_x86_64 ttm snd_hda_core crypto_simd drm_kms_helper snd_hwdep cryptd glue_helpe                                r intel_cstate intel_rapl_perf pcspkr
[549694.795061]  intel_wmi_thunderbolt drm mxm_wmi snd_pcm fb_sys_fops syscopyar                                ea snd_timer sysfillrect sysimgblt input_leds snd cdc_acm soundcore mei_me mei m                                ac_hid vhost_net vhost tap nfsd ib_iser rdma_cm iw_cm ib_cm auth_rpcgss nfs_acl                                 iscsi_tcp lockd libiscsi_tcp grace libiscsi scsi_transport_iscsi sunrpc nct6775                                 hwmon_vid coretemp ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zlua(PO) zavl                                (PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs xor zstd_compress raid6_pq lib                                crc32c mlx4_ib ib_uverbs ib_core mlx4_en hid_generic usbkbd usbmouse usbhid hid                                 mpt3sas raid_class ahci igb i2c_algo_bit qla2xxx i2c_i801 lpc_ich libahci scsi_t                                ransport_sas dca e1000e nvme_fc nvme_fabrics arcmsr mlx4_core scsi_transport_fc                                 wmi
[549694.803576] CR2: 0000000000000014
[549694.804280] ---[ end trace 95c4cdc70b373214 ]---
[549694.804993] RIP: 0010:keyring_gc_check_iterator+0x30/0x40
[549694.805684] Code: 48 83 e7 fc b8 01 00 00 00 48 89 e5 f6 87 80 00 00 00 21 7                                5 19 48 8b 57 58 48 39 16 7c 05 48 85 d2 7f 0b 48 8b 87 a0 00 00 00 <0f> b6 40 1                                4 5d c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55
[549694.807155] RSP: 0018:ffff9a6f8aff7db8 EFLAGS: 00010246
[549694.807885] RAX: 0000000000000000 RBX: ffff8c78f54e7600 RCX: ffff9a6f8aff7e2                                0
[549694.808611] RDX: 0000000000000000 RSI: ffff9a6f8aff7e20 RDI: ffff8c7eb720990                                0
[549694.809342] RBP: ffff9a6f8aff7db8 R08: 0000000000000010 R09: 000000000000000                                0
[549694.810066] R10: 000000000000000f R11: 0000000000000000 R12: ffff8c78f54e769                                0
[549694.810785] R13: ffffffffa8e27dd0 R14: ffff8c7eb7209d00 R15: ffff8c78f54e767                                8
[549694.811496] FS:  0000000000000000(0000) GS:ffff8c835f9c0000(0000) knlGS:0000                                000000000000
[549694.812209] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[549694.812895] CR2: 0000000000000014 CR3: 0000001f32d14002 CR4: 00000000001626e                                0

 

[leofabri]

Were you able to solve the problem? What was the cause of it?