WebGUI not displaying storage, but not only please help

Feb 13, 2024
9
0
1
Hi
I got weird situation. Cluster suddenly stopped displaying storage (only local is showing), not showing any users and I can't migrate vm's around.
Cluster had beforehand nfs mounted, backup server and iSCSI. When I'm logged to console I can see that multipath is showing all targets available, and pvs showing correctly iscsi storage:
Code:
PV                 VG         Fmt  Attr PSize    PFree 
/dev/mapper/mpatha VG-sa01hol lvm2 a--   <10.28t  <5.65t
/dev/sda3          pve        lvm2 a--  <222.00g <16.00g

however, nfs is gone and backup is gone.

I can see users in /etc/pve/pve/user.cfg, but again they not shown in webGUI and (for example) monitoring user is not authenticating - monitoring system reporting auth failure.

Is anybody able to help with this one?

Code:
proxmox-ve: 7.4-1 (running kernel: 5.15.158-2-pve)
pve-manager: 7.4-19 (running version: 7.4-19/f98bf8d4)
pve-kernel-5.15: 7.4-15
pve-kernel-5.15.158-2-pve: 5.15.158-2
pve-kernel-5.15.102-1-pve: 5.15.102-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx4
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.4
libproxmox-backup-qemu0: 1.3.1-1
libproxmox-rs-perl: 0.2.1
libpve-access-control: 7.4.3
libpve-apiclient-perl: 3.2-2
libpve-common-perl: 7.4-2
libpve-guest-common-perl: 4.2-4
libpve-http-server-perl: 4.3.0
libpve-rs-perl: 0.7.7
libpve-storage-perl: 7.4-4
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.2-2
lxcfs: 5.0.3-pve1
novnc-pve: 1.4.0-1
openvswitch-switch: 2.15.0+ds1-2+deb11u5
proxmox-backup-client: 2.4.7-1
proxmox-backup-file-restore: 2.4.7-1
proxmox-kernel-helper: 7.4-1
proxmox-mail-forward: 0.1.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.7.4
pve-cluster: 7.3-3
pve-container: 4.4-7
pve-docs: 7.4-2
pve-edk2-firmware: 3.20230228-4~bpo11+3
pve-firewall: 4.3-5
pve-firmware: 3.6-6
pve-ha-manager: 3.6.1
pve-i18n: 2.12-1
pve-qemu-kvm: 7.2.10-1
pve-xtermjs: 4.16.0-2
qemu-server: 7.4-6
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+3
vncterm: 1.7-1
zfsutils-linux: 2.1.15-pve1
 
If anybody got slightest idea where to start, please share. All vm's are in state that they can't be rebooted, as they will loose connectivity to storage. Also they can't be backed up, again because of error accessing iscsi storage.
 
Did some digging, and all looks ok to me.

Code:
sqlite3 /root/config.db 'PRAGMA integrity_check'
ok

As the situation is "hanging" dangerously, meaning I can't reboot any of the vm or nodes. I'm glueing up cluster next to existing one and will restore vm's from backup. In the meantime any input much appreciated.