Proxmox web gui not working(and more); VM works

jaceqp

Well-Known Member
May 28, 2018
98
7
48
45
Hi there...
I've installed Proxmox with ZFS on an old server (1x 4core cpu, 12GB RAM, 2x HDD SATA 1TB).
I've installed OpenMediaVault on it as a single VM.

Now after... lets say... 2 months I noticed some weired behavior in gui. Everything worked fine except missing icon in left panel VM/storage list (exclamation mark icons instead of default ones).
Now, after 2-3 days I can no longer access to PM web qui (page not found).
Funny (or not) is that VM itself still works fine. After a reboot the problem remains.
At least I can still remote access via SHH command line...

But there's more, I tried to do some troubleshooting but for example executing 'systemctl' shows:
Code:
# systemctl
Failed to list units: Connection timed out

By the way, version:
Code:
# pveversion
pve-manager/5.2-1/0fcd7879 (running kernel: 4.15.17-1-pve)

I wasn't doing any modifications for weeks. No updates what so ever...

Any ideas?
 
does dmesg/syslog say anything? maybe a broken disk ?

if even systemctl fails, i am not sure what can be done...
 
If you still can get to it via ssh.
As fast as you can get the all config out. Do a vm backup and get it off the machine.
I lost whole setup like this.
Depending on your hardware it can be your psu
 
I've checked ZFS status (seems fine):
Code:
# zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
rpool   928G   184G   744G         -    22%    19%  1.00x  ONLINE  -

# zpool status
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 1h11m with 0 errors on Sun Dec  9 01:35:56 2018
config:

        NAME        STATE     READ WRITE CKSUM
        rpool       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sda2    ONLINE       0     0     0
            sdb2    ONLINE       0     0     0

errors: No known data errors

I'm also using ZFS filesystem snapshot solution to make full/differential disk backups:

Code:
# zfs get written
NAME                                                               PROPERTY  VAL                UE    SOURCE
rpool                                                              written   0                        -
rpool@zfs-auto-snap_monthly-2018-11-01-0552                        written   128                K     -
rpool@zfs-auto-snap_monthly-2018-12-01-0552                        written   0                        -
rpool@zfs-auto-snap_daily-2018-12-03-0525                          written   0                        -
rpool@zfs-auto-snap_daily-2018-12-04-0525                          written   0                        -
rpool@zfs-auto-snap_daily-2018-12-05-0525                          written   0                        -
rpool@zfs-auto-snap_daily-2018-12-06-0525                          written   0                        -
rpool@zfs-auto-snap_daily-2018-12-07-0525                          written   0                        -
rpool@zfs-auto-snap_daily-2018-12-08-0525                          written   0                        -
rpool@zfs-auto-snap_daily-2018-12-09-0525                          written   0                        -
rpool@zfs-auto-snap_daily-2018-12-10-0525                          written   0                        -
rpool@zfs-auto-snap_daily-2018-12-11-0525                          written   0                        -
rpool@zfs-auto-snap_daily-2018-12-12-0525                          written   0                        -
rpool@zfs-auto-snap_daily-2018-12-13-0525                          written   0                        -
rpool@zfs-auto-snap_daily-2018-12-14-0525                          written   0                        -
rpool@zfs-auto-snap_daily-2018-12-15-0525                          written   0                        -
rpool@zfs-auto-snap_daily-2018-12-16-0525                          written   0                        -
rpool/ROOT                                                         written   96K                      -
rpool/ROOT/pve-1                                                   written   5.1                4M    -
rpool/ROOT/pve-1@zfs-auto-snap_monthly-2018-11-01-0552             written   6.3                7G    -
rpool/ROOT/pve-1@zfs-auto-snap_monthly-2018-12-01-0552             written   6.1                3M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-03-0525               written   4.9                8M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-04-0525               written   2.8                1M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-05-0525               written   3.0                4M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-06-0525               written   2.6                0M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-07-0525               written   3.0                9M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-08-0525               written   3.1                1M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-09-0525               written   3.1                7M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-10-0525               written   4.6                4M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-11-0525               written   2.9                8M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-12-0525               written   5.4                9M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-13-0525               written   2.3                0M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-14-0525               written   3.1                3M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-15-0525               written   3.0                9M    -
rpool/ROOT/pve-1@zfs-auto-snap_daily-2018-12-16-0525               written   3.2                9M    -
rpool/data                                                         written   0                        -
rpool/data@zfs-auto-snap_monthly-2018-11-01-0552                   written   96K                      -
rpool/data@zfs-auto-snap_monthly-2018-12-01-0552                   written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-03-0525                     written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-04-0525                     written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-05-0525                     written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-06-0525                     written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-07-0525                     written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-08-0525                     written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-09-0525                     written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-10-0525                     written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-11-0525                     written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-12-0525                     written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-13-0525                     written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-14-0525                     written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-15-0525                     written   0                        -
rpool/data@zfs-auto-snap_daily-2018-12-16-0525                     written   0                        -
rpool/disks                                                        written   0                        -
rpool/disks@zfs-auto-snap_monthly-2018-11-01-0552                  written   96K                      -
rpool/disks@zfs-auto-snap_monthly-2018-12-01-0552                  written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-03-0525                    written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-04-0525                    written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-05-0525                    written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-06-0525                    written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-07-0525                    written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-08-0525                    written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-09-0525                    written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-10-0525                    written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-11-0525                    written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-12-0525                    written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-13-0525                    written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-14-0525                    written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-15-0525                    written   0                        -
rpool/disks@zfs-auto-snap_daily-2018-12-16-0525                    written   0                        -
rpool/disks/vm-100-disk-1                                          written   589                M     -
rpool/disks/vm-100-disk-1@zfs-auto-snap_monthly-2018-11-01-0552    written   3.3                3G    -
rpool/disks/vm-100-disk-1@zfs-auto-snap_monthly-2018-12-01-0552    written   0                        -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-03-0525      written   0                        -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-04-0525      written   0                        -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-05-0525      written   0                        -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-06-0525      written   0                        -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-07-0525      written   0                        -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-08-0525      written   0                        -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-09-0525      written   0                        -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-10-0525      written   0                        -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-11-0525      written   0                        -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-12-0525      written   756                M     -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-13-0525      written   461                M     -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-14-0525      written   428                M     -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-15-0525      written   412                M     -
rpool/disks/vm-100-disk-1@zfs-auto-snap_daily-2018-12-16-0525      written   419                M     -
rpool/disks/vm-101-disk-1                                          written   970                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_monthly-2018-11-01-0552    written   13.                3G    -
rpool/disks/vm-101-disk-1@zfs-auto-snap_monthly-2018-12-01-0552    written   1.9                5G    -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-03-0525      written   960                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-04-0525      written   682                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-05-0525      written   518                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-06-0525      written   507                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-07-0525      written   661                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-08-0525      written   582                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-09-0525      written   758                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-10-0525      written   591                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-11-0525      written   475                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-12-0525      written   590                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-13-0525      written   450                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-14-0525      written   518                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-15-0525      written   455                M     -
rpool/disks/vm-101-disk-1@zfs-auto-snap_daily-2018-12-16-0525      written   800                M     -
rpool/iso                                                          written   1.8                9G    -
rpool/storage                                                      written   0                        -
rpool/storage@zfs-auto-snap_monthly-2018-11-01-0552                written   96K                      -
rpool/storage@zfs-auto-snap_monthly-2018-12-01-0552                written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-03-0525                  written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-04-0525                  written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-05-0525                  written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-06-0525                  written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-07-0525                  written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-08-0525                  written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-09-0525                  written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-10-0525                  written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-11-0525                  written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-12-0525                  written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-13-0525                  written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-14-0525                  written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-15-0525                  written   0                        -
rpool/storage@zfs-auto-snap_daily-2018-12-16-0525                  written   0                        -
rpool/storage/vm-101-disk-1                                        written   133                M     -
rpool/storage/vm-101-disk-1@zfs-auto-snap_monthly-2018-11-01-0552  written   135                G     -
rpool/storage/vm-101-disk-1@zfs-auto-snap_monthly-2018-12-01-0552  written   655                M     -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-03-0525    written   112                M     -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-04-0525    written   53.                2M    -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-05-0525    written   48.                8M    -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-06-0525    written   29.                7M    -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-07-0525    written   53.                3M    -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-08-0525    written   45.                6M    -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-09-0525    written   79.                0M    -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-10-0525    written   60.                0M    -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-11-0525    written   54.                6M    -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-12-0525    written   46.                0M    -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-13-0525    written   38.                4M    -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-14-0525    written   38.                0M    -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-15-0525    written   46.                9M    -
rpool/storage/vm-101-disk-1@zfs-auto-snap_daily-2018-12-16-0525    written   58.                8M    -
rpool/swap                                                         written   7.5                7G    -