GUI login failed. SSH works

[naters210]

I cannot login to the GUI but when I SSH in with same credentials it works. I've been searching the forums for others with the same problem, and I see that those with remote mounts not working then they can't login. I did setup nfs shares for one of my containers, but when I look at the storage.cfg file, they are not listed.

Any guidance would be appreciated!

proxmox-ve: 7.3-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.3-3 (running version: 7.3-3/c3928077)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-8
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.2-12
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.5-6
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-1
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1


Journal file /var/log/journal/63b79848ccdb4c3a946d3d643eb0781b/system.journal is truncated, ignoring file.
-- Journal begins at Mon 2023-02-20 22:22:19 EST, ends at Fri 2023-03-24 07:24:17 EDT. --
Mar 24 06:37:35 pve1 pveproxy[192142]: worker exit
Mar 24 06:37:35 pve1 pveproxy[987]: worker 192142 finished
Mar 24 06:37:35 pve1 pveproxy[987]: starting 1 worker(s)
Mar 24 06:37:35 pve1 pveproxy[987]: worker 494359 started
Mar 24 06:37:44 pve1 pveproxy[192239]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1791.
Mar 24 06:37:44 pve1 pveproxy[192239]: error writing access log
Mar 24 06:45:32 pve1 pveproxy[192239]: worker exit
Mar 24 06:45:32 pve1 pveproxy[987]: worker 192239 finished
Mar 24 06:45:32 pve1 pveproxy[987]: starting 1 worker(s)
Mar 24 06:45:32 pve1 pveproxy[987]: worker 495442 started
Mar 24 06:45:35 pve1 pveproxy[206184]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1791.
Mar 24 06:45:35 pve1 pveproxy[206184]: error writing access log
Mar 24 06:48:16 pve1 pveproxy[206184]: worker exit
Mar 24 06:48:16 pve1 pveproxy[987]: worker 206184 finished
Mar 24 06:48:16 pve1 pveproxy[987]: starting 1 worker(s)
Mar 24 06:48:16 pve1 pveproxy[987]: worker 495812 started
Mar 24 06:48:25 pve1 pveproxy[494359]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1791.
Mar 24 06:48:25 pve1 pveproxy[494359]: error writing access log
Mar 24 06:48:50 pve1 pveproxy[494359]: worker exit
Mar 24 06:48:50 pve1 pveproxy[987]: worker 494359 finished
Mar 24 06:48:50 pve1 pveproxy[987]: starting 1 worker(s)
Mar 24 06:48:50 pve1 pveproxy[987]: worker 495885 started
Mar 24 06:48:59 pve1 pveproxy[495442]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1791.
Mar 24 06:48:59 pve1 pveproxy[495442]: error writing access log

 

[Matthias.]

well, seems you don't have enough space left. Free some up and it should work again
What's output of df -h?

 

[naters210]

That may help yes. I assume they might be my backups as I couldn't login for a few days. How can I do that through SSH? Sorry, I'm still new with working in the command line.

Filesystem Size Used Avail Use% Mounted on
udev 7.8G 0 7.8G 0% /dev
tmpfs 1.6G 160M 1.4G 11% /run
/dev/mapper/pve-root 94G 94G 0 100% /
tmpfs 7.8G 40M 7.8G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/fuse 128M 24K 128M 1% /etc/pve
tmpfs 1.6G 0 1.6G 0% /run/user/0

 

[Matthias.]

You can use e.g. du or ncdu (but not without space to install) to locate large files. /var/tmp is something you might want to clean up.
PVE Backups themselves are in /var/lib/vz/dump. ISO files in /var/lib/vz/template/iso. See https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_storage_configuration

 

[naters210]

Thanks. Removing the backups in /var/lib/vz/dump.

Filesystem Size Used Avail Use% Mounted on
udev 7.8G 0 7.8G 0% /dev
tmpfs 1.6G 157M 1.4G 10% /run
/dev/mapper/pve-root 94G 13G 77G 14% /
tmpfs 7.8G 40M 7.8G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/fuse 128M 24K 128M 1% /etc/pve
tmpfs 1.6G 0 1.6G 0% /run/user/0

Do I have to delete items in the /run directory to free up space on those partitions?

 

[Matthias.]

Can you login now?

No. They also have plenty of space left.

 

[naters210]

No same error. Here is my journal file now if that helps.

-- Journal begins at Mon 2023-02-20 22:22:19 EST, ends at Fri 2023-03-24 08:06:21 EDT. --
Mar 24 07:41:06 pve1 pveproxy[495442]: worker exit
Mar 24 07:41:06 pve1 pveproxy[987]: worker 495442 finished
Mar 24 07:41:06 pve1 pveproxy[987]: starting 1 worker(s)
Mar 24 07:41:06 pve1 pveproxy[987]: worker 502922 started
Mar 24 07:41:15 pve1 pveproxy[495812]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1791.
Mar 24 07:41:15 pve1 pveproxy[495812]: error writing access log
Mar 24 08:04:25 pve1 pveproxy[495812]: worker exit
Mar 24 08:04:25 pve1 pveproxy[987]: worker 495812 finished
Mar 24 08:04:25 pve1 pveproxy[987]: starting 1 worker(s)
Mar 24 08:04:25 pve1 pveproxy[987]: worker 506048 started

 

[Matthias.]

Huh. maybe you need to restart pveproxy? systemctl restart pveproxy

 

[naters210]

After doing that, I tried logging into the GUI and still receive same error.

-- Journal begins at Mon 2023-02-20 22:22:19 EST, ends at Fri 2023-03-24 08:23:47 EDT. --
Mar 24 07:41:06 pve1 pveproxy[495442]: worker exit
Mar 24 07:41:06 pve1 pveproxy[987]: worker 495442 finished
Mar 24 07:41:06 pve1 pveproxy[987]: starting 1 worker(s)
Mar 24 07:41:06 pve1 pveproxy[987]: worker 502922 started
Mar 24 07:41:15 pve1 pveproxy[495812]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 1791.
Mar 24 07:41:15 pve1 pveproxy[495812]: error writing access log
Mar 24 08:04:25 pve1 pveproxy[495812]: worker exit
Mar 24 08:04:25 pve1 pveproxy[987]: worker 495812 finished
Mar 24 08:04:25 pve1 pveproxy[987]: starting 1 worker(s)
Mar 24 08:04:25 pve1 pveproxy[987]: worker 506048 started
Mar 24 08:23:17 pve1 systemd[1]: Stopping PVE API Proxy Server...
Mar 24 08:23:17 pve1 pveproxy[987]: received signal TERM
Mar 24 08:23:17 pve1 pveproxy[987]: server closing
Mar 24 08:23:17 pve1 pveproxy[495885]: worker exit
Mar 24 08:23:17 pve1 pveproxy[506048]: worker exit
Mar 24 08:23:17 pve1 pveproxy[502922]: worker exit
Mar 24 08:23:17 pve1 pveproxy[987]: worker 495885 finished
Mar 24 08:23:17 pve1 pveproxy[987]: worker 506048 finished
Mar 24 08:23:17 pve1 pveproxy[987]: worker 502922 finished
Mar 24 08:23:17 pve1 pveproxy[987]: server stopped
Mar 24 08:23:18 pve1 systemd[1]: pveproxy.service: Succeeded.
Mar 24 08:23:18 pve1 systemd[1]: Stopped PVE API Proxy Server.
Mar 24 08:23:18 pve1 systemd[1]: pveproxy.service: Consumed 20.222s CPU time.
Mar 24 08:23:18 pve1 systemd[1]: Starting PVE API Proxy Server...
Mar 24 08:23:19 pve1 pvecm[508575]: unable to open file '/etc/pve/priv/authorized_keys.tmp.508577' - Input/output error
Mar 24 08:23:19 pve1 pveproxy[508579]: starting server
Mar 24 08:23:19 pve1 pveproxy[508579]: starting 3 worker(s)
Mar 24 08:23:19 pve1 pveproxy[508579]: worker 508580 started
Mar 24 08:23:19 pve1 pveproxy[508579]: worker 508581 started
Mar 24 08:23:19 pve1 pveproxy[508579]: worker 508582 started
Mar 24 08:23:19 pve1 systemd[1]: Started PVE API Proxy Server.

 

[vanlueckn]

I have the same error, do you still have this issue?

 

[Dunuin]

Did you check if your root filesystem is full? This is the most common case why the pveproxy would fail.