We deployed 9 physical servers at various sites in our environment all running Proxmox as the hypervisor. None of the servers are clustered. This was a couple months ago and we haven't really had any need to go into Proxmox in that time. I wanted to check on something last week and discovered that I am no longer able to sign into the web GUI. All our VMs are still happily running. The first thing I noticed is that when I navigate to the login page, the realm field is empty with nothing to select (usually I see Linux PAM standard authentication, Proxmox VE authentication server, and our domain name which I set up for AD auth). Obviously the sign-in goes nowhere without a realm. When I refresh the page a couple times the realms come back, however I am still unable to login either using PAM (root) or our AD connector. I know that I have the correct root password for PAM as I am able to login via ssh. I saw some posts about this being a caching issue, so I tried Chrome, Edge, and Firefox (on two different computers) with the same results, so I know it's not my browser.
This issue is happening on all 9 servers so I'm concerned we've done something wrong when we set them up. I did see a post about apt upgrade breaking things and this is what was used when the servers were built (as opposed to apt dist-upgrade). When I attempt to install anything now I get:
systemctl status pvestatd.service shows:
And journalctl -xeu pvestatd.service shows:
I have seen a couple error messages relating to "Unable to load access control list: No buffer space available", however I've confirmed there is plenty of RAM available as well as disk space.
As I have access to SSH, please let me know any logs that would be helpful to see.
Here's the versioning:
This issue is happening on all 9 servers so I'm concerned we've done something wrong when we set them up. I did see a post about apt upgrade breaking things and this is what was used when the servers were built (as opposed to apt dist-upgrade). When I attempt to install anything now I get:
Code:
Setting up pve-manager (8.1.4) ...
LVM configuration valid.
Job for pvestatd.service failed.
See "systemctl status pvestatd.service" and "journalctl -xeu pvestatd.service" for details.
dpkg: error processing package pve-manager (--configure):
installed pve-manager package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
pve-manager
E: Sub-process /usr/bin/dpkg returned an error code (1)systemctl status pvestatd.service shows:
Code:
● pvestatd.service - PVE Status Daemon
Loaded: loaded (/lib/systemd/system/pvestatd.service; enabled; preset: enabled)
Active: active (running) since Thu 2023-12-07 11:25:20 EST; 1 month 22 days ago
Process: 2246566 ExecReload=/usr/bin/pvestatd restart (code=exited, status=105)
Main PID: 1074 (pvestatd)
Tasks: 1 (limit: 76890)
Memory: 159.9M
CPU: 3h 50min 59.356s
CGroup: /system.slice/pvestatd.service
└─1074 pvestatd
Jan 29 08:59:06 STP-HV systemd[1]: Reload failed for pvestatd.service - PVE Status Daemon.
Jan 29 09:38:41 STP-HV systemd[1]: Reloading pvestatd.service - PVE Status Daemon...
Jan 29 09:38:41 STP-HV pvestatd[2242371]: Unable to load access control list: No buffer space available
Jan 29 09:38:41 STP-HV systemd[1]: pvestatd.service: Control process exited, code=exited, status=105/n/a
Jan 29 09:38:41 STP-HV systemd[1]: Reload failed for pvestatd.service - PVE Status Daemon.
Jan 29 09:45:07 STP-HV pvestatd[1074]: auth key pair too old, rotating..
Jan 29 10:07:54 STP-HV systemd[1]: Reloading pvestatd.service - PVE Status Daemon...
Jan 29 10:07:55 STP-HV pvestatd[2246566]: Unable to load access control list: No buffer space available
Jan 29 10:07:55 STP-HV systemd[1]: pvestatd.service: Control process exited, code=exited, status=105/n/a
Jan 29 10:07:55 STP-HV systemd[1]: Reload failed for pvestatd.service - PVE Status Daemon.And journalctl -xeu pvestatd.service shows:
Code:
Jan 29 10:07:54 STP-HV systemd[1]: Reloading pvestatd.service - PVE Status Daemon...
░░ Subject: A reload job for unit pvestatd.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A reload job for unit pvestatd.service has begun execution.
░░
░░ The job identifier is 46219.
Jan 29 10:07:55 STP-HV pvestatd[2246566]: Unable to load access control list: No buffer space available
Jan 29 10:07:55 STP-HV systemd[1]: pvestatd.service: Control process exited, code=exited, status=105/n/a
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecReload= process belonging to unit pvestatd.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 105.
Jan 29 10:07:55 STP-HV systemd[1]: Reload failed for pvestatd.service - PVE Status Daemon.
░░ Subject: A reload job for unit pvestatd.service has finished
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A reload job for unit pvestatd.service has finished.
░░
░░ The job identifier is 46219 and the job result is failed.I have seen a couple error messages relating to "Unable to load access control list: No buffer space available", however I've confirmed there is plenty of RAM available as well as disk space.
As I have access to SSH, please let me know any logs that would be helpful to see.
Here's the versioning:
Code:
root@STP-HV:/# pveversion -v
proxmox-ve: 8.1.0 (running kernel: 6.2.16-19-pve)
pve-manager: not correctly installed (running version: 8.1.4/ec5affc9e41f1d79)
proxmox-kernel-helper: 8.1.0
pve-kernel-6.2: 8.0.5
proxmox-kernel-6.5: 6.5.11-7
proxmox-kernel-6.5.11-7-pve-signed: 6.5.11-7
proxmox-kernel-6.2.16-20-pve: 6.2.16-20
proxmox-kernel-6.2: 6.2.16-20
proxmox-kernel-6.2.16-19-pve: 6.2.16-19
pve-kernel-6.2.16-3-pve: 6.2.16-3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.0.7
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.1.0
libpve-guest-common-perl: 5.0.6
libpve-http-server-perl: 5.0.5
libpve-network-perl: 0.9.5
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.0.5
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve4
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.1.2-1
proxmox-backup-file-restore: 3.1.2-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.1.3
pve-cluster: 8.0.5
pve-container: 5.0.8
pve-docs: 8.1.3
pve-edk2-firmware: 4.2023.08-3
pve-firewall: 5.0.3
pve-firmware: 3.9-1
pve-ha-manager: 4.0.3
pve-i18n: 3.2.0
pve-qemu-kvm: 8.1.2-6
pve-xtermjs: 5.3.0-3
qemu-server: 8.0.10
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.2-pve1
Last edited:
