No webgui interface, but with access via ssh

berg_rj · Sep 3, 2024

Friends, initially I report the following problem. initially proxmox accessed via ssh but gave a password error in the webgui. the two VMs that were on it were working fine. After restarting the GUI interface stopped working and the vms did not go up. I ran some commands I saw on the forum and got the following results:
root@pve03:~# systemctl status pve-cluster.service
× pve-cluster.service - The Proxmox VE cluster filesystem
Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Tue 2024-09-03 09:46:09 -03; 37min ago
Process: 11456 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)
CPU: 24ms

Sep 03 09:46:09 pve03 systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Sep 03 09:46:09 pve03 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
Sep 03 09:46:09 pve03 systemd[1]: pve-cluster.service: Start request repeated too quickly.
Sep 03 09:46:09 pve03 systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Sep 03 09:46:09 pve03 systemd[1]: Failed to start pve-cluster.service - The Proxmox VE cluster filesystem.

root@pve03:~# journalctl -f
Sep 03 08:57:16 pve03 pveproxy[1506]: starting 1 worker(s)
Sep 03 08:57:16 pve03 pveproxy[1506]: worker 7376 started
Sep 03 08:57:16 pve03 pveproxy[1506]: worker 7375 finished
Sep 03 08:57:16 pve03 pveproxy[1506]: worker 7374 finished
Sep 03 08:57:16 pve03 pveproxy[1506]: starting 2 worker(s)
Sep 03 08:57:16 pve03 pveproxy[7376]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Sep 03 08:57:16 pve03 pveproxy[1506]: worker 7377 started
Sep 03 08:57:16 pve03 pveproxy[1506]: worker 7378 started
Sep 03 08:57:16 pve03 pveproxy[7377]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Sep 03 08:57:16 pve03 pveproxy[7378]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Sep 03 08:57:21 pve03 pveproxy[7376]: worker exit
Sep 03 08:57:21 pve03 pveproxy[1506]: worker 7376 finished
Sep 03 08:57:21 pve03 pveproxy[1506]: starting 1 worker(s)
Sep 03 08:57:21 pve03 pveproxy[7377]: worker exit
Sep 03 08:57:21 pve03 pveproxy[7378]: worker exit
Sep 03 08:57:21 pve03 pveproxy[1506]: worker 7382 started
Sep 03 08:57:21 pve03 pveproxy[1506]: worker 7378 finished
Sep 03 08:57:21 pve03 pveproxy[1506]: starting 1 worker(s)
Sep 03 08:57:21 pve03 pveproxy[1506]: worker 7383 started
Sep 03 08:57:21 pve03 pveproxy[1506]: worker 7377 finished
Sep 03 08:57:21 pve03 pveproxy[1506]: starting 1 worker(s)
Sep 03 08:57:21 pve03 pveproxy[7382]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Sep 03 08:57:21 pve03 pveproxy[1506]: worker 7384 started
Sep 03 08:57:21 pve03 pveproxy[7383]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Sep 03 08:57:21 pve03 pveproxy[7384]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Sep 03 08:57:26 pve03 pveproxy[7382]: worker exit
Sep 03 08:57:26 pve03 pveproxy[7383]: worker exit
Sep 03 08:57:26 pve03 pveproxy[7384]: worker exit
Sep 03 08:57:26 pve03 pveproxy[1506]: worker 7382 finished
Sep 03 08:57:26 pve03 pveproxy[1506]: starting 1 worker(s)
Sep 03 08:57:26 pve03 pveproxy[1506]: worker 7402 started
Sep 03 08:57:26 pve03 pveproxy[1506]: worker 7383 finished
Sep 03 08:57:26 pve03 pveproxy[1506]: starting 1 worker(s)
Sep 03 08:57:26 pve03 pveproxy[1506]: worker 7403 started
Sep 03 08:57:26 pve03 pveproxy[1506]: worker 7384 finished
Sep 03 08:57:26 pve03 pveproxy[1506]: starting 1 worker(s)
Sep 03 08:57:26 pve03 pveproxy[1506]: worker 7404 started
Sep 03 08:57:26 pve03 pveproxy[7402]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Sep 03 08:57:26 pve03 pveproxy[7403]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Sep 03 08:57:26 pve03 pveproxy[7404]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.

Is there anything you can do to restore or at least copy the vms to another place?
Thanks

bbgeek17 · Sep 3, 2024

Hi @berg_rj ,

Are you running a cluster? If you are, is it healthy? How many nodes are in it? How many are up? Do you have space on the root disk of each node?
What is the output (in text encoded with CODE tags) of the following commands:
lsblk
pvecm status
systemctl|grep pve
systemctl status pveproxy
df -h

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

berg_rj · Sep 3, 2024

Hi bbgeek17,
root@pve03:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 894.3G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 1G 0 part
└─sda3 8:3 0 893.3G 0 part
sdb 8:16 0 894.3G 0 disk
├─sdb1 8:17 0 1007K 0 part
├─sdb2 8:18 0 1G 0 part
└─sdb3 8:19 0 893.3G 0 part
sr0 11:0 1 1.2G 0 rom
zd0 230:0 0 320G 0 disk
├─zd0p1 230:1 0 549M 0 part
└─zd0p2 230:2 0 319.5G 0 part
zd16 230:16 0 800G 0 disk
├─zd16p1 230:17 0 549M 0 part
├─zd16p2 230:18 0 194.8G 0 part
└─zd16p3 230:19 0 604.7G 0 part

root@pve03:~# pvecm status
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused

systemctl|grep pve
● pve-cluster.service loaded failed failed The Proxmox VE cluster filesystem
● pve-firewall.service loaded failed failed Proxmox VE firewall
● pve-guests.service loaded failed failed PVE guests
● pve-ha-crm.service loaded failed failed PVE Cluster HA Resource Manager Daemon
● pve-ha-lrm.service loaded failed failed PVE Local HA Resource Manager Daemon
pve-lxc-syscalld.service loaded active running Proxmox VE LXC Syscall Daemon
pvebanner.service loaded active exited Proxmox VE Login Banner
pvedaemon.service loaded active running PVE API Daemon
pvefw-logger.service loaded active running Proxmox VE firewall logger
pvenetcommit.service loaded active exited Commit Proxmox VE network changes
pveproxy.service loaded active running PVE API Proxy Server
● pvescheduler.service loaded failed failed Proxmox VE scheduler
● pvestatd.service loaded failed failed PVE Status Daemon
pve-storage.target loaded active active PVE Storage Target
pve-daily-update.timer loaded active waiting Daily PVE download activities
root@pve03:~#

systemctl status pveproxy
● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
Active: active (running) since Tue 2024-09-03 07:49:34 -03; 3h 14min ago
Process: 1504 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=111)
Process: 1505 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
Main PID: 1506 (pveproxy)
Tasks: 4 (limit: 33447)
Memory: 147.8M
CPU: 10min 23.122s
CGroup: /system.slice/pveproxy.service
├─ 1506 pveproxy
├─17202 "pveproxy worker"
├─17204 "pveproxy worker"
└─17205 "pveproxy worker"

Sep 03 11:04:10 pve03 pveproxy[17200]: worker exit
Sep 03 11:04:10 pve03 pveproxy[1506]: worker 17200 finished
Sep 03 11:04:10 pve03 pveproxy[1506]: starting 1 worker(s)
Sep 03 11:04:10 pve03 pveproxy[1506]: worker 17204 started
Sep 03 11:04:10 pve03 pveproxy[17201]: worker exit
Sep 03 11:04:10 pve03 pveproxy[1506]: worker 17201 finished
Sep 03 11:04:10 pve03 pveproxy[17204]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Sep 03 11:04:10 pve03 pveproxy[1506]: starting 1 worker(s)
Sep 03 11:04:10 pve03 pveproxy[1506]: worker 17205 started
Sep 03 11:04:10 pve03 pveproxy[17205]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
root@pve03:~#

root@pve03:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 14G 0 14G 0% /dev
tmpfs 2.8G 1.5M 2.8G 1% /run
rpool/ROOT/pve-1 877G 2.3G 875G 1% /
tmpfs 14G 0 14G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
rpool/var-lib-vz 926G 52G 875G 6% /var/lib/vz
rpool 875G 128K 875G 1% /rpool
rpool/ROOT 875G 128K 875G 1% /rpool/ROOT
rpool/data 875G 128K 875G 1% /rpool/data
tmpfs 2.8G 0 2.8G 0% /run/user/0
root@pve03:~#

berg_rj · Sep 3, 2024

bbgeek17 said:
Hi @berg_rj ,

Are you running a cluster? If you are, is it healthy? How many nodes are in it? How many are up? Do you have space on the root disk of each node?
What is the output (in text encoded with CODE tags) of the following commands:
lsblk
pvecm status
systemctl|grep pve
systemctl status pveproxy
df -h

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

berg_rj · Sep 3, 2024

Are you running a cluster? no.
How many nodes are in it? It is an independent server, it does not have any nodes.
Do you have space on the root disk of each node? yes.

thanks

bbgeek17 · Sep 3, 2024

What are the outputs of the following commands (try to use CODE tags available in the menu of the edit box, the icon looks as such </>):

hostname
cat /etc/hosts
hostnamectl
curl -sk https://localhost:8006

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

berg_rj · Sep 3, 2024

Code:

Linux pve03 6.5.11-4-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.11-4 (2023-11-20T10:19Z) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Sep  3 12:13:33 2024
root@pve03:~# hostname
pve03
root@pve03:~#
root@pve03:~#
root@pve03:~# cat /etc/hosts
10.0.105.27 pve03.cejamrj.org.br pve03

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
root@pve03:~#
root@pve03:~#
root@pve03:~# hostnamectl
 Static hostname: pve03
       Icon name: computer-server
         Chassis: server 
      Machine ID: 3cc7d5dd15c544faabfa3f0cf0f1500b
         Boot ID: 66607e1c90004f88babed263c40cfeff
Operating System: Debian GNU/Linux 12 (bookworm)
          Kernel: Linux 6.5.11-4-pve
    Architecture: x86-64
 Hardware Vendor: Intel Corporation
  Hardware Model: S2600CP
Versão do firmware: SE5C600.86B.02.05.0004.051120151007
root@pve03:~# curl -sk https://localhost:8006

in command "curl -sk https://localhost:8006" I got no response. sorry for the delay

bbgeek17 · Sep 3, 2024

What happened to this line in /etc/hosts?
127.0.0.1 localhost.localdomain localhost

What happens when you restart pveproxy: systemctl restart pveproxy

Is this the right IP? 10.0.105.27
Does it match "ip a" and /etc/network/interfaces ?

Is there anything in the log after pveproxy restart?

What is the output of : cat /etc/corosync/corosync.conf

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

berg_rj · Sep 3, 2024

Code:

root@pve03:~# systemctl restart pveproxy
root@pve03:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 00:1e:67:49:96:9c brd ff:ff:ff:ff:ff:ff
    altname enp7s0f0
3: enp7s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000
    link/ether 00:1e:67:49:96:9d brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:1e:67:49:96:9d brd ff:ff:ff:ff:ff:ff
    inet 10.0.105.27/16 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::21e:67ff:fe49:969d/64 scope link
       valid_lft forever preferred_lft forever
root@pve03:~# cat /etc/corosync/corosync.conf
cat: /etc/corosync/corosync.conf: No such file or directory
root@pve03:~# /etc/network/interfaces
-bash: /etc/network/interfaces: Permission denied
root@pve03:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface enp7s0f1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 10.0.105.27/16
        gateway 10.0.101.250
        bridge-ports enp7s0f1
        bridge-stp off
        bridge-fd 0

iface eno0 inet manual


source /etc/network/interfaces.d/*
root@pve03:~# systemctl status pveproxy
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
     Active: active (running) since Tue 2024-09-03 16:07:35 -03; 2min 26s ago
    Process: 1972 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=111)
    Process: 1982 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 2010 (pveproxy)
      Tasks: 4 (limit: 33447)
     Memory: 147.7M
        CPU: 8.870s
     CGroup: /system.slice/pveproxy.service
             ├─2010 pveproxy
             ├─2198 "pveproxy worker"
             ├─2199 "pveproxy worker"
             └─2200 "pveproxy worker"

Sep 03 16:09:57 pve03 pveproxy[2010]: worker 2195 finished
Sep 03 16:09:57 pve03 pveproxy[2010]: starting 2 worker(s)
Sep 03 16:09:57 pve03 pveproxy[2010]: worker 2198 started
Sep 03 16:09:57 pve03 pveproxy[2010]: worker 2199 started
Sep 03 16:09:57 pve03 pveproxy[2010]: worker 2197 finished
Sep 03 16:09:57 pve03 pveproxy[2010]: starting 1 worker(s)
Sep 03 16:09:57 pve03 pveproxy[2010]: worker 2200 started
Sep 03 16:09:57 pve03 pveproxy[2198]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Sep 03 16:09:57 pve03 pveproxy[2199]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.
Sep 03 16:09:57 pve03 pveproxy[2200]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at /usr/share/perl5/PVE/APIServer/AnyEvent.pm line 2009.

bbgeek17 · Sep 3, 2024

What happens when you run this manually?
/usr/bin/pvecm updatecerts --silent

What about:
/usr/bin/pvecm updatecerts --silent --force true

What about: pveproxy start --debug 1

Have you rebooted recently?

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

berg_rj · Sep 3, 2024

bbgeek17 said:
Have you rebooted recently?

Yes, for try acess interface web

Code:

root@pve03:~# pvesm status
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused
root@pve03:~# /usr/bin/pvecm updatecerts --silent
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused
root@pve03:~# /usr/bin/pvecm updatecerts --silent --force true
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused
root@pve03:~# pveproxy start --debug 1
start failed - can't acquire lock '/var/run/pveproxy/pveproxy.pid.lock' - daemon already started (pid = 2010)
root@pve03:~#

bbgeek17 · Sep 3, 2024

review this thread
https://forum.proxmox.com/threads/unable-to-load-access-control-list-connection-refused.72245/page-2

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Search

Search

No webgui interface, but with access via ssh

berg_rj

New Member

bbgeek17

Distinguished Member

berg_rj

New Member

berg_rj

New Member

berg_rj

New Member

bbgeek17

Distinguished Member

berg_rj

New Member

bbgeek17

Distinguished Member

berg_rj

New Member

bbgeek17

Distinguished Member

berg_rj

New Member

bbgeek17

Distinguished Member

We value your privacy