Help with Proxmox Recovery

[ridel]

I've gotten myself into a little bind, I was trying to create a container with around 50gb of storage, the command went through and i started using it. Somehow after trying to install a program it seems that 50g space was full so I made it 100g, I thought that was less than the left over amount of storage, turns out it was not and my node broke. I was able to SSH into the pve node and realized the disk space was full, so I CDed into the container storage and deleted the harddrive of the broken ct, I thought my issues would be fixed, it did not, the gui still does not work and ive tried everything i can think of.

root@proxmox:~# journalctl -xe
░░
░░ The unit pve-cluster.service has entered the 'failed' state with result 'exit-code'.
Mar 19 23:28:15 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.
░░ Subject: A start job for unit pve-cluster.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit pve-cluster.service has finished with a failure.
░░
░░ The job identifier is 2756 and the job result is failed.
Mar 19 23:28:17 proxmox pveproxy[1154]: worker exit
Mar 19 23:28:17 proxmox pveproxy[906]: worker 1154 finished
Mar 19 23:28:17 proxmox pveproxy[906]: starting 1 worker(s)
Mar 19 23:28:17 proxmox pveproxy[906]: worker 1164 started
Mar 19 23:28:17 proxmox pveproxy[1155]: worker exit
Mar 19 23:28:17 proxmox pveproxy[1164]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) >
Mar 19 23:28:17 proxmox pveproxy[1156]: worker exit
Mar 19 23:28:17 proxmox pveproxy[906]: worker 1155 finished
Mar 19 23:28:17 proxmox pveproxy[906]: starting 1 worker(s)
Mar 19 23:28:17 proxmox pveproxy[906]: worker 1165 started
Mar 19 23:28:17 proxmox pveproxy[906]: worker 1156 finished
Mar 19 23:28:17 proxmox pveproxy[906]: starting 1 worker(s)
Mar 19 23:28:17 proxmox pveproxy[906]: worker 1166 started
Mar 19 23:28:17 proxmox pveproxy[1165]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) >
Mar 19 23:28:17 proxmox pveproxy[1166]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) >
lines 3082-3106/3106 (END)

root@proxmox:~# systemctl status -l pve-cluster
● pve-cluster.service - The Proxmox VE cluster filesystem
     Loaded: loaded (/lib/systemd/system/pve-cluster.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2023-03-19 23:26:28 PDT; 47s ago
    Process: 863 ExecStart=/usr/bin/pmxcfs (code=exited, status=255/EXCEPTION)
        CPU: 10ms

Mar 19 23:26:28 proxmox systemd[1]: pve-cluster.service: Scheduled restart job, restart counter is at 5.
Mar 19 23:26:28 proxmox systemd[1]: Stopped The Proxmox VE cluster filesystem.
Mar 19 23:26:28 proxmox systemd[1]: pve-cluster.service: Start request repeated too quickly.
Mar 19 23:26:28 proxmox systemd[1]: pve-cluster.service: Failed with result 'exit-code'.
Mar 19 23:26:28 proxmox systemd[1]: Failed to start The Proxmox VE cluster filesystem.

are the logs for the cluster that isnt working.

root@proxmox:~# pct list
ipcc_send_rec[1] failed: Connection refused
ipcc_send_rec[2] failed: Connection refused
ipcc_send_rec[3] failed: Connection refused
Unable to load access control list: Connection refused
root@proxmox:~#

happens whenever i try to use pct list or qm list and thus forth, ive tried manually doing the certs using pvecm updatecerts and the above happens.

 

[pille99]

as for the error messages - its the connection.
something is wrong with the ssl keys
try
https://pve.proxmox.com/wiki/Proxmox_SSL_Error_Fixing

 

[ridel]

Tried that, still receiving this error,

● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2023-03-25 18:38:29 PDT; 1min 17s ago
    Process: 325670 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=111)
    Process: 325671 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 325676 (pveproxy)
      Tasks: 4 (limit: 38272)
     Memory: 132.9M
        CPU: 2.898s
     CGroup: /system.slice/pveproxy.service
             ├─325676 pveproxy
             ├─325748 pveproxy worker
             ├─325749 pveproxy worker
             └─325750 pveproxy worker

Mar 25 18:39:44 proxmox pveproxy[325747]: worker exit
Mar 25 18:39:44 proxmox pveproxy[325748]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key>
Mar 25 18:39:44 proxmox pveproxy[325676]: worker 325746 finished
Mar 25 18:39:44 proxmox pveproxy[325676]: starting 1 worker(s)
Mar 25 18:39:44 proxmox pveproxy[325676]: worker 325749 started
Mar 25 18:39:44 proxmox pveproxy[325676]: worker 325747 finished
Mar 25 18:39:44 proxmox pveproxy[325676]: starting 1 worker(s)
Mar 25 18:39:44 proxmox pveproxy[325676]: worker 325750 started
Mar 25 18:39:44 proxmox pveproxy[325749]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key>
Mar 25 18:39:44 proxmox pveproxy[325750]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key>

 

[pille99]

what happen if you do
ssh proxmoxserver ?

 

[ridel]

~ $ ssh proxmox
ssh: Could not resolve hostname proxmox: No address associated with hostname
~ $

though i am able to ssh the ip directly and the hosts file has the correct definitions.

 

[pille99]

check your hostfile
run the command on IP
ssh IP_Proxmox
if the same messages comes up, your DNS, Hostfile is incorect
did you change the node names ?

 

[ridel]

The node names never changed, all this happened when the server had 0 bytes of storage, as for me sshing to the IP that is working and I have been doing that for logs

 

[pille99]

check the files itself. maybe linux deleted the content of the files. it can happen, if file is in use but run out of space. than it crashes the files or delete the content.

 

[pille99]

just add your configs here

 

[ridel]

GNU nano 5.4       /etc/hosts
127.0.0.1 localhost.localdomain localhost
10.0.1.111 proxmox.denserver.local proxmox
# The following lines are desirable for IPv6 ca>

#::1     ip6-localhost ip6-loopback
#fe00::0 ip6-localnet
#ff00::0 ip6-mcastprefix
#ff02::1 ip6-allnodes
#ff02::2 ip6-allrouters
#ff02::3 ip6-allhosts

Here is the hosts file

 

[pille99]

how many servers you have ?
can you add all hosts from all servers ?

 

[ridel]

That server is my only proxmox server