I was trying to rejoin an existing cluster that was deleted on another node (already another thread). I cant access GUI on multiple broswers, even tried changing IP address but still no luck. I can access thru SSH. Here are some commands run:
root@pve1:/etc/pve# curl -v -k https://10.90.255.11
* Trying 10.90.255.11:443...
* connect to 10.90.255.11 port 443 failed: Connection refused
* Failed to connect to 10.90.255.11 port 443 after 0 ms: Couldn't connect to server
* Closing connection 0
curl: (7) Failed to connect to 10.90.255.11 port 443 after 0 ms: Couldn't connect to server
root@pve1:~# journalctl -xe
░░ Subject: A start job for unit corosync.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit corosync.service has finished successfully.
░░
░░ The job identifier is 845.
Jul 15 15:54:16 pve1 pmxcfs[8309]: [status] notice: update cluster info (cluster name proxcluster01, version = 2)
Jul 15 15:54:16 pve1 pmxcfs[8309]: [dcdb] notice: members: 2/8309
Jul 15 15:54:16 pve1 pmxcfs[8309]: [dcdb] notice: all data is up to date
Jul 15 15:54:16 pve1 pmxcfs[8309]: [status] notice: members: 2/8309
Jul 15 15:54:16 pve1 pmxcfs[8309]: [status] notice: all data is up to date
Jul 15 15:54:41 pve1 pvecm[8320]: got timeout when trying to ensure cluster certificates and base file hierarchy is set>
Jul 15 15:54:42 pve1 pveproxy[8401]: starting server
Jul 15 15:54:42 pve1 pveproxy[8401]: starting 3 worker(s)
Jul 15 15:54:42 pve1 pveproxy[8401]: worker 8402 started
Jul 15 15:54:42 pve1 pveproxy[8401]: worker 8403 started
Jul 15 15:54:42 pve1 pveproxy[8401]: worker 8404 started
Jul 15 15:54:42 pve1 systemd[1]: Started pveproxy.service - PVE API Proxy Server.
░░ Subject: A start job for unit pveproxy.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit pveproxy.service has finished successfully.
░░
░░ The job identifier is 737.
Jul 15 15:54:42 pve1 pveproxy[8404]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at >
Jul 15 15:54:42 pve1 pveproxy[8402]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at >
Jul 15 15:54:42 pve1 pveproxy[8403]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at >
root@pve1:~# pvecm updatecerts --force
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
got timeout when trying to ensure cluster certificates and base file hierarchy is set up - no quorum (yet) or hung pmxcfs?
root@pve1:~# journalctl -u pve-cluster
Jun 16 15:46:30 pve1 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
Jun 16 15:46:30 pve1 pmxcfs[1008]: [main] notice: resolved node name 'pve1' to '10.90.255.11' for default node IP addre>
Jun 16 15:46:30 pve1 pmxcfs[1008]: [main] notice: resolved node name 'pve1' to '10.90.255.11' for default node IP addre>
Jun 16 15:46:32 pve1 systemd[1]: Started pve-cluster.service - The Proxmox VE cluster filesystem.
Jun 17 10:25:10 pve1 systemd[1]: Stopping pve-cluster.service - The Proxmox VE cluster filesystem...
Jun 17 10:25:10 pve1 pmxcfs[1069]: [main] notice: teardown filesystem
Jun 17 10:25:10 pve1 pmxcfs[1069]: [main] notice: exit proxmox configuration filesystem (0)
Jun 17 10:25:10 pve1 systemd[1]: pve-cluster.service: Deactivated successfully.
Jun 17 10:25:10 pve1 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
Jun 17 10:25:10 pve1 systemd[1]: pve-cluster.service: Consumed 32.979s CPU time.
-- Boot c19e8691e94f4f2a9a3569a38a86103a --
Jun 17 10:25:55 pve1 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
Jun 17 10:25:55 pve1 pmxcfs[1029]: [main] notice: resolved node name 'pve1' to '10.90.255.11' for default node IP addre>
Jun 17 10:25:55 pve1 pmxcfs[1029]: [main] notice: resolved node name 'pve1' to '10.90.255.11' for default node IP addre>
Jun 17 10:25:56 pve1 systemd[1]: Started pve-cluster.service - The Proxmox VE cluster filesystem.
Jun 17 11:41:10 pve1 systemd[1]: Stopping pve-cluster.service - The Proxmox VE cluster filesystem...
Jun 17 11:41:10 pve1 pmxcfs[1093]: [main] notice: teardown filesystem
Jun 17 11:41:20 pve1 systemd[1]: pve-cluster.service: State 'stop-sigterm' timed out. Killing.
Jun 17 11:41:20 pve1 systemd[1]: pve-cluster.service: Killing process 1093 (pmxcfs) with signal SIGKILL.
Jun 17 11:41:20 pve1 systemd[1]: pve-cluster.service: Killing process 1095 (n/a) with signal SIGKILL.
Jun 17 11:41:20 pve1 systemd[1]: pve-cluster.service: Main process exited, code=killed, status=9/KILL
Jun 17 11:41:20 pve1 systemd[1]: pve-cluster.service: Failed with result 'timeout'.
Jun 17 11:41:20 pve1 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
Jun 17 11:41:20 pve1 systemd[1]: pve-cluster.service: Consumed 2.486s CPU time.
Jun 17 11:41:20 pve1 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
Jun 17 11:41:20 pve1 pmxcfs[13219]: [main] notice: resolved node name 'pve1' to '10.90.255.11' for default node IP addr>
Jun 17 11:41:20 pve1 pmxcfs[13219]: [main] notice: resolved node name 'pve1' to '10.90.255.11' for default node IP addr>
Jun 17 11:41:20 pve1 pmxcfs[13224]: [quorum] crit: quorum_initialize failed: 2
Jun 17 11:41:20 pve1 pmxcfs[13224]: [quorum] crit: can't initialize service
root@pve1:/etc/pve# curl -v -k https://10.90.255.11
* Trying 10.90.255.11:443...
* connect to 10.90.255.11 port 443 failed: Connection refused
* Failed to connect to 10.90.255.11 port 443 after 0 ms: Couldn't connect to server
* Closing connection 0
curl: (7) Failed to connect to 10.90.255.11 port 443 after 0 ms: Couldn't connect to server
root@pve1:~# journalctl -xe
░░ Subject: A start job for unit corosync.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit corosync.service has finished successfully.
░░
░░ The job identifier is 845.
Jul 15 15:54:16 pve1 pmxcfs[8309]: [status] notice: update cluster info (cluster name proxcluster01, version = 2)
Jul 15 15:54:16 pve1 pmxcfs[8309]: [dcdb] notice: members: 2/8309
Jul 15 15:54:16 pve1 pmxcfs[8309]: [dcdb] notice: all data is up to date
Jul 15 15:54:16 pve1 pmxcfs[8309]: [status] notice: members: 2/8309
Jul 15 15:54:16 pve1 pmxcfs[8309]: [status] notice: all data is up to date
Jul 15 15:54:41 pve1 pvecm[8320]: got timeout when trying to ensure cluster certificates and base file hierarchy is set>
Jul 15 15:54:42 pve1 pveproxy[8401]: starting server
Jul 15 15:54:42 pve1 pveproxy[8401]: starting 3 worker(s)
Jul 15 15:54:42 pve1 pveproxy[8401]: worker 8402 started
Jul 15 15:54:42 pve1 pveproxy[8401]: worker 8403 started
Jul 15 15:54:42 pve1 pveproxy[8401]: worker 8404 started
Jul 15 15:54:42 pve1 systemd[1]: Started pveproxy.service - PVE API Proxy Server.
░░ Subject: A start job for unit pveproxy.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit pveproxy.service has finished successfully.
░░
░░ The job identifier is 737.
Jul 15 15:54:42 pve1 pveproxy[8404]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at >
Jul 15 15:54:42 pve1 pveproxy[8402]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at >
Jul 15 15:54:42 pve1 pveproxy[8403]: /etc/pve/local/pve-ssl.key: failed to load local private key (key_file or key) at >
root@pve1:~# pvecm updatecerts --force
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
waiting for pmxcfs mount to appear and get quorate...
got timeout when trying to ensure cluster certificates and base file hierarchy is set up - no quorum (yet) or hung pmxcfs?
root@pve1:~# journalctl -u pve-cluster
Jun 16 15:46:30 pve1 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
Jun 16 15:46:30 pve1 pmxcfs[1008]: [main] notice: resolved node name 'pve1' to '10.90.255.11' for default node IP addre>
Jun 16 15:46:30 pve1 pmxcfs[1008]: [main] notice: resolved node name 'pve1' to '10.90.255.11' for default node IP addre>
Jun 16 15:46:32 pve1 systemd[1]: Started pve-cluster.service - The Proxmox VE cluster filesystem.
Jun 17 10:25:10 pve1 systemd[1]: Stopping pve-cluster.service - The Proxmox VE cluster filesystem...
Jun 17 10:25:10 pve1 pmxcfs[1069]: [main] notice: teardown filesystem
Jun 17 10:25:10 pve1 pmxcfs[1069]: [main] notice: exit proxmox configuration filesystem (0)
Jun 17 10:25:10 pve1 systemd[1]: pve-cluster.service: Deactivated successfully.
Jun 17 10:25:10 pve1 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
Jun 17 10:25:10 pve1 systemd[1]: pve-cluster.service: Consumed 32.979s CPU time.
-- Boot c19e8691e94f4f2a9a3569a38a86103a --
Jun 17 10:25:55 pve1 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
Jun 17 10:25:55 pve1 pmxcfs[1029]: [main] notice: resolved node name 'pve1' to '10.90.255.11' for default node IP addre>
Jun 17 10:25:55 pve1 pmxcfs[1029]: [main] notice: resolved node name 'pve1' to '10.90.255.11' for default node IP addre>
Jun 17 10:25:56 pve1 systemd[1]: Started pve-cluster.service - The Proxmox VE cluster filesystem.
Jun 17 11:41:10 pve1 systemd[1]: Stopping pve-cluster.service - The Proxmox VE cluster filesystem...
Jun 17 11:41:10 pve1 pmxcfs[1093]: [main] notice: teardown filesystem
Jun 17 11:41:20 pve1 systemd[1]: pve-cluster.service: State 'stop-sigterm' timed out. Killing.
Jun 17 11:41:20 pve1 systemd[1]: pve-cluster.service: Killing process 1093 (pmxcfs) with signal SIGKILL.
Jun 17 11:41:20 pve1 systemd[1]: pve-cluster.service: Killing process 1095 (n/a) with signal SIGKILL.
Jun 17 11:41:20 pve1 systemd[1]: pve-cluster.service: Main process exited, code=killed, status=9/KILL
Jun 17 11:41:20 pve1 systemd[1]: pve-cluster.service: Failed with result 'timeout'.
Jun 17 11:41:20 pve1 systemd[1]: Stopped pve-cluster.service - The Proxmox VE cluster filesystem.
Jun 17 11:41:20 pve1 systemd[1]: pve-cluster.service: Consumed 2.486s CPU time.
Jun 17 11:41:20 pve1 systemd[1]: Starting pve-cluster.service - The Proxmox VE cluster filesystem...
Jun 17 11:41:20 pve1 pmxcfs[13219]: [main] notice: resolved node name 'pve1' to '10.90.255.11' for default node IP addr>
Jun 17 11:41:20 pve1 pmxcfs[13219]: [main] notice: resolved node name 'pve1' to '10.90.255.11' for default node IP addr>
Jun 17 11:41:20 pve1 pmxcfs[13224]: [quorum] crit: quorum_initialize failed: 2
Jun 17 11:41:20 pve1 pmxcfs[13224]: [quorum] crit: can't initialize service
Last edited: