I had attempted to update (through the web GUI) my nodes yesterday. Node 1 and 2 went without issue. Node 3 failed to update (I forget the error, I didn't screenshot and it was last night) and my first thought was "try turning it off and on again!" So I rebooted and it failed to boot suggesting I run fsck. I made a Proxmox USB and booted the rescue environment and managed to run "e2fsck /dev/sda" and it found a lot of errors, all of which I hit "Y" to correct.
Rebooted again and now I get to the root login. It is unreachable by IP, and is not reporting as connected to the cluster. Attempting to login as root gets me a variety of errors, from just flashing black for a second before ending up back at the login. Showing pvedaemon start failures (forgive me on the lack of exact verbiage, it flashed quickly once and wasn't prepared to take a picture.)
Attempting to reboot by hitting ctrl alt del on the keyboard got me the attached errors.
I am prepared to accept this node as a total loss. I have a new boot SSD arriving today. I have read the documentation (https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node) and just want to make sure I got the right before I screw up my entire cluster. I do not use ceph in my cluster. I do not have any replication jobs.
My idea is as follows:
1. Install new SSD
2. Install proxmox and give node a completely new name (In this case it will be "pve3-1" to replace the old "pve3") and give it the same static IP's as before. WebUI and corosync network both getting the same IP's as before. (This is something I am unsure about because the cluster web gui references those IP's directly. I can use new IP's if need be.)
3. Join "pve3-1" to the cluster in the GUI
4. Migrate all VM's to "pve" (This is probably unnecessary, but I am just being some kind of paranoid)
5. Delete all HA rules
6. On one of the other nodes run "pvecm delnode pve3" and safely ignore the "could not kill node (error = cs_err_not_exist)" error.
Have I got it right?
I appreciate everyone who chimes in and helps.
Rebooted again and now I get to the root login. It is unreachable by IP, and is not reporting as connected to the cluster. Attempting to login as root gets me a variety of errors, from just flashing black for a second before ending up back at the login. Showing pvedaemon start failures (forgive me on the lack of exact verbiage, it flashed quickly once and wasn't prepared to take a picture.)
Attempting to reboot by hitting ctrl alt del on the keyboard got me the attached errors.
I am prepared to accept this node as a total loss. I have a new boot SSD arriving today. I have read the documentation (https://pve.proxmox.com/wiki/Cluster_Manager#_remove_a_cluster_node) and just want to make sure I got the right before I screw up my entire cluster. I do not use ceph in my cluster. I do not have any replication jobs.
My idea is as follows:
1. Install new SSD
2. Install proxmox and give node a completely new name (In this case it will be "pve3-1" to replace the old "pve3") and give it the same static IP's as before. WebUI and corosync network both getting the same IP's as before. (This is something I am unsure about because the cluster web gui references those IP's directly. I can use new IP's if need be.)
3. Join "pve3-1" to the cluster in the GUI
4. Migrate all VM's to "pve" (This is probably unnecessary, but I am just being some kind of paranoid)
5. Delete all HA rules
6. On one of the other nodes run "pvecm delnode pve3" and safely ignore the "could not kill node (error = cs_err_not_exist)" error.
Have I got it right?
I appreciate everyone who chimes in and helps.
Attachments
Last edited: