Unexpected behavior when joining non-identical node to cluster

kriebz

New Member
Jun 26, 2024
1
0
1
I don't know if this should just be a "heads up", or if there's a good official answer to this. Doing a test setup at work. Install one server on a Dell R740, almost all defaults from the installer. Using hardware RAID, one giant RAID 6 is the only storage. Installer creates "local-lvm" storage using the "data" LV in VG "pve".

Build a second server, this one has the "back drives" configured as a hardware RAID 1, plus the "front drives", much like the other. Installer offers no flexibility for storage configuration, so... fine. Install PVE, then create a new VG "pve2" with the big raid, create a thin volume called "data", then use the GUI to remove the local-lvm storage the installer made, make a new one with the same name, just use the other LV (which is in a different VG)...

Fast forward, power down and move the server to its new home, I join the 2nd server to the cluster, everything looks fine. Boss migrates a VM from the 1st server to the second and 1) it fills the local-lvm storage and 2) it corrupts the disk for the VM. Triage that, and realize that when it joined the cluster, the cluster-wide `/etc/pve/storage.conf` reconfigured the 2nd host to use the original settings for the local-lvm storage, so now it was using the (much smaller) spare space on the boot drive. Furthermore, instead of failing intelligently, or preventing the move in the fist place, the migration just wrecked the VM disk.

Now, both of these can be prevented by good practices, but they also both seem like things that should be caught and generate warnings.
 
and realize that when it joined the cluster, the cluster-wide `/etc/pve/storage.conf`
Well..., the very first line in https://pve.proxmox.com/pve-docs/pve-admin-guide.html#pvecm_join_node_to_cluster says

> " All existing configuration in /etc/pve is overwritten when joining acluster. "

Best practice to handle a very complex system is to read the documentation first... ;-)

Sorry, no harm intended. Some things you know only after trampling into a pitfall. Been there, done that.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!