Joining a new node to a cluster removes local-lvm and creates phantom datastores

[runbgp]

I'm having a hard time understanding or finding good reasoning behind this behavior, but it has become extremely problematic and is reproducible every time. In this case, I have 3 nodes that are installed with ZFS. They all have local, local-zfs, and 2 have a specific datastore for a local NVMe SSD. If I attempt to join a new server to the cluster that does not have an identical storage configuartion (installed on a hardware RAID array) that only has local and local-lvm, local-lvm is "removed" and is not usable, and a phantom datastore is created that is identical to the naming scheme for the local NVMe lvm-thin on other hosts.

On the cluster level, local-zfs has the nodes field explicity set for the hosts with ZFS.



I first had to remove the phantom NVMe datastore from the cluster level and then set it up on the host (this new host has a NVMe SSD as well). That is now fine. I have the following questions:

1. How do I recover local-lvm on this host and make use of it again?
2. Why is Proxmox overwriting storage configurations with seemingly no reason on new cluster members?
3. Why are phantom datastores being created (e.g. a datstore is created, has question mark icon because it doesn't actually exist)? This seems extremely flawed in behavior as frequently clusters are ran with hardware that is not identical.

Host info
Proxmox VE 7.2-11
Linux 5.15.60-2-pve #1 SMP PVE 5.15.60-2 (Tue, 04 Oct 2022 16:52:28 +0200)
pve-manager/7.2-11/b76d3178

Below are screenshots of the storage config on the new host. Any insight is appreciated.

root@us-btd1-h5:~# lvdisplay
  --- Logical volume ---
  LV Path                /dev/pve/swap
  LV Name                swap
  VG Name                pve
  LV UUID                AOaDfH-orRd-LABH-Pr3a-0Xzx-ZAzO-AxAZ3G
  LV Write Access        read/write
  LV Creation host, time proxmox, 2022-10-15 20:17:18 -0500
  LV Status              available
  # open                 2
  LV Size                8.00 GiB
  Current LE             2048
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:0
 
  --- Logical volume ---
  LV Path                /dev/pve/root
  LV Name                root
  VG Name                pve
  LV UUID                9VSNs2-6QUR-9Md2-CBT0-0FSe-xvEg-70AJMt
  LV Write Access        read/write
  LV Creation host, time proxmox, 2022-10-15 20:17:18 -0500
  LV Status              available
  # open                 1
  LV Size                96.00 GiB
  Current LE             24576
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1
 
  --- Logical volume ---
  LV Name                data
  VG Name                pve
  LV UUID                hsSeOm-3lHa-5z7H-mqJE-zQEv-xQ1G-snOqUp
  LV Write Access        read/write
  LV Creation host, time proxmox, 2022-10-15 20:17:35 -0500
  LV Pool metadata       data_tmeta
  LV Pool data           data_tdata
  LV Status              available
  # open                 0
  LV Size                <754.11 GiB
  Allocated pool data    0.00%
  Allocated metadata     0.25%
  Current LE             193051
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:4
 
  --- Logical volume ---
  LV Name                us-btd1-h5-local-nvme
  VG Name                us-btd1-h5-local-nvme
  LV UUID                jtucTa-5Sgd-bcGA-Rckc-z9aW-ML0m-6ob2Af
  LV Write Access        read/write
  LV Creation host, time us-btd1-h5, 2022-10-15 20:43:20 -0500
  LV Pool metadata       us-btd1-h5-local-nvme_tmeta
  LV Pool data           us-btd1-h5-local-nvme_tdata
  LV Status              available
  # open                 0
  LV Size                <1.43 TiB
  Allocated pool data    0.00%
  Allocated metadata     0.15%
  Current LE             373884
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:7

root@us-btd1-h5:~# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content backup,vztmpl,iso

zfspool: local-zfs
        pool rpool/data
        content images,rootdir
        nodes us-btd1-h2,us-btd1-h3,us-btd1-h1
        sparse 1

lvmthin: us-btd1-h2-local-nvme
        thinpool us-btd1-h2-local-nvme
        vgname us-btd1-h2-local-nvme
        content rootdir,images
        nodes us-btd1-h2

lvmthin: us-btd1-h1-local-nvme
        thinpool us-btd1-h1-local-nvme
        vgname us-btd1-h1-local-nvme
        content rootdir,images
        nodes us-btd1-h1

lvmthin: us-btd1-h5-local-nvme
        thinpool us-btd1-h5-local-nvme
        vgname us-btd1-h5-local-nvme
        content images,rootdir
        nodes us-btd1-h5

 

[runbgp]

I was able to get local-lvm visible in the GUI again by adding the following to storage.cfg on the host:

lvmthin: local-lvm
        thinpool data
        vgname pve
        content images,rootdir

 

[runbgp]

Anyone have any input on this?

 

[fabian]

this is expected behaviour when joining a cluster - all the existing configuration in /etc/pve is lost and the one from the cluster is used. this is why there's a check for any configured guests on the joining node prior to joining - those would be lost as well!

while the storage.cfg file wouldn't be too hard to merge in practice, there are lots of other config files and settings where this would not be trivial. the expectation is to join only empty, freshly installed nodes to a cluster, ideally with a similar base setup so that no fixups are necessary.

your storage.cfg problem can be fixed like this (all over the GUI!):
- limit the local-zfs storage to mark it as only available for the "old" part of the cluster (Datacenter -> Storage -> Edit -> "Nodes")
- re-add the local-lvm storage on the freshly joined node, limited to that node (connect to GUI on joined node, Datacenter -> Storage -> Add -> LVM-Thin -> ID 'local-lvm', volume group 'pve', thin pool 'data', nodes only that single node)

of course, the same can be done by editing the config file or using pvesm

 

[btspce]

Hello
I think we have the same problem and need clarification. We are currently creating a cluster with 3 identical servers. Local nvme for boot and a local scsi array named "vg0" on every node which is configured as LVM-Thin. When we joined the first node it's local "vg0" disappeared. I can't just add it again under datacenter storage because id "vg0" is already in use. But my understanding is that we need the same id if moving guest vm's around or they would not boot after migration.

Proxmox 8.0.4 on all nodes.
bit2: Main node with vm's in production.
bit1: Joined node where "vg0" disapperad.

All nodes have /dev/sdc as the scsi array "vg0" LVM-Thin pool

As we have running hosts in production on the main cluster node bit2 with data on its local "vg0" I want to be completely sure on how to do this and which solution that is the best.

Can we simply edit "vg0" under datacenter>storage and add the joining bit1 node to it while everything is running and it would be availible on bit1 again?

Or do we have to do this on every node?:
- limit the local-zfs storage on every node to itself (Datacenter -> Storage -> Edit -> "Nodes")
- re-add the local-lvm storage on the freshly joined node, limited to that node (connect to GUI on joined node, Datacenter -> Storage -> Add -> LVM-Thin -> ID 'vg0', volume group 'vg0', thin pool 'vg0', nodes only that single node)

 

[btspce]

For others that find this thread.
If the local storage of the nodes are identical you can just add the node to the local lvm-thin storage which is what we did. Nothing happened to the running vm's on the other host and "vg0" was immediately availible on bit1 to use after we added the node.