Added a new node but "local-lvm" storage unknown - due to different volume group name?

[marcosscriven]

I just added a new node to my cluster. Usually, I have both the LVM and LVM-thin storage on the same SSD.

On my new node I wanted to have a one small SSD for just proxmox root, and a second for the VM data. So what I've ended up with is:

nvme0n1           259:0    0   3.6T  0 disk
├─data-data_tmeta 252:2    0  15.9G  0 lvm
│ └─data-data     252:4    0   3.6T  0 lvm
└─data-data_tdata 252:3    0   3.6T  0 lvm
  └─data-data     252:4    0   3.6T  0 lvm
nvme1n1           259:1    0 931.5G  0 disk
nvme2n1           259:2    0 119.2G  0 disk
├─nvme2n1p1       259:3    0  1007K  0 part
├─nvme2n1p2       259:4    0     1G  0 part /boot/efi
└─nvme2n1p3       259:5    0 118.2G  0 part
  ├─pve-swap      252:0    0     8G  0 lvm  [SWAP]
  └─pve-root      252:1    0  39.6G  0 lvm  /
 
  root@pve-pc:~# lvs
  LV   VG   Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data data twi-a-tz--  <3.61t             0.00   0.19
  root pve  -wi-ao---- <39.56g
  swap pve  -wi-ao----   8.00g

Is this the wrong way to go about it? Do I need to change the VG name on the root proxmox disk to something other than "pve", then use the "pve" name for the VG on the separate SSD?

 

[bbgeek17]

No, the underlying VG group name is not important. If you are planning this for migration - the Storage Pool name must match.
You didnt post actual failure scenario, but based on the subject of the thread the system is complaining about missing "local-lvm" which is a Storage Pool name, not an LVM name.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[marcosscriven]

it says "no such logical volume pve/local-lvm". Maybe it's just because the thin pool is called "data"?

But that happened when I created the thin pool.

 

[bbgeek17]

it says "no such logical volume pve/local-lvm".

It does not say that. It says "no such logical volume pve/data".

It means you did not restrict your Storage Pool to a specific node and/or marked it shared. The storage, especially local one where underlying structure differs, should be set to be specific to a particular node.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[marcosscriven]

To take a step back - how do I get it to behave the same as my other nodes - where I just went with the default of having both LVM and LVM-thin on the same disk?

Storage as the cluster level seems to want the volume group to be call "pve" everwhere? But what about the situation where the node I'm adding I want that to be on a separate SSD? Should I have just named the VG "pve" on the new node, rather than "data"?

 

[bbgeek17]

What is the context of your /etc/pve/storage.cfg ? Use text and CODE tags.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[marcosscriven]

root@pve-pc:~# cat  /etc/pve/storage.cfg
dir: local
    path /var/lib/vz
    content vztmpl,iso,backup

lvmthin: local-lvm
    thinpool data
    vgname pve
    content rootdir,images

pbs: pbs
    datastore backups
    server 192.168.0.3
    content backup
    fingerprint ...
    prune-backups keep-all=1
    username root@pam

 

[bbgeek17]

lvmthin: local-lvm thinpool data vgname pve content rootdir,images

PVE expects this storage to be available across all nodes with that exact configuration.

You can use "pvesm set local-lvm --nodes [node1,node2]" to tell PVE that only those nodes have this storage.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[marcosscriven]

Thanks for your help - but I'd like to do the opposite, and make the storage on this new node match the others.

Is that not possible from a logical level, if the underlying storage is instead on a different SSD to the pve/lvm volume?

 

[bbgeek17]

but I'd like to do the opposite, and make the storage on this new node match the others.

Plug this into google "lvm rename volume group", read a few articles and proceed with caution to rename the default volume on new node. Then you can re-use the name.
You may also need to adjust your /etc/fstab.
Note that this is an advanced procedure outside of normal PVE installation and configuration. I'd recommend you install a virtual/nested PVE and try it there first.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[marcosscriven]

Thanks - but here's the issue:

 vgs
  VG   #PV #LV #SN Attr   VSize    VFree
  data   1   1   0 wz--n-   <3.64t 376.00m
  pve    1   2   0 wz--n- <118.24g <70.68g

I already have a volume group called "pve" - and as I suspected, I can't renamed the 'data' one:

lvm vgrename DnVIZ9-rr2V-FGnZ-Dsn4-eeW3-8qSI-ngIVrK pve
  /dev/pve: already exists in filesystem

So what does one do in the even you want `pve/lvm` and `pve/lvm-local` on one disk on most nodes, but on separate disks on just one node?

 

[bbgeek17]

So what does one do in the even you want `pve/lvm` and `pve/lvm-local` on one disk on most nodes, but on separate disks on just one node?

You have:
node1: vgnameA,poolA
node2: vgnameA,poolA
node3: vgnameA,poolA AND vgnameB/poolB

To achieve your goal, without getting into whether its a worthy one, you would need to:
rename on node3: vgnameA to vgnameC, poolA to poolC, vgnameB to vgnameA, poolB to poolA

Then, you'd be able to have "local-lvm" storage pool that points to vgnameA/poolA and is valid on all 3 nodes.

Again, test before doing it on live system. Including making sure it reboots.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[marcosscriven]

Thanks again - but wouldn't that just then move the problem to the `local` storage?

 

[bbgeek17]

Thanks again - but wouldn't that just then move the problem to the `local` storage?

Can you clarify your concern? "local" is a Directory storage pool pointing to /var/lib/vz directory. Its present on all nodes in 100% of cases.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[marcosscriven]

It seems odd to be forced to move root and swap off the “PVE” VG, just to be able to have “local-lvm” on a different SSD.

So I’d be trading one quirk for another.

What’s the recommended way of doing this? I can’t believe I’m the only one that would want the Proxmox root on its own disk, but then not have to faff juggling the volume groups.

Perhaps I should start again and abandon the idea of doing this.