Promox 9 LVM share between nodes issue

[tessierp]

Hi,

I have the following problem on Proxmox 9.

I did some research to try and fix the following problem but can't find a solution. I have 3 proxmox servers part of a cluster so 3 nodes. On each node I have two SSDs available for VMs. I have initialized the disks as LVM. In the Datacenter I enabled the LVM and they are all shared for all nodes to access; purpose of this is to allow migration and cloning between nodes. Here is the configuration



Even though they are shared, disks from proxmox node PTR1-PVE-1 have a status of unknown. As you can see with the image below, only the local SSDs are seen. The ones from the other node, node 1, have a status of unknown.



Even 'pvesm status' gives me an inactive status



All this to say, because node 1 can't see node 2 LVMs (inactive) migration and cloning is not possible.

I have tried every possible solution I could find such as sharing them through NFS, etc but I can't get the LVMs on one node to show as active.

Could anyone tell me what I'm missing?

 

[bbgeek17]

Hi @tessierp ,

You are confusing an attribute that declares to PVE that the storage is already shared with actually creating shared storage.

Please read the following paragraph:
https://pve.proxmox.com/pve-docs/ch...uest hook scripts-,shared,-Indicate that this

shared
Indicate that this is a single storage with the same contents on all nodes (or all listed in the nodes option). It will not make the contents of a local storage automatically accessible to other nodes, it just marks an already shared storage as such!

You have _local_ disks, they are not visible across all nodes - not shared. Your options are: implement Ceph or use ZFS/replication.

Cheers

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[tessierp]

Hi @tessierp ,

You are confusing an attribute that declares to PVE that the storage is already shared with actually creating shared storage.

Please read the following paragraph:
https://pve.proxmox.com/pve-docs/chapter-pvesm.html#:~:text=guest hook scripts-,shared,-Indicate that this

shared
Indicate that this is a single storage with the same contents on all nodes (or all listed in the nodes option). It will not make the contents of a local storage automatically accessible to other nodes, it just marks an already shared storage as such!

You have _local_ disks, they are not visible across all nodes - not shared. Your options are: implement Ceph or use ZFS/replication.

Cheers

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Could be me but the way this is phrased is a bit confusing to me. I noticed that I was able to migrate VMs to another node but it remained on the disk of origin.. So I can run a VM on a node sitting on another node's disk but I cannot transfer from a node's disk to another because the disk itself isn't shared. Is that what I'm supposed to understand?

 

[fabian]

no. in PVE you have two types of storages: local and shared. local storage means that the contents of that storage are only available per-node (like a local directory, ZFS, LVM on a local disk, ..). shared means that by virtue of the storage itself, its contents is identical on all nodes (like ceph, NFS, CIFS, LVM on top of an iSCSI disk accessible by all nodes, ..).

for local storages, if a node fails the guest volume is gone/unavailable and you cannot recover it except from backups. for shared storages, you can enable HA and recover the guest, since the volumes are available on every node. migration is possible for both local and shared storages, as long as both source and target nodes are online.

having the disk on one node while the VM is running on another is not possible at all, except if you make one of your nodes an NFS server or some similar setup - but that is very easy to do wrong and doesn't provide resilience in any fashion.