[SOLVED] Can not add shared storage to cluster node

[muekno]

I build a virtual test cluster. Did this not the first time.
Made three virtual PVEs
made first a cluster, joint second, joint third everything look fine as it should
added disk with ZFS to first node, ok added second uncheck "add storage" works fine
the added storage is seen under storage with sane name
did the same on the third node. It is seen under disks ZFS but
it is not listed as storage under the node where the other storages are
Rmove des disk grome the node and addes again, still not seen
Where can I check whats going wrong

 

[UdoB]

it is not listed as storage under the node where the other storages are

Check: "Datacenter --> Storage --> <your ZFS pool> --> Edit --> Dialog opens ---> Nodes".

This is a multi-select(!) drop-down menu. Deleting all entries implies "All (No restrictions)" to tell PVE that this storage is available on all nodes.

 

[muekno]

Check: "Datacenter --> Storage --> <your ZFS pool> --> Edit --> Dialog opens ---> Nodes".

This is a multi-select(!) drop-down menu. Deleting all entries implies "All (No restrictions)" to tell PVE that this storage is available on all nodes.

Yes thank you, I tried a third time and then it worked, but it took nearly 45 second until the storage apeared, while normal it took less than 5 seconds

 

[bbgeek17]

added disk with ZFS to first node

ZFS storage cannot be used as shared storage.

Keep in mind that when you select "shared" that attribute just informs PVE that the underlying storage is already available to all nodes via some other (external method). PVE does not make the storage shared.
ZFS cannot be used in shared storage application as its not compatible with concurrent multi-host access.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[muekno]

ZFS storage cannot be used as shared storage.

Keep in mind that when you select "shared" that attribute just informs PVE that the underlying storage is already available to all nodes via some other (external method). PVE does not make the storage shared.
ZFS cannot be used in shared storage application as its not compatible with concurrent multi-host access.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Every description says create storage disk with ZFS that works fine

 

[bbgeek17]

Every description says create storage disk with ZFS that works fine

Not in conjunction with "shared" storage.
Shared means that the disk, with the file system (if file system is used), is available on multiple hosts at the same time for concurrent access.

Based on your description, it appears that you have local disks on each host that you formatted with ZFS. That is similar to "local-lvm". You will be able to move VM from one node to another but it will be a fully copy. This is not shared storage.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[UdoB]

You need to setup replication. https://pve.proxmox.com/pve-docs/chapter-pvesr.html

Every few minutes (or every hour or ...) you can replicate a VM from the node it is running on to the other two.

Then you can do a) quick live migration and b) High-Availability.

This is "a poor guys shared storage". For a lot of setups it works quite well. But @bbgeek17 is right, of course: it ist not really shared.


---
Edit: here: https://pve.proxmox.com/pve-docs/chapter-pvesm.html#_storage_types it is NOT listed as "shared".

 

[bbgeek17]

of course: it ist not really shared.

Agreed.

The issue with marking "local" storage as "shared" is that during VM migration the PVE will expect for VM image (disk/ZFS slice) to exist on remote side because you, @muekno, told it so via "shared" attribute. The image, of course, is not there so the migration will fail.

Replication is the appropriate technology to use with local ZFS on each side.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[muekno]

Agreed.

The issue with marking "local" storage as "shared" is that during VM migration the PVE will expect for VM image (disk/ZFS slice) to exist on remote side because you, @muekno, told it so via "shared" attribute. The image, of course, is not there so the migration will fail.

Replication is the appropriate technology to use with local ZFS on each side.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Correct, this name "shared" storage is not the same physical disk, it is a separate disk on each cluster node.