One ZFS storage shown twice with two names, why?

[T1nue]

Today I created a cluster and added a 2nd node to it. Now I have a strange mixup with my storage:



On node "k2", storage "virtualmachines" and "virtualmachines-lhotse" both point to a local ZFS formatted disk. The content is 100% identical, it's just a duplication in the GUI that I can't explain.

On node "lhotse", Storage "virtualmachines-lhotse" points to a local ZSF formatted disk. There is no duplication here.

Why is this happening? How can I debug / fix this? I can explain what I did, but it's a bit complicated.

 

[aaron]

What is the storage configuration? Can you post the oyutput of cat /etc/pve/storage.cfg?

 

[T1nue]

Thanks for your reply. This is the output (edited a bit on the NFS share):

dir: local
    path /var/lib/vz
    content iso,vztmpl
    shared 0

lvmthin: local-lvm
    thinpool data
    vgname pve
    content images,rootdir

zfspool: virtualmachines
    pool virtualmachines
    content rootdir,images
    mountpoint /virtualmachines
    nodes k2

nfs: Synology
    export /volume1/Proxmox-Backup
    path /mnt/pve/Synology
    server xx.xxx.xxx.xxx
    content backup
    options vers=4.1
    prune-backups keep-last=7

zfspool: virtualmachines-lhotse
    pool virtualmachines
    content rootdir,images
    mountpoint /virtualmachines
    sparse 0

 

[aaron]

If you look at it, you will see that virtualmachines and virtualmachines-lhotse use both the same pool and dataset.
virtualmachines is limited to the node k2.

I would go and check which storages are used by your VMs more. You can do that by checking the first part of the VMs disks config as that references the storage.

If you only have a few VMs on k2 that use the virtualmachines storage, consider to use the "Move Disk" to move them to either virtualmachines-lhotse or temporarily to the local-lvm one. Then remove the virtualmachines storage from the Datacenter -> Storage panel. Don't worry, no data will be destroyed, just the entry in the /etc/pve/storage.cfg file removed.

 

[T1nue]

Actually they do not use the same pool. I started with two independent nodes. Both have a local ZFS (pool) named "virtualmachines", which is on the local SSD of each node. On k2 I had a dataset also named "virtualmachines" pointing to this ZFS. On node lhotse I did not yet add a dataset.

Only then I defined a cluster, and added the two machines together. I think Proxmox is confused that both nodes each have a local ZFS pool with the same name. Since I wanted to also access the pool "virtualmachines" of node "Lhotse", I tried to add another dataset. I could not create one named "virtualmachines"; The error mentioned that a dataset with this name already exists.

So I added a dataset named "virtualmachines-lhotse", pointing to the pool "virtualmachines". This resulted in the configuration posted above.

Or am I totally wrong, and my cluster now "magically" has a ZFS pool spanning both nodes transparently? I don't think so, because if I check on each node "Disks", "ZFS", then I see both individual file systems, and they are not pooled in any way.

 

[aaron]

Or am I totally wrong, and my cluster now "magically" has a ZFS pool spanning both nodes transparently? I don't think so, because if I check on each node "Disks", "ZFS", then I see both individual file systems, and they are not pooled in any way.

No it does not. The storage config just tells Proxmox VE what storages are there.
If you have the same ZFS pool and do not limit it to a certain node, it will show up, but will be local to that node. Like the "local" and "local-lvm" storages are configured without a node limitation and shows you the storage local to that node.

So either limit the "...-lothse" to that node (top right if you edit the storage, or remove it. to have the "virtualmachines" storage available on both, since both use the same pool name locally, by removing the node limitation for the "virtualmachines" storage.

 

[T1nue]

Thank you, I think I got it now! Instead of trying to add another "virtualmachines" pool for lhotse, I should have removed the "k2" node limitation on the existing definition. Supposedly if I had added the pool on lhotse before joining the nodes, then this would have been performed by the cluster join magic, like it did with "local" and "local-lvm".

I have done what you recommended, and now have one single "virtualmachines" entry, just like "local-lvm". Perfect!

 

[aaron]

Doing it this way, knowing that ZFS is local storage, also make it possible, if you want to use the VM Replication feature in the future. It needs a ZFS storage with the same name on both nodes.

 

[T1nue]

Good to know, thanks! So far I only manually moved a VM, but replication looks like something to look into.