Understanding ZFS: added new zfs in existing zfs pool

[rakurtz]

Hello together,

i thought i'd understood ZFS quite enough but no i am not so sure. On in my cluster consisting of two node, each having a local ZFS RaidZ2, called data (zpool), which uses all of the disks in that server except the one, Proxmox itself is installed on.

I wanted to add another "zfs" storage through the WebGui and gave it the ID anotherdata and as ZFS POOL i chose data (the one from above). To my surprise i can now see both of the storages appearing under my node(s), but in an instant all the VM disks which were shown under content of data before, now appear (only there!) under anotherdata. VMs are still running.

This was not what i wanted. I thought i could add another storage which i wanted to use to separate disks of different users by assigning different user rights to data and anotherdata.

No i don't know how to reverse to the state i head before since i am afraid to delete all VM Images when i delete anotherdata.
So i deactivated anotherdata, but the VM-Images didn't appear under the content of data. The VMs were still running without any problems, but their images weren't shown anywhere in the cluster (at least not in the WebGui).

Can i delete anotherdata or would that delete all the VM-Images suddenly stored there?! What kind of storage is a new zfs inside an existing zpool?
Does it make sense to move a VM-Image from anotherdata back to data?

thx!

 

[fiona]

Hi,

Hello together,

i thought i'd understood ZFS quite enough but no i am not so sure. On in my cluster consisting of two node, each having a local ZFS RaidZ2, called data (zpool), which uses all of the disks in that server except the one, Proxmox itself is installed on.

I wanted to add another "zfs" storage through the WebGui and gave it the ID anotherdata and as ZFS POOL i chose data (the one from above). To my surprise i can now see both of the storages appearing under my node(s), but in an instant all the VM disks which were shown under content of data before, now appear (only there!) under anotherdata. VMs are still running.

yes, PVE might get confused when you use the very same pool for two different storages. But there's no need to panic. The storage configuration in PVE is really just configuration. You should be fine just deleting the new storage again, but let's make sure everything is fine first (see below). To make extra sure, you can also just edit the /etc/pve/storage.cfg manually and remove the anotherdata entry.

This was not what i wanted. I thought i could add another storage which i wanted to use to separate disks of different users by assigning different user rights to data and anotherdata.

I suppose you need to create sub-datasets for this, otherwise you won't get the desired separation.

No i don't know how to reverse to the state i head before since i am afraid to delete all VM Images when i delete anotherdata.
So i deactivated anotherdata, but the VM-Images didn't appear under the content of data. The VMs were still running without any problems, but their images weren't shown anywhere in the cluster (at least not in the WebGui).

This is what I cannot reproduce. For me the images show up for both storages (even if I disable the new one). Could you run the following:

pveversion -v
zfs list
pvesm list data
cat /etc/pve/storage.cfg

Can i delete anotherdata or would that delete all the VM-Images suddenly stored there?! What kind of storage is a new zfs inside an existing zpool?
Does it make sense to move a VM-Image from anotherdata back to data?

You shouldn't need to move them, because the real location didn't change. The VM images should still be just ZFS volumes within data, as the zfs list command hopefully shows.

thx!

 

[rakurtz]

Good morning Fabian,

thank you for your quick reply. Interestingly i found out, being logged in as root i can see all the images in both storages. Being logged in as the user (also set as admin user) who created the anotherdata storage, the files are only shown in anotherdata.

zfs list

shows all images in data.

in deleted anotherdata via

nano /etc/pve/storage.cfg

now. It seams that the reason i didn't see the images within content of data was connected to some user permission issues. After explicitly giving permission to that user, it showed all images as normal.

Learned something. Thanks!

 

[Dunuin]

"zdevs" are your physical drives.
"zpool" is your "raid array".
"datasets" are like folders which can branch and be inherited to structure stuff (but they aren't really folder, each dataset is its own filesystem).
"zvols" are logical block devices.

Your VMs virtual harddiscs are zvols that should be directly stored on that zpool if you didn't used datasets before, so it should look like this:
zdevs -> zpool -> zvols

If you want to organize it so each zvol can inherit the rights you need to create datasets so it looks like this:
zdev -> zpool -> datasets -> zvols