Directories Under ZFS POOL

[heres_johnny]

Forgive me if any of my technical references are incorrect, but I am relatively new to Proxmox & ZFS and this is my first post, I inherited a new server build with Proxmox pre-installed and I'm trying to get it configured with the basics.

In addition to the 2 mirrored drives for the OS, the server came with 5-840GB SSDs unallocated, and I'm attempting to create a pool using ZFS across all 5 SSDs and then divide into directories for 1) ISOs, 2) VM storage, and 3) snapshots.

So far I created the pool ("POOL1") using ZFS RAID-Z2, but I'm unable to create the directories under that. I figured this should be fairly straight-forward but I'm not able to find posts or videos with this scenario (although, I'm sure there are many). Maybe I'm taking the wrong approach.

I don't expect to use too much ISO storage, so I don't want to dedicate an entire disk to it. I also don't know the disk cost of snapshots in Proxmox but I understand it is relatively low, so I'd like to set up for that, if feasible. I've been a Windows/VMware administrator for several years and I'm making the move to open source, so I understand the concepts and benefits behind Proxmox and ZFS, just not the execution.

With the SSDs that I have, am I going about this the best way? If so, I just need help to get over the hump. If not, any guidance is appreciated.

 

[guletz]

I'm attempting to create a pool using ZFS across all 5 SSDs and then divide into directories for 1) ISOs,

Hi,

Thease are need to be datasets and not directory(with ISO exception), and could be created only with zfs create ..... from command line. Then add from storage as zfs.

Good luck / Bafta !

 

[aaron]

First, if you plan to store VM disks on that pool, consider using mirror vdevs (RAID1, RAID10) and not any raidz. There is a section in the documentation about this. The TL;DR is that VM disks are not using the more common file system datasets but volume (zvol) datasets which provide a block device. For these, the parity can use up quite a bit of space which is usually not expected. Plus the performance for multiple VMs is better with mirror vdevs than with any raidz.

As @guletz already mentioned, if you plan to split it up even more, create sub datasets, e.g. zfs create <pool>/isos and zfs create <pool>/guest_storage.

Since a Storage of the type ZFS on the Proxmox VE level can only store VM and container disks, you will need to create a storage of the type "Directory" and point it to the path of the dataset. The path then is for example /<pool name>/isos (keep in mind the "/" at the beginning in this case. Additionally you should configure the is_mountpoint parameter for that storage. It tells Proxmox VE that something needs to be mounted at that path before the storage can be activated.

pvesm set <storage> --is_mountpoint yes

Snapshots are based on the functionality of the underlying storage and are not dedicated files. For ZFS this means that the datasets of the VM disks get snapshotted with the means provided by ZFS. You can list snapshots by zfs list -t snapshot

 

[aaron]

One more thing, datasets for VM disks will not show up in the directory tree, but will be visible with a zfs list and are exposed in the /dev path as /dev/zdX and as /dev/zvol/<pool>/<dataset>

 

[heres_johnny]

Wow. I was much further off base than I thought. It makes sense now why I wasn't able to find documentation to support what I was trying to do. I'll rethink my approach with the guidance you've both provided and post my updates (hopefully) shortly.

 

[Dunuin]

And you can't store snapshots anywhere else because they are just a part of the filesystem. And snapshots can get very big if your VMs are write heavy or you keep them for too long. Its not unusual that a snapshot can be a multiple of the size of the VM itself. ZFS is a copy-on-write filesystem so if you edit or delete a file nothing will be deleted if there is a snapshot. So if you create a 10GB file, and edit 10 times half of the file, the file will only consume 10GB of the virtual disk but the snapshot will grow by 60GB.
So they are more useful for automated hourly/daily backup that are only kept for a day or a week (cv4pve-snapshot is a usefull script for that).
For long term backups I would recommend to use vzdump or better "Proxmox Backup Server" instead.

 

[heres_johnny]

Thanks to everyone who took the time to reply Your level of detail has been very helpful.
Looking at this configuration again, I'm going to re-think the disk layout. As it turns out, I expect that I will have 3 Windows VMs on each physical server (2 total), and maybe 3 of either Linux VMs or containers. On one virtual host, there will be a VM running Exchange Server, and on the other virtual host, there will be a VM running MS SQL server. I don't know what my predecessor had in mind, but I don't think I have the capacity to host storage locally on the 5 available SSDs.
We have 2 TrueNAS R20s to be unboxed and configured as well, but I'm trying to get my head wrapped around how I'm going to get the virtual hosts configured before anything.
Unfortunately, I'm only able to research and test during small pockets of time throughout my day, and it seems to really drag out the process.
I'll jump back in here with updates when I can.

 

[heres_johnny]

It looks as though I was over-thinking the initial setup. I ended up creating a RAID 10 across 4 disks and making the 5th a hot spare. Not really difficult at all.

As for the Exchange and SQL DB data, apparently my predecessor intended to create iSCSI targets on the TrueNAS box and connect drives in the VM, which is outside the scope of this forum, so I'll end it at that.

Thanks for all the feedback and direction.