Recommended FileSystem for Single SSD Boot Drive?

[Allister]

Hey there..

I'm building a 3 node cluster for Proxmox. Each node has a single Samsung 870 EVO SSD for the boot drive. It will have a separate Samsung 980 Pro 2TB drive for the VMs and LXCs which will be zfs. I'm just not sure what the right choice would be for the standalone boot drive.

What would be the right filesystem to choose for the standalone boot drive?

 

[louie1961]

nothing wrong with going with zfs, even in a single drive arrangement

 

[Allister]

nothing wrong with going with zfs, even in a single drive arrangement

Really? I read that ZFS has a tendency to write so much to the drives that it kills consumer grade drives fairly quick. Is that not the case? Or is that Proxmox in general does a lot of writes and has nothing to do with the file system type?

 

[news]

Why are you not use a 2 drive raid1 - zfs mirror setup ?
For ZFS - yes there many iops for the redudancy and checksum for you data.
So you must consider a only 3 year live time.

for you Samsung 980 Pro 2TB
TWB 1.200 TB is given

Let the zpool trim <pool> command run in a cron job and set

zfs atime=off <dataset>

for less read and write.

But for Proxmox Backupserver the dataset must have

set  zfs atime=on <dataset>

i read i this forum.

 

[leesteken]

Really? I read that ZFS has a tendency to write so much to the drives that it kills consumer grade drives fairly quick. Is that not the case? Or is that Proxmox in general does a lot of writes and has nothing to do with the file system type?

Proxmox does write a lot and ZFS magnifies that greatly, so yes, it will wear consumer drive sometimes quickly. But you are already planning to run your VMs on a consumer drive with ZFS.
A Proxmox installation will run fine from a (slow, rotating) hard disk, which can handle the many small writes better. If you use a whole 870EVO just for Proxmox (which is only a few GB), you'll probably be fine. I ran Proxmox and VMs (with little IO) with ZFS on 970EVO for years (65TB written, 6% worn) but I disabled PVE cluster services.

 

[louie1961]

Hey there..

I'm building a 3 node cluster for Proxmox. Each node has a single Samsung 870 EVO SSD for the boot drive. It will have a separate Samsung 980 Pro 2TB drive for the VMs and LXCs which will be zfs. I'm just not sure what the right choice would be for the standalone boot drive.

What would be the right filesystem to choose for the standalone boot drive?

This hasn't been my experience. I have a N100 NUC box unit with a single 1TB Sabrent Rocket m.2 NVMe in it, formatted in ZFS. That drive holds the Proxmox OS as well as three VMs, running two instances of Openmediavault and one instance of Debian as a host for Ansible. Been running it for a few months now and it shows 0% wearout. I have an HP mini G9 with a pair of Team Group SSD 1TB m.2 NVMe drives in a mirror, pretty much with a similar situation (running the Proxmox OS and all of the VMs in that mirror). That machine has been online for about a year now, and also shows 0% wearout.

I don't run in a cluster however, so I can't say how that might affect it, and I also don't store any data on my Proxmox hosts. All the data for my VMs and containers sits in separate NFS shares served up by my Synology.

Also, with my setup, if a Proxmox host dies, it is pretty easy for me to replace it. I have all my VMs and containers backed up to a separate TrueNAS machine and to AWS Glacier daily (or every two hours in the case of my publicly facing wordpress sites), and all of my data backed up to a Synology NAS that also backs up to AWS Glacier. So its a low risk experiment for me to run consumer drives.

 

[news]

I use everytime zfs, lowes secure is raid1 - zfs mirror.

One PM VE Server has ex. 4x SSD with zfs raidz1.
And an other PM VE Server has 4x SSD with zfs - mirror0 2 x SSD and mirror1 2 x SSD.

It depends on.

I Always save per automatic snapshot and rsync the dirs /root and /etc for proxmox ve recovery reason.

 

[Kingneutron]

If you're not sure, there's nothing wrong with LVM+ext4. It's easy to backup and you can set noatime on all non-zfs filesystems.

 

[Kingneutron]

Why are you not use a 2 drive raid1 - zfs mirror setup ?

ZFS boot/root mirror is actually a more complex setup than LVM+ext4. Yes it's nice from a redundancy standpoint but it requires reading up on a fair amount of documentation. Sysadmin needs to know the right procedures to follow when a drive fails, to properly restore the EFI boot on both drives. And it's recommended to use 2 different drive manufacturers or models so they don't fail around the same time.

I just tested this last night in a VM (migrating a mirrored PVE zfs boot/root to smaller drives) and will be writing up my findings on the reddit forum $soon.

For me, backing up and restoring an ext4 rootfs is dead easy, I have scripts for it and can probably get back up and running in a couple of hours. And Veeam agent for linux will actually recreate the LVM for you (NOT the lvm-thin however!)

ZFS on root just requires more consideration and has a bit of a learning curve, you don't want to just blindly set it up and then **not test** a DR procedure. System needs to be able to boot from either mirror drive with the other one missing. And then you need to have the docs handy for when a drive fails and needs to be replaced, it doesn't all happen automatically.

Sorry, don't want to derail the thread but I had to respond ;-)

 

[louie1961]

If you're not sure, there's nothing wrong with LVM+ext4. It's easy to backup and you can set noatime on all non-zfs filesystems.

I actually think BTRFS in a single drive setup is probably a decent set up as well. I would do like Synology does however, and rely on MDADM and LVM for any raid or volume management functions, and just use BTRFS for the file system.