Which Filesystem for Hardware Raid 1 SSD drives?

[toben88]

Hello,

My understanding is some file systems will keep writing logs on the drives all the time. So if a drive is SSD and hardware RAID 1 this should be avoided because SSDs can only have so many writes before they start marking sectors as bad.

What is the best filesystem to hold the VMs in this scenario?
EXT4, ZFS, or XFS?

Thanks!

 

[Dunuin]

ZFS shouldn't be used on top of HW raid. I would go with LVM-Thin, then you got a block-level storage that supports thin-provisioning and snapshots and it skip the write amplification of nested filesystems.
Then you still got the choice what filesystem to use inside your VMs. If I remeber right, when I did some benchmarks both etx4 and xfs got a similar write amplification when doing async sequential writes. But on sync random writes xfs wrote less, because the metadata was more efficient.

But in general: The more reliable a filesystem is, the more it will write (additional copies of metadata, additional checksums, writing everything twice because of journaling, more sync writes and less async write, ...). I personally wouldn't choose a less reliable filesystem or more insecure options just to save some SSD wear. Its better to buy some robust SSDs that can handle the addtional writes.

 

[toben88]

ZFS shouldn't be used on top of HW raid. I would go with LVM-Thin, then you got a block-level storage that supports thin-provisioning and snapshots and it skip the write amplification of nested filesystems.
Then you still got the choice what filesystem to use inside your VMs. If I remeber right, when I did some benchmarks both etx4 and xfs got a similar write amplification when doing async sequential writes. But on sync random writes xfs wrote less, because the metadata was more efficient.

But in general: The more reliable a filesystem is, the more it will write (additional copies of metadata, additional checksums, writing everything twice because of journaling, more sync writes and less async write, ...). I personally wouldn't choose a less reliable filesystem or more insecure options just to save some SSD wear. Its better to buy some robust SSDs that can handle the additional writes.

I am shipping the server to a colo facility and the plan is to not physcially touch the server for a few years unless there is a drive failure. At that point the hardware RAID should hold and I plan to mail in a replacement SSD drives as needed. So I do actually want to limit my ssd writes when it makes sense.

From your previous post it appears LVM-thin with xfs is the way to go for the filesystem holding my vm's?

 

[jdancer]

I am shipping the server to a colo facility and the plan is to not physcially touch the server for a few years unless there is a drive failure. At that point the hardware RAID should hold and I plan to mail in a replacement SSD drives as needed. So I do actually want to limit my ssd writes when it makes sense.

From your previous post it appears LVM-thin with xfs is the way to go for the filesystem holding my vm's?

LVM-thin/XFS is a good choice.

 

[_gabriel]

there is no FS choice to be done with lvmthin from host-pve.
you'll choose FS in your VMs (even each vdisk can have different FS).

 

[leesteken]

If you are shipping this system to Mars (or something), setup a cluster with multiple systems for reliability and don't focus on the SSDs so much. Failures can happen on any part of the system(s).

 

[toben88]

According to this article - I should use ext4 with journaling shut off. https://www.phoronix.com/review/linux_raid_fs4

You may think that is crazy but my server is being setup to run 24/7 and never be turned off.

All drives are SSD Hardware RAID1, Redundant power supplies, 256G Ram, Dual Xeon. Server to be placed in a colo facility that I will not be able to physically service.

Boot drive - 500GB SSDs w hardware RAID1 only running proxmox.
Data1 drive - 1TB SSDs w hardware RAID1 only for VMs.
Data 2 drive - 2 TB SSDs w hardware RAID1 for additional VMs.
Data 3 drive - 2 TB SSDs with hardware RAID1 for backup of critical VMs. If any non critical VMs can't fit in this location they will be backed up offsite via the cluster anyway.

Drives are hot swappable upon a failure by mailing replacement drives to colo facility.

I will setup on machine VM backups to other drives and a cluster to backup to a home location, but a full system colo power off is to be avoided at all costs.

If there is a redundant SSD failure on a RAID1 I should still be able to recover unless it is the boot drive. And the boot drive is only running proxmox. No VMS there.

Any other opinions for filesystems to use on the data drives/VM drives in this situation?

Thanks!

 

[Dunuin]

According to this article - I should use ext4 with journaling shut off. https://www.phoronix.com/review/linux_raid_fs4

Yes, and that is one of the points I wouldn't recommend. The journal needs additional writes but it's not there just for fun. Disabling it is terrible for data integrity. A failing disks because of too much writes isn't the only case where you can lose a complete filesystem.

 

[toben88]

Yes, and that is one of the points I wouldn't recommend. The journal needs additional writes but it's not there just for fun. Disabling it is terrible for data integrity. A failing disks because of too much writes isn't the only case where you can lose a complete filesystem.

Doesn't hardware raid1 eliminate this risk?

 

[Dunuin]

Raid1 will only make sure both disks always contain the same data, so you got one copy in case a single disk fails. A HW raid won't check if your data got corrupted, it won't fix corrupted data and in case there is a computation error it will write corrupted data to both disks at the same time. And in case of a kernel crash or hardware failures both disks will lose cached data at the same time. You usually then would use the journal to fix your corrupted filesystem, so not great if you disable that.