I have some filesystem questions regarding PVE, ZFS and how they interact with VMs.
The storage system I'm trying to achieve consists of 5 physical disks:
Disks 1 and 2 contain little data outside of booting and running Proxmox itself. They're mirrored in case of a disk failure.
Disks 3 and 4 are what I actually care about.
For the RAID0 striping performance, I joined a portion of Disk 3 and Disk 4 together as one volume group and then created logical volumes on top of it with varying stripe sizes. It uses XFS (not a COW filesystem) and LVM. See this thread. I've mounted the logical volumes as directories in PVE. This part was successful.
For the RAID1 mirroring reliability, I'd like to use a ZFS mirror for reliability and integrity. But I don't understand how already having a ZFS mirror on the PVE disks plays into all this. How exactly does this work?
What's the smartest way to go about it?
Some questions:
I'd prefer to keep using LVM and XFS for the RAID0 portion of Disk 3 + Disk 4, but I'm not married to any one way of setting up.
(My reason for splitting Disk 3 + Disk 4 into RAID0 and RAID1 portions instead of adding a Disk 6 + Disk 7 was purely for cost, btw. Having Disk 3 + Disk 4 are already enough for the expected workload)
The storage system I'm trying to achieve consists of 5 physical disks:
| Physical Disk | Purpose | RAID |
|---|---|---|
| Disk 1 + Disk 2 | PVE + mirror | ZFS mirror |
| Disk 3 + Disk 4 | VMs and container storage | a mix of striping and mirroring (but not RAID 10) |
| Disk 5 | backups, snapshots, ISO files, templates, etc. | no striping nor mirroring |
Disks 1 and 2 contain little data outside of booting and running Proxmox itself. They're mirrored in case of a disk failure.
Disks 3 and 4 are what I actually care about.
For the RAID0 striping performance, I joined a portion of Disk 3 and Disk 4 together as one volume group and then created logical volumes on top of it with varying stripe sizes. It uses XFS (not a COW filesystem) and LVM. See this thread. I've mounted the logical volumes as directories in PVE. This part was successful.
For the RAID1 mirroring reliability, I'd like to use a ZFS mirror for reliability and integrity. But I don't understand how already having a ZFS mirror on the PVE disks plays into all this. How exactly does this work?
What's the smartest way to go about it?
Some questions:
- If I make a new ZFS mirror using disk space on Disk 3 + Disk 4, who exactly does the ultimate work of writing to physical disk, PVE or the VM's OS?
- Would the ZFS mirror on Disk 1 + Disk 2 impact performance of the VMs if I make a ZFS mirror on Disk 3 + Disk 4 (outside of the usual ZFS performance penalties)?
- Would I be creating some kind of zpool inside of another zpool by doing it this way?
- Would it be easier if I ditched ZFS on the PVE and used something else to mirror Disk 1 + Disk 2 instead?
I'd prefer to keep using LVM and XFS for the RAID0 portion of Disk 3 + Disk 4, but I'm not married to any one way of setting up.
(My reason for splitting Disk 3 + Disk 4 into RAID0 and RAID1 portions instead of adding a Disk 6 + Disk 7 was purely for cost, btw. Having Disk 3 + Disk 4 are already enough for the expected workload)
Last edited: