[SOLVED] [Virtualizing PDM] Host PVE Node Uses ZFS Storage. When installing PDM inside VM, EXT4 or ZFS?

[SInisterPisces]

Hello,

I've never virtualized Proxmox VE, Proxmox Backup Server, or Proxmox Datacenter Manager. In fact, I've never installed PDM, and I'd like to test it in a VM.

Normally, I use ext4 inside my VMs; this is a home server hobbyist/home office environment, and using ext4 in my guests has always been fine--and the VM virtual disk lives on a ZFS pool, so it's getting the benefits of ZFS on the underlying storage.

But, I'm very used to using ZFS in PVE and PBS for the advantages it provides. I'm not sure whether it's the preferred filesystem for PDM. Is there a benefit to running PDM with ZFS inside the guest? If there is, are there any special considerations I should be aware of when using ZFS inside a zVol/virtual disk?

(Years ago, this topic seems to have been complex enough when talking about general purpose VMs that most people seemed to default to not using ZFS inside the VM without a very good reason. Curious if that has changed.)

 

[UdoB]

I am using ext4 for everything. (There are valid reasons to use other filesystems, but those are not included in my zoo.)

The only reason for ZFS inside a guest would be if there is payload software utilizing specific ZFS features by itself. I do not see that for PDM (or PBS or ...).

The reasons for "do not stack ZFS onto another" are still the same, I believe.

Just my two €¢..., your-mileage-may-vary - as usual

 

[Impact]

I avoid running ZFS inside my guests. Only on the node. PDM runs in a debian CT for me.

 

[SInisterPisces]

Thank you both.

The reasons for "do not stack ZFS onto another" are still the same, I believe.

Is there a relatively modern write-up about why this is a bad idea (and maybe some use cases when you might actually want to do it)?

I've gotten to the point where I'm pretty wary of taking any articles about ZFS older than ~2020 years old as still being completely applicable to modern ZFS. A lot of the older rules (avoiding consumer SSDs, always using ECC, 1 GB RAM per 1 TB of storage) either seem to be outdated or come with a lot of caveats now.

It's a shame so much ancient stuff (hello, 2013 ZFS docs) still gets surfaced at the top of a lot of "how do I ZFS"-style searches.

 

[UdoB]

Is there a relatively modern write-up about why this is a bad idea

Sorry, I don't have any current pointers. But the concept of the unwanted, massive write-amplification is still true, right?

There were some "historical" misunderstandings and there are some newer improvements, yes. But "consumer SSDs" are still bad; "always ECC" never was true (but it is still recommended!) and "1 GB/TB" was always just a rule of thumb.

It's a shame so much ancient stuff (hello, 2013 ZFS docs)

I prefer to call it stable

 

[J-Rod]

I have various scenarios where ZFS is running on the host and in a VM. All of my PVE hosts are consuming storage via ZFS, for compression, RAID and error correction.

Then I have a TrueNAS VM, although currently primarily for testing. I also have some PBS VMs and of course they were set up with ZFS to allow for a ZFS special device for metadata. I run PBS in VMs so that I can use the remaining storage for other purposes. I could not see giving up all of the enterprise SSD capacity to just the special device, or on the flip side running out of capacity and having to go through hardware gymnastics to increase that. When it's a VM, just hot add some more and you're done.

I've also hot migrated my main PBS VM twice already to different hosts as I came in for a landing on the hardware selection.

I am using EXT4 for my PDM VM and generally for other Linux VMs because there is no specific reason to need ZFS and there IS overhead to running ZFS in general.

 

[Johannes S]

It really depends on the usecase. For example for OPNsense ( see https://forum.opnsense.org/index.php?topic=44159.0 by meyergru ) it makes sense to use zfs inside the vm to have snapshots and due to the relative low write amount of opnsense the increased wear load is tolerable.

Otherwise ext4 has the benefit of being more universal ( nearly any Linux box can read it ) and often enough more robust. It's drawback is the reduced performance for some usecases ( many small files, databases iirc, propably pbs too ) compared to xfs and even zfs. For this reason for a database it might be worth it to benchmark xfs versus ext4, especially if they are on a zfs-backed virtual vm disc. Even on bare metal xfs might be better since databases implement some of zfs features ( replication, checksum etc ) on their own.
Following talk is a nice overview on the history and differences of Unix filesystems:
https://programm.froscon.org/2025/events/3238.html

It's based on several blog posts of the speaker:
https://blog.koehntopp.info/2023/05/05/50-years-in-filesystems-1974/
https://blog.koehntopp.info/2023/05/06/50-years-in-filesystems-1984/
https://blog.koehntopp.info/2023/05/12/50-years-in-filesystems-1994/
https://blog.koehntopp.info/2023/05/15/50-years-in-filesystems-vnodes/
https://blog.koehntopp.info/2023/05/17/50-years-in-filesystems-towards-2004-lfs/

He also wrote one specific about btrfs/zfs/Copy-on-write in German:

https://blog.koehntopp.info/2009/02/06/neue-ideen-in-dateisystemen-oder-btrfs-in-fedora-11/

 

[UdoB]

I have various scenarios where ZFS is running on the host and in a VM.

Yeah, that's fine as long it is useful inside of the VM

Then I have a TrueNAS VM, although currently primarily for testing.

For pure testing it it fine - as there are definitely other rules than for a productive system. I am sure you know the recommendation to pass-through a controller for this, before going productive. (Pass-through single disks is not sufficient. Other topic...)

I also have some PBS VMs and of course they were set up with ZFS to allow for a ZFS special device for metadata.

That doesn't make sense. You are using virtual disks from the host to build ZFS inside that VM? Then you add virtual disks from the host to create a Special Device in the VM? Sorry, that feels... wrong. (And PBS it itself is filesystem agnostic, as far as I know. Definitely the .chunk-Store is.)

Maybe I do understand it wrongly. The same here:

I run PBS in VMs so that I can use the remaining storage for other purposes.

You have virtual disks from the host, so large that "other purposes", handled inside the VM makes sense?

I could not see giving up all of the enterprise SSD capacity to just the special device,

A Special Device is best added to the main pool on the host. All pool-consumers will gain benefit from it. (¹)

When is too large (which is often the case) then you configure "special_small_blocks". It may be tricky to guess the right setting and it only confirms the optimal setting after some time of usage as it is only relevant when new data written. (I do not like the idea to partition a disk and use it for different tasks. But that's possibly only me.)

or on the flip side running out of capacity and having to go through hardware gymnastics to increase that. When it's a VM, just hot add some more and you're done.

For a VM this is true for most filesystems. For ext4 there is "resize2fs" and it works both on hardware and inside a VM. ;-)

I've also hot migrated my main PBS VM twice already to different hosts as I came in for a landing on the hardware selection.

Fine! Again this works independent of the filesystem used inside of the VM.

Either I have interpreted you post #6 wrong (which may be!) or you have some area for optimization
In any case: if it works for you and it fulfills your expectations, then it is fine. Being "sub-optimal" is usually not a catastrophe.

----
(¹) this is true for datasets. ZVOLs are a different beast...

 

[Onslow]

Otherwise ext4 has the benefit of being more universal ( nearly any Linux box can read it ) and often enough more robust. It's drawback is the reduced performance for some usecases ( many small files, databases iirc, propably pbs too ) compared to ext4 and even zfs.

Hi, I have read this paragraph a few times and still can't understand which fs it is about ;-(.
Ext4 compared to ext4?
Maybe you meant other fs than ext4 in the first sentence ("ext4 has the benefit") or maybe you meant other fs than ext4 in the second sentence ("compared to ext4")?

 

[J-Rod]

For maximum virtual disk placement flexibility, my first choice would typically be for PVE to handle the storage, especially with ZFS and Ceph being fully supported.

A little home lab history: previously, I was running Hyper-V. I had some very nice storage arrangements via hardware RAID with SSD caching, storage spaces with tiering and compression/dedupe, as well as 30-second shared nothing replication. But there were management limitations due to not having an SCVM license. The inability to hot migrate a VM between replication pairs was also extremely annoying.

For multiple decades of production (general use Hyper-V and vSphere, 50+ servers, 500+ VMs, 10+ PB), I also have had the hypervisor consume the storage first. Presenting it to the VM instead adds many restrictions, such as when the time comes to upgrade/replace/move hardware around.

The reasons TrueNAS gives for wanting to have the storage controller passed through don't really apply when running PVE on ZFS.

 

[Johannes S]

Hi, I have read this paragraph a few times and still can't understand which fs it is about ;-(.
Ext4 compared to ext4?

It was a typo, I corrected. Thanks for the hint I meant that ext4 is more robust than xfs e.g. in case of a power failure while xfs is said to perform better with databases and similiar workloads. Another difference is, that ext4 can shrink filesystems, xfs can't. In enterprise you usually have some emergency power supply and don't need to shrink your fikesystems so in that cases xfs make sense. I would always do a benchmark though.

 

[Johannes S]

The reasons TrueNAS gives for wanting to have the storage controller passed through don't really apply when running PVE on ZFS.

Why? Not trying to be a jerk, I'm genuinely interested

 

[J-Rod]

TrueNAS states that they need to talk to the disks directly to be able to see the SMART data, detect and correct errors, manage scrubs and rebuilds, etc. But of course PVE has the same ZFS available, so can do all of that as well. Also, TrueNAS dropped FreeBSD and now is based on Debian, just like PVE. Granted, PVE lacks a GUI for such storage management tasks, but TrueNAS does still require a bit of knowledge to operate, versus a "true" NAS device from Synology and etc.

PCIe passthrough comes with various challenges and problems, such as the hypervisor trying to do things with the disks when it shouldn't, or other random hardware issues due to oddities with how a certain system might improperly handle passthrough.

With passthrough you lose a lot of the advantages of running a VM, such as snapshots, cloning, hot migrations, etc.

 

[UdoB]

That's interesting. Basically it's a great example of having the same information as everybody and coming to the exact opposite conclusion.

While personally I would not go that road: if it works for you, it is fine!

(( It may depend on the actual use case: for a small TrueNAS with low usage I might possibly think about it. For actual large bulk-storage (which is the usual use case, I think) I can not see a way to convince me... ))

 

[J-Rod]

I don't think my position is unusual. Check various forums and you'll see plenty of people running TrueNAS as a VM, and even without passthrough. Virtualizing FreeBSD historically has been slightly more challenging than Linux, due to the reduced popularity, but typically it's been a solvable problem (m0n0wall back in the day anyone??) However, now that TrueNAS is on Debian, there is really no remaining logical reason that it can't run just fine as a VM, the same as any other Linux VM.

To be blunt, TrueNAS may wish to think they are special, but they aren't, and the fact that they offer their own hypervisor with TrueNAS perhaps means they could muddy the discussion due to an inherent conflict of interest.

From a support perspective, ANY company will find fault with the idea of running their product in a VM, with the typical claim being that they don't know what is going on underneath. Of course this is a valid point in general, but historically it's fallen on deaf ears due to virtualization being mature, critical, popular, etc. If you go around telling customers they have to run your product bare metal to get support, they probably are going to go with a competitor instead. AND, bare metal is actually a massively larger minefield of random hardware that a customer could be trying to make your product work on, compared to the handful of available hypervisors.


It's likely that a lot of these similar discussions will eventually drop into the real reason for the conversation, and that is the pros/cons of storing large amounts of data in a perhaps single, large virtual disk. Of course there are various limits at the hypervisor level depending on local storage, iSCSI or NFS (62TB/64TB for example), then limits of the filesystem volume size and then single file size limits. After that, you have to consider backups, replication, DR, recovery times, maintenance windows, hardware/software migrations. So it's a lengthy conversation now, and there are MANY variables to consider.

I have run VMs with hundreds of TBs of storage per VM. General use file sharing, NVR and backups were the primary use cases, although once I did have a digital document storage VM that got completely out of hand due to the product being a bit rubbish and storing all images as uncompressed TIFFs. For NVR and backups, I could attach another virtual disk and let the software go at it. For file sharing, I would break up the share points by department or some other logical separation if possible, or use directory junctions or symlinks. For the rubbish document storage, I used spanning via windows dynamic disks.

After some hundreds of TBs, the conversation perhaps moves to other options, such as a SAN, scale-out SDS or parallel file systems like BeeGFS or Lustre. Although there is this: https://www.reddit.com/r/Proxmox/comments/1jq1znm/how_to_create_a_virtual_disk_128tb/

 

[Johannes S]

To be blunt, TrueNAS may wish to think they are special, but they aren't, and the fact that they offer their own hypervisor with TrueNAS perhaps means they could muddy the discussion due to an inherent conflict of interest.

The most referenced texts ( https://www.truenas.com/community/r...guide-to-not-completely-losing-your-data.212/ and https://www.truenas.com/community/t...nas-in-production-as-a-virtual-machine.12484/ ) were not written by a staff member of them but a community member (jgreco) though. He also wrote a piece on running TrueNAS in a VM for a small environment or a testing environment (so similiar usecase as one of yours) so I don't think that he is too dogmatical on this:
https://www.truenas.com/community/t...ative-for-those-seeking-virtualization.26095/

Regarding your point that ProxmoxVE itself also can use S.M.A.R.T and friends the same way as TrueNAS I would agree but let's he honest: ProxmoxVEs GUI doesn't have much integration for it on the same level as TrueNAS (e.G. for configuring the amount and schedule of S.M.A.R.T check jobs, scrubs etc). Now this isn't a big deal if you are comfortable in using the CLI (which I am and you propably too) but again let's be honest: Most homelabbers/clickops admins don't use TrueNAS for being an enterprise-grade NAS but for having a GUI for everything together with ZFS. For these people I think the old recommendation (use a passedthrough controller) still holds true. Personally I think, that virtualizing TrueNAS is pointless if in the end you don't use it's features for managing the storage. If you just want to have a Linux based fileserver with a GUI, you could also setup an lxc (for cifs shares) or VM (for nfs and cifs) with WebMin or Cockpit and call it a day.