Configuration check - EXT4 for Proxmox / VMs / LXCs on single NVME SSD, ZFS on 2xHDD for file storage

tauhammerhead

New Member
Oct 15, 2025
3
2
3
Hi,

I'm adding some HDDs to my home server and wanted to double-check my proposal makes sense before I hit go. Please can you advise if this seems sensible or if I am barking up the wrong tree?

Hardware:
  • HP EliteDesk 800 G3,
  • i7-7700 CPU,
  • 32GB non-ECC RAM,
  • 1x512GB NVME SSD
  • 2x4GB Seagate Exos 7E8 HDD
My Proxmox / VMs and LXCs all run on ext4 on my single 512gb NVME SSD. I've just added 2xHDDs as file/media/photo storage to replace an old separate WD MyCloud Drive. My thoughts were to set up the HDDs as a ZFS pool in Proxmox in RAID1 configuration. I'd have broadly four datasets:
  1. tank/PVE-Backups - for daily/weekly/monthly backups of the VMs and LXCs.
  2. tank/PVE-templates - downloaded copies of distros and templates
  3. tank/Files - My main file/media storage which will be shared with a turnkey File Server (if I can figure out the permissions) and a couple of VMs/LXCs
  4. tank/Other-Backups - Backup location (also shared by the samba file server in 3) for other network devices to back-up to.
I will back-up 1,3 and 4 to a separate local external HDD (probably use rsync on the redundant WD MyCloud drive) and a remote linux computer using a VPN (ideally encrypted) - I've heard of Proxmox Backup Server suggested for this latter bit.
  1. Given I am using ZFS only for file storage, is it worthwhile the extra learning curve? From what I have read, the checksumming and self-repairing is very attractive, is snapshotting of file storage useful? I don't have ECC RAM but, from what I have read, there seem to be some ways to mitigate the risk of not having this.
  2. Would mounting the HDDs in EXT4 in Proxmox and then passing through to something like TrueNAS be simpler?
  3. What is the best way to back-up the file storage if I move to ZFS? I am not so concerned about downtime but having a back-up system that is simple and reliable is important. The number of posts I have read whilst trying to research this of people being locked out of the zfs pool or not being able to re-access their data fills me with absolute dread!!!
Thanks in advance.
 
1) There is not much of a learning curve
2) No
3) ZFS send and receive on another host with ZFS. Otherwise you can zfs send backups and snapshots to tar or other targets as well (so you have continuous backups).
 
  • Like
Reactions: Johannes S
Given I am using ZFS only for file storage, is it worthwhile the extra learning curve? From what I have read, the checksumming and self-repairing is very attractive, is snapshotting of file storage useful? I don't have ECC RAM but, from what I have read, there seem to be some ways to mitigate the risk of not having this.

Yes and you don't need ECC Ram see this article: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

It quotes a forum post by ZFS developer Matthew Ahrens:
There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. Actually, ZFS can mitigate this risk to some degree if you enable the unsupported ZFS_DEBUG_MODIFY flag (zfs_flags=0x10). This will checksum the data while at rest in memory, and verify it before writing to disk, thus reducing the window of vulnerability from a memory error.

I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS. https://arstechnica.com/civis/threa...esystem-on-linux.1235679/page-4#post-26303271

Regarding whether ZFS is worth it:

I would even use ZFS for the host since it allows more flexibility which space can be used for VM/lxcs and which for other data ( e.g. iso images). My Elitedesk has three SATA slots did you consider to install a SATA SSD? Then you could build a large pool out of the hdds with the ssds as special device which would gives you the most flexibility at the cost of not seperating data and os.
Would mounting the HDDs in EXT4 in Proxmox and then passing through to something like TrueNAS be simpler?

No! For TrueNAS ( or any NAS os ) it's recommended to add all discs via a dedicated storage adapter to avoid data loss: https://www.truenas.com/community/r...guide-to-not-completely-losing-your-data.212/

BTW: If you mainly want to use this host as NAS you could also install TrueNAS bare metal and run everything from it.
What is the best way to back-up the file storage if I move to ZFS? I am not so concerned about downtime but having a back-up system that is simple and reliable is important. The number of posts I have read whilst trying to research this of people being locked out of the zfs pool or not being able to re-access their data fills me with absolute dread!!!

It depends on your Budget and Hardware options. Do you want to use cloud storage as offsite backup or external discs or a combination of both? Do you happen to have another Server ( e.G. ProxmoxBackupServer or an old NAS) with or without ZFS support? Do you want everything ( vms/lxcs operating system and data) in one place or are you fine with seperating vm/lxc and data backups to save some money?
 
  • Like
Reactions: UdoB
Thanks @guruevi and @Johannes S - It is helpful to have someone verify things I have read or thought.

I dabbled with bare metal Proxmox on a 2019 Macbook AIr which was surprisingly capable once I had gotten around the broken screen and T2 security chip. SInce then I've been running a whole host of different LXCs and VMs. However, it was a bit janky at times so this is my foray into something more capable and stable and I am overwhelmed by the options now.

I run a whole host of different internal services (incl. Home Assistant, Immich, Mealie, Jellyfin and an iTunes Backup). I think I only need a Turnkey FIle Server rather than a NAS OS so network devices can access certain datasets - it was just that permissions set-up was a bit cumbersome and I don't think I have it quite right hence my question on whether it might be quicker and simpler to pass it all through to one VM (albeit less efficient on resources).

The crux of what I am concerned about is this: our house burns down, thief nicks the server, home server catastrophically fails. What is the simplest back-up method so that I can buy new equipment, reinstall proxmox and copy back the data from my backups. SInce this morning, I've tried Proxmox Backup Server and that looks great....for LXCs and VMs but I can't see a way of snapshotting or backing up the actual data on my HDDs

Maybe I am not searching with the right terms (very possible) but it looks like PBS can't do this bit?

Backing my media/files + the latest VM/LXC backup to an ext4 system seems the simplest solution whereby I can access the data from a variety of other platforms/OS's in the case that I can't get the system up and running again or there is a delay in replacing equipment.
 
  • Like
Reactions: UdoB
If your Macbook is still working it would be a good option for a ProxmoxBackupServer. If the data is on VM/lxcs discs a backup of the vm/lxc will backup it too. And PBS allows to sync backups between different servers, so If you happen to have an offsite ProxmoxBackupServer (as a vserver or a Proxmox Backup Server cloud provider) this would be a fetchall solution. On the other hand vservers storage tends to be limited or getting expensive, Proxmox Backup Servers as cloud (like the ones provided by tuxis.nl or Inett) cost more than regular Cloud 3 storage (e.G. backblaze S3 or Hetzners storage box) so it might be a good idea to seperate these things. It would also mean that you would need a working PBS to restore everything although you might only be interested in the raw data. Alternatively you could also ask a friend or family member whether you might put a remote PBS in their LAN (that approach obviouvsly would need fast Internet on both sites)

I'm doing a mixed setup I described here (PBS on a vserver for the vms and lxcs, data backup of my notebook and NAS files with restic to a hetzner storagebox): https://forum.proxmox.com/threads/b...er-with-incremental-chunks.157965/post-723568

one thing changed though: I upgraded the hardware for my main ProxmoxVE server, it's precedessor now serves as an additional ProxmoxBackupServer and TrueNAS replication target in my home lan.


Another option for an affordable offsite backup might be to buy some external UBS SSDs or HDDs (depending on your budget, I would get a SSD if possible), add them as removable datastore in PBS (see: https://pbs.proxmox.com/docs/storage.html#removable-datastores ). You would then have just a few snapshots in your PBS (like let's say from this week) but would regulary attach the removable datastore to sync your backups to it. This external disc could also serve as storage for another backup tool like restic or rsnapshot (or any other backup tool you are comfortable with).
Now the basic idea is that you would always have at least two discs, one of them you will store outside of your place (e.g. in your office, at a friends or familly members place) while the other one is used for the current backups in your setup. From time to time (ideally weekly but could be less depending on your paranoia level and possible exchange times) you would swap them. This could be even extending by buying another two discs to also have monthly swap. This is a lot more affordable than anything needing another computer or cloud hosting (for storage or a vserver), but needs manual intervention (I always forget to do my sync to my USB disc, luckily my main backup is on my vserver and storagebox ;)).
 
have at least two discs, one of them you will store outside of your place (e.g. in your office, at a friends or familly members place) while the other one is used for the current backups in your setup. From time to time (ideally weekly but could be less depending on your paranoia level and possible exchange times) you would swap them.
@tauhammerhead : Ideally have these backups encrypted on the external disks and never have them both in the same place / car / aeroplane / home / office in the same moment :).
 
Right, I have done some playing around and think I now have a solution. I have bought another EliteDesk and HDDs because I could get them at a very good price. Here is the proposal:

Main Proxmox Server (1)2. Proxmox Backup Server (2)PBS VM hosted on Main Proxmox Server (3)
Hardware
  • HP Elitedesk 800 G3
  • 32GB non-ECC RAM
  • 1 x 512GB NVME SSD
  • 2 x 4GB Exos 7e8 HDD
  • HP Elitedesk 800 G3
  • 8GB non-ECC RAM
  • 1 x 512GB NVME SSD
  • 2 x 4GB Exos 7e8 HDD
  • VM on Main server
  • 1 x 4GB Exos 7e8 HDD
  • 3.5" HDD powered USB caddy
  • Recommended hardware provisions as per Proxmox
Firmware/Software
  • Root on LVM on NVME SSD
  • VMs (including 3) and Containers on LVM-thin on NVME SSD
  • Longer-term file/media storage on HDDs set up in ZFS RAID1 configuration
  • PBS bare metal on standard file system (Ext4 or NFS) on NVME SSD
  • 'Hot' datastore on the NVME.
  • 'Warm' datastore on the HDDs set up in ZFS RAID1 configuration
  • PBS in VM with USB passthrough of the HDD caddy.
  • SIngle 'Cold' datastore set up as a removeable datastore based on the USB HDD caddy.
Acts asFirst back-upSecond back-upThird back-up moved to remote location

Here is how it might work:
  1. 1 is chugging away quite happily. Taking a daily backup of VMs and Containers overnight and saving to the HDDs.
  2. Every hour of the day, 2 wakes up and takes a backup of the VMs and Containers on 1 then goes back to sleep.
  3. Overnight, 2 wakes up and takes a backup of the ZFS datasets on 1's HDDs then goes back to sleep.
  4. Sometime each day, 2 carries out prune and garbage collection jobs.
  5. Sometime every fortnight, 1 and 2 carry out scrubbing on their ZFS pools.
  6. Every 1-6 months, collect my remote HDD, insert it into the USB caddy and plug into 1 then boot up 3. Run a 'sync' job where 2 is the 'remote' and I am replicating the 'Warm' datastore only. Once finished, power down 3, disconnect the HDD and take to secure location.
Does this sound sensible? Should I be setting up the 'cold' HDD as a single ZFS HDD or something different?

Things for me still to figure out (putting here more for my own sanity):
  • Encryption - Good shout @Onslow . Looks like I can do this in PBS and also on my cold storage so will investigate further.
  • Dataset backup - I have got proxmox-backup-client working on my datasets but restoring from this looks a bit more tricky. Investigate ZFS send and receive commands with backups/snapshots.
  • ZFS Dataset Permissions in LXCs and VMs -Trying to share different datasets with different LXCs/VMs with varying levels of permission
  • Restoration - Test restoring proxmox config, ZFS datasets and VMs from both 2 and 3.
  • Automation - Investigate longer term monitoring, alarms, alerts, and power saving measures.
 
  • Like
Reactions: Johannes S
Does this sound sensible? Should I be setting up the 'cold' HDD as a single ZFS HDD or something different?

This sounds sensible, I would try to aim to replciate to the cold store at least once a month but right now I don't do it myself so who am I to judge ;)
One thing to consider: For the cold storage it might be faster to use ext4 or XFS instead of ZFS (afik they are also the only options offered by PBS in the GUI for creating removable datastores), on the other hand ZFS has a parameter copies which would allow you to have redundancy even on your single-disc (although in a limited way see https://forums.truenas.com/t/the-zfs-copies-features-is-useless-change-my-mind/5075/ Ignore the click-baity title, in the thread several people explains why for a single-disc copies might still have some use).
 
Without some longer thinking I can't see any error in your conception, so it's probably OK :).

To that schema I would add backuping of the important files of the hypervisor and bare metal PBS.
I have read quite many posts that it isn't worth doing... That it's easier to reinstall a clean hypervisor and restore only the VMs... My opinion is different. I know from my own experience how much thinking and considering various options I did. And noted them in text files in a separate place, to not have to reinvent them later. Even such easy stuff like the shell prompt and bash aliases which substantially help to avoid serious errors. And a list additional packages, and customized configs of them...

I would backup /etc and /root
Even not sophisticated solution will do. Like a tar file.

Edit: or proxmox-backup-client :)

A good approach when creating a backup solution is giving only minimal permissions for the user which automatically creates backups. So it can create (and restore) backups, but not delete nor prune them. The role not higher than DatastoreBackup (plus optionally DatastoreAudit). Retension settings of backup jobs should NOT be configured in PVE, only in PBS. With ransomware protection in mind.

Encryption - Good shout @Onslow . Looks like I can do this in PBS and also on my cold storage so will investigate further.

Remember to preserve any encryption keys in a safe place (NOT with the backups). Without them, the backups will be useless!
E.g. an USB stick in the safe box and additionally a paper copy (see https://pve.proxmox.com/pve-docs/pve-admin-guide.html#storage_pbs_encryption ).

Good luck!
 
Last edited:
  • Like
Reactions: Johannes S