Configuration check - EXT4 for Proxmox / VMs / LXCs on single NVME SSD, ZFS on 2xHDD for file storage

[tauhammerhead]

Hi,

I'm adding some HDDs to my home server and wanted to double-check my proposal makes sense before I hit go. Please can you advise if this seems sensible or if I am barking up the wrong tree?

Hardware:

• HP EliteDesk 800 G3,
• i7-7700 CPU,
• 32GB non-ECC RAM,
• 1x512GB NVME SSD
• 2x4GB Seagate Exos 7E8 HDD

My Proxmox / VMs and LXCs all run on ext4 on my single 512gb NVME SSD. I've just added 2xHDDs as file/media/photo storage to replace an old separate WD MyCloud Drive. My thoughts were to set up the HDDs as a ZFS pool in Proxmox in RAID1 configuration. I'd have broadly four datasets:

1. tank/PVE-Backups - for daily/weekly/monthly backups of the VMs and LXCs.
2. tank/PVE-templates - downloaded copies of distros and templates
3. tank/Files - My main file/media storage which will be shared with a turnkey File Server (if I can figure out the permissions) and a couple of VMs/LXCs
4. tank/Other-Backups - Backup location (also shared by the samba file server in 3) for other network devices to back-up to.

I will back-up 1,3 and 4 to a separate local external HDD (probably use rsync on the redundant WD MyCloud drive) and a remote linux computer using a VPN (ideally encrypted) - I've heard of Proxmox Backup Server suggested for this latter bit.

1. Given I am using ZFS only for file storage, is it worthwhile the extra learning curve? From what I have read, the checksumming and self-repairing is very attractive, is snapshotting of file storage useful? I don't have ECC RAM but, from what I have read, there seem to be some ways to mitigate the risk of not having this.
2. Would mounting the HDDs in EXT4 in Proxmox and then passing through to something like TrueNAS be simpler?
3. What is the best way to back-up the file storage if I move to ZFS? I am not so concerned about downtime but having a back-up system that is simple and reliable is important. The number of posts I have read whilst trying to research this of people being locked out of the zfs pool or not being able to re-access their data fills me with absolute dread!!!

Thanks in advance.

 

[guruevi]

1) There is not much of a learning curve
2) No
3) ZFS send and receive on another host with ZFS. Otherwise you can zfs send backups and snapshots to tar or other targets as well (so you have continuous backups).

 

[Johannes S]

Given I am using ZFS only for file storage, is it worthwhile the extra learning curve? From what I have read, the checksumming and self-repairing is very attractive, is snapshotting of file storage useful? I don't have ECC RAM but, from what I have read, there seem to be some ways to mitigate the risk of not having this.

Yes and you don't need ECC Ram see this article: https://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/

It quotes a forum post by ZFS developer Matthew Ahrens:

There's nothing special about ZFS that requires/encourages the use of ECC RAM more so than any other filesystem. If you use UFS, EXT, NTFS, btrfs, etc without ECC RAM, you are just as much at risk as if you used ZFS without ECC RAM. Actually, ZFS can mitigate this risk to some degree if you enable the unsupported ZFS_DEBUG_MODIFY flag (zfs_flags=0x10). This will checksum the data while at rest in memory, and verify it before writing to disk, thus reducing the window of vulnerability from a memory error.

I would simply say: if you love your data, use ECC RAM. Additionally, use a filesystem that checksums your data, such as ZFS. https://arstechnica.com/civis/threa...esystem-on-linux.1235679/page-4#post-26303271

Regarding whether ZFS is worth it:

[TUTORIAL] Thread 'FabU: This is just a small setup with limited resources and only a few disks, should I use ZFS at all?'

Jan 3, 2025

Should I use ZFS at all?​

For once this requires a disclaimer first: I am using ZFS nearly everywhere, where it is easily possible, though exceptions do exists. In any case I am definitely biased pro ZFS.

That said..., the correct answer is obviously: “Yes, of course” ;-)

Integrity​

ZFS assures integrity. It will deliver exactly the very same data when you read it as was written at some point of time in the past.

“But all filesystems do this, right?” Well, there is more to it. ZFS works hard to actively assure the correctness. An...

• UdoB
• consumer class enterprise class integrity mirror raidz storage zfs
• Replies: 1
• Forum: Proxmox VE: Installation and configuration

• 

I would even use ZFS for the host since it allows more flexibility which space can be used for VM/lxcs and which for other data ( e.g. iso images). My Elitedesk has three SATA slots did you consider to install a SATA SSD? Then you could build a large pool out of the hdds with the ssds as special device which would gives you the most flexibility at the cost of not seperating data and os.

Would mounting the HDDs in EXT4 in Proxmox and then passing through to something like TrueNAS be simpler?

No! For TrueNAS ( or any NAS os ) it's recommended to add all discs via a dedicated storage adapter to avoid data loss: https://www.truenas.com/community/r...guide-to-not-completely-losing-your-data.212/

BTW: If you mainly want to use this host as NAS you could also install TrueNAS bare metal and run everything from it.

What is the best way to back-up the file storage if I move to ZFS? I am not so concerned about downtime but having a back-up system that is simple and reliable is important. The number of posts I have read whilst trying to research this of people being locked out of the zfs pool or not being able to re-access their data fills me with absolute dread!!!

It depends on your Budget and Hardware options. Do you want to use cloud storage as offsite backup or external discs or a combination of both? Do you happen to have another Server ( e.G. ProxmoxBackupServer or an old NAS) with or without ZFS support? Do you want everything ( vms/lxcs operating system and data) in one place or are you fine with seperating vm/lxc and data backups to save some money?

 

[tauhammerhead]

Thanks @guruevi and @Johannes S - It is helpful to have someone verify things I have read or thought.

I dabbled with bare metal Proxmox on a 2019 Macbook AIr which was surprisingly capable once I had gotten around the broken screen and T2 security chip. SInce then I've been running a whole host of different LXCs and VMs. However, it was a bit janky at times so this is my foray into something more capable and stable and I am overwhelmed by the options now.

I run a whole host of different internal services (incl. Home Assistant, Immich, Mealie, Jellyfin and an iTunes Backup). I think I only need a Turnkey FIle Server rather than a NAS OS so network devices can access certain datasets - it was just that permissions set-up was a bit cumbersome and I don't think I have it quite right hence my question on whether it might be quicker and simpler to pass it all through to one VM (albeit less efficient on resources).

The crux of what I am concerned about is this: our house burns down, thief nicks the server, home server catastrophically fails. What is the simplest back-up method so that I can buy new equipment, reinstall proxmox and copy back the data from my backups. SInce this morning, I've tried Proxmox Backup Server and that looks great....for LXCs and VMs but I can't see a way of snapshotting or backing up the actual data on my HDDs

Maybe I am not searching with the right terms (very possible) but it looks like PBS can't do this bit?

Backing my media/files + the latest VM/LXC backup to an ext4 system seems the simplest solution whereby I can access the data from a variety of other platforms/OS's in the case that I can't get the system up and running again or there is a delay in replacing equipment.

 

[Johannes S]

If your Macbook is still working it would be a good option for a ProxmoxBackupServer. If the data is on VM/lxcs discs a backup of the vm/lxc will backup it too. And PBS allows to sync backups between different servers, so If you happen to have an offsite ProxmoxBackupServer (as a vserver or a Proxmox Backup Server cloud provider) this would be a fetchall solution. On the other hand vservers storage tends to be limited or getting expensive, Proxmox Backup Servers as cloud (like the ones provided by tuxis.nl or Inett) cost more than regular Cloud 3 storage (e.G. backblaze S3 or Hetzners storage box) so it might be a good idea to seperate these things. It would also mean that you would need a working PBS to restore everything although you might only be interested in the raw data. Alternatively you could also ask a friend or family member whether you might put a remote PBS in their LAN (that approach obviouvsly would need fast Internet on both sites)

I'm doing a mixed setup I described here (PBS on a vserver for the vms and lxcs, data backup of my notebook and NAS files with restic to a hetzner storagebox): https://forum.proxmox.com/threads/b...er-with-incremental-chunks.157965/post-723568

one thing changed though: I upgraded the hardware for my main ProxmoxVE server, it's precedessor now serves as an additional ProxmoxBackupServer and TrueNAS replication target in my home lan.


Another option for an affordable offsite backup might be to buy some external UBS SSDs or HDDs (depending on your budget, I would get a SSD if possible), add them as removable datastore in PBS (see: https://pbs.proxmox.com/docs/storage.html#removable-datastores ). You would then have just a few snapshots in your PBS (like let's say from this week) but would regulary attach the removable datastore to sync your backups to it. This external disc could also serve as storage for another backup tool like restic or rsnapshot (or any other backup tool you are comfortable with).
Now the basic idea is that you would always have at least two discs, one of them you will store outside of your place (e.g. in your office, at a friends or familly members place) while the other one is used for the current backups in your setup. From time to time (ideally weekly but could be less depending on your paranoia level and possible exchange times) you would swap them. This could be even extending by buying another two discs to also have monthly swap. This is a lot more affordable than anything needing another computer or cloud hosting (for storage or a vserver), but needs manual intervention (I always forget to do my sync to my USB disc, luckily my main backup is on my vserver and storagebox ).

 

[Onslow]

have at least two discs, one of them you will store outside of your place (e.g. in your office, at a friends or familly members place) while the other one is used for the current backups in your setup. From time to time (ideally weekly but could be less depending on your paranoia level and possible exchange times) you would swap them.

@tauhammerhead : Ideally have these backups encrypted on the external disks and never have them both in the same place / car / aeroplane / home / office in the same moment .