[SOLVED] Yet another new user with filesystem questions

[garyd9]

I recently discovered proxmox and have decided to migrate my existing "home lab" from hyper-v server to proxmox. Why? Well... why not? (The machine currently running hyper-v is on it's last legs and will be retired after I migrate.)

My new "server" is a mini PC with a i5-1250P with 64GB of RAM (non-ECC), but only has a single nvme slot populated with a 2TB consumer NVME (Samsung 990 pro.) I'm painfully aware that this provides no redundancy (other than backups I'll be making to a NAS that doesn't support ZFS.)

As with so many other people, I'm trying to decide on the best option for a filesystem. I _think_ my choices are ZFS and EXT4. I've read countless threads and posts on this topic, but I really haven't found good advice for my situation: I don't have the option of redundancy and I don't have the option of getting an enterprise grade NVME.

Taking the above into account, my guess is that I'd be better off with EXT4 as it would reduce the wear and tear on the NVME drive. If I choose ext4 over zfs, what would I lose in terms of functionality?

Is there a way to install proxmox on a new system with ext4 on lvm-thin (in order to have the ability to snapshot containers)? If so, what options would I select during the install?

Thanks in advance
Gary

 

[BobhWasatch]

I believe the default installation on a machine with only a single disk will give you ext4 + lvm-thin. Rootfs on ext4, VM's and CT's as block devices on lvm-thin. You do get snapshots with that configuration, including for containers.

What you lose is a lot of write amplification Honestly, ZFS doesn't buy you much on a machine with only one disk. Replication I guess, but you need another machine to make that useful.

 

[garyd9]

I believe the default installation on a machine with only a single disk will give you ext4 + lvm-thin. Rootfs on ext4, VM's and CT's as block devices on lvm-thin. You do get snapshots with that configuration, including for containers.

What you lose is a lot of write amplification Honestly, ZFS doesn't buy you much on a machine with only one disk. Replication I guess, but you need another machine to make that useful.

Thank you. Just installed it using the default ext4 filesystem. It does appear to have created it on lvm-thin with no swap. (I really hope with 64GB RAM on a light home lab I won't need swap space.)

Take care
Gary

 

[BobhWasatch]

For reasons that aren't relevant here I installed mine on top of Debian so there is a small swap partition. If there is in fact no swap (check with "free") I would recommend creating just a small swap partition of 1 GB or so out of the lvm, or even a swap file on the ext4. Reason is it gives the kernel more flexibility in how it manages memory. What tends to happen after a while is some little used blocks end up in swap, which frees up memory for better uses.

In general though 64 GB should be fine for a home lab. I've only got 32 and run 5 VM's and 2 CT's, plus the occasional VM I create to try something out.

Good luck!

 

[ubu]

I would use ZFS:
It gives you Compression, Snapshots, optional Replication and I my opinion is easiest to manage. If you have a single disk now you can mirror it easily later.
Just mount storage from your NAS and use Proxmox backup server on that storage.

If your disk does you install proxmox on a new disk, Install PBS, hook up your NAS storage and restore your VMs

 

[garyd9]

For reasons that aren't relevant here I installed mine on top of Debian so there is a small swap partition. If there is in fact no swap (check with "free") I would recommend creating just a small swap partition of 1 GB or so out of the lvm, or even a swap file on the ext4.

It appears I was mistaken. There's a small 8GB swap set up. I thought the field was blank in the installer, but I guess I was wrong.

 

[garyd9]

I would use ZFS:
It gives you Compression, Snapshots, optional Replication and I my opinion is easiest to manage. If you have a single disk now you can mirror it easily later.
Just mount storage from your NAS and use Proxmox backup server on that storage.

If your disk does you install proxmox on a new disk, Install PBS, hook up your NAS storage and restore your VMs

compression: not needed

snapshots: I think I have them with ext4 on lvm-thin.

replication: I only have a single pve instance/node with no intention to ever add another.

mirroring: As I mentioned above, I only have a single nvme slot. The only other way I'd have to add another drive would be via USB3 (which wouldn't make a very good mirror.)

backing up: I don't understand... I'm already able to back up containers and VM's to my NAS via the backup stuff built into proxmox ve. What would I gain with Proxmox Backup Server? Is using ZFS a prerequisite to PBS?

 

[Dunuin]

What you lose is a lot of write amplification Honestly, ZFS doesn't buy you much on a machine with only one disk. Replication I guess, but you need another machine to make that useful.

Write amplification and the ability to tell you when your data got corrupted so you can restore it from the (month old) backup (as ZFS canT fix corrupted data without a mirror/parity data).

Is using ZFS a prerequisite to PBS?

No

What would I gain with Proxmox Backup Server?

Only a fraction of the space is consumed when storing multiple backups of a guest because of the deduplication. Encryption. Ransomware protection. Data integrity checks. Incremental and therefore way faster backups that consume less bandwidth. Zero-trust concept, so you can do offsite backups to some datacenter and even the admins there won't be able to access your data.

Lets say you got a 1TB VM that changes 10GB per day and you want to keep 10 daily backups. With PBS this would probably consume something like 1,1TB in total while VZDump backups would consume the full 10TB. And with VZDump you always need to backup the full 1 TB. With PBS you only need to backup the 1TB once. All backups after that will be incremental any only need to backup those 10GB that changed. And its way faster to backup 10GB than 1TB

 

[garyd9]

Lets say you got a 1TB VM that changes 10GB per day and you want to keep 10 daily backups. With PBS this would probably consume something like 1,1TB in total while VZDump backups would consume the full 10TB. And with VZDump you always need to backup the full 1 TB. With PBS you only need to backup the 1TB once. All backups after that will be incremental any only need to backup those 10GB that changed. And its way faster to backup 10GB than 1TB

Sounds interesting. I wonder if I can set up PBS as a VM on the host that it would be backing up VM's from. Currently fighting some other interesting issues with proxmox. From within a windows VM, I can SEND TCP data over the network interface at full line speed (1 Gbps), but can only receive at 20-25% speed. It impacts the host OS as well (and probably also also containers.) Searches in this forum reveal that many others have similar issues, unresolved, related to network speeds. It appears the issue showed up with a kernel update somewhere in the proxmox v7 timeframe. Why is that relevant? Well, I wouldn't want a backup server running in a VM that can only receive data at a reduced rate.

 

[ubu]

You can set up PBS in a VM, LXC or even install it on your PVE host. It does incremental deduplicated virtual fullbackup, so you need much less space. You are correct, your setup provides the same features minus compression and checksums for your data.

 

[Dunuin]

But keep in mind: No running PBS -> no restore possible

So not that useful to run PBS + PVE on the same server if thats your only PVE node.

 

[garyd9]

But keep in mind: No running PBS -> no restore possible

So not that useful to run PBS + PVE on the same server if thats your only PVE node.

If I used a PBS running as a VM or container on the PVE host, backing up to a NAS, and the host drive dies, would it be possible/feasible to newly install PBS on a different machine and use that to restore on a new drive from the NAS backups?

 

[Dunuin]

If I used a PBS running as a VM or container on the PVE host, backing up to a NAS, and the host drive dies, would it be possible/feasible to newly install PBS on a different machine and use that to restore on a new drive from the NAS backups?

If you don't care about downtime, yes.

 

[garyd9]

So.. just an update on this thread (because I love kicking myself.)

I was an idiot and didn't realize the mini PC I'm using has 2 nvme slots running at PCIe4x4... so I ran out and bought another 2TB NVME, backed up the 2 VM's and 1 container I already set up, installed the second NVME, re-installed proxmox using ZFS RAID1 (mirroring) and restored the backups from USB. I'm 'okay' with the extra wear on the NVME's if I actually get some advantage from it... (Any tips on tuning ZFS?)

While doing that, I also spent quite a bit of time troubleshooting my networking issue. It turns out the problem is the linux bridge. This was confirmed by copying files from Windows to the proxmox host (using scp) and watching the network usage. When proxmox was using the bridge device for connectivity, I'd copy at a max of around 215mbps. When proxmox was using the native ethernet device (enp130s0 or something), I'd copy at 913mbps.

That's nearly a 5x difference in speed.

Switching from the linux bridge to the OVS bridge also fixes the problem.

Even with OVS, the problem appears to be gone, but after several tests, it comes back. Thats with or without bonding.

 

[Dunuin]

(Any tips on tuning ZFS?)

There isn't much to tune. Basically all stuff that people recommend to reduce wear will increase the risk of losing data. Same for performance.
Best thing to get good performance and long disk life is to buy proper Enterprise SSDs with PLP.