Configuration Advice in Proxmox VE

[aidenpryde]

Hello,

I'm new to Proxmox, and it's kinda overwhelming for me as I just have an Unraid server which as I know is a prosumer solution and doesn't have things like ZFS or snapshotting atm.

The purpose of this Proxmox node is as a backup server for Unraid's data via a Truenas VM, and as a test platform for other VMs that I can destroy later and not care about, but the backup guest would stick around through that.

I've got Truenas set up in Proxmox and everything is running fine, but I got to looking around, and I don't think I've set up my configuration in the correct way and all these new filesystems and what should be used for what has got me scratching my head.

What I did initially:

When I installed Proxmox I installed it on a couple of 120GB SSDs as a ZFS mirror for some redundancy. I only allocated 60GB to it as I had read that I could overprovision these consumer grade drives to get more life out of them by not using the whole disk.

Once in Proxmox I set up another ZFS mirror with a couple more consumer 500GB SSDs, and that's where the Truenas VM is located. I then assigned an HBA over to the VM so that it could see the SMART data on the HDDs and send reporting if there's any issue to my email address (as at the time I hadn't figured out how to get Proxmox to do it, and there's apparently a command line/conf file editing way to do that).

My Questions:

1. There are two other file systems besides ZFS that Proxmox uses (LVM and LVM Thin), and I keep reading that people use them to store VMs. My question is what filesystem should I use to store my VMs? I don't have another node to do replication, and so I thought disk level redundancy via ZFS was a good idea and from what I can tell LVM and LVM Thin don't work that way.

2. I'm very confused by caching with Proxmox. It's stupid simple in Unraid, but I'm trying to understand things like SLOG and L2Arc and it's confusing to me. I thus far haven't seen a tutorial to even use those features in Proxmox and I'm having trouble finding anything in the GUI about them. Where, how, and should I use these features? I keep reading stuff about offloading stuff to HDDs because it will wear out SSDs.

3. Should VMs be located on HDDs instead of SSDs? This is somewhat related to question #2 as I was concerned about I/O speed on the guests, but I read stuff about wearing out SSDs really quickly.

What hardware I have available:

2x 120GB SATA SSDs
2x 500GB SATA SSDs
2x 500GB NVMe SSDs
10x 4TB HDDs
I even have 2x 16GB SATADOMs, but read I shouldn't use those for Proxmox.

 

[guletz]

(LVM and LVM Thin

Hi,

There are two other file systems besides ZFS that Proxmox uses (LVM and LVM Thin)

This are not filesystems(LVM). Think as a containers where you can create many "partitions" and then format with a FileSystem like ext4, xfs and so on.

Where, how, and should I use these features?

Using command line!

Should VMs be located on HDDs instead of SSDs? This is somewhat related to question #2 as I was concerned about I/O speed on the guests, but I read stuff about wearing out SSDs really quickly.

This depends of your load! Also could be mixed(a zfs pool with HDD only, another zfs pool with SSD only, or even mixed - aka a single pool with HDDs+SSDs= see special devices). Anyway, consumer SSDs are not good to be used with zfs(wear out is faster), but any professional/Data-Center SSDs are "friendly" with zfs.

As a general advice, use SSDs(pool) only for the VM that really need this(like DBs) and only for the folder where you need it, and not for entire OS file-system.

Good luck / Bafta !

 

[Dunuin]

If a consumer grade SSD would'n live long because of the write amplification that result in a lot of small writes a HDD would possibly also get into troubles because it can't handle the IOPS. In that case you should consider a enterprise grade SSD.

If you want mirroring and don't want ZFS you could install proxmox ontop of a debian. That way you can use mdraid software raid with ext4 or xfs. But keep in mind that proxmox doesn't support mdraid and in the last days with the proxmox 6.3 update I saw some people with mdraid setups that lost data. Because it isn't supported no one checks if a proxmox kernel update will crash it before releasing it. But my mdraid array is still working fine and might be the only option if you want encryption on your mirrored boot drives.

 

[aidenpryde]

So, I did a little more reading on LVM, and it seems as though a lot of people write about it in terms of having a hardware RAID. I'm certainly not interested in that in a homelab environment, and the increased throughput I'd get seems to not be worth it.

Unlike Unraid, I'm seeing very little in the way of community tutorials on how to set up these features. Information is all over the place and conflicted.

Does anyone think I should use those SATA 500GBs to store the VM VHDs?

I know datacenter SSDs are recommended, but I don't have any right now, and I feel I can afford to lose those drives as the Truenas VDEVs will survive that, and since they'd be mirrored, and I have a UPS connected to this I'm halfway protected from power failures.

 

[guletz]

Does anyone think I should use those SATA 500GBs to store the VM VHDs?

Yes. It is the only option that you have using your hardware!