New Proxmox Hardware Configuration for ZFS (DL160 Gen10)

Syloc

New Member
Jan 2, 2021
4
0
1
Hi everyone!

It's been for several years that I've been using Proxmox as a simple VM Host, and I'm now going to use it with ZFS to enable Trim function and others functionnalities.

I'm using a DL160 gen10 with : - 4x2To SATA SSD in z-raid10 running 2 VM
- 2x1To SATA SSD in HW RAID1 running PROXMOX OS
- 2x4To SATA HDD in HW RAID1 dedicated to Snapshots and backups
- 1 NVMe 512Go SSD for ZFS caching
- 64Go RAM, at most running at 50% of capabilities in heavy use

I mentionned that I wanted to use TRIM functions, so some might be suprised to see that I use a HW RAID 1 for Proxmox. Simply because excepted the logs files, the caching must be used on NVMe SSD and nothing else runs on PROXMOX Drive.

I'm free to edit the configuration by now, so I'm simply showing you my configuration because I'm quite a beginner with ZFS optimization, so I'd be really grateful that you share me any advice/matters concerning my setup. I'm hear to learn, so do not hesitate to tell me what's wrong and what you'd recommend me to do.

Thank you. :)
 
What type/ grade of SSDs / NVMe have you got?
The 2*4TB HDDs are a no brainer for me: separate pool in Mirror Configuration ;)
 
What type/ grade of SSDs / NVMe have you got?
The 2*4TB HDDs are a no brainer for me: separate pool in Mirror Configuration ;)
- The 2To SSD are TLC DC500R from Kingston. I don't expect a tons of writing as it'll run an program through RDS gateway with a low amounts of writing operations.
The 1 To SSD are last MLC 860 Pro from Samsung with 1Go cache. So no worries to run the Proxmox drive for a while.
The 512 NVMe SSD is an old PM from Samsung that I got from my Lenovo laptop.

- Do you mean that I should also include the HDD in the ZFS pool instead of HW?
I didn't mention that the 2*1To SSD (Proxmox) and 2*4To HDD are linked directly to the Raid card. If i don't get them in a Raid despite being linked to the HW drive, won't I face any issue with ZFS?

I don't know btw if proxmox drive can't healthily handle itself and RAID1 with the same 1To SATA SSD or if it's better to only set an HW RAID in this case. I won't get any TRIM support and integrity checksum if I use HW raid.

Thank you very much for your return! :D
 
Last edited:
won't I face any issue with ZFS?
You will. However you have not mentioned that they are in RAID.
Nevertheless I would try to move to ZFS over all, just because it provides benefits like checksumming.
You can try to set your RAID-card into "JBOD-mode" and then you should be able to use it safely with ZFS.

About all the SSDs it really depends. I am a big fan of ZFS and hence would try to put everything under the umbrella of it.
I was asking about the NVMe because that could be used as a SLOG (separate Log device) for the other pools, hence at least relieve the other SSDs from some writes and speed up the HDDs.
On the other hand I personally think that this is likely not datacenter-grade at all if it is "borrowed" from a Laptop.
So I think I'd either replace that with 2x smaller M2-based cards (I am guessing) and boot the system from that or just make an ISO/temporary/whatever storage which is not redundant.
The other three "classes" could easily go into 3 discrete pools in a mirror configuration.
 
Nevertheless I would try to move to ZFS over all, just because it provides benefits like checksumming.
You can try to set your RAID-card into "JBOD-mode" and then you should be able to use it safely with ZFS.

Meanwhile, I did it, and I got no special problem to mount them for ZFS as I mount the disks with AHCI mode instead of RAID mode in the BIOS.

I was asking about the NVMe because that could be used as a SLOG (separate Log device) for the other pools, hence at least relieve the other SSDs from some writes and speed up the HDDs.

That's what I've been thinking when I told "ZFS caching", I meant SLOG. Even if it's a "laptop" SSD, it's still NVMe in MLC and IOPS are not bad, so I hope it'll should be enough. Sadly, I didn't thought about zfs at the beginning and can't no more buy an OPTANE for this server now..
Compared to SATA, an NVMe card could be enough otherwise shoudl I use ZeusRAM or Optane for SLOG absolutelly?

So I think I'd either replace that with 2x smaller M2-based cards (I am guessing) and boot the system from that or just make an ISO/temporary/whatever storage which is not redundant.

I'm not sure to understand. Do you mean to dedicate PROXMOX INSTALL DRIVE to 2 NVMe drive in mirror zpool to fully support the SLOG elsewhere to grab it onto a random Drive meant to suffer and die about writing?

For know, I have been using 1To in mirror zpool (Proxmox Install) / 4To in raid 10 pool (VM Storage) / 4To HDD in raid mirror for backup and it's working pretty well!

I couldn't use the M.2 as I need the card to plug it in the motherboard's PCIe slot that I'll take tomorrow.

Thank very much for you answer!
 
still NVMe in MLC and IOPS are not bad, so I hope it'll should be enough.
I don't question that. In fact 512gb is waaaay too much. Your SLOg will not need to.be more than a few GB (4-8)
So technically you can partition the device and use it as slog or readcache for.multiple pools.

otherwise shoudl I use ZeusRAM or Optane for SLOG absolutelly?
That's a bit too much IMHO. We are talking about.only a few disks. No need for that IMHO.
m not sure to understand. Do you mean to dedicate PROXMOX INSTALL DRIVE to 2 NVMe drive in mirror zpool
PVE only needs little space. I am running it from an 8GB data flashdisk. That's a little low but for sure you don't need 512gb. That was my idea behind of it.
4To HDD in raid mirror for backup and it's working pretty well!
I'd bring this into ZFS as well, simply to get checksums etc.
But if it works fine for you then all is OK. In the end I only can express opinions and I don't feel offended if you choose not to follow mine. We are grown ups are we ;)?
 
I don't question that. In fact 512gb is waaaay too much. Your SLOg will not need to.be more than a few GB (4-8)
So technically you can partition the device and use it as slog or readcache for.multiple pools.

Copy! I did replace the slogs with 2 32Go nvme sticks to get them in RAID1 in case of faillures.

That's a bit too much IMHO. We are talking about.only a few disks. No need for that IMHO.

I did a lot of research with no conclusive answers as I always wish to reach the lowest latency for the RDS Server, so I added an 300Go Optane 900p to a L2ARC. I don't know if it would be effective in this case or fi it most be directly configure with pci-passtrough into the concerned vm.

PVE only needs little space. I am running it from an 8GB data flashdisk. That's a little low but for sure you don't need 512gb. That was my idea behind of it.

That's what I've been thinking firstly, but I didn't see any recommendation to do this from Proxmox (contrary to ESXi datasheets) and others tells they wouldn't recommend it as Proxmox write more than what we imagine (I'm suspicious about that).

I'd bring this into ZFS as well, simply to get checksums etc.
But if it works fine for you then all is OK. In the end I only can express opinions and I don't feel offended if you choose not to follow mine. We are grown ups are we ;)?

That's what I've done. I hope to reach the best performances as ZFS 2.0 should be come soon into Proxmox with the new compression.

Sorry for my late answer. And thank you again for your help. :)
 
I did a lot of research with no conclusive answers as I always wish to reach the lowest latency for the RDS Server, so I added an 300Go Optane 900p to a L2ARC. I don't know if it would be effective in this case or fi it most be directly configure with pci-passtrough into the concerned vm.
I have skipped L2ARC completely in my new setup. Reason for me was that I have read at some place that L2ARC uses a lot of ARC. The bigger it is, the more it consumes. and in the end it is much slower than ARC. So I decided to completely abandon this concept. arcstat and arcsummary are tools, which might help you identifying your usage of L2ARC and if it is beneficial.

That's what I've been thinking firstly, but I didn't see any recommendation to do this from Proxmox (contrary to ESXi datasheets) and others tells they wouldn't recommend it as Proxmox write more than what we imagine (I'm suspicious about that).
8GB do work. I can tell you from my own experience. It is a bit limited / scarce but it works, if you clean up the system on recent basis.
I have created a little script helper for this purpose. You can find it here:
https://forum.proxmox.com/threads/lib-modules-filling-up-hard-drive.39199/
What did not work for me and what I can't recommend is to use USB thumb drives. I have purchased an 8GB SATA SSD and that works just fine for me.

That's what I've done. I hope to reach the best performances as ZFS 2.0 should be come soon into Proxmox with the new compression.
ZFS 2.0 has recently been brought into PVE:

Code:
#pveversion
pve-manager/6.3-6/2184247e (running kernel: 5.4.98-1-pve)

# sudo zfs --version
zfs-2.0.3-pve2
zfs-kmod-2.0.3-pve1

Sorry for my late answer. And thank you again for your help. :)
no worries ;)
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!