Best use of 4x NVMe drives

P

pjalm

New Member
Apr 21, 2025
13
0
1
Hi everyone.

I am looking for some advice on the best use/setup of 4x SSDs for use with Proxmox.

I have 3x 1TB PM981 SSDs and 1x 512GB PM981 SSD and I want to put in the PVE server as I need the SAS SSD for another project.

What is the best option to use when installing Proxmox?

I have the following setup:

HP z840 (PVE Server)
  • 2x Xeon E5-2697 v3 (14 cores, 28 threads each)
  • 4x 64GB SkHynix ECC DDR4 RAM
  • 4x Seagate 12TB EXOS SAS HDDs
  • 1x 3.84GB SAS SSD
  • 2x Intel 10Gbe NICs
HP z600 (PBS Server)
  • 2x Xeon (6 cores, 12 threads each)
  • 6x 16GB ECC DDR3 RAM
  • 1x 256GB SATA SSD
  • 2x Seagate 4TB EXOS SAS HDDs
  • 1x HP SureStore LTO7 SAS Tape Drive
  • 1x LSI SAS HBA
  • 2x Intel 10Gbe NICs

The following is running on the PVE server
  • addguard
  • homebridge
  • onlyoffice
  • plex
  • pelican-wings
  • pelican-panel
  • homepage
  • meshcentral
  • OMV
  • HAOS
  • Windows 11 VMs
  • Linux VMs

I run Pelican with several game servers running and a few Linux and Windows VMs for app testing as I am an app developer.
OMV is used for my file sharing which has a dedicated SAS HBA passthrough with the SAS HDDs.

Thx in advance for any help.
 
Last edited:
I have the HP Z Turbo Drive Quad Pro PCIe NVMe SSD Adapter, thats not the issue, I just want to know what the best option to chose when installing PVE with these drives. Thx
 
I was told ZFS is bad for NVMe drives.

ZFS is not inherently "bad" for NVMe drives in terms of destroying them, but it can significantly bottleneck their performance and accelerate wear if improperly configured. While ZFS provides superior data integrity, its write amplification can cause excessive wear on consumer NVMe drives, and it often achieves only 50-80% of native drive speed due to, overhead
Click to expand...

My question is what option? there is ZFS (RAID0), ZFS (RAID1), ZFS (RAID10), ZFS (RAIDZ-1), ZFS (RAIDZ-2) and ZFS (RAIDZ-3) so not really obvious.

One other question, will I be able to expand by adding another drive later using ZFS or will I need to reinstall everything from scratch?
 
Last edited:
default ext4/Lvmthin for datastore will be fine and the fastest.
e.g:
500G for PVE and iso (custom sizes at install : disk = 400G and maxvz = 0 to not create the default local-lvm Datastore on the 500G )
1 Lvmthin datastore per 1TB NVMe
Restore backup from PBS in case of emergency.
 
Last edited:
_gabriel said:
default ext4/Lvmthin for datastore will be fine and the fastest.
e.g:
500G for PVE and iso (custom sizes at install : disk = 400G and maxvz = 0 to not create the default local-lvm Datastore on the 500G )
1 Lvmthin datastore per 1TB NVMe
Restore backup from PBS in case of emergency.
Click to expand...
I don't understand, how do I do this in the installer?

Also there will be zero ISOs on this server, I should have mentioned I have a iVentoy server with all my ISO so I just do PXE boot on VMs or real machines.
 
Last edited:
Sizes can be set with "Options" button during install.
if you don't need iso, keep installing PVE on the 500GB disk ( installer will create a ~380G local-lvm Datastore ).
Just remember if you reinstalling PVE, the default local-lvm Datastore will be erased.
PVE installer always erase the "Target disk" selected during installation. Others disks are left of course.
Once PVE installed, you can make Lvmthin datastores on your otherdisks.
 
_gabriel said:
Sizes can be set with "Options" button during install.
if you don't need iso, keep installing PVE on the 500GB disk ( installer will create a ~380G local-lvm Datastore ).
Just remember if you reinstalling PVE, the default local-lvm Datastore will be erased.
PVE installer always erase the "Target disk" selected during installation. Others disks are left of course.
Once PVE installed, you can make Lvmthin datastores on your otherdisks.
Click to expand...

If I understand correctly you are saying I can not install PVE on a multidisk setup for boot?
 
_gabriel said:
no, I suggest you one way.
ZFS is the way for multi boot disks, but it's slower because the overhead for checksum and integrity, even more with none datacenter drives.
ZFS is enterprise oriented.
Click to expand...
I see that makes sense, thx
 
You must log in or register to reply here.
Top Bottom
Back