Looking for suggestions for a ZFS server that can have ssd and nvme and local high capacity (for backup) storage

mmenaz

Active Member
Jun 25, 2009
828
20
38
Northern east Italy
Forgive my ignorance in "high level hardware" but I've only experience with my "self made" Proxmox server at home and I've been asked by a friend that runs a small IT business about a Proxmox server for his customers.
My idea is that I will boot from 2 ZFS raid1 small DC Sata SSD, have a VM dedicated ZFS Storage as Sata SSD or NVMe Raid1 or Raid10, and at least 2 Raid1 high capacity HDD for local backup.
This sounds a basic requirements to me (my home server is more or less like that), but looking in major brend sites (Dell, HP), I was not able to understand if these needs can be met (of course without a stellar budget, here, Italy, business are really small in size). Seems that if you want feature 'A' you can't have 'B' and so on. Also storage seems "all 3.5" or "all 2.5", do you select "3.5" and then use 2.5 to 3.5 adapters?
So, based on your experience, I would love to have suggestions about HW that can be currently purcased (and not obsolete as in many posts here or self assembled) and has these features:
- works with Proxmox and ZFS
- decent CPU range performance (index 10 - 20.000)
- can be expanded with >= 64GB RAM but a max of 256GB is more than enough
- one cpu socket is enough (and don't want to have to add "second CPU" just to have NVMe or something like that)
- has bays for at least 2 x 3.5" Sata HDD (big capacity for local backup), and "n" 2.5" SSD or U.2 NVMe for VM storage + 2 x 2.5" SSD for Proxmox installation (as stated above)
- can have at least 4 disks NVMe U.2 (we will use of course DC ones)
- preferably form factor 2U 19" (1U is too noisy!) or Tower
- enough PCIe slots for at least i.e. 1 x 4 1Gbit card and one dual port 10 or 40GBit (or 100Gbit) card (if I want a cluster or fast connection with switch / other host)
- reduntant power supply option would be nice to have
- storage (disks) can be purcased separately (i.e. buy a Dell server but buy lower cost storage somewhere else)

In short I would love to have an "universal" server that can be equiped with basic storage (OS and backup) and have additional Sata and/or NVMe ZFS storage for VMs.
(ZFS implies a MB that has raliable on board sata or a tested HBA IT mode).

Please tell me also if my assumptions are wrong, i.e. I've found not 2.5" HD high capacity that are ok for server usage, only laptop ones.
Also I would need the exact list of what you are using (i.e. Server Model X, with controller Y1 for Sata end Y2 for NVMe, and optionale backplane X for nvme, cables, etc.).

If this can't be achieved with only one server model and different expansion options, 2 models (Sata and NVMe) can be considered.
Finally, looking at Dell site I see in some "NVMe and PCIe Storage Adapters" section that as adapter is listed "SAS 12Gbps HBA External Controller", what do they mean for "external" and why "SAS" and how can a 12Gbps be enough? Or is one controller for each NVMe disk (still below NVMe theoretichal bandwidth)?
Thanks a lot!
 

LnxBil

Famous Member
Feb 21, 2015
6,292
777
163
Saarland, Germany
- one cpu socket is enough (and don't want to have to add "second CPU" just to have NVMe or something like that)
unfortunately, that's how that works. For a fast storage system you need a lot of PCIe lanes and they come only from more CPUs.

Please tell me also if my assumptions are wrong, i.e. I've found not 2.5" HD high capacity that are ok for server usage, only laptop ones.
HDDs: Yes, SFF (small form factor, 2.5") are only fast, not big. You get LFF (large form factor, 3.5") NL-SAS (neal line) spinners that are basically large normal drives with enterprise firmware and SAS.
only the formfactor of 2.5" is "laptop like", the disks are higher (12mm) and are normally 10k or 15k, whereas laptop drives 5,4k or 7,2k rpms have.

PCIe Storage Adapters" section that as adapter is listed "SAS 12Gbps HBA External Controller", what do they mean for "external" and why "SAS" and how can a 12Gbps be enough?
That can be a 2U shelf with one Mini-SAS (4x SAS 12 Gbps channels) to the controller and then e.g. 24x SFF or 12x LFF disks. This is the best way to combine what you want to have. I do not know of any server than can house a multiple U.2 SFF and LFF, so the external shelf is a good way to go.

I personally haven't seen LFF in server for a very, very long time. Most use SFF HDD for everything due to the much more improved IOPS over LFFs HDDs and also for the same formfactor in SSD and HDD in one case/slot.

I would recommend to have an external backup server, perferably with PBS for deduplication. Local backup is useless.
 
Last edited:

mmenaz

Active Member
Jun 25, 2009
828
20
38
Northern east Italy
I would recommend to have an external backup server, perferably with PBS for deduplication. Local backup is useless.
PBS is a good solution, but costly and for a fast restore you also need fast connections. With a local backup you simply move the backup storage in the new server and do a fast restore, so local backup is not useless if you can move it :)
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!