Hardware buying guide

[MaXX99]

Hello All,

I am in the process of setting up a new ProxMox VE Server. I already have some hardware but am looking for some insights how to finalize my build.

The system will be used in a SOHO/HomeLab setting and run following applications and VMs:

• A PfSense instance on a 400 Mbps Internet line running a few VPNs + multiple VLANs
• General file serving
• A homeassistant Server
• A Linux VM running some image manipulation on 24MP RAW images using GIMP/Darktable/Digikam + remote display
• Idem as above system but running windows (not all applications are available on Linux unfortunately)
• A Linux VM for general use office apps + remote display
• A few Docker containers (or VMs ... some testing is needed)
  • 2x Odoo
  • a few MariaDB's
  • Asterisk
  • ZoneMinder (8x 2 to 8 MP security camera's with image analysis)
  • ...
• Backup to Synology NAS and/or Backblaze

Requirements and uses will grow over time of course

Currently I already have the following for my build:

Dell PowerEdge T440

• CPU: 1x Xeon Silver 4110 (Dual CPU Capable)
• RAM: 1x 8Gb 2666Mhz (Up to 16 DIMM slots available)
• Disk: 1x 1Tb 7200rpm SAS3
• NIC: 2x onboard 1 GbE

That is ofcourse the basic system. I have already purchased a few extra's in order to accomodate Proxmox and it's VMs :

• NIC: 1x Intel I350-T4v2 (Quad 1 GbE)
• RAM: None yet (advice needed)
• Disk: None Yet (advice needed)
• SSD: 1x Intel Optane 900p 280 Gb PCIe
• Other:
  • 8 Drive caddies

As you noticed I still have some key components (RAM/Disk & SSD) missing where I would like to get your advice upon.
disks.
Electricity isn't cheap in Belgium and budgets are finite, so power consumption and cost play a lot in the hardware choices.

Memory is easy ... The more the better of course, but my budget isn't infinite and DDR4 Reg ECC Memory is f*$&%§g expensive. I was thinking of starting with the included 8 Gb and buying some extra RAM as prices drop a bit. (now 2x Samsung M393A4K40CB2-CTD 32 Gb costs around 682 €)
With 1 CPU is have 10 DIMM slots available. With the second CPU installed that number grows to 16, however I don't really think I will be needing the second CPU anytime soon and expect to be needing RAM a lot earlier.

Storage requirements and recommendations raise a lot of questions though ....

I purchased the Optane drive (only one for now --> €€€) to host the ZIL,L2ARC and Swap volumes as these drives are known to have a huge write endurance & performance.
Proxmox Boot: Should i boot from the Optane ? Boot from the spinning disk pool ? Or better purchase 1 or 2 more SSD's to boot from ?

I could for example drop in a PCIe to 2x M2 adapter to accomodate 2 extra SSD's and use these as boot devices ? (expected cost : approx 280 €)​

Disk Pool : I still will be needing some spinning rust also of course to accomodate data (and VMs).
The T440 has 8 hot-swap bays connected to a SAS3 backplane. The included 1Tb disk is quite useless I think, so i am planning to leave it out right away and dropping in some SATA or SAS.
Since ZFS RAIDZ or striped mirror vdev's can't be restriped afterwards, I am thinking of either going for:

• Toshiba DT01ACA300 7200rpm SATA (Idle Power: 5.2W / 70 €)
• Toshiba E300 5920 rpm SATA (Idle Power 4.2W / 70 €)
  
• 8 x WD Red 4Tb : 5400rpm SATA +/- 120 €/ drive) This is a cheaper and slower option but would allow for one of the following setups
  • 3x Striped Mirror + 2 Spares (12 TB Net, No parity, secure unless the second disk breaks down during rebuild)
  • 4x Striped Mirror, no spares (16 Tb Net, No parity, No added security)
  • RAIDZ2 6D+2P (+/- 24 Tb Net, slower but most secure leaving time to purchase a replacement drive.
  • RAIDZ1 6D+1P+1Spare
• 4 x HGST He10 SAS3 7200 rpm (+/- 330 € / drive) : Performance option leaving room to upgrade
  • 2x Striped Mirror (20 Tb Net)
  • RAIDZ1 3D+1P (+/- 27 Tb Net)
• 2x HGST He10 in mirror (this would be my preferred option should there be an effective way to restripe data after adding disks). I did fine some suggestions and a script which could achive this feat here : serverfault.com/questions/341552/zfs-redistribute-zvol-over-all-disks-in-the-zpool

I'm always open to other ideas and/or options of course.

​

 

[MaXX99]

No-one who could assist me a bit in my disk purchases and architecture in order to get a good starting point for my proxmox environment. It would be a shame to have to learn the hard way spending money on inappropriate hardware. Especially with all the first hand knowledge available on this forum

 

[alexskysilk]

none of the questions you asked have a objective answer. since your use case is basically to play around, you may as well learn the hard way...

If I were you, I'd play around on an existing pc (or one you buy on ebay for a few €) until I had a better understanding of what I'm trying to accomplish, and what is acceptable to me.

 

[MaXX99]

Nonetheless, there are some guidelines, procedures and best practises for machine architectures.They exist in my level of experstise (enterprise storage and SAN) and they sure do exist for proxmox. But well, compared to an enterprise setup my small office needs are indeed just 'playing around'. That still doesn't mean that my data isn't valuable to me or funds are just waiting to be spent on useless hardware just for fun ....

I went for a second Optane 900p so I can use them for the ZIL and L2ARC volumes and the boot setup.

• mirrored for the boot vols
• Mirrored for the ZIL (25 Gb)
• Striped for the L2ARC (150 Gb)

I also purchased 2 x HGST HE10 SAS3 10Tb 4Kn drives with the idea of striping them afterwards as storage needs grow and addiitional mirrors are needed.

Memory is still at the base 8Gb for the time being untill all hardware has been delivered and put together. It will suffise for the base OS and the first machines to go onto the proxmox setup. I have bugetted to add 64 gb soon.