Help for a new server/workstation

[technewbie]

Hi everyone,

I recently started learning and using proxmox on my home lab.

I'm trying to build a server/workstation for machine learning and deep learning and maybe some gaming.

My system build is as following;

AMD Threadripper Pro 3995WX 64 bit - 64 cores,
Asus WRX80 SAGE wifi Motherboard,
32 GB 3200 Mhz DDR4 ECC RAM x 8 units total 256GB,
Nvidia RTX 3090 Turbo 24 GB Reference Graphics Cards x 4 units,

As for storage I'm thinking of;

2 units of NVME drives 1TB each with RAID0 (High speed RAID) ZFS for proxmox installation and OSes of separate VMs (High speed - small size VDisks),
1 unit of 256 GB NVME for SWAP partition,
4 units of 2TB 2.5" SSDs with RAID10 (High speed and redundancy) ZFS as seperate attached disks for VMs (Medium speed - medium size VDisks),
4 units of 10TB 3.5" HDDs with RAID10 (High speed and redundancy) ZFS as seperate attached disks for VMs (Low speed - large size VDisks)

I'm gonna create VMs with high speed VDisks and attach each VM multiple Medium speed and Low speed VDisks for AI training.

The motherboard has 3 NVME and 8 SATA ports and that's why I thought of this RAID and Disk combination but I couldn't find if the motherboard or Proxmox support the RAID and disk combination I'm thinking of. Is this configuration logical to you and if I can do it ?

It's perfect that Proxmox supports PCIE passthrough and GPU Virtualization as I can assign these resources physically or virtually to create more AI Training VMs.

I'm open to advices and new ideas. I would be glad if I could find a solution for RAID because I don't want to use closed source virtualization softwares like VMware etc.

I hope I could express what I'm thinking because I'm not a native English speaker.

Thanks

 

[Stefan_R]

Now that's a high-spec workstation! The storage setup in general is certainly possible and seems to make sense, if it fits your requirements. Here's some general pointers:

• With ZFS, there's no need to check for compatibility of the motherboard etc..., as everything happens in software - it will work on any machine, provided the disks are compatible (i.e. same size and type within a zRAID or stripe/mirror config). Tradeoff being a little bit of CPU and (with ARC caching) quite some RAM consumption, but seeing your specs that is probably negligible entirely
• The setup you describe is possible, with the exception of the swap partition you should be able to configure all of it via the installer or the web GUI even.
• Some motherboards have restrictions on which NVMe slots are available when other ports are used (SATA, etc...) due to PCIe lane assignments. Might not be relevant for such a high-end board, but be sure to check the manual thoroughly.
• On the same line of thought, make sure the PCIe slots for your GPUs are at least x8 (better x16), often not all slots are equal, especially if you need to use spread out ones (consider the height of the GPUs)
• The RTX 3090 is not officially capable of NVIDIA GRID-style multi-assignment - that is reserved for Quadro cards. This means you can only ever assign one GPU to one VM (or the host), not multiple. With 4 cards that's probably fine though?

Good luck!

 

[technewbie]

Now that's a high-spec workstation! The storage setup in general is certainly possible and seems to make sense, if it fits your requirements. Here's some general pointers:

• With ZFS, there's no need to check for compatibility of the motherboard etc..., as everything happens in software - it will work on any machine, provided the disks are compatible (i.e. same size and type within a zRAID or stripe/mirror config). Tradeoff being a little bit of CPU and (with ARC caching) quite some RAM consumption, but seeing your specs that is probably negligible entirely
• The setup you describe is possible, with the exception of the swap partition you should be able to configure all of it via the installer or the web GUI even.
• Some motherboards have restrictions on which NVMe slots are available when other ports are used (SATA, etc...) due to PCIe lane assignments. Might not be relevant for such a high-end board, but be sure to check the manual thoroughly.
• On the same line of thought, make sure the PCIe slots for your GPUs are at least x8 (better x16), often not all slots are equal, especially if you need to use spread out ones (consider the height of the GPUs)
• The RTX 3090 is not officially capable of NVIDIA GRID-style multi-assignment - that is reserved for Quadro cards. This means you can only ever assign one GPU to one VM (or the host), not multiple. With 4 cards that's probably fine though?

Good luck!

Hi Stefan,

Thanks for the informaiton but the information you've given has created new questions in my mind

With a system and RAID setup like that how much RAM and CPU cores should I reserve for the host macine ? (Btw, I'm using Proxmox bare machine. No desktop environments etc. VNCing from another machine I mean)

Now i sould learn about ARC caching.

Is ARC caching kind of like special SWAP for ZFS ?

So far I haven't had any troubles with the disk RAID array, motherboard is supporting.

No problem with the GPUs as the PCIE lanes on board are 16x lanes.

I will update you about the development.

Thanks again

 

[Dunuin]

ARC is the read cache of ZFS. So its more comparable to the linux page files than to swap. Rule of thumb is 4GB + 1GB per 1TB of storage for your ARC. So 54 GB RAM for your ARC for your storage setup. If you dont manually limit it it will use up to 50% of your hosts RAM (so 128GB) for caching.

Its also recommended to use enterprise SSD especially if using ZFS. So you want some U.2 NVMe and not M.2 SSD. You can buy U.2 to M.2 adapter cables.

And keep in kind that your 8 SATA ports probably wont offer the full performance if they are provided by the mainboards chipset. Check how the chipset is connected to the CPU. All stuff like SATA, NICs, soundcard, USB, maybe some PCIe slots that is connected to the chipset have to use this link between CPU and chipset.
Here on my servers mainboard this link is only 4 PCIe 3.0 lanes so it is bottlenecking and I would need a multiple of that if I would like to use all USB/NICs/SATA and so on with full speed at the same time. If you want full SATA bandwidth you probably need to use a HBA card.

And if you got alot of sync writes it might also be usefull to use a partition of that swap drive as a SLOG (zfs write cache). And for SLOG you want a drive with a really low latency so something like a Intel Optane P5800X might be a good choice.

Also keep in mind that raid1 isn't only there so you don't loose data if a drive dies. Its also there to ensure data integrity. One of the great benefits of ZFS is that it can detect data corruption caused by bit rot. But if you don't got any parity data (raid1/raid10/raid5/raid6) it can only tell you that your data got corrupted but can't do anything against it. With for example raid10 it would be able to repair it so you got a selfhealing filesystem. So ZFSs checksumming is like the ECC of your memory but for your drives and not the RAM. So using raid0 without parity is like using RAM without ECC.