Hi,
I built a home server without difficulty thanks to Proxmox :
CPU: Threadripper 1920X
RAM: 64GB ECC
MB: Asrock X399 Professional Gaming @ BIOS 3.30
GPU: Inno3D 1050Ti
HBA: SAS9305-16i
ZFS Pool: HGST He10 10TB * 3 + WD Red 3TB *3s
GPU and USB passthrough works very well, since there are many tutorials available I'm not going to detail here.
However, there are several small things which makes the configuration not perfect.
So I made some tweaking(hacking) scripts.
CPU pinning
Even if NUMA is specified in the VM configuration file, since there is no fixed mapping between vCPUs and Host CPUs, cross-node access can still be happened. In my case it reflects in Aida64 memory benchmark with random speed drop down (from 46GB/s to 35GB/s) and latency increse (form 86ns to 140ns).
This script by ayufan handles the cpu pinning very well.
I made some adjustments so you can specify which cores are used explicitly.
Example: assign core 0-3 to VM100:
Change CPU scheduler to FIFO
Multiple threads indicate SCHD_FIFO can reduce latency (lag spikes).
Run custom command after VM start up
I hacked QemuServer to run these tweaks at startup. (Use at your own risk !)
Reboot, then in your VM config file add:
Disclaimer
These tweaks are provided as-are, and there are no guarantees that they fit your purposes or that they are bug-free. Use it at your own risk!
I built a home server without difficulty thanks to Proxmox :
CPU: Threadripper 1920X
RAM: 64GB ECC
MB: Asrock X399 Professional Gaming @ BIOS 3.30
GPU: Inno3D 1050Ti
HBA: SAS9305-16i
ZFS Pool: HGST He10 10TB * 3 + WD Red 3TB *3s
GPU and USB passthrough works very well, since there are many tutorials available I'm not going to detail here.
However, there are several small things which makes the configuration not perfect.
So I made some tweaking(hacking) scripts.
Code:
github.com/HiFiPhile/PVE_TweaksCPU pinning
Even if NUMA is specified in the VM configuration file, since there is no fixed mapping between vCPUs and Host CPUs, cross-node access can still be happened. In my case it reflects in Aida64 memory benchmark with random speed drop down (from 46GB/s to 35GB/s) and latency increse (form 86ns to 140ns).
This script by ayufan handles the cpu pinning very well.
Code:
gist.github.com/ayufan/37be5c0b8fd26113a8beI made some adjustments so you can specify which cores are used explicitly.
Example: assign core 0-3 to VM100:
Code:
pve_cpu_pinning.sh 100 0,1,2,3Change CPU scheduler to FIFO
Multiple threads indicate SCHD_FIFO can reduce latency (lag spikes).
Code:
pve_schd_fifo.sh 100Run custom command after VM start up
I hacked QemuServer to run these tweaks at startup.
(Use at your own risk !)
Code:
patch /usr/share/perl5/PVE/QemuServer.pm ./QemuServer_PostCommand.patch
Code:
postcommand: /home/your_tweaks.shDisclaimer
These tweaks are provided as-are, and there are no guarantees that they fit your purposes or that they are bug-free. Use it at your own risk!
Last edited: