Proxmox 8.0 Tuning and best practices for production/colocated servers.

[Shlee]

Started reading every Wiki page and a bunch of forum posts.. but the "best practices" for a stable production server are a little vague.

SuperMicro H12DSI-N6
2 x AMD EPYC 7642
16 x 32GB ECC 3200 DDR4 ECC

2 x Micron 5300 MAX 960GB SATA (9000TBW)
1 x Micron 5300 PRO 480GB M.2 (1324TBW) currently unused
8 x WD Red SA500 SSD 1TB SATA (600TBW)
4 x Kingston KC3000 NVMe (800TBW) on a PCIE bifurcated card

Intel NICs for network.

Does anybody have any recommendations for setting for sysctl? network? Kernel tweaks? or Do I need to pay somebody for this kind of detail? and support?

Is `init_on_alloc=0` recommended as a GRUB_CMDLINE_LINUX_DEFAULT? Should I tweak the fs.file-max? fs.nr_open? net.ipv4.tcp_keepalive_time? net.ipv4.tcp_congestion_control = scalable? as those are included in some of the tuning guides.

1) I want to have a stable, but performative environment... What are the best practices for modifing a freshly installed Proxmox?
2) For storage. I've mirrored the 5300 MAX for the boot drive... RAIDZ2 the reds and RAIDZ10 the Kingstons.... but ZFS feels like it needs to be tweaked for best performance.
Considering the NVMe storage exists to service my Postgres DBs.. Any recommendations for that storage setup?

I've seen:
* https://gist.github.com/sergey-dryabzhinsky/bcc1a15cb7d06f3d4606823fcc834824 (from PVE3?)
* https://github.com/brcak-zmaj/Proxmox_Tuning_Playbook

 

[LnxBil]

Does anybody have any recommendations for setting for sysctl? network? Kernel tweaks? or Do I need to pay somebody for this kind of detail? and support?

That depends on what you want to archieve. PVE comes with a lot of sane defaults for most users. If you ran into problems, you can still tweak to your liking. I'd only do that on errors. Most users do not know their actual requirement until you hit a wall or never.

Is `init_on_alloc=0` recommended as a GRUB_CMDLINE_LINUX_DEFAULT? Should I tweak the fs.file-max? fs.nr_open? net.ipv4.tcp_keepalive_time? net.ipv4.tcp_congestion_control = scalable? as those are included in some of the tuning guides.

If you run a lot of containers, you may want to look into them IF you reach the current limits. For KVM/QEMU VMs, there are not as many files open then with containers, so you run later in those limits.

I'd also consider KSM, which only works for KVM/QEMU VMs and not containers. ZFS performs better with datasets (when it known the actual recordsize up to 128k) then if it does not know what is going on with volblocksize 8 with respect to the padding overhead with raidzX. In a performance setting, I'd always and exclusively go with RAID10 (or stripped mirrors in the ZFS sense). Also a single ZFS performs better than multiple ZFSs due to the split ARC.

960 GB bootdrive? what a waste ... you need on ZFS less then 4G due to the compression.
I would buy 14/16 same devices and use them in a stripped mirror. Will be 3-8 times faster than your setup. If that's still not enough, get small Optane NVMe for SLOG and sync writes. This will - for those special workloads - increase the throughput and IOPS (e.g. databases).

As nice this CPU is, performance-wise I would also not use a highly packed CPU, I'd go with less cores, more speed but that depends heavily on the used software. We optimize here often for best single thread performance due to our used requirements (mostly non-parallel databases), yours may be different. We almost exclusively use the highest frequency possible and still do not have 100% cpu utilization, so this works totally fine for us. If we need more compute power, we add more nodes (horizontal scaling).

We also optimize first for high availability, is this not a requirement for your setup?

Considering the NVMe storage exists to service my Postgres DBs. Any recommendations for that storage setup?

critical summary about a lot of optimizations for PostgreSQL on ZFS (directly without another filesystem).