PVE Best practices and White paper

[cayden]

Hi Experts,
Hope you're doing well.
I’m reaching out to check if we have any BP or white paper related documents for the design and deployment phases？thanks.

 

[jdancer]

Like with any deployment, start with a proof-of-concept testbed.

As you know, separate OS from data. So, use small drives to mirror the OS. I use ZFS RAID-1. Rest of bigger drives for VMs/data. Strongly suggest to use a IT/HBA-mode storage controller. I use a Dell HBA330 in production with no issues.

If servers in a cluster, use homogeneous hardware, ie, same CPU, memory, networking, storage controller, storage, firmware, etc.

I use the following optimizations learned through trial-and-error. YMMV.

    Set SAS HDD Write Cache Enable (WCE) (sdparm -s WCE=1 -S /dev/sd[x])
    Set VM Disk Cache to None if clustered, Writeback if standalone
    Set VM Disk controller to VirtIO-Single SCSI controller and enable IO Thread & Discard option
    Set VM CPU Type to 'Host' for Linux and 'x86-64-v2-AES' on older CPUs/'x86-64-v3' on newer CPUs for Windows
    Set VM CPU NUMA
    Set VM Networking VirtIO Multiqueue to 1
    Set VM Qemu-Guest-Agent software installed and VirtIO drivers on Windows
    Set VM IO Scheduler to none/noop on Linux
    Set Ceph RBD pool to use 'krbd' option

 

[cayden]

Like with any deployment, start with a proof-of-concept testbed.

As you know, separate OS from data. So, use small drives to mirror the OS. I use ZFS RAID-1. Rest of bigger drives for VMs/data. Strongly suggest to use a IT/HBA-mode storage controller. I use a Dell HBA330 in production with no issues.

If servers in a cluster, use homogeneous hardware, ie, same CPU, memory, networking, storage controller, storage, firmware, etc.

I use the following optimizations learned through trial-and-error. YMMV.

    Set SAS HDD Write Cache Enable (WCE) (sdparm -s WCE=1 -S /dev/sd[x])
    Set VM Disk Cache to None if clustered, Writeback if standalone
    Set VM Disk controller to VirtIO-Single SCSI controller and enable IO Thread & Discard option
    Set VM CPU Type to 'Host' for Linux and 'x86-64-v2-AES' on older CPUs/'x86-64-v3' on newer CPUs for Windows
    Set VM CPU NUMA
    Set VM Networking VirtIO Multiqueue to 1
    Set VM Qemu-Guest-Agent software installed and VirtIO drivers on Windows
    Set VM IO Scheduler to none/noop on Linux
    Set Ceph RBD pool to use 'krbd' option

jdancer,
Thank you so much for your detailed and helpful response! I really appreciate you taking the time to explain this.
Regarding the solutions you mentioned, could you please point me to any official documentation or sources where these are discussed?
Understanding the official context would help me in applying them correctly and learning more about the underlying principles,thanks.