Advice Needed on VM Boot Drive Sizes and SWAP Partition Setup

[PapaGigas]

Hi Proxmox Community,

I'm currently setting up a Proxmox environment and could use some guidance regarding the optimal sizes for VM boot drives and the best practices for creating a SWAP partition. Here is the breakdown of my planned setup:

Storage Setup:

• 6x p1600x/118GB drives in RAID10
  
  
• Total usable storage: 354GB (329.6882GiB)

Planned Allocations:

• 32GiB for SWAP and L2ARC
  
  
• 16GiB for FreeBSD running OPNsense (sharing 2 NICs with PVE)
  
  
• 16GiB for Debian running TrueNAS (with onboard SATA controllers passthrough)
  
  
• 128GiB for Debian running CasaOS (with GPU passthrough)
  
  
• 128GiB for Windows 11 running Hyper-V (with USB3 controller, NVMe's and GPU passthrough)

Remaining Storage:

• Approximately 9.6882GiB for running PVE + PBS

Additional Hyper-V Setup:

• The Hyper-V instance will manage 3 additional Windows 11 VMs.
  
  
• GPU passthrough to share the GPU across these VMs.
  
  
• Each of the 3 Windows 11 VMs will use a dedicated Samsung 970 Evo Plus as a boot drive.

Questions:

1. Optimal Boot Drive Sizes: Given the above setup, what would be the recommended sizes for the boot drives of each VM? Specifically, I want to ensure that each VM has sufficient space for the OS and any essential services without over-allocating and wasting valuable storage.
   
   
2. SWAP Partition Setup: What is the best method for creating the SWAP partition in this setup? Should I allocate a dedicated SWAP partition on the Proxmox host, or would it be more efficient to create individual SWAP files for each VM? Additionally, any advice on the recommended SWAP size relative to the VM's RAM would be appreciated.

Additional Context:

• Performance and stability are top priorities, especially for VMs handling network and storage tasks.
  
  
• The Hyper-V setup is specifically designed to leverage GPU passthrough for enhanced performance across multiple Windows 11 VMs, each with dedicated NVMe boot drives.

Any advice or recommendations from the community would be greatly appreciated. Thank you in advance for your help!

 

[Kingneutron]

You don't want to swap at all, if possible. Swap should be no more than ~1-2GB. Stuff the server with as much RAM as it can hold / you can afford, and set swappiness to 1. VM guests can be allowed to swap internally, but again - you don't want more than ~1GB of vswap in the guest either bc it will slow down performance.

Put this in /etc/rc.local and run /etc/rc.local at the command prompt (will also take effect next reboot)

# we dont want to swap much if at all possible
echo 1 > /proc/sys/vm/swappiness

# hopefully better multitasking I/O performance
echo 20 > /proc/sys/vm/dirty_ratio

# Try to keep at least 100MB of free RAM at all times
echo 100000 > /proc/sys/vm/min_free_kbytes

# Default 100 - try more aggressively to reclaim inodes, etc from cache
echo 160 > /proc/sys/vm/vfs_cache_pressure

20-22GB rootfs minimum is what I usually give to 64-bit OS these days, with ext4/XFS you can use a swapfile instead of a dedicated partition.

With proxmox host I recommend 30-40GB rootfs minimum for ISOs and housekeeping. (Don't forget, you need space for logs and package upgrades.) Install to zfs boot/root and you'll have compression as well.

You don't want to run low on disk space bc resizing is a hassle, and it's all thin-provisioned anyway. You can run proxmox rootfs on as little as 32-64GB, but you'll want larger SSD disk (256GB-1TB) so the cells don't wear out as fast - and to have a bit for lvm-thin or a ZFS partition at the end.

Don't forget, you still need disk space / NAS outside the server for backups.

 

[PapaGigas]

I'm only talking about boot drives, and I'm using Optane (6 drives in RAID10). I'm setting up TrueNAS with 14 SSDs and 2 HDDs (with an Optane drive for metadata), all SATA drives. Additionally, the system has 32 cores (64 threads) and 128GB of RAM.

This is the configuration I have in mind:

• OPNsense: 4 CPUs, 8GB RAM, 16GiB Optane
  
  
• TrueNAS: 8 CPUs, 16GB RAM, 16GiB Optane (boot), 2TB SSD RAID10 (4x Kingston + 1 spare), 4TB SSD RAID10 (8x Samsung + 1 spare), 4TB HDD RAID1 (2x Seagate), 118GB Optane (metadata)
  
  
• CasaOS: 16 CPUs, 32GB RAM, 128GiB Optane (boot), NVIDIA Tesla P4, shares storage from TrueNAS
  
  
• Windows 11 with Hyper-V: 24 CPUs, 48GB RAM, 128GiB Optane (boot), 4x 500GB NAND, NVIDIA 4070, shares storage from TrueNAS
  
  
• Each VM inside Hyper-V: 6 CPUs, 12GB RAM, 500GB NAND (boot), shares GPU with Hyper-V, and shares storage from TrueNAS
  
  
• Proxmox: min. 4 CPUs, 12GB RAM (including ARC and KSM sharing), 32GiB + approximately 9.6882GiB for running PVE + PBS (including SWAP and L2ARC), shares storage from TrueNAS

PS - I'm still using 4x 970 EVO Plus (RAID10) in this system, but I'm replacing them with the Optane drives and giving them to the Windows 11 VMs.