i am trying to tune a linux VM (UBUNTU 22.04) for the best possible LLM performance and i am wondering if there is anything i can do for better cpu/ram performance besides the basic options like setting cpu to host, etc?
any tips/tricks/suggestions towards tweaking proxmox or ubuntu for better vm llm performance welcome
i am not really worried about security in general if it in any way stands in the way of perforance, mitigations (already disabled all of them), etc, i just want the most possible performance, it is a local box homelab type of server running behind a firewall and all it runs is media/games/llm & AI/docker and random self hosting projects.
i have a 7GB vram vgpu profile setup in the ubuntu vm but with only 7GB of vram on the profile, it usually ends up offloading many models to the cpu/ram too, even if only in part and the perfomance really tanks when this happens, far more than it should even, i end up seeing things like 0.3-1.5 tokens / second and even my laptop with a slower cpu running pure cpu can get 3-5 t/s out of the same models, so i should be seeing around 8 tokens/s or more from this cpu and i would expect at least 4-5 t/s in a vm because of performance loss but instead it often is far worse.
hardware:
MOBO: Gigabyte x299 UD4 pro
CPU: 16 x Intel(R) Core(TM) i7-7820X CPU @ 3.60GHz
RAM: 96GB DDR4 (x8 sticks)
GPU 1: INTEL ARC A310 4GB
GPU 2: Nvidia Tesla P4 8GB
vm .conf
i was also reading this Linux Lazy Unmap Flush "LUF" Reducing TLB Shootdowns By 97%, Faster AI LLM Performance hoping possibly
any tips/tricks/suggestions towards tweaking proxmox or ubuntu for better vm llm performance welcome
i am not really worried about security in general if it in any way stands in the way of perforance, mitigations (already disabled all of them), etc, i just want the most possible performance, it is a local box homelab type of server running behind a firewall and all it runs is media/games/llm & AI/docker and random self hosting projects.
i have a 7GB vram vgpu profile setup in the ubuntu vm but with only 7GB of vram on the profile, it usually ends up offloading many models to the cpu/ram too, even if only in part and the perfomance really tanks when this happens, far more than it should even, i end up seeing things like 0.3-1.5 tokens / second and even my laptop with a slower cpu running pure cpu can get 3-5 t/s out of the same models, so i should be seeing around 8 tokens/s or more from this cpu and i would expect at least 4-5 t/s in a vm because of performance loss but instead it often is far worse.
hardware:
MOBO: Gigabyte x299 UD4 pro
CPU: 16 x Intel(R) Core(TM) i7-7820X CPU @ 3.60GHz
RAM: 96GB DDR4 (x8 sticks)
GPU 1: INTEL ARC A310 4GB
GPU 2: Nvidia Tesla P4 8GB
vm .conf
Code:
agent: 1
cpu: host,flags=+pdpe1gb;+hv-tlbflush;+hv-evmcs;+aes
balloon: 0
bios: seabios
boot: order=virtio0;net0
cores: 16
cpu: host,flags=+pdpe1gb;+hv-tlbflush;+hv-evmcs;+aes
hostpci0: mapping=vGPU-Nvidia,mdev=nvidia-69
kvm: 1
machine: pc-q35-9.0,viommu=virtio
memory: 43024
meta: creation-qemu=9.0.2,ctime=1740017094
name: Ubuntu
net0: virtio=BC:24:11:83:AD:1D,bridge=vmbr0,firewall=1
numa: 1
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=95c62090-ec6f-4239-a9ec-d1a8848b102e
sockets: 1
tablet: 0
virtio0: SSD:vm-107-disk-0,backup=0,discard=on,iothread=1,size=4T
virtio1: NVME:107/vm-107-disk-0.qcow2,backup=0,discard=on,iothread=1,size=16G
virtio2: zfs-ai:vm-107-disk-0,aio=threads,backup=0,discard=on,iothread=1,size=4T
vmgenid: f7df888c-ba98-4339-921a-56d01b965ba3i was also reading this Linux Lazy Unmap Flush "LUF" Reducing TLB Shootdowns By 97%, Faster AI LLM Performance hoping possibly
+hv-tlbflush could help yield better results, but that causes the following error kvm: Hyper-V paravirtualized TLB flush (hv-tlbflush) requires Hyper-V VP_INDEX MSR (hv-vpindex) and if hv-vpindex is added to the .conf file manually proxmox clears the whole CPU config and resets to default with the following messages in journalctl
Code:
Feb 23 02:14:07 prox pvedaemon[2196698]: vm 107 - unable to parse value of 'cpu' - VM-specific CPU flags must be a subset of: pcid, spec-ctrl, ibpb, ssbd, virt-ssbd, amd-ssbd, amd-no-ssb, pdpe1gb, md-clear, hv-tlbflush, hv-evmcs, a>
Feb 23 02:14:09 prox pveproxy[2179567]: vm 107 - unable to parse value of 'cpu' - VM-specific CPU flags must be a subset of: pcid, spec-ctrl, ibpb, ssbd, virt-ssbd, amd-ssbd, amd-no-ssb, pdpe1gb, md-clear, hv-tlbflush, hv-evmcs, aes
Feb 23 02:14:12 prox pvestatd[2751]: vm 107 - unable to parse value of 'cpu' - VM-specific CPU flags must be a subset of: pcid, spec-ctrl, ibpb, ssbd, virt-ssbd, amd-ssbd, amd-no-ssb, pdpe1gb, md-clear, hv-tlbflush, hv-evmcs, aes
Feb 23 02:14:14 prox pvedaemon[2188733]: vm 107 - unable to parse value of 'cpu' - VM-specific CPU flags must be a subset of: pcid, spec-ctrl, ibpb, ssbd, virt-ssbd, amd-ssbd, amd-no-ssb, pdpe1gb, md-clear, hv-tlbflush, hv-evmcs, a>
Feb 23 02:14:15 prox pveproxy[2166887]: vm 107 - unable to parse value of 'cpu' - VM-specific CPU flags must be a subset of: pcid, spec-ctrl, ibpb, ssbd, virt-ssbd, amd-ssbd, amd-no-ssb, pdpe1gb, md-clear, hv-tlbflush, hv-evmcs, aes
Feb 23 02:14:23 prox pvedaemon[2166352]: vm 107 - unable to parse value of 'cpu' - VM-specific CPU flags must be a subset of: pcid, spec-ctrl, ibpb, ssbd, virt-ssbd, amd-ssbd, amd-no-ssb, pdpe1gb, md-clear, hv-tlbflush, hv-evmcs, a>
Feb 23 02:14:30 prox pveproxy[2185637]: vm 107 - unable to parse value of 'cpu' - VM-specific CPU flags must be a subset of: pcid, spec-ctrl, ibpb, ssbd, virt-ssbd, amd-ssbd, amd-no-ssb, pdpe1gb, md-clear, hv-tlbflush, hv-evmcs, aes
Feb 23 02:15:06 prox pveproxy[2198169]: vm 107 - unable to parse value of 'cpu' - VM-specific CPU flags must be a subset of: pcid, spec-ctrl, ibpb, ssbd, virt-ssbd, amd-ssbd, amd-no-ssb, pdpe1gb, md-clear, hv-tlbflush, hv-evmcs, aes