PCI Passthrough RAM Issue

[Possums]

Hello all,

I am attempting to set up Proxmox with PCI passthrough for GPUs.
The actual passthrough works perfectly, but only with low RAM values.

As soon as I allocate more than about 24G of ram to a VM, the VM will refuse to start.
Pressing the start button results in in either a perpetual spinning circle or a time-out error.

Without the GPU passed through, any RAM amount works fine.
Has anyone encountered a similar issue?

VM Config:

args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
balloon: 0
bios: ovmf
bootdisk: virtio0
cores: 16
cpu: host,hidden=1,flags=+pcid;+spec-ctrl
efidisk0: local-lvm:vm-101-disk-1,size=128K
hostpci0: 21:00,pcie=1
hugepages: 2
ide2: local:iso/archlinux-2019.01.01-x86_64.iso,media=cdrom
machine: q35
memory: 24000
name: Arch
net0: virtio=76:FA:28:6C:95:AA,bridge=vmbr0
numa: 1
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=0371aae5-72aa-4a7e-9638-a5e9525c25d6
sockets: 4
virtio0: local-lvm:vm-101-disk-0,cache=writeback,size=500Gvmgenid: 13010caa-133e-4255-8d32-4a9f059b8c65​

 

[dcsapak]

with pci passthrough, qemu has to allocate/reserve the memory on start (for the pci memory mappings) this can take a while depending on how full your memory is

 

[Possums]

Thanks for your help!

It turns out that was the issue.
I managed to boot the VM (64GB ram) by starting it from the terminal and waiting about 15-20 minutes.
This was with no other VMs running.

Are there any methods to speed this process up?

Edit: With a second boot of the VM, the boot time is now reduced to ~3 minutes, a reasonable time frame.

 

[dcsapak]

Are there any methods to speed this process up?

this depends largely what is inside your memory and how your system uses it
but have a look at hugepages, maybe they help

 

[mx781]

@Possums was there a flag you passed to qm to avoid it from timing out?

I am encountering the same issue, but after roughly 10 minutes, the VM simply stops. Launcing via qm start, it responds after 30secs with "start failed", but qm status keeps replying with "running". While running, the memory size is growing in a linear fashion up to around 180GB. I am passing through an NVIDIA GPU, and have 256GB of RAM in total.

I'm running proxmox on ZFS - might the ARC target size play into this at all?

 

[Possums]

I ended up using terminal commands to start the vm. Just go into the error log in the Proxmox web interface, and copy the command that times out; it should start with 'usr/bin/kvm.' My system only has 128GB of ram, so it is possible that you are experiencing a different issue. My VM's also run on a LVM disk, so I'm not the most familiar with ZFS.

 

[dumdum]

To have VM with high amount of RAM and to reduce the start up time, do the below.

1. Enable hugepages of 2MB

$ qm set {vmid} --hugepages 2

2. Remove "Memory" from hotplug option