QEMU Guest Agent not working with GPU passthrough.

[bluezeta]

Hi everyone.

So, I've have been having some issues with GPU passthrough on my Proxmox server. I did had it working beforehand and was working fine for a couple of weeks. Then, it started to not work for these past couple of hours. I don't what the cause exactly is and how to fix it. For whatever reason when I add in my GPU into one of the VM's, I started noticing that my ram usage spikes up and hangs there.



However, when I remove the GPU, this issue goes away. I've used this guide in order to get GPU Passthrough to work properly for me.
https://www.youtube.com/watch?v=S6jQx4AJlFw&t=332s

Here is what I have for /etc/default/grub. Afterwards I ran update-grub.

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt"
GRUB_CMDLINE_LINUX=""

/etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

/etc/modprobe.d/blacklist.conf.
blacklist.conf

blacklist radeon
blacklist nouveau
blacklist nvidia
blacklist nvidiafb

/etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1b80,10de:10f0 disable_vga=1

Then I did update-initramfs -u

/etc/pve/qemu-server/100.conf

agent: 1
bios: ovmf
boot: order=virtio0;ide2;net0;ide0
cores: 8
cpu: host,hidden=1,flags=+pcid
efidisk0: local-lvm:vm-100-disk-3,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:07:00,pcie=1
ide0: none,media=cdrom
ide2: none,media=cdrom
machine: pc-q35-6.2
memory: 16384
meta: creation-qemu=6.2.0,ctime=1664684829
name: Win11
net0: virtio=F6:9B:5A:3F:97:27,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: win11
scsihw: virtio-scsi-pci
smbios1: uuid=336adb50-43a5-4831-8585-32f0bc552a61
sockets: 1
tpmstate0: local-lvm:vm-100-disk-4,size=4M,version=v2.0
vga: none
virtio0: local-lvm:vm-100-disk-5,cache=writethrough,size=256G
vmgenid: f92b96a8-81e7-42cb-80b2-9aa00a7e3f22

I tried going over the tutorial once again to make sure I had everything setup properly. However, I did notice when using this line of code

dmesg | grep -e DMAR -e IOMMU

No output text appeared after I used that code. I know this means something is not working correctly, but I don't know what exactly. I checked if my bios settings was setup properly for virtualization since I had the IOMMU and SV-T enabled. I don't know if there is something I also have to setup within the bios. I have an ASROCK b450M Pro 4 if anyone knows how to properly setup virtualization on that motherboard.

 

[leesteken]

For whatever reason when I add in my GPU into one of the VM's, I started noticing that my ram usage spikes up and hangs there.

PCI(e) devices can do DMA from and to any part of the VM memory at any time without the CPU knowing about it. Therefore, all VM memory must be pinned in to actual host RAM. That's why ballooning won't work for VMs with passthrough.
To make Proxmox display the memory usage accoring to the VM, you need to have QEMU Guest Agent installed inside the VM and enabled in the VM setting in Proxmox.
I can see you enabled QEMU Guest Agent for the VM in Proxmox, but it is not running inside the VM. Maybe you did not install it yet, or maybe it had an update and did not restart properly, or maybe something else is wrong. Either way, Proxmox cannot show any less memory usage because it cannot get the memory usage data from the QEMU Guest Agent.

PS: Your VM is using just as much memory as before, PCIe passthrough appears to be working just as before, only the reporting display/graph in Proxmox is different (not broken) because QEMU Guest Agent is not running (see your screenshot). Unless something else is wrong, maybe you can change the title of this thread?

EDIT: I guess I misunderstood the issue.

 

[bluezeta]

PCI(e) devices can do DMA from and to any part of the VM memory at any time without the CPU knowing about it. Therefore, all VM memory must be pinned in to actual host RAM. That's why ballooning won't work for VMs with passthrough.
To make Proxmox display the memory usage accoring to the VM, you need to have QEMU Guest Agent installed inside the VM and enabled in the VM setting in Proxmox.
I can see you enabled QEMU Guest Agent for the VM in Proxmox, but it is not running inside the VM. Maybe you did not install it yet, or maybe it had an update and did not restart properly, or maybe something else is wrong. Either way, Proxmox cannot show any less memory usage because it cannot get the memory usage data from the QEMU Guest Agent.


PS: Your VM is using just as much memory as before, PCIe passthrough appears to be working just as before, only the reporting display/graph in Proxmox is different (not broken) because QEMU Guest Agent is not running (see your screenshot). Unless something else is wrong, maybe you can change the title of this thread?

Hello and thanks for the reply. Looks like what was causing the issue was my motherboard settings were reset.

 

[bluezeta]

PCI(e) devices can do DMA from and to any part of the VM memory at any time without the CPU knowing about it. Therefore, all VM memory must be pinned in to actual host RAM. That's why ballooning won't work for VMs with passthrough.
To make Proxmox display the memory usage accoring to the VM, you need to have QEMU Guest Agent installed inside the VM and enabled in the VM setting in Proxmox.
I can see you enabled QEMU Guest Agent for the VM in Proxmox, but it is not running inside the VM. Maybe you did not install it yet, or maybe it had an update and did not restart properly, or maybe something else is wrong. Either way, Proxmox cannot show any less memory usage because it cannot get the memory usage data from the QEMU Guest Agent.

PS: Your VM is using just as much memory as before, PCIe passthrough appears to be working just as before, only the reporting display/graph in Proxmox is different (not broken) because QEMU Guest Agent is not running (see your screenshot). Unless something else is wrong, maybe you can change the title of this thread?

EDIT: I guess I misunderstood the issue.

Hello again. The problem seems to be appearing again unexpectedly. I do have QEMU Guest Agent enabled in Proxmox in the Options tab. I've also installed the VirtIO drivers (virtio-win-0.1.225-2) on the windows 11 VM and restarted the machine. I had this VM working before with GPU detected, however after turning on the VM a second time, this issue seems to have appeared again.

Drivers I've installed on Windows VM
https://fedorapeople.org/groups/vir...ownloads/archive-virtio/virtio-win-0.1.225-2/

 

[leesteken]

Hello again. The problem seems to be appearing again unexpectedly. I do have QEMU Guest Agent enabled in Proxmox in the Options tab. I've also installed the VirtIO drivers (virtio-win-0.1.225-2) on the windows 11 VM and restarted the machine. I had this VM working before with GPU detected, however after turning on the VM a second time, this issue seems to have appeared again.

The issue of the memory usage/graph not corresponding with inside the VM and the QEMU Guest Agent not runnig, or passthrough not working?

Drivers I've installed on Windows VM
https://fedorapeople.org/groups/vir...ownloads/archive-virtio/virtio-win-0.1.225-2/

Did you install the QEMU Guest Agent? I think it's not (only) a driver but (also) a program that needs to be installed. (Sorry but I have little experience with Windows.)

 

[bluezeta]

The issue of the memory usage/graph not corresponding with inside the VM and the QEMU Guest Agent not runnig, or passthrough not working?

Did you install the QEMU Guest Agent? I think it's not (only) a driver but (also) a program that needs to be installed. (Sorry but I have little experience with Windows.)

Whenever I add my GPU to the VM, the QEMU Guest Agent and memory usage/graph problem shows up. However if I remove my graphics card from the VM, then the problem disappears.

Looks like I originally didn't have the QEMU Guest Agent installed. But even after I've installed it, the issue still seems to pop up.

PS C:\Users\Jacob> Get-Service QEMU-GA

Status   Name               DisplayName
------   ----               -----------
Running  QEMU-GA            QEMU Guest Agent

 

[bluezeta]

Update on the issue. I found this forum post that had a similar issue to mine.
https://forum.proxmox.com/threads/p...twork-when-pci-passthrough-is-enabled.109013/
Looks like it has something to do Linux kernel (5.15.30-2-pve). Therefore I to use Linux kernel 5.13.19-6-pve and that has seem to resolve my issue.

 

[Neobin]

5.15.30-2-pve

This is the initial version from the 7.2-ISO; so it looks to me, that you never (properly) updated that PVE-host. See here: [1] how to.

After you did that and in the case that the most recent 5.15 kernel does not already fix your problem, you could also try the opt-in 5.19 kernel: [2].

[1] https://forum.proxmox.com/threads/im-unable-to-upload-files-to-my-proxmox-server.114541/#post-495356
[2] https://forum.proxmox.com/threads/opt-in-linux-5-19-kernel-for-proxmox-ve-7-x-available.115090