Exclude IOMMU groups or PCIe devices from VM possible?

M

Muffel

New Member
May 22, 2024
7
0
1
Hi,
I just have created this test machine, and while I haven't added any USB controller, hub or device to it as hardware, I can see these devices within the VM using lspci. Same goes for the audio device, and potentially other one. Is there a way to exclude PCIe devices/IOMMU groups from being passed to a VM?

root@nas /mnt $% lspci
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Device 1234:1111 (rev 02)
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03)
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03)
00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03)
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03)
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
00:1c.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1c.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:03.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:04.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
06:08.0 Communication controller: Red Hat, Inc. Virtio console
06:0c.0 SCSI storage controller: Red Hat, Inc. Virtio block device
06:0d.0 SCSI storage controller: Red Hat, Inc. Virtio block device
06:0e.0 SCSI storage controller: Red Hat, Inc. Virtio block device
06:12.0 Ethernet controller: Red Hat, Inc. Virtio network device
09:01.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
 
Hi,

these are all just virtualized/emulated hardware devices and are expected to be present. Nothing of this is actual pass-through'd hardware, as that must be done explicitly.

I'd suggest reading upon the chapter QEMU/KVM Virtual Machines in our admin guide, that explains all that in depth :)
 
The problem is:

It doesn't matter if emulated or PT. As soon the hardware is visible in the VM, a driver is attached to it. And while Linux has some nice features to (re)enable power reduction settings, other systems don't have such features.

In my case Proxmox runs at 7W idle, with the hardware package down to C8 all the time, but as soon as I launch Opnsense as VM, power consumption goes up to 16W, and if I also launch a Debian VM, I am at 28W. Powertop on Proxmox shows the CPU to sit idle at C10, but the hardware access of the two VMs causes the HW package never to reach even C2.

And because I can't drop the two VMs, I am now looking in ways to limit the access to the underlying hardware. Hardware - no matter if emulated or PT - which is not under control of a VM - should act the same way as if the WM is not running. And as starter points I have I dentified the various USB connectors, Sound, and potentially more HW which I don't need in any VM and want to be excluded.

Here is the Debian VM again - and it definitely attaches drivers to all kind of resources...

root@nas /home/becks $% lspci -k
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
Subsystem: Red Hat, Inc. QEMU Virtual Machine
00:01.0 VGA compatible controller: Device 1234:1111 (rev 02)
Subsystem: Red Hat, Inc. Device 1100
Kernel driver in use: bochs-drm
Kernel modules: bochs
00:1a.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4 (rev 03)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Kernel driver in use: uhci_hcd
Kernel modules: uhci_hcd
00:1a.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5 (rev 03)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Kernel driver in use: uhci_hcd
Kernel modules: uhci_hcd
00:1a.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6 (rev 03)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Kernel driver in use: uhci_hcd
Kernel modules: uhci_hcd
00:1a.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2 (rev 03)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci
00:1b.0 Audio device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller (rev 03)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
00:1c.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
Subsystem: Red Hat, Inc. Device 0000
Kernel driver in use: pcieport
00:1c.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
Subsystem: Red Hat, Inc. Device 0000
Kernel driver in use: pcieport
00:1c.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
Subsystem: Red Hat, Inc. Device 0000
Kernel driver in use: pcieport
00:1c.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
Subsystem: Red Hat, Inc. Device 0000
Kernel driver in use: pcieport
00:1d.0 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1 (rev 03)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Kernel driver in use: uhci_hcd
Kernel modules: uhci_hcd
00:1d.1 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2 (rev 03)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Kernel driver in use: uhci_hcd
Kernel modules: uhci_hcd
00:1d.2 USB controller: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3 (rev 03)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Kernel driver in use: uhci_hcd
Kernel modules: uhci_hcd
00:1d.7 USB controller: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1 (rev 03)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Kernel driver in use: ehci-pci
Kernel modules: ehci_pci
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Kernel driver in use: lpc_ich
Kernel modules: lpc_ich
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Kernel driver in use: ahci
Kernel modules: ahci
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
Subsystem: Red Hat, Inc. QEMU Virtual Machine
Kernel driver in use: i801_smbus
Kernel modules: i2c_i801
05:01.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:02.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:03.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
05:04.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
06:08.0 Communication controller: Red Hat, Inc. Virtio console
Subsystem: Red Hat, Inc. Device 0003
Kernel driver in use: virtio-pci
06:0c.0 SCSI storage controller: Red Hat, Inc. Virtio block device
Subsystem: Red Hat, Inc. Device 0002
Kernel driver in use: virtio-pci
06:0d.0 SCSI storage controller: Red Hat, Inc. Virtio block device
Subsystem: Red Hat, Inc. Device 0002
Kernel driver in use: virtio-pci
06:0e.0 SCSI storage controller: Red Hat, Inc. Virtio block device
Subsystem: Red Hat, Inc. Device 0002
Kernel driver in use: virtio-pci
06:12.0 Ethernet controller: Red Hat, Inc. Virtio network device
Subsystem: Red Hat, Inc. Device 0001
Kernel driver in use: virtio-pci
09:01.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI
Subsystem: Red Hat, Inc. Device 0008
Kernel driver in use: virtio-pci
 
Here is another one:

Powertop, run within the VM. Despite running powertop --auto-tune the system keeps USB EHCI Controller #1,#2 and USB UHCI #1 to #6 at 100%. Why - I don't know. But I want to avoid fighting the same game whenever I add a new VM and try to put devices into sleep mode, while at the same time it would be easier to do this on Proxmox level and exclude unneeded hardware from the VMs.

PowerTOP 2.15 Overview Idle stats Frequency stats Device stats Tunables WakeUp
Usage Device name
8.6% CPU misc
8.6% CPU core
8.6% DRAM
4380 pkts/s Network interface: enp6s18 (virtio_net)
100.0% PCI Device: Red Hat, Inc. QEMU PCIe Root port
100.0% PCI Device: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #2
100.0% PCI Device: Red Hat, Inc. Virtio SCSI
100.0% PCI Device: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #4
100.0% PCI Device: Red Hat, Inc. Virtio network device
100.0% PCI Device: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #6
100.0% PCI Device: Intel Corporation 82801I (ICH9 Family) HD Audio Controller
100.0% PCI Device: Red Hat, Inc. QEMU PCIe Root port
100.0% PCI Device: Red Hat, Inc. QEMU PCI-PCI bridge
100.0% PCI Device: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3
100.0% PCI Device: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2
100.0% PCI Device: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1
100.0% PCI Device: Intel Corporation 82801IB (ICH9) LPC Interface Controller
100.0% PCI Device: Intel Corporation 82801 PCI Bridge
100.0% PCI Device: Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1
100.0% PCI Device: Red Hat, Inc. QEMU PCI-PCI bridge
100.0% PCI Device: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
100.0% PCI Device: Red Hat, Inc. QEMU PCIe Root port
100.0% PCI Device: Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #5
100.0% PCI Device: Red Hat, Inc. Virtio console
100.0% PCI Device: Red Hat, Inc. Virtio block device
100.0% PCI Device: Device 1234:1111
100.0% PCI Device: Red Hat, Inc. Virtio block device
100.0% PCI Device: Red Hat, Inc. Virtio block device
100.0% PCI Device: Red Hat, Inc. QEMU PCIe Root port
4.9 pkts/s nic:vethdd4e917
4.9 pkts/s nic:veth35065de
1.0 pkts/s nic:br-99f5c992c195
0.5 pkts/s nic:veth6954a7c
0.5 pkts/s nic:veth1a5935e
0.0 pkts/s nic:vethcadb334
0.0 pkts/s nic:veth65714bd
0.0 pkts/s nic:br-e517b37a6ffe
 
How about dropping all the virtualization and running all your stuff bare metal? Then you can enjoy power-efficiency.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back