Passed through GPU still consuming power when the VM is not started

[a9udn9u]

I recently bought a "smart" plug from Amazon for monitoring the power draw of my home server. It reports ~78w when the server is doing nothing, no VM nor container started, just Proxmox idling. I was expecting the idling power draw to be much lower, maybe 30-40w. I tried to measure individual component power consumption as much as I can and here are the breakdowns I got:

	Power Draw	Data Source
3.5" HDD x 1	3w	I sent a sleep command to the HDD using `hdparm`, power draw from wall dropped by 3w.
SATA SSD x 3 + NVMe x 2	10w	`hdparm` doesn't work on these drives, but they should consume less power than the HDD so I estimated 2w per drive.
Case and CPU fans x 5	<5w	I unplugged one case fan, it appeared to reduce less than 1w, but it's within the fluctuation.
CPU	<12w	This is the package wattage reported by `turbostat`, I have an Intel 12700K.
GPU	38w	I pulled the GPU out, total power draw dropped by 38w. It's a Sapphire Pulse RX 6700 XT
Total from wall	78w	Measured by KASA smart plug KP115

Other components consumes the rest 10 watts, I guess those are motherboard, RAM, motherboard integrated WiFi modules, etc.

GPU obviously pulled the most power, which is strange because I disabled my GPU in Proxmox so I can pass it to a VM (which is not started during my tests), and I noticed that the GPU fans keep spinning even though it's not doing anything, when I start the Windows VM, at idle those fans don't spin and the AMD software reports <5w power draw, I guess it's the Windows AMD driver doing the right thing by telling the GPU to take a rest when there's no work.

Is there any way to let GPU draw less power when it's not actually in use in the host OS?

 

[apoc]

GPU obviously pulled the most power, which is strange because I disabled my GPU in Proxmox...

That does not mean that the PCIe device isn't powered. It just means that the kernel / host OS does not grab it.
I would really be surprised if the card wouldn't pull power. 38W seems reasonable for an idle graphics card. Have you checked what the card pulls under full load?

According to the following page there is the technically possibility to disable a port, but it also says that a reboot is likely needed to re-enable it.
https://bbs.archlinux.org/viewtopic.php?id=222735

Bottom line - I am not sure if your wish it technically feasible.

 

[a9udn9u]

That does not mean that the PCIe device isn't powered. It just means that the kernel / host OS does not grab it.
I would really be surprised if the card wouldn't pull power. 38W seems reasonable for an idle graphics card. Have you checked what the card pulls under full load?

According to the following page there is the technically possibility to disable a port, but it also says that a reboot is likely needed to re-enable it.
https://bbs.archlinux.org/viewtopic.php?id=222735

Bottom line - I am not sure if your wish it technically feasible.

Thanks for replying!

I don't expect 0w power draw, but sub-10w like under Windows. In the Windows vm when AMD driver kicks in, the GPU fans stop spinning, this is when the GPU is used by the vm albeit litely. But when vm stopped, the fans start to spin although it's technically "disabled" in the host OS. There got to be something going on in the host OS that can be optimized, I just need to find where to start looking.

 

[leesteken]

Either return the GPU to the proper Linux driver (instead of vfio-pci) so it can idle the card, or run a small Windows VM with the same drivers. Proxmox does not handle this as you deliberately passed the device to a VM. but it can be arranged via hookscripts when your GPU resets properly. It might be easiest with least power draw to just keep the Windows VM running and doing nothing.

 

[a9udn9u]

Either return the GPU to the proper Linux driver (instead of vfio-pci) so it can idle the card, or run a small Windows VM with the same drivers. Proxmox does not handle this as you deliberately passed the device to a VM. but it can be arranged via hookscripts when your GPU resets properly. It might be easiest with least power draw to just keep the Windows VM running and doing nothing.

Helpful suggestion as always! I'll try out the small VM method tonight, I guess a small linux VM with proper driver also works? Those are easier to do than Windows IMO.

 

[leesteken]

I'll try out the small VM method tonight, I guess a small linux VM with proper driver also works? Those are easier to do than Windows IMO.

I agree since you have a AMD GPU (of a generation that should reset properly) and any small up-to-date Linux has drivers already included. NVidia would be more work.

 

[Dunuin]

Is there any way to let GPU draw less power when it's not actually in use in the host OS?

Some weeks ago I also created a small Debian VM with Nvidia drivers that I could run instead of my Win VMs when I don't need the GPU. But at least with my GPU I wasn't able to measure any power drop when booting that Debian VM with PCI pasthrough. But my GPU only got a TDP of 19W, so maybe that power save mode can't do much anyway.

 

[apoc]

I don't expect 0w power draw, but sub-10w like under Windows.

Imho In this Situation the system is in a different state. As you mention the driver is loaded and can control the card.
If the OS is not started it is more likely to be in the state as it is during POST and BIOS initialisation phase.
If you bring the system (host) to that state you will likely see the same power draw of the card.
Again from my perspective it is a wrong comparison as the situations are fundamentally different.

I am not sure what happens if you set the OS to sleep (from the OS level). I'd expect this to preserve the power state of the GPU. Maybe this is something you haven't tried yet?

 

[a9udn9u]

Some weeks ago I also created a small Debian VM with Nvidia drivers that I could run instead of my Win VMs when I don't need to GPU. But at least with my GPU I wasn't able to measure any power drop when booting that Debian VM with PCI pasthrough. But my GPU only got a TDP of 19W, so maybe that power save mode can't do much anyway.

I tried my GPU last night, my power consumption dropped by about 20w, the VM + idle GPU consumes about 18w now. Still higher than I'd like but the drop is nice. Some GPUs have low power mode (not sure what's the official term for it), maybe yours don't? Does your GPU fan spin at all at idle?

 

[a9udn9u]

I am not sure what happens if you set the OS to sleep (from the OS level). I'd expect this to preserve the power state of the GPU. Maybe this is something you haven't tried yet?

Interesting point! However I have to share my GPU in multiple VMs so I can't leave one VM on sleep while holding the GPU.

 

[Dunuin]

I tried my GPU last night, my power consumption dropped by about 20w, the VM + idle GPU consumes about 18w now. Still higher than I'd like but the drop is nice. Some GPUs have low power mode (not sure what's the official term for it), maybe yours don't? Does your GPU fan spin at all at idle?

It doesn't even got a fan, so hard to tell^^

 

[a9udn9u]

I was able to get another 11w back by not handling the Intel iGPU to VFIO.

blacklist i915

options vfio_pci ids=####:####

I had modprope config to disable the iGPU and accidentally discovered that if I leave it to the host OS, power consumption drops. I think it's the same exact issue, for both iGPU and dGPU, VFIO can not properly tell them to enter low power mode at idle.

 

[leesteken]

I had modprope config to disable the iGPU and accidentally discovered that if I leave it to the host OS, power consumption drops. I think it's the same exact issue, for both iGPU and dGPU, VFIO can not properly tell them to enter low power mode at idle.

As expected, VFIO does not know anything about the PCI(e) devices and is only for passthrough. The host drivers or the VM drivers for that specific device can put it in lower power mode. You can return the devices to the host by unbinding them (from vfio-pci) and binding them to their specific driver.

#!/bin/bash
function Unbind {
    if [ -e "/sys/bus/$2/devices/$1/driver/." ]
    then
        echo "Unbind $1 from bus $2..."
        echo "$1" > "/sys/bus/$2/devices/$1/driver/unbind" && sleep 3
    fi
}
function BindTo {
    echo "Bind $1 to driver $3 on bus $2..."
    echo "$1" >"/sys/bus/$2/drivers/$3/bind" && sleep 3
}
function RebindTo {
    Unbind "$1" "$2" && BindTo "$1" "$2" "$3"
}
GPU='0000:0b:00.0'
RebindTo "${GPU}" pci amdgpu

 

[klizso]

Ahogy az várható volt, a VFIO nem tud semmit a PCI(e) eszközökről, és csak áthaladásra szolgál. Az adott eszközhöz tartozó gazdagép illesztőprogramok vagy virtuális gép- illesztőprogramok alacsonyabb energiafogyasztású módba állíthatják. Az eszközöket visszaküldheti a gazdagépnek, ha megszünteti a kötődést (a vfio-pci-ből), és hozzárendeli őket az adott illesztőprogramhoz.

#!/bin/bash
function Unbind {
    if [ -e "/sys/bus/$2/devices/$1/driver/." ]
    akkor
        echo "Unbind $1 from bus $2..."
        echo "$1" > "/sys/bus/$2/devices/$1/driver/unbind" && sleep 3
    fi
}
function BindTo {
    echo "Bind $1 to driver $3 on bus $2..."
    echo "$1" >"/sys/bus/$2/drivers/$3/bind" && sleep 3
}
function RebindTo {
    "$1" "$2" kötés feloldása && kötés "$1" "$2" "$3"
}
GPU='0000:0b:00.0'
RebindTo "${GPU}" pci amdgpu

Ha jól értem, a fenti módszerrel a virtuális gép egyszerűen visszaadja a kártya irányítását a proxmoxnak?

Nagyon érdekelne ez a lehetőség, mert a gépben lévő plusz GPU-t csak 1 operációs rendszerhez szeretném használni (ami csak használatkor indul el és leáll).

Szóval szeretném, ha a kártya minél kevesebbet fogyasztana.