Passed through GPU still consuming power when the VM is not started

a9udn9u

New Member
Nov 9, 2022
12
2
3
I recently bought a "smart" plug from Amazon for monitoring the power draw of my home server. It reports ~78w when the server is doing nothing, no VM nor container started, just Proxmox idling. I was expecting the idling power draw to be much lower, maybe 30-40w. I tried to measure individual component power consumption as much as I can and here are the breakdowns I got:

Power DrawData Source
3.5" HDD x 13wI sent a sleep command to the HDD using `hdparm`, power draw from wall dropped by 3w.
SATA SSD x 3 + NVMe x 210w`hdparm` doesn't work on these drives, but they should consume less power than the HDD so I estimated 2w per drive.
Case and CPU fans x 5<5wI unplugged one case fan, it appeared to reduce less than 1w, but it's within the fluctuation.
CPU<12wThis is the package wattage reported by `turbostat`, I have an Intel 12700K.
GPU38wI pulled the GPU out, total power draw dropped by 38w. It's a Sapphire Pulse RX 6700 XT
Total from wall78wMeasured by KASA smart plug KP115

Other components consumes the rest 10 watts, I guess those are motherboard, RAM, motherboard integrated WiFi modules, etc.

GPU obviously pulled the most power, which is strange because I disabled my GPU in Proxmox so I can pass it to a VM (which is not started during my tests), and I noticed that the GPU fans keep spinning even though it's not doing anything, when I start the Windows VM, at idle those fans don't spin and the AMD software reports <5w power draw, I guess it's the Windows AMD driver doing the right thing by telling the GPU to take a rest when there's no work.

Is there any way to let GPU draw less power when it's not actually in use in the host OS?
 
Last edited:

apoc

Famous Member
Oct 13, 2017
1,012
148
88
GPU obviously pulled the most power, which is strange because I disabled my GPU in Proxmox...
That does not mean that the PCIe device isn't powered. It just means that the kernel / host OS does not grab it.
I would really be surprised if the card wouldn't pull power. 38W seems reasonable for an idle graphics card. Have you checked what the card pulls under full load?

According to the following page there is the technically possibility to disable a port, but it also says that a reboot is likely needed to re-enable it.
https://bbs.archlinux.org/viewtopic.php?id=222735

Bottom line - I am not sure if your wish it technically feasible.
 

a9udn9u

New Member
Nov 9, 2022
12
2
3
That does not mean that the PCIe device isn't powered. It just means that the kernel / host OS does not grab it.
I would really be surprised if the card wouldn't pull power. 38W seems reasonable for an idle graphics card. Have you checked what the card pulls under full load?

According to the following page there is the technically possibility to disable a port, but it also says that a reboot is likely needed to re-enable it.
https://bbs.archlinux.org/viewtopic.php?id=222735

Bottom line - I am not sure if your wish it technically feasible.
Thanks for replying!

I don't expect 0w power draw, but sub-10w like under Windows. In the Windows vm when AMD driver kicks in, the GPU fans stop spinning, this is when the GPU is used by the vm albeit litely. But when vm stopped, the fans start to spin although it's technically "disabled" in the host OS. There got to be something going on in the host OS that can be optimized, I just need to find where to start looking.
 

leesteken

Famous Member
May 31, 2020
2,655
577
118
Either return the GPU to the proper Linux driver (instead of vfio-pci) so it can idle the card, or run a small Windows VM with the same drivers. Proxmox does not handle this as you deliberately passed the device to a VM. but it can be arranged via hookscripts when your GPU resets properly. It might be easiest with least power draw to just keep the Windows VM running and doing nothing.
 
Last edited:

a9udn9u

New Member
Nov 9, 2022
12
2
3
Either return the GPU to the proper Linux driver (instead of vfio-pci) so it can idle the card, or run a small Windows VM with the same drivers. Proxmox does not handle this as you deliberately passed the device to a VM. but it can be arranged via hookscripts when your GPU resets properly. It might be easiest with least power draw to just keep the Windows VM running and doing nothing.
Helpful suggestion as always! I'll try out the small VM method tonight, I guess a small linux VM with proper driver also works? Those are easier to do than Windows IMO.
 

leesteken

Famous Member
May 31, 2020
2,655
577
118
I'll try out the small VM method tonight, I guess a small linux VM with proper driver also works? Those are easier to do than Windows IMO.
I agree since you have a AMD GPU (of a generation that should reset properly) and any small up-to-date Linux has drivers already included. NVidia would be more work.
 

Dunuin

Famous Member
Jun 30, 2020
9,006
2,313
156
Germany
Is there any way to let GPU draw less power when it's not actually in use in the host OS?
Some weeks ago I also created a small Debian VM with Nvidia drivers that I could run instead of my Win VMs when I don't need the GPU. But at least with my GPU I wasn't able to measure any power drop when booting that Debian VM with PCI pasthrough. But my GPU only got a TDP of 19W, so maybe that power save mode can't do much anyway.
 
Last edited:

apoc

Famous Member
Oct 13, 2017
1,012
148
88
I don't expect 0w power draw, but sub-10w like under Windows.
Imho In this Situation the system is in a different state. As you mention the driver is loaded and can control the card.
If the OS is not started it is more likely to be in the state as it is during POST and BIOS initialisation phase.
If you bring the system (host) to that state you will likely see the same power draw of the card.
Again from my perspective it is a wrong comparison as the situations are fundamentally different.

I am not sure what happens if you set the OS to sleep (from the OS level). I'd expect this to preserve the power state of the GPU. Maybe this is something you haven't tried yet?
 

a9udn9u

New Member
Nov 9, 2022
12
2
3
Some weeks ago I also created a small Debian VM with Nvidia drivers that I could run instead of my Win VMs when I don't need to GPU. But at least with my GPU I wasn't able to measure any power drop when booting that Debian VM with PCI pasthrough. But my GPU only got a TDP of 19W, so maybe that power save mode can't do much anyway.
I tried my GPU last night, my power consumption dropped by about 20w, the VM + idle GPU consumes about 18w now. Still higher than I'd like but the drop is nice. Some GPUs have low power mode (not sure what's the official term for it), maybe yours don't? Does your GPU fan spin at all at idle?
 

a9udn9u

New Member
Nov 9, 2022
12
2
3
I am not sure what happens if you set the OS to sleep (from the OS level). I'd expect this to preserve the power state of the GPU. Maybe this is something you haven't tried yet?
Interesting point! However I have to share my GPU in multiple VMs so I can't leave one VM on sleep while holding the GPU.
 

Dunuin

Famous Member
Jun 30, 2020
9,006
2,313
156
Germany
I tried my GPU last night, my power consumption dropped by about 20w, the VM + idle GPU consumes about 18w now. Still higher than I'd like but the drop is nice. Some GPUs have low power mode (not sure what's the official term for it), maybe yours don't? Does your GPU fan spin at all at idle?
It doesn't even got a fan, so hard to tell^^
 

a9udn9u

New Member
Nov 9, 2022
12
2
3
I was able to get another 11w back by not handling the Intel iGPU to VFIO.

Code:
blacklist i915

options vfio_pci ids=####:####

I had modprope config to disable the iGPU and accidentally discovered that if I leave it to the host OS, power consumption drops. I think it's the same exact issue, for both iGPU and dGPU, VFIO can not properly tell them to enter low power mode at idle.
 

leesteken

Famous Member
May 31, 2020
2,655
577
118
I had modprope config to disable the iGPU and accidentally discovered that if I leave it to the host OS, power consumption drops. I think it's the same exact issue, for both iGPU and dGPU, VFIO can not properly tell them to enter low power mode at idle.
As expected, VFIO does not know anything about the PCI(e) devices and is only for passthrough. The host drivers or the VM drivers for that specific device can put it in lower power mode. You can return the devices to the host by unbinding them (from vfio-pci) and binding them to their specific driver.
Bash:
#!/bin/bash
function Unbind {
    if [ -e "/sys/bus/$2/devices/$1/driver/." ]
    then
        echo "Unbind $1 from bus $2..."
        echo "$1" > "/sys/bus/$2/devices/$1/driver/unbind" && sleep 3
    fi
}
function BindTo {
    echo "Bind $1 to driver $3 on bus $2..."
    echo "$1" >"/sys/bus/$2/drivers/$3/bind" && sleep 3
}
function RebindTo {
    Unbind "$1" "$2" && BindTo "$1" "$2" "$3"
}
GPU='0000:0b:00.0'
RebindTo "${GPU}" pci amdgpu
 
  • Like
Reactions: Neobin and Dunuin

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!