Unable to start ubuntu VM trying to passthrough a google coral TPU

epicurean

Member
Jun 21, 2021
44
1
13
54
I have a google coral tpu in wyse 5070 (CPU J4125) that I am trying to passthrough to ubuntu VM. The VM cannot start if I add the coral TPU.
The wyse is running proxmox 8.02
This is the error message
vfio-pci 0000:02:00.0: unable to change power state from D3cold to D0, device inaccessible

Any help appreciated! thanks
 
is this related to google coral TPU being a A+E M.2 or mini pcie format?
Probably not. Maybe it's your motherboard or motherboard BIOS? Or the device does not reset properly and maybe early binding to vfio-pci helps?
Alternatlively, run Ubuntu as a container and load the driver on the Proxmox host like some other people on this forum appear to have done?

EDIT: Maybe this thread can help you?
 
Last edited:
Just wanna update that I can at least get the ubuntu VM to boot by

by adding host kernel boot argument pcie_port_pm=off to /etc/default/grub
 
  • Like
Reactions: leesteken
the coral stuff is bit tricky.. there is few post about the way to pass on esxi, i think from 2-3 weeks ago. dig over there to see setting to have into your win vm. Of course if you new to passthrough, try with any other device first.
 
@epicurean, any luck? I am literally doing the same thing on Wyse 5070 and hitting the same issue.

I finally got it to boot up by adding this to kernel cmd line:

pcie_port_pm=off

This is my entire cmdline:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 modprobe.blacklist=apex,gasket vfio-pci.ids=1ac1:089a kvm.ignore_msrs=1 pcie_aspm=off pcie_port_pm=off"

I still need to see how much did the power draw increase but now my VMs see the TPU.
 
@epicurean, any luck? I am literally doing the same thing on Wyse 5070 and hitting the same issue.

I finally got it to boot up by adding this to kernel cmd line:

pcie_port_pm=off

This is my entire cmdline:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1 modprobe.blacklist=apex,gasket vfio-pci.ids=1ac1:089a kvm.ignore_msrs=1 pcie_aspm=off pcie_port_pm=off"

I still need to see how much did the power draw increase but now my VMs see the TPU.
OMG Thank you!!!

In my case :

GRUB_CMDLINE_LINUX="quiet intel_iommu=on pcie_aspm=off pcie_port_pm=off consoleblank=300"

and it finally works!
 
wow it works!!!!

I asked ChatGPT what does it mean it says:

In the GRUB configuration file for Proxmox, the parameter pcie_port_pm=off is a kernel boot option that controls the power management behavior of PCIe ports. Specifically, this setting disables the power management feature for PCIe ports.

What This Parameter Does:​

  • PCIe Port Power Management: Normally, PCIe devices can enter low-power modes when idle through power management, reducing power consumption, which is especially useful in mobile devices or energy-efficient servers.
  • Disabling Power Management: By adding pcie_port_pm=off to the GRUB configuration, you're instructing the Linux kernel not to automatically manage the power of PCIe ports. This results in the PCIe devices remaining fully powered and not transitioning into low-power states.

Why Use pcie_port_pm=off?​

In some cases, PCIe device power management can lead to instability, particularly in certain hardware combinations or high-performance environments. For example:
  1. Incompatible Hardware: Some PCIe devices may not fully support power management, potentially causing them to malfunction or fail to recover from low-power states.
  2. Server Stability: In server environments, stability is often prioritized over energy savings, so disabling power management can reduce the chance of errors or unexpected behavior.
  3. Performance Needs: In scenarios where maximum performance is required from PCIe devices, avoiding low-power states is a reasonable choice to ensure optimal device performance.

Application in Proxmox:​

Proxmox, as a virtualization platform, may host several PCIe devices like network cards, storage controllers, etc. Disabling power management prevents these devices from entering low-power modes, which could introduce latency or instability in virtualized environments.
This parameter is typically added to the GRUB_CMDLINE_LINUX_DEFAULT line in the /etc/default/grub file. After updating GRUB and rebooting the system, the setting takes effect.

Conclusion:​

pcie_port_pm=off disables power management for PCIe ports during boot. In environments like servers or virtualization platforms (e.g., Proxmox), this can help improve stability and performance, especially if certain hardware is incompatible with power-saving features. However, it may result in slightly higher power consumption.

4o
 
wow it works!!!!

I asked ChatGPT what does it mean it says:

In the GRUB configuration file for Proxmox, the parameter pcie_port_pm=off is a kernel boot option that controls the power management behavior of PCIe ports. Specifically, this setting disables the power management feature for PCIe ports.

What This Parameter Does:​

  • PCIe Port Power Management: Normally, PCIe devices can enter low-power modes when idle through power management, reducing power consumption, which is especially useful in mobile devices or energy-efficient servers.
  • Disabling Power Management: By adding pcie_port_pm=off to the GRUB configuration, you're instructing the Linux kernel not to automatically manage the power of PCIe ports. This results in the PCIe devices remaining fully powered and not transitioning into low-power states.

Why Use pcie_port_pm=off?​

In some cases, PCIe device power management can lead to instability, particularly in certain hardware combinations or high-performance environments. For example:
  1. Incompatible Hardware: Some PCIe devices may not fully support power management, potentially causing them to malfunction or fail to recover from low-power states.
  2. Server Stability: In server environments, stability is often prioritized over energy savings, so disabling power management can reduce the chance of errors or unexpected behavior.
  3. Performance Needs: In scenarios where maximum performance is required from PCIe devices, avoiding low-power states is a reasonable choice to ensure optimal device performance.

Application in Proxmox:​

Proxmox, as a virtualization platform, may host several PCIe devices like network cards, storage controllers, etc. Disabling power management prevents these devices from entering low-power modes, which could introduce latency or instability in virtualized environments.
This parameter is typically added to the GRUB_CMDLINE_LINUX_DEFAULT line in the /etc/default/grub file. After updating GRUB and rebooting the system, the setting takes effect.

Conclusion:​

pcie_port_pm=off disables power management for PCIe ports during boot. In environments like servers or virtualization platforms (e.g., Proxmox), this can help improve stability and performance, especially if certain hardware is incompatible with power-saving features. However, it may result in slightly higher power consumption.

4o
i don't follow .. You mean instead of reading the manual, you ask gpt to condense info, skip the detail, skip example and skip deep explanation ??
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!