How to use one GPU as passthrough and another identical GPU in LXCs?

8192K · Oct 16, 2025

I have two exactly identical GPUs (Nvidia). I'd like to pass one through to a VM and use the other one in several LXCs. In order to use passthrough, I have to blacklist the nvidia/nouveau drivers rendering the LXC one unusable.

How can I achieve this?

uzumo · Oct 16, 2025

With Radeon, you don't need to add it to the blacklist if you unbind the driver, but is that not possible with NVIDIA?

I don't have one, so I can't test it or provide guidance.

8192K · Oct 16, 2025

Can you be more precise, please? Radeon passthrough works without adding the driver to the blacklist? I don't think so, honestly, as I had a Radeon in there until recently.
What exactly do you mean by "unbind"?

uzumo · Oct 16, 2025

8192K said:
What exactly do you mean by "unbind"?

Code:

echo "0000:04:00.0" > /sys/bus/pci/drivers/"driver name"/unbind 2>/dev/null

This refers to using a command to remove the driver “driver name” bound to pci “0000:04:00.0”.

A blacklist prevents the driver from being used from the start, whereas a command initially allows the device to operate with the driver. The command is then executed to remove the driver from a specific PCI device afterward.

Therefore, you can remove the driver from a specific device without adding it to the blacklist.

U

Post in thread 'GPU Passthrough For Windows 11 Not Working (kvm: vfio: Unable to power on device, stuck in D3)'

Aug 1, 2025

I have been using it for several months with the following settings and have not encountered any problems.

Due to a bug in vfio, the latest kernel is required.

It is not necessary to specify vBIOS. Perhaps it is not a ROM that should be specified.
Specifying disable_idle_d3=1 will cause the problem to worsen.

There will be no problems even if you restart or shut down the virtual machine.
When the virtual machine is stopped, amdgpu is used, but when it is started, vfio-pci is used.

・intel (Please do not include unnecessary options)
sed -i '1s/$/ quiet iommu=pt...

0000:04:00.0 and amdgpu according to your environment.

*amdgpu is the driver name.
*The value to be specified can be confirmed using lspci -nnk.

If audio devices require similar processing, you can add that line.

Code:

mkdir -p /var/lib/vz/snippets
nano /var/lib/vz/snippets/gpu_reset.sh
---
#!/bin/bash

phase="$2"
echo "Phase is $phase"
if [ "$phase" == "pre-start" ]; then
echo "0000:04:00.0" > /sys/bus/pci/drivers/amdgpu/unbind 2>/dev/null
sleep 2
echo 8 > /sys/bus/pci/devices/0000:04:00.0/resource2_resize
sleep 2
elif [ "$phase" == "post-stop" ]; then
sleep 5
echo "0000:04:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind 2>/dev/null
sleep 2
echo "0000:04:00.0" > /sys/bus/pci/drivers/amdgpu/bind 2>/dev/null
sleep 2
fi
---
chmod +x /var/lib/vz/snippets/gpu_reset.
qm set VMID -hookscript local:snippets/gpu_reset.sh

All settings added for the following PCI passthrough must be removed.

blacklist XXX
/etc/modprobe.d/vfio.conf
grub kernel parameters

8192K said:
Radeon passthrough works without adding the driver to the blacklist?

I'm running this script on the following hardware:

Radeon RX 9070 XT
Ryzen integrated GPU
Intel integrated GPU

Therefore, I don't think NVIDIA can't do the same thing.

8192K said:
I don't think so, honestly, as I had a Radeon in there until recently

The person who created the script in the first thread, as well as those who posted after the aforementioned thread, are functioning similarly.

9qwrog1y · Oct 24, 2025

uzumo said:
Code:

echo "0000:04:00.0" > /sys/bus/pci/drivers/"driver name"/unbind 2>/dev/null

This refers to using a command to remove the driver “driver name” bound to pci “0000:04:00.0”.

A blacklist prevents the driver from being used from the start, whereas a command initially allows the device to operate with the driver. The command is then executed to remove the driver from a specific PCI device afterward.

Therefore, you can remove the driver from a specific device without adding it to the blacklist.

U

Post in thread 'GPU Passthrough For Windows 11 Not Working (kvm: vfio: Unable to power on device, stuck in D3)'

Aug 1, 2025

I have been using it for several months with the following settings and have not encountered any problems.

Due to a bug in vfio, the latest kernel is required.

It is not necessary to specify vBIOS. Perhaps it is not a ROM that should be specified.
Specifying disable_idle_d3=1 will cause the problem to worsen.

There will be no problems even if you restart or shut down the virtual machine.
When the virtual machine is stopped, amdgpu is used, but when it is started, vfio-pci is used.

・intel (Please do not include unnecessary options)
sed -i '1s/$/ quiet iommu=pt...

Click to expand...

uzumo

0000:04:00.0 and amdgpu according to your environment.

*amdgpu is the driver name.
*The value to be specified can be confirmed using lspci -nnk.

If audio devices require similar processing, you can add that line.

Code:

mkdir -p /var/lib/vz/snippets nano /var/lib/vz/snippets/gpu_reset.sh --- #!/bin/bash phase="$2" echo "Phase is $phase" if [ "$phase" == "pre-start" ]; then echo "0000:04:00.0" > /sys/bus/pci/drivers/amdgpu/unbind 2>/dev/null sleep 2 echo 8 > /sys/bus/pci/devices/0000:04:00.0/resource2_resize sleep 2 elif [ "$phase" == "post-stop" ]; then sleep 5 echo "0000:04:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind 2>/dev/null sleep 2 echo "0000:04:00.0" > /sys/bus/pci/drivers/amdgpu/bind 2>/dev/null sleep 2 fi --- chmod +x /var/lib/vz/snippets/gpu_reset. qm set VMID -hookscript local:snippets/gpu_reset.sh

All settings added for the following PCI passthrough must be removed.

blacklist XXX
/etc/modprobe.d/vfio.conf
grub kernel parameters

I'm running this script on the following hardware:

Radeon RX 9070 XT
Ryzen integrated GPU
Intel integrated GPU

Therefore, I don't think NVIDIA can't do the same thing.

The person who created the script in the first thread, as well as those who posted after the aforementioned thread, are functioning similarly.

hi
This script cannot execute successfully — it seems to hang during the echo command.

root@pve:~# fuser /var/lock/qemu-server/lock-124.conf
/run/lock/qemu-server/lock-124.conf: 15247
root@pve:~# ps -ef | grep 15247
root 15247 2637 5 17:33 ? 00:00:00 /usr/bin/perl /usr/sbin/qm cleanup 124 1 1
root 15251 15247 0 17:33 ? 00:00:00 /bin/bash /var/lib/vz/snippets/gpu_reset.sh 124 post-stop
root 15289 4432 0 17:33 pts/0 00:00:00 grep 15247

root@pve:~# cat /var/lib/vz/snippets/gpu_reset.sh
#!/bin/bash

phase="$2"
echo "Phase is $phase"
if [ "$phase" == "pre-start" ]; then
echo "0000:da:00.0" > /sys/bus/pci/drivers/amdgpu/unbind 2>/dev/null
sleep 2
echo 8 > /sys/bus/pci/devices/0000:da:00.0/resource2_resize
sleep 2
elif [ "$phase" == "post-stop" ]; then
sleep 5
echo "0000:da:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind 2>/dev/null
sleep 2
echo "0000:da:00.0" > /sys/bus/pci/drivers/amdgpu/bind 2>/dev/null >>>>>>>>>>>>>>>hanging
sleep 2
fi

What else can I do / What other areas can I adjust?

uzumo · Oct 24, 2025

I believe I heard you were using NVIDIA., may I understand that this refers to the settings when using Radeon?

I suspect you're getting an error when reverting to the amdgpu driver that binds at OS startup. If you don't need the display after stopping the VM, you might consider removing the line below.

Does dmesg show anything?

Code:

echo "0000:da:00.0" > /sys/bus/pci/drivers/amdgpu/bind 2>/dev/null
sleep 2

There were also instances where scripts were used with NVIDIA, but I don't know the details.

A

Post in thread 'Passthrough RTX 6000/5090 CPU Soft BUG lockup, D3cold to D0, after guest shutdown'

Jul 17, 2025

tytanick said:
@adolfotregosa oh i get it, i thought it was requirement to get it stable. Fow low idle yeah that os ok. I will be looking into switching between nvidia and vfio but honestly first thing that came to my mind is to fire up vm with nvidia driver istead of messing in proxmox.
Do you have some ready solition that unbinds nvidia and binds vfio automatically when new vm need some gpus ?
I will look into making some scripts but maybe we can use the reset script that was needed to reset gpu state after vm shutdown for older gpus.

Have a look at the attachment:

My notes tell me...

9qwrog1y · Oct 24, 2025

uzumo said:
I believe I heard you were using NVIDIA., may I understand that this refers to the settings when using Radeon?

I suspect you're getting an error when reverting to the amdgpu driver that binds at OS startup. If you don't need the display after stopping the VM, you might consider removing the line below.

Does dmesg show anything?

Code:

echo "0000:da:00.0" > /sys/bus/pci/drivers/amdgpu/bind 2>/dev/null sleep 2

There were also instances where scripts were used with NVIDIA, but I don't know the details.

A

Post in thread 'Passthrough RTX 6000/5090 CPU Soft BUG lockup, D3cold to D0, after guest shutdown'

Jul 17, 2025

tytanick said:

@adolfotregosa oh i get it, i thought it was requirement to get it stable. Fow low idle yeah that os ok. I will be looking into switching between nvidia and vfio but honestly first thing that came to my mind is to fire up vm with nvidia driver istead of messing in proxmox.
Do you have some ready solition that unbinds nvidia and binds vfio automatically when new vm need some gpus ?
I will look into making some scripts but maybe we can use the reset script that was needed to reset gpu state after vm shutdown for older gpus.

Click to expand...

Have a look at the attachment:

My notes tell me...

adolfotregosa

hi
i use AMD 9070xt

Code:

da:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 [Radeon RX 9070/9070 XT/9070 GRE] (rev c0)
        Subsystem: Sapphire Technology Limited Navi 48 XTX [Sapphire Pulse Radeon RX 9070 XT]
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
da:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 48 HDMI/DP Audio Controller
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

qm start dmesg

Code:

[Fri Oct 24 19:06:43 2025] amdgpu 0000:da:00.0: amdgpu: amdgpu: finishing device.
[Fri Oct 24 19:06:43 2025] [drm] pre_validate_dsc:1627 MST_DSC dsc precompute is not needed
[Fri Oct 24 19:06:44 2025] [drm] amdgpu: ttm finalized
[Fri Oct 24 19:06:46 2025] pci 0000:da:00.0: BAR 0 [mem 0x80000000000-0x803ffffffff 64bit pref]: releasing
[Fri Oct 24 19:06:46 2025] pci 0000:da:00.0: BAR 2 [mem 0x80400000000-0x804001fffff 64bit pref]: releasing
[Fri Oct 24 19:06:46 2025] pcieport 0000:d9:00.0: bridge window [mem 0x80000000000-0x805ffffffff 64bit pref]: releasing
[Fri Oct 24 19:06:46 2025] pcieport 0000:d8:00.0: bridge window [mem 0x80000000000-0x805ffffffff 64bit pref]: releasing
[Fri Oct 24 19:06:46 2025] pcieport 0000:d7:00.0: bridge window [mem 0x80000000000-0x805ffffffff 64bit pref]: releasing
[Fri Oct 24 19:06:46 2025] pcieport 0000:d7:00.0: bridge window [mem 0x80000000000-0x805ffffffff 64bit pref]: assigned
[Fri Oct 24 19:06:46 2025] pcieport 0000:d8:00.0: bridge window [mem 0x80000000000-0x805ffffffff 64bit pref]: assigned
[Fri Oct 24 19:06:46 2025] pcieport 0000:d9:00.0: bridge window [mem 0x80000000000-0x805ffffffff 64bit pref]: assigned
[Fri Oct 24 19:06:46 2025] pci 0000:da:00.0: BAR 0 [mem 0x80000000000-0x803ffffffff 64bit pref]: assigned
[Fri Oct 24 19:06:46 2025] pci 0000:da:00.0: BAR 2 [mem 0x80400000000-0x8040fffffff 64bit pref]: assigned
[Fri Oct 24 19:06:46 2025] pcieport 0000:d7:00.0: PCI bridge to [bus d8-da]
[Fri Oct 24 19:06:46 2025] pcieport 0000:d7:00.0:   bridge window [io  0xf000-0xffff]
[Fri Oct 24 19:06:46 2025] pcieport 0000:d7:00.0:   bridge window [mem 0xfbd00000-0xfbefffff]
[Fri Oct 24 19:06:46 2025] pcieport 0000:d7:00.0:   bridge window [mem 0x80000000000-0x805ffffffff 64bit pref]
[Fri Oct 24 19:06:46 2025] pcieport 0000:d8:00.0: PCI bridge to [bus d9-da]
[Fri Oct 24 19:06:46 2025] pcieport 0000:d8:00.0:   bridge window [io  0xf000-0xffff]
[Fri Oct 24 19:06:46 2025] pcieport 0000:d8:00.0:   bridge window [mem 0xfbd00000-0xfbdfffff]
[Fri Oct 24 19:06:46 2025] pcieport 0000:d8:00.0:   bridge window [mem 0x80000000000-0x805ffffffff 64bit pref]
[Fri Oct 24 19:06:46 2025] pcieport 0000:d9:00.0: PCI bridge to [bus da]
[Fri Oct 24 19:06:46 2025] pcieport 0000:d9:00.0:   bridge window [io  0xf000-0xffff]
[Fri Oct 24 19:06:46 2025] pcieport 0000:d9:00.0:   bridge window [mem 0xfbd00000-0xfbdfffff]
[Fri Oct 24 19:06:46 2025] pcieport 0000:d9:00.0:   bridge window [mem 0x80000000000-0x805ffffffff 64bit pref]
[Fri Oct 24 19:06:48 2025] VFIO - User Level meta-driver version: 0.3
[Fri Oct 24 19:06:48 2025] vfio-pci 0000:da:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[Fri Oct 24 19:06:48 2025] vfio-pci 0000:da:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[Fri Oct 24 19:06:48 2025] vfio-pci 0000:da:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[Fri Oct 24 19:06:48 2025] vfio-pci 0000:da:00.0: resetting
[Fri Oct 24 19:06:48 2025] vfio-pci 0000:da:00.0: reset done
[Fri Oct 24 19:06:48 2025] kauditd_printk_skb: 115 callbacks suppressed
[Fri Oct 24 19:06:48 2025] audit: type=1400 audit(1761304008.231:127): apparmor="DENIED" operation="capable" class="cap" profile="swtpm" pid=21920 comm="swtpm" capability=21  capname="sys_admin"
[Fri Oct 24 19:06:48 2025] tap124i0: entered promiscuous mode
[Fri Oct 24 19:06:49 2025] vmbr0: port 2(fwpr124p0) entered blocking state
[Fri Oct 24 19:06:49 2025] vmbr0: port 2(fwpr124p0) entered disabled state
[Fri Oct 24 19:06:49 2025] fwpr124p0: entered allmulticast mode
[Fri Oct 24 19:06:49 2025] fwpr124p0: entered promiscuous mode
[Fri Oct 24 19:06:49 2025] vmbr0: port 2(fwpr124p0) entered blocking state
[Fri Oct 24 19:06:49 2025] vmbr0: port 2(fwpr124p0) entered forwarding state
[Fri Oct 24 19:06:49 2025] fwbr124i0: port 1(fwln124i0) entered blocking state
[Fri Oct 24 19:06:49 2025] fwbr124i0: port 1(fwln124i0) entered disabled state
[Fri Oct 24 19:06:49 2025] fwln124i0: entered allmulticast mode
[Fri Oct 24 19:06:49 2025] fwln124i0: entered promiscuous mode
[Fri Oct 24 19:06:49 2025] fwbr124i0: port 1(fwln124i0) entered blocking state
[Fri Oct 24 19:06:49 2025] fwbr124i0: port 1(fwln124i0) entered forwarding state
[Fri Oct 24 19:06:49 2025] fwbr124i0: port 2(tap124i0) entered blocking state
[Fri Oct 24 19:06:49 2025] fwbr124i0: port 2(tap124i0) entered disabled state
[Fri Oct 24 19:06:49 2025] tap124i0: entered allmulticast mode
[Fri Oct 24 19:06:49 2025] fwbr124i0: port 2(tap124i0) entered blocking state
[Fri Oct 24 19:06:49 2025] fwbr124i0: port 2(tap124i0) entered forwarding state
[Fri Oct 24 19:06:52 2025] vfio-pci 0000:da:00.0: resetting
[Fri Oct 24 19:06:52 2025] vfio-pci 0000:da:00.0: reset done
[Fri Oct 24 19:06:52 2025] vfio-pci 0000:da:00.0: resetting
[Fri Oct 24 19:06:52 2025] vfio-pci 0000:da:00.1: resetting
[Fri Oct 24 19:06:52 2025] vfio-pci 0000:da:00.0: reset done
[Fri Oct 24 19:06:52 2025] vfio-pci 0000:da:00.1: reset done

qm shutdown dmesg

Code:

[Fri Oct 24 19:08:09 2025]  zd1072: p1 p2 p3 p4
[Fri Oct 24 19:08:09 2025]  zd176: p1 p2
[Fri Oct 24 19:08:09 2025]  zd144: p1 p2
[Fri Oct 24 19:08:09 2025] tap124i0: left allmulticast mode
[Fri Oct 24 19:08:09 2025] fwbr124i0: port 2(tap124i0) entered disabled state
[Fri Oct 24 19:08:09 2025] fwbr124i0: port 1(fwln124i0) entered disabled state
[Fri Oct 24 19:08:09 2025] vmbr0: port 2(fwpr124p0) entered disabled state
[Fri Oct 24 19:08:09 2025] fwln124i0 (unregistering): left allmulticast mode
[Fri Oct 24 19:08:09 2025] fwln124i0 (unregistering): left promiscuous mode
[Fri Oct 24 19:08:09 2025] fwbr124i0: port 1(fwln124i0) entered disabled state
[Fri Oct 24 19:08:09 2025] fwpr124p0 (unregistering): left allmulticast mode
[Fri Oct 24 19:08:09 2025] fwpr124p0 (unregistering): left promiscuous mode
[Fri Oct 24 19:08:09 2025] vmbr0: port 2(fwpr124p0) entered disabled state
[Fri Oct 24 19:08:09 2025] vfio-pci 0000:da:00.0: resetting
[Fri Oct 24 19:08:09 2025] vfio-pci 0000:da:00.1: resetting
[Fri Oct 24 19:08:10 2025] pcieport 0000:d9:00.0: broken device, retraining non-functional downstream link at 2.5GT/s
[Fri Oct 24 19:08:11 2025] pcieport 0000:d9:00.0: retraining failed
[Fri Oct 24 19:08:11 2025] ------------[ cut here ]------------
[Fri Oct 24 19:08:11 2025] WARNING: CPU: 12 PID: 21925 at drivers/pci/pcie/bwctrl.c:168 pcie_set_target_speed+0x1ad/0x280
[Fri Oct 24 19:08:11 2025] Modules linked in: veth vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd tcp_diag inet_diag ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables iptable_filter nf_tables softdog sunrpc binfmt_misc bonding tls nfnetlink_log intel_rapl_msr intel_rapl_common intel_uncore_frequency intel_uncore_frequency_common skx_edac skx_edac_common nfit x86_pkg_temp_thermal intel_powerclamp ipmi_ssif coretemp amdgpu kvm_intel kvm amdxcp gpu_sched snd_hda_codec_hdmi drm_panel_backlight_quirks drm_buddy irdma irqbypass drm_ttm_helper snd_hda_intel polyval_clmulni ttm snd_intel_dspcfg polyval_generic snd_intel_sdw_acpi drm_exec ghash_clmulni_intel drm_suballoc_helper snd_hda_codec ice sha256_ssse3 drm_display_helper sha1_ssse3 snd_hda_core aesni_intel cec snd_hwdep gnss crypto_simd snd_pcm cmdlinepart ib_uverbs cryptd rc_core snd_timer video spi_nor rapl snd ast ib_core sch_fq_codel soundcore intel_cstate ses mei_me mtd i2c_algo_bit enclosure pcspkr scsi_transport_sas mei intel_pch_thermal
[Fri Oct 24 19:08:11 2025]  joydev acpi_power_meter input_leds ipmi_si acpi_ipmi ipmi_devintf mac_hid ipmi_msghandler acpi_pad zfs(PO) spl(O) vhost_net vhost vhost_iotlb tap efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic xor raid6_pq usbmouse hid_generic usbkbd usbhid hid dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio i40e xhci_pci libie i2c_i801 spi_intel_pci i2c_mux xhci_hcd ahci megaraid_sas spi_intel i2c_smbus ioatdma lpc_ich libahci wmi dca
[Fri Oct 24 19:08:11 2025] CPU: 12 UID: 0 PID: 21925 Comm: kvm Tainted: P           O       6.14.11-4-pve #1
[Fri Oct 24 19:08:11 2025] Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE
[Fri Oct 24 19:08:11 2025] Hardware name: O.E.M VA-FD-DS1001-205/, BIOS 4.3 04/19/2024
[Fri Oct 24 19:08:11 2025] RIP: 0010:pcie_set_target_speed+0x1ad/0x280
[Fri Oct 24 19:08:11 2025] Code: 0f b6 75 d4 4c 89 e7 e8 b1 d8 fe ff 85 c0 78 74 49 8b 7c 24 18 48 85 ff 74 05 e8 ce 08 fe ff 41 bc 01 00 00 00 e9 75 ff ff ff <0f> 0b bb ea ff ff ff eb b0 89 f0 b9 3f 00 00 00 49 c7 c7 ff ff ff
[Fri Oct 24 19:08:11 2025] RSP: 0018:ffffcf583739f8e8 EFLAGS: 00010202
[Fri Oct 24 19:08:11 2025] RAX: 00000000000000eb RBX: ffff8c718a39d000 RCX: 0000000000000000
[Fri Oct 24 19:08:11 2025] RDX: 0000000000000001 RSI: 00000000000000ff RDI: ffff8c718a39d000
[Fri Oct 24 19:08:11 2025] RBP: ffffcf583739f918 R08: 0000000000000000 R09: 0000000000000000
[Fri Oct 24 19:08:11 2025] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000045
[Fri Oct 24 19:08:11 2025] R13: ffffcf583739f92c R14: ffff8c71805d0000 R15: ffff8c718a39d0c8
[Fri Oct 24 19:08:11 2025] FS:  0000000000000000(0000) GS:ffff8c70df600000(0000) knlGS:0000000000000000
[Fri Oct 24 19:08:11 2025] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Fri Oct 24 19:08:11 2025] CR2: 00007ee6f7d07990 CR3: 0000001a6b638003 CR4: 00000000007726f0
[Fri Oct 24 19:08:11 2025] PKRU: 55555554
[Fri Oct 24 19:08:11 2025] Call Trace:
[Fri Oct 24 19:08:11 2025]  <TASK>
[Fri Oct 24 19:08:11 2025]  pcie_failed_link_retrain.cold+0x65/0xf5
[Fri Oct 24 19:08:11 2025]  pci_bridge_wait_for_secondary_bus+0x18d/0x2d0
[Fri Oct 24 19:08:11 2025]  ? pci_reset_secondary_bus+0x72/0xa0
[Fri Oct 24 19:08:11 2025]  pci_bridge_secondary_bus_reset+0x47/0x80
[Fri Oct 24 19:08:11 2025]  __pci_reset_bus+0x77/0xe0
[Fri Oct 24 19:08:11 2025]  pci_reset_bus+0x60/0x2e0
[Fri Oct 24 19:08:11 2025]  vfio_pci_core_disable+0x329/0x450 [vfio_pci_core]
[Fri Oct 24 19:08:11 2025]  vfio_pci_core_close_device+0x64/0xd0 [vfio_pci_core]
[Fri Oct 24 19:08:11 2025]  vfio_df_close+0x5c/0xb0 [vfio]
[Fri Oct 24 19:08:11 2025]  vfio_df_group_close+0x37/0x80 [vfio]
[Fri Oct 24 19:08:11 2025]  vfio_device_fops_release+0x22/0x50 [vfio]
[Fri Oct 24 19:08:11 2025]  __fput+0xea/0x2d0
[Fri Oct 24 19:08:11 2025]  ____fput+0x15/0x20
[Fri Oct 24 19:08:11 2025]  task_work_run+0x5d/0xa0
[Fri Oct 24 19:08:11 2025]  do_exit+0x329/0xa90
[Fri Oct 24 19:08:11 2025]  ? wake_up_state+0x10/0x20
[Fri Oct 24 19:08:11 2025]  do_group_exit+0x34/0x90
[Fri Oct 24 19:08:11 2025]  __x64_sys_exit_group+0x18/0x20
[Fri Oct 24 19:08:11 2025]  x64_sys_call+0x141e/0x2310
[Fri Oct 24 19:08:11 2025]  do_syscall_64+0x7e/0x170
[Fri Oct 24 19:08:11 2025]  ? filemap_map_pages+0x56a/0x6e0
[Fri Oct 24 19:08:11 2025]  ? kvm_mmu_notifier_invalidate_range_end+0x131/0x1b0 [kvm]
[Fri Oct 24 19:08:11 2025]  ? node_is_toptier+0x42/0x60
[Fri Oct 24 19:08:11 2025]  ? ptep_set_access_flags+0x4a/0x70
[Fri Oct 24 19:08:11 2025]  ? wp_page_reuse+0x97/0xc0
[Fri Oct 24 19:08:11 2025]  ? do_wp_page+0x81a/0xd50
[Fri Oct 24 19:08:11 2025]  ? ___pte_offset_map+0x1c/0x1a0
[Fri Oct 24 19:08:11 2025]  ? __handle_mm_fault+0xb43/0x1040
[Fri Oct 24 19:08:11 2025]  ? asm_sysvec_apic_timer_interrupt+0x1b/0x20
[Fri Oct 24 19:08:11 2025]  ? __count_memcg_events+0xc0/0x160
[Fri Oct 24 19:08:11 2025]  ? count_memcg_events.constprop.0+0x2a/0x50
[Fri Oct 24 19:08:11 2025]  ? handle_mm_fault+0x22d/0x350
[Fri Oct 24 19:08:11 2025]  ? do_user_addr_fault+0x5e9/0x7e0
[Fri Oct 24 19:08:11 2025]  ? arch_exit_to_user_mode_prepare.isra.0+0x22/0x120
[Fri Oct 24 19:08:11 2025]  ? irqentry_exit_to_user_mode+0x2d/0x1d0
[Fri Oct 24 19:08:11 2025]  ? irqentry_exit+0x43/0x50
[Fri Oct 24 19:08:11 2025]  ? exc_page_fault+0x96/0x1e0
[Fri Oct 24 19:08:11 2025]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[Fri Oct 24 19:08:11 2025] RIP: 0033:0x7ee6fb9ff295
[Fri Oct 24 19:08:11 2025] Code: Unable to access opcode bytes at 0x7ee6fb9ff26b.
[Fri Oct 24 19:08:11 2025] RSP: 002b:00007ffc3e2e3688 EFLAGS: 00000206 ORIG_RAX: 00000000000000e7
[Fri Oct 24 19:08:11 2025] RAX: ffffffffffffffda RBX: 00007ee6fbb0afe8 RCX: 00007ee6fb9ff295
[Fri Oct 24 19:08:11 2025] RDX: 00000000000000e7 RSI: ffffffffffffda28 RDI: 0000000000000000
[Fri Oct 24 19:08:11 2025] RBP: 0000000000000000 R08: 0000000000000030 R09: 0000000000000000
[Fri Oct 24 19:08:11 2025] R10: 0000000000000073 R11: 0000000000000206 R12: 00007ee6fbb09680
[Fri Oct 24 19:08:11 2025] R13: 00007ee6fdf51c80 R14: 00000000000002cb R15: 00007ee6fbb0b000
[Fri Oct 24 19:08:11 2025]  </TASK>
[Fri Oct 24 19:08:11 2025] ---[ end trace 0000000000000000 ]---
[Fri Oct 24 19:08:11 2025] pcieport 0000:d9:00.0: Data Link Layer Link Active not set in 100 msec
[Fri Oct 24 19:08:11 2025] vfio-pci 0000:da:00.0: reset done
[Fri Oct 24 19:08:12 2025] vfio-pci 0000:da:00.1: reset done
[Fri Oct 24 19:08:12 2025] vfio-pci 0000:da:00.1: Unable to change power state from D0 to D3hot, device inaccessible
[Fri Oct 24 19:08:12 2025] vfio-pci 0000:da:00.0: Unable to change power state from D0 to D3hot, device inaccessible
[Fri Oct 24 19:08:15 2025] vfio-pci 0000:da:00.0: Unable to change power state from D3cold to D0, device inaccessible
[Fri Oct 24 19:08:15 2025] vfio-pci 0000:da:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[Fri Oct 24 19:08:17 2025] amdgpu 0000:da:00.0: Unable to change power state from D3cold to D0, device inaccessible
[Fri Oct 24 19:08:17 2025] [drm] initializing kernel modesetting (IP DISCOVERY 0x1002:0x7550 0x1DA2:0xE490 0xC0).
[Fri Oct 24 19:08:17 2025] [drm] register mmio base: 0xFBD00000
[Fri Oct 24 19:08:17 2025] [drm] register mmio size: 524288
[Fri Oct 24 19:08:17 2025] amdgpu 0000:da:00.0: amdgpu: trn=2 ACK should not assert! wait again !
[Fri Oct 24 19:08:17 2025] amdgpu 0000:da:00.0: amdgpu: trn=2 ACK should not assert! wait again !
[Fri Oct 24 19:08:17 2025] amdgpu 0000:da:00.0: amdgpu: trn=2 ACK should not assert! wait again !
[Fri Oct 24 19:08:17 2025] amdgpu 0000:da:00.0: amdgpu: trn=2 ACK should not assert! wait again !
[Fri Oct 24 19:08:17 2025] amdgpu 0000:da:00.0: amdgpu: trn=2 ACK should not assert! wait again !
[Fri Oct 24 19:08:17 2025] amdgpu 0000:da:00.0: amdgpu: trn=2 ACK should not assert! wait again !
[Fri Oct 24 19:08:17 2025] amdgpu 0000:da:00.0: amdgpu: trn=2 ACK should not assert! wait again !
[Fri Oct 24 19:08:17 2025] amdgpu 0000:da:00.0: amdgpu: trn=2 ACK should not assert! wait again !
[Fri Oct 24 19:08:17 2025] amdgpu 0000:da:00.0: amdgpu: trn=2 ACK should not assert! wait again !
[Fri Oct 24 19:08:17 2025] amdgpu 0000:da:00.0: amdgpu: trn=2 ACK should not assert! wait again !
[Fri Oct 24 19:08:22 2025] xgpu_nv_mailbox_trans_msg: 2481 callbacks suppressed

uzumo · Oct 24, 2025

I didn't realize you weren't the original poster of this thread.

You should start a new thread.

Hijacking threads is not okay.

9qwrog1y · Oct 24, 2025

sorry!!!

uzumo · Oct 24, 2025

I can't answer that here.

8192K · 2025-11-25T10:41:13+0100

This is OP. It turned out to be easy.
After unblacklisting, I added a file /etc/initramfs-tools/scripts/init-top/bind_vfio.sh
Contents:

Bash:

#!/bin/sh
PREREQS=""
case $1 in
  prereqs)
    echo "$PREREQS"
    exit 0
  ;;
esac

# Change these IDs to match the GPU for the VM
DEVS="0000:4b:00.0 0000:4b:00.1"

for DEV in $DEVS; do
  echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

then

Bash:

chmod +x /etc/initramfs-tools/scripts/init-top/bind_vfio.sh
update-initramfs -u

and after reboot lspci -nnk -d 10de: should list one GPU used by nvidia and the other by vfio-pc. The one listed as "nvidia" is the one for the LXC.

Works perfectly!

Search

Search

How to use one GPU as passthrough and another identical GPU in LXCs?

8192K

Member

uzumo

Active Member

8192K

Member

uzumo

Active Member

Post in thread 'GPU Passthrough For Windows 11 Not Working (kvm: vfio: Unable to power on device, stuck in D3)'

9qwrog1y

New Member

Post in thread 'GPU Passthrough For Windows 11 Not Working (kvm: vfio: Unable to power on device, stuck in D3)'

uzumo

Active Member

Post in thread 'Passthrough RTX 6000/5090 CPU Soft BUG lockup, D3cold to D0, after guest shutdown'

9qwrog1y

New Member

Post in thread 'Passthrough RTX 6000/5090 CPU Soft BUG lockup, D3cold to D0, after guest shutdown'

uzumo

Active Member

9qwrog1y

New Member

uzumo

Active Member

8192K

Member

We value your privacy