GPU (Radeon) and SAS-Controller Passthrough not possible at the same time

[mcflym]

Hi,

my setup:

Proxmox: 6.1-5

Board: Supermicro x10sl7-f with Xeon 1230v3

GPU: Radeon 5500 XT --> GPU-VM
SAS-Controller: Onboard --> Storage-VM

GPU-VM is Windows 10
Storage-VM is debian

I can't use both pcie devices at the same time. After a while the GPU-VM is going into suspended mode in proxmox. If I resume the VM - it stopps. If I then start the VM again, the GPU gets code 43 inside the VM. The Storage-VM is doing fine the whole time.

If I only use the GPU-VM the GPU-VM works fine.

Any ideas? Thank you in advance!

Here are my settings:

GPU-VM conf:

bios: ovmf
bootdisk: virtio0
cores: 2
cpu: host
efidisk0: vm:100/vm-100-disk-1.raw,size=128K
hostpci0: 03:00,pcie=1,x-vga=1
machine: q35
memory: 8192
name: steamserver
net0: virtio=4A:8F:9C:E0:E9:7E,bridge=vmbr1,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=0ca99945-1c85-4dc4-9c60-f974dba990b8
sockets: 1
virtio0: vm:100/vm-100-disk-2.raw,size=60G
virtio2: /dev/disk/by-id/ata-OOS2000G128M_00000000,size=1953514584K
vmgenid: 786777f6-f730-4d9a-8988-d7d0fe807224

Storage-VM conf:

boot: c
bootdisk: virtio0
cores: 2
cpu: Haswell-noTSX
hostpci0: 04:00,pcie=1
machine: q35
memory: 8192
name: omv
net0: virtio=D2:73:C0:03:6D:F8,bridge=vmbr0
numa: 0
ostype: l26
sockets: 1
startup: order=1,up=60
virtio0: vm:102/vm-102-disk-1.raw,size=5G

root@proxmox:~# lsmod
Module                  Size  Used by
tcp_diag               16384  0
inet_diag              24576  1 tcp_diag
veth                   28672  0
ebtable_filter         16384  0
ebtables               36864  1 ebtable_filter
ip_set                 53248  0
ip6table_raw           16384  0
iptable_raw            16384  0
ip6table_filter        16384  0
ip6_tables             32768  2 ip6table_filter,ip6table_raw
nls_utf8               16384  1
cifs                 1003520  0
libarc4                16384  1 cifs
fscache               372736  1 cifs
iptable_filter         16384  0
bpfilter               32768  0
softdog                16384  2
nfnetlink_log          20480  1
nfnetlink              16384  3 ip_set,nfnetlink_log
intel_rapl_msr         20480  0
intel_rapl_common      24576  1 intel_rapl_msr
x86_pkg_temp_thermal    20480  0
intel_powerclamp       20480  0
coretemp               20480  0
kvm_intel             249856  8
kvm                   655360  1 kvm_intel
ipmi_ssif              32768  0
joydev                 28672  0
input_leds             16384  0
drm_vram_helper        20480  0
ttm                   106496  1 drm_vram_helper
snd_hda_intel          53248  0
drm_kms_helper        184320  0
crct10dif_pclmul       16384  1
snd_hda_codec         139264  1 snd_hda_intel
snd_hda_core           90112  2 snd_hda_intel,snd_hda_codec
crc32_pclmul           16384  0
drm                   491520  3 drm_kms_helper,drm_vram_helper,ttm
snd_hwdep              20480  1 snd_hda_codec
fb_sys_fops            16384  1 drm_kms_helper
syscopyarea            16384  1 drm_kms_helper
sysfillrect            16384  1 drm_kms_helper
hid_generic            16384  0
usbmouse               16384  0
ghash_clmulni_intel    16384  0
usbkbd                 16384  0
sysimgblt              16384  1 drm_kms_helper
snd_pcm               102400  3 snd_hda_intel,snd_hda_codec,snd_hda_core
snd_timer              36864  1 snd_pcm
usbhid                 57344  0
snd                    86016  5 snd_hwdep,snd_hda_intel,snd_hda_codec,snd_timer,snd_pcm
hid                   131072  2 usbhid,hid_generic
aesni_intel           372736  0
soundcore              16384  1 snd
aes_x86_64             20480  1 aesni_intel
zfs                  3887104  6
crypto_simd            16384  1 aesni_intel
cryptd                 24576  2 crypto_simd,ghash_clmulni_intel
glue_helper            16384  1 aesni_intel
zunicode              331776  1 zfs
ipmi_si                61440  0
intel_cstate           20480  0
zlua                  143360  1 zfs
ipmi_devintf           20480  0
intel_rapl_perf        20480  0
ie31200_edac           16384  0
zavl                   16384  1 zfs
pcspkr                 16384  0
ipmi_msghandler       102400  3 ipmi_devintf,ipmi_si,ipmi_ssif
icp                   278528  1 zfs
mac_hid                16384  0
intel_pch_thermal      16384  0
zcommon                86016  2 zfs,icp
znvpair                81920  2 zfs,zcommon
spl                   106496  5 zfs,icp,znvpair,zcommon,zavl
vhost_net              32768  2
vhost                  49152  1 vhost_net
tap                    24576  1 vhost_net
ib_iser                53248  0
rdma_cm                61440  1 ib_iser
iw_cm                  49152  1 rdma_cm
ib_cm                  57344  1 rdma_cm
ib_core               299008  4 rdma_cm,iw_cm,ib_iser,ib_cm
iscsi_tcp              24576  0
libiscsi_tcp           32768  1 iscsi_tcp
libiscsi               57344  3 libiscsi_tcp,iscsi_tcp,ib_iser
scsi_transport_iscsi   110592  5 libiscsi_tcp,iscsi_tcp,ib_iser,libiscsi
vfio_pci               49152  3
vfio_virqfd            16384  1 vfio_pci
irqbypass              16384  16 vfio_pci,kvm
sunrpc                393216  1
vfio_iommu_type1       28672  2
vfio                   32768  10 vfio_iommu_type1,vfio_pci
ip_tables              28672  2 iptable_filter,iptable_raw
x_tables               40960  7 ebtables,ip6table_filter,ip6table_raw,iptable_filter,ip6_tables,iptable_raw,ip_tables
autofs4                45056  2
ahci                   40960  5
i2c_i801               32768  0
libahci                32768  1 ahci
lpc_ich                24576  0
igb                   221184  0
i2c_algo_bit           16384  1 igb
dca                    16384  1 igb
video                  49152  0

iommu-groups:

root@proxmox:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:07:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.3
/sys/kernel/iommu_groups/15/devices/0000:04:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:1c.0
/sys/kernel/iommu_groups/13/devices/0000:03:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/21/devices/0000:09:00.2
/sys/kernel/iommu_groups/11/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/18/devices/0000:08:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.4
/sys/kernel/iommu_groups/16/devices/0000:06:00.0
/sys/kernel/iommu_groups/16/devices/0000:05:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:1c.2
/sys/kernel/iommu_groups/14/devices/0000:03:00.1
/sys/kernel/iommu_groups/4/devices/0000:00:1a.0
/sys/kernel/iommu_groups/22/devices/0000:09:00.3
/sys/kernel/iommu_groups/12/devices/0000:02:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:01.1
/sys/kernel/iommu_groups/20/devices/0000:09:00.1
/sys/kernel/iommu_groups/10/devices/0000:00:1f.2
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.3
/sys/kernel/iommu_groups/10/devices/0000:00:1f.6
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/19/devices/0000:09:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1d.0

lspci:

root@proxmox:~# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C222 Series Chipset Family Server Essential SKU LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Device 1478 (rev c5)
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Device 1479
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 7340 (rev c5)
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio
04:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
06:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)
07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
08:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
09:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
09:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
09:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
09:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

 

[Alwin]

Could you resolve this?

I can't use both pcie devices at the same time.

Did you try the GPU passthrough alone?

After a while the GPU-VM is going into suspended mode in proxmox.

Any message in the logs?