[SOLVED] passthrough of usb-controller causes ethernet to stop

L

liebero3

Member
Apr 27, 2017
18
0
6
43
Hello,

I'm a novice in virtualisation and don't know any further with a problem I'm having.

I have a working windows 10 vm with gpu passthrough. But if I pass through a second pci, no matter if I choose the onboard asmedia or a pci usb card (both of them have their own iommu group) the ethernet of my host stops working right in the moment I start the vm.

Does anyone have any idea, what could cause this behaviour? I'm willing to post every log you like, but since I don't have any idea, what the problem could be, I don't know what log could help you.

My cpu is a ryzen 1700 and my mainboard is the gigabyte x370 gaming 5.

Thanks in advance,
Till
 
I am pretty sure it is a fault on my side. But I have no clue where.

The moment I start the vm with the usb controller passed through the ethernet loses connection. Its still working in a way, its sending whois requests to the router. But the moment the vm starts the lights on the ethernet port go out and restarting the network service does not help, just a restart brings back network connection.

Do you use the m2 port? I have a ssd there and the gpu i am passing through is in the x8 slot, the second from the bottom. Maybe its a problem of lane capacity, because ethernet, 2nd gpu and nvme are connected? I read in the manual, that they are all connected
 
Sry for doubleposting, but I found time to gather some informations, which might help helping me.

Here is my lspci output

Code: 
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1450
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 1451
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a804
02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43b9 (rev 02)
02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b5 (rev 02)
02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02)
03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
03:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
04:00.0 USB controller: ASMedia Technology Inc. Device 1343
05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
06:00.0 Ethernet controller: Qualcomm Atheros Device e0b1 (rev 10)
08:00.0 VGA compatible controller: NVIDIA Corporation G86 [GeForce 8400 GS] (rev a1)
09:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710B] (rev a1)
09:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
0a:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456
0a:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 145c
0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
0b:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0b:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457

This is the last seconds till I start the vm from tcpdump:

Code: 
13:46:17.281762 IP 185.54.150.28.https > pve.fritz.box.45676: Flags [R.], seq 232938313, ack 2584655648, win 0, length 0
13:46:17.281896 IP pve.fritz.box.59643 > fritz.box.domain: 29021+ PTR? 28.150.54.185.in-addr.arpa. (44)
13:46:17.427666 IP fritz.box.domain > pve.fritz.box.59643: 29021 NXDomain 0/1/0 (107)
13:46:18.365849 c0:25:06:ec:35:00 (oui Unknown) > 00:b0:52:00:00:01 (oui Unknown), ethertype Unknown (0x88e1), length 60:
        0x0000:  0068 a000 b052 2c0d 1277 0000 0000 0000  .h...R,..w......
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
13:46:18.365859 c0:25:06:ec:35:00 (oui Unknown) > Broadcast, ethertype Unknown (0x88e1), length 60:
        0x0000:  0000 a000 b052 2c0d 1277 0000 0000 0000  .....R,..w......
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
13:46:18.365861 c0:25:06:ec:35:00 (oui Unknown) > Broadcast, ethertype Unknown (0x8912), length 60:
        0x0000:  0170 a000 0000 1f84 0da3 97a2 5553 bef1  .p..........US..
        0x0010:  fcf9 796b 5214 13e9 e200 0000 0000 0000  ..ykR...........
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
13:46:20.367953 c0:25:06:ec:35:00 (oui Unknown) > 00:b0:52:00:00:01 (oui Unknown), ethertype Unknown (0x88e1), length 60:
        0x0000:  0068 a000 b052 2c0d 1277 0000 0000 0000  .h...R,..w......
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
13:46:20.367965 c0:25:06:ec:35:00 (oui Unknown) > Broadcast, ethertype Unknown (0x88e1), length 60:
        0x0000:  0000 a000 b052 2c0d 1277 0000 0000 0000  .....R,..w......
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
13:46:20.367968 c0:25:06:ec:35:00 (oui Unknown) > Broadcast, ethertype Unknown (0x8912), length 60:
        0x0000:  0170 a000 0000 1f84 0da3 97a2 5553 bef1  .p..........US..
        0x0010:  fcf9 796b 5214 13e9 e200 0000 0000 0000  ..ykR...........
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
13:46:22.369032 c0:25:06:ec:35:00 (oui Unknown) > 00:b0:52:00:00:01 (oui Unknown), ethertype Unknown (0x88e1), length 60:
        0x0000:  0068 a000 b052 2c0d 1277 0000 0000 0000  .h...R,..w......
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
13:46:22.369043 c0:25:06:ec:35:00 (oui Unknown) > Broadcast, ethertype Unknown (0x88e1), length 60:
        0x0000:  0000 a000 b052 2c0d 1277 0000 0000 0000  .....R,..w......
        0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
13:46:22.369045 c0:25:06:ec:35:00 (oui Unknown) > Broadcast, ethertype Unknown (0x8912), length 60:
        0x0000:  0170 a000 0000 1f84 0da3 97a2 5553 bef1  .p..........US..
        0x0010:  fcf9 796b 5214 13e9 e200 0000 0000 0000  ..ykR...........
        0x0020:  0000 0000 0000 0000 0000 0000 0000       ..............
13:46:23.178611 IP dsp.adfarm1.adition.com.https > pve.fritz.box.41906: Flags [P.], seq 57193017:57193048, ack 3850339201, win 245, length 31
13:46:23.178642 IP pve.fritz.box.41906 > dsp.adfarm1.adition.com.https: Flags [.], ack 31, win 342, length 0
13:46:23.178646 IP dsp.adfarm1.adition.com.https > pve.fritz.box.41906: Flags [F.], seq 31, ack 1, win 245, length 0
13:46:23.178751 IP pve.fritz.box.48952 > fritz.box.domain: 31213+ PTR? 118.159.114.85.in-addr.arpa. (45)
13:46:23.192451 IP fritz.box.domain > pve.fritz.box.48952: 31213 1/0/0 PTR dsp.adfarm1.adition.com. (82)
13:46:23.221877 IP pve.fritz.box.41906 > dsp.adfarm1.adition.com.https: Flags [.], ack 32, win 342, length 0
13:46:23.395706 IP pve.fritz.box.41906 > dsp.adfarm1.adition.com.https: Flags [R.], seq 1, ack 32, win 342, length 0
tcpdump: pcap_loop: The interface went down
663 packets captured
5633 packets received by filter
4970 packets dropped by kernel

And here is my configuration file of the windows vm:

Code: 
bios: ovmf
bootdisk: scsi0
cores: 8
efidisk0: local-lvm:vm-100-disk-2,size=128K
hostpci0: 09:00,x-vga=on,pcie=1
hostpci1: 04:00,pcie=1
ide0: local:iso/virtio-win-0.1.160.iso,media=cdrom,size=315276K
ide2: local:iso/Win10_1803_German_x64.iso,media=cdrom
machine: q35
memory: 8192
net0: virtio=42:69:13:3C:BA:24,bridge=vmbr0
numa: 0
ostype: win10
scsi0: local-lvm:vm-100-disk-1,cache=writethrough,size=64G
scsihw: virtio-scsi-pci
smbios1: uuid=bad0258f-81ba-4fe1-900d-372707a286cc
sockets: 1

Anyone any ideas what could be wrong?
 
can you list your iommu groups? e.g. with this script
Code: 
#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
   n=${d#*/iommu_groups/*}; n=${n%%/*}
   printf 'IOMMU Group %s ' "$n"
   lspci -nns "${d##*/}"
done;
 
You were right, it was that easy. I mentioned, that I am a novice? :)

I actually thought, that the Groups were the part before the colon in the lspci output.

I changed to the other usb-controller in Group 20 and everything works perfect!

Thanks a lot!


Code: 
IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 10 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 11 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 12 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 12 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
IOMMU Group 13 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
IOMMU Group 13 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
IOMMU Group 13 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
IOMMU Group 13 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
IOMMU Group 13 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
IOMMU Group 13 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
IOMMU Group 13 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]
IOMMU Group 14 01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a804]
IOMMU Group 15 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b9] (rev 02)
IOMMU Group 15 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b5] (rev 02)
IOMMU Group 15 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b0] (rev 02)
IOMMU Group 15 03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 15 03:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 15 03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 15 03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 15 04:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:1343]
IOMMU Group 15 05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 15 06:00.0 Ethernet controller [0200]: Qualcomm Atheros Device [1969:e0b1] (rev 10)
IOMMU Group 16 08:00.0 VGA compatible controller [0300]: NVIDIA Corporation G86 [GeForce 8400 GS] [10de:0422] (rev a1)
IOMMU Group 17 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208 [GeForce GT 710B] [10de:128b] (rev a1)
IOMMU Group 17 09:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU Group 18 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 19 0a:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
IOMMU Group 1 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 20 0a:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:145c]
IOMMU Group 21 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 22 0b:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 23 0b:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457]
IOMMU Group 2 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 3 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 4 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 5 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 6 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 7 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 8 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 9 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
 
I had this problem today....Found that the newer kernel's have the ACS patch included for PCI passthrough / iommu groups. Just make sure you use the full line in your /etc/default/grub: pcie_acs_override=downstream,multifunction

For an AMD CPU:
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nomodeset"

It has to have the multifunction to work. Update your grub etc..

Now mine doesn't lockup.
This should probably be in the pcipassthrough wiki.
It took me several hours to find this out.

Thanks for the group script dcsapak!
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back