PCIe passthrough problems on HPE microserver gen10

herrJones

Member
Mar 10, 2018
4
0
6
42
Hi all,

I have a problem with passing my pcie DVB-S card to a VM.
When I boot the VM, network connections are disabled. The only option I have is to reboot the server (a HPE microserver gen10 with X3421 APU and 16GB RAM)

In order to do this I followed the guidelines in the PCI-passthrough wiki article.

output: dmesg | grep -e AMD -e amd
Code:
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.15.17-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt
[    0.000000]   AMD AuthenticAMD
[    0.000000] RAMDISK: [mem 0x33ab3000-0x35d50fff]
[    0.000000] ACPI: ASF! 0x00000000DD5B5080 0000D6 (v32 AMD    SB700ASF 00000001 TFSM 000F4240)
[    0.000000] ACPI: IVRS 0x00000000DD5B51A0 0000D0 (v02 AMD    AGESA    00000001 AMD  00000000)
[    0.000000] ACPI: SSDT 0x00000000DD5B5270 000854 (v01 AMD    AGESA    00000001 AMD  00000001)
[    0.000000] ACPI: SSDT 0x00000000DD5B5AC8 00888F (v02 AMD    AGESA    00000002 MSFT 04000000)
[    0.000000] ACPI: CRAT 0x00000000DD5BE358 000550 (v01 AMD    AGESA    00000001 AMD  00000001)
[    0.000000] ACPI: SSDT 0x00000000DD5BE8A8 001492 (v01 AMD    CPMDFIGP 00000001 INTL 20120913)
[    0.000000] ACPI: SSDT 0x00000000DD5BFD40 00165E (v01 AMD    CPMCMN   00000001 INTL 20120913)
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.15.17-1-pve root=/dev/mapper/pve-root ro quiet amd_iommu=on iommu=pt
[    0.046222] Spectre V2 : Mitigation: Full AMD retpoline
[    0.060000] smpboot: CPU0: AMD Opteron(tm) X3421 APU (family: 0x15, model: 0x60, stepping: 0x1)
[    0.060000] Performance Events: Fam15h core perfctr, AMD PMU driver.
[    1.421279] AMD-Vi: IOMMU performance counters supported
[    1.424145] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.424147] AMD-Vi: Extended features (0x37ef22294ada):
[    1.424156] AMD-Vi: Interrupt remapping enabled
[    1.424156] AMD-Vi: virtual APIC enabled
[    1.424457] AMD-Vi: Lazy IO/TLB flushing enabled
[    1.424558] amd_uncore: AMD NB counters detected
[    1.424816] perf: AMD IBS detected (0x000007ff)
[    1.424825] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    3.584263] AMD0020:00: ttyS4 at MMIO 0xfedc6000 (irq = 10, base_baud = 3000000) is a 16550A
[    3.663014] [drm] amdgpu kernel modesetting enabled.
[    4.133859] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

Config 102 VM:
Code:
usb0: host=2040:8268
hostpci0: host=02:00,pcie=1
agent: 1
autostart: 0
balloon: 512
bootdisk: scsi0
cores: 2
ide2: local:iso/install-amd64-minimal-20180116T214503Z.iso,media=cdrom,size=303M
machine: q35
memory: 4096
name: htsGentoo
net0: virtio=8A:3A:AC:B2:8A:40,bridge=vmbr0
numa: 0
onboot: 0
ostype: l26
scsi0: local-lvm:vm-100-disk-1,iothread=1,size=32G
scsihw: virtio-scsi-single
smbios1: uuid=70a9fbcb-8fb8-4f78-81c2-4f485b12d8c8
sockets: 1

relevant output from lspci -v -n
Code:
02:00.0 0400: 18c3:0720 (rev 01)
        Subsystem: 18c3:dd00
        Physical Slot: 2
        Flags: bus master, fast devsel, latency 0, IRQ 5
        Memory at fe810000 (32-bit, non-prefetchable) [size=64K]
        Memory at fe800000 (64-bit, non-prefetchable) [size=64K]
        Capabilities: [40] Power Management version 2
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [58] Express Endpoint, MSI 00
        Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00
        Capabilities: [400] Virtual Channel
        Kernel driver in use: vfio-pci
        Kernel modules: ngene

When booting the VM, network connections are all disabled (so config site, ssh sessions are sudddenly unreachable).
from the kernel log:
Code:
May 22 22:23:36 pve qm[31267]: <root@pam> starting task UPID:pve:00007A24:000E5968:5B047C48:qmstart:102:root@pam:
May 22 22:23:37 pve kernel: [ 9403.865997] vmbr0: port 1(enp3s0f0) entered disabled state
May 22 22:23:37 pve kernel: [ 9403.866190] device enp3s0f0 left promiscuous mode
May 22 22:23:37 pve kernel: [ 9403.866198] vmbr0: port 1(enp3s0f0) entered disabled state
May 22 22:23:37 pve kernel: [ 9403.930155] ata9.00: disabled
May 22 22:23:38 pve kernel: [ 9404.998106] device tap102i0 entered promiscuous mode
May 22 22:23:38 pve kernel: [ 9405.024104] vmbr0: port 1(tap102i0) entered blocking state
May 22 22:23:38 pve kernel: [ 9405.024108] vmbr0: port 1(tap102i0) entered disabled state
May 22 22:23:38 pve kernel: [ 9405.024290] vmbr0: port 1(tap102i0) entered blocking state
May 22 22:23:38 pve kernel: [ 9405.024292] vmbr0: port 1(tap102i0) entered forwarding state
May 22 22:23:43 pve qm[31267]: <root@pam> end task UPID:pve:00007A24:000E5968:5B047C48:qmstart:102:root@pam: OK

from the syslog:
Code:
May 22 22:17:22 pve pveproxy[2076]: starting 1 worker(s)
May 22 22:17:22 pve pveproxy[2076]: worker 30615 started
May 22 22:18:00 pve systemd[1]: Starting Proxmox VE replication runner...
May 22 22:18:00 pve systemd[1]: Started Proxmox VE replication runner.
May 22 22:19:00 pve systemd[1]: Starting Proxmox VE replication runner...
May 22 22:19:00 pve systemd[1]: Started Proxmox VE replication runner.
May 22 22:20:00 pve systemd[1]: Starting Proxmox VE replication runner...
May 22 22:20:00 pve systemd[1]: Started Proxmox VE replication runner.
May 22 22:21:00 pve systemd[1]: Starting Proxmox VE replication runner...
May 22 22:21:00 pve systemd[1]: Started Proxmox VE replication runner.
May 22 22:21:30 pve systemd[1]: Started Session 6 of user root.
May 22 22:21:32 pve systemd[1]: Started Getty on tty2.
May 22 22:21:42 pve systemd[1]: Started Session 7 of user root.
May 22 22:21:45 pve systemd[1]: Started Getty on tty3.
May 22 22:21:55 pve systemd[1]: Started Session 8 of user root.
May 22 22:22:00 pve systemd[1]: Starting Proxmox VE replication runner...
May 22 22:22:00 pve systemd[1]: Started Proxmox VE replication runner.
May 22 22:22:29 pve systemd[1]: Started Getty on tty4.
May 22 22:23:00 pve systemd[1]: Starting Proxmox VE replication runner...
May 22 22:23:00 pve systemd[1]: Started Proxmox VE replication runner.
May 22 22:23:36 pve qm[31267]: <root@pam> starting task UPID:pve:00007A24:000E5968:5B047C48:qmstart:102:root@pam:
May 22 22:23:36 pve qm[31268]: start VM 102: UPID:pve:00007A24:000E5968:5B047C48:qmstart:102:root@pam:
May 22 22:23:37 pve kernel: [ 9403.865997] vmbr0: port 1(enp3s0f0) entered disabled state
May 22 22:23:37 pve kernel: [ 9403.866190] device enp3s0f0 left promiscuous mode
May 22 22:23:37 pve kernel: [ 9403.866198] vmbr0: port 1(enp3s0f0) entered disabled state
May 22 22:23:37 pve kernel: [ 9403.930155] ata9.00: disabled
May 22 22:23:37 pve systemd[1]: Started 102.scope.
May 22 22:23:37 pve systemd-udevd[31309]: Could not generate persistent MAC address for tap102i0: No such file or directory
May 22 22:23:38 pve kernel: [ 9404.998106] device tap102i0 entered promiscuous mode
May 22 22:23:38 pve kernel: [ 9405.024104] vmbr0: port 1(tap102i0) entered blocking state
May 22 22:23:38 pve kernel: [ 9405.024108] vmbr0: port 1(tap102i0) entered disabled state
May 22 22:23:38 pve kernel: [ 9405.024290] vmbr0: port 1(tap102i0) entered blocking state
May 22 22:23:38 pve kernel: [ 9405.024292] vmbr0: port 1(tap102i0) entered forwarding state
May 22 22:23:43 pve qm[31267]: <root@pam> end task UPID:pve:00007A24:000E5968:5B047C48:qmstart:102:root@pam: OK
May 22 22:24:00 pve systemd[1]: Starting Proxmox VE replication runner...
May 22 22:24:00 pve systemd[1]: Started Proxmox VE replication runner.
May 22 22:25:00 pve systemd[1]: Starting Proxmox VE replication runner...
May 22 22:25:00 pve systemd[1]: Started Proxmox VE replication runner.
May 22 22:26:00 pve systemd[1]: Starting Proxmox VE replication runner...
May 22 22:26:00 pve systemd[1]: Started Proxmox VE replication runner.
May 22 22:27:00 pve systemd[1]: Starting Proxmox VE replication runner...
May 22 22:27:00 pve systemd[1]: Started Proxmox VE replication runner.

It's a standalone node, so replication is probably not needed (? in fact: I have no idea)

Some suggestions?

Thanx,

Jan
 

mf14v

New Member
May 15, 2018
3
0
1
Wild guess (haven't used PCI passthrough on proxmox yet):
can you try

hostpci0: host=02:00.0,pcie=1

instead of 02:00?

Perhaps 02:00 gives more devices to the VM than just the DVB card?
 

herrJones

Member
Mar 10, 2018
4
0
6
42
Hi all,

Still, whenever I try to boot my vm machine with these parameters, the general network stops, so I have to connect to my console in the attic.

If anyone has a suggestion to get the network back alive without rebooting the server, that'd save already heaps of time. It may be a microserver, but it takes the 'normal' server time to boot :)

-- Jan
 

bobmc

Active Member
May 17, 2018
239
37
33
64
I know IOMMU is reported found, but is it actually enabled in the BIOS?
 

herrJones

Member
Mar 10, 2018
4
0
6
42
after starting a vm with passthrough, it seems I lose my physical network adapters

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp3s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP mode DEFAULT group default qlen 1000
link/ether 98:f2:b3:e6:3c:f4 brd ff:ff:ff:ff:ff:ff
3: enp3s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr1 state UP mode DEFAULT group default qlen 1000
link/ether 98:f2:b3:e6:3c:f5 brd ff:ff:ff:ff:ff:ff
4: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 98:f2:b3:e6:3c:f4 brd ff:ff:ff:ff:ff:ff
5: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 98:f2:b3:e6:3c:f5 brd ff:ff:ff:ff:ff:ff

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
4: vmbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
5: vmbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 5e:11:88:05:a2:84 brd ff:ff:ff:ff:ff:ff
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!