Lost host console display if IOMMU is on

[NaokiS28]

Hi all,

Firstly I understand its not a major problem since everything is controlled through the web ui but I noticed on my server (admittedly a very old server board) that if IOMMU is turned on, I will lose the onboard GPU output after 20 seconds. This isn’t usually an issue since Im only passing NICs and HBAs through but I was really struggling to figure out what was wrong with the network interface and couldn’t access the console to diagnose it properly.

After a while I did manage to get into the recovery mode to figure out it was the IOMMU causing it but Im just wondering if there is anything I can do so that proxmox always uses the onboard gpu for the console output? Or if not, if its possible to enable console redirection over to COM1?

Thanks

 

[leesteken]

Intel Broadwell intergrated graphics is known to have problems with IOMMU. Try intel_iommu=on,igfx_off instead of intel_iommu=on.
Proxmox is based on Debian GNU/Linux and can use a serial console with something like console=ttyS0, but I would have to search for the details.

EDIT: Maybe you can share your IOMMU groups? for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU group %s ' "$n"; lspci -nns "${d##*/}"; done

 

[NaokiS28]

Thanks, I will try that.

And here they are

IOMMU group 0 00:00.0 Host bridge [0600]: Intel Corporation 5520 I/O Hub to ESI Port [8086:3406] (rev 22)
IOMMU group 10 00:14.3 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers [8086:3438] (rev 22)
IOMMU group 11 00:15.0 PIC [0800]: Intel Corporation 7500/5520/5500/X58 Trusted Execution Technology Registers [8086:342f] (rev 22)
IOMMU group 12 00:16.0 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3430] (rev 22)
IOMMU group 13 00:16.1 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3431] (rev 22)
IOMMU group 14 00:16.2 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3432] (rev 22)
IOMMU group 15 00:16.3 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3433] (rev 22)
IOMMU group 16 00:16.4 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:3429] (rev 22)
IOMMU group 17 00:16.5 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342a] (rev 22)
IOMMU group 18 00:16.6 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342b] (rev 22)
IOMMU group 19 00:16.7 System peripheral [0880]: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device [8086:342c] (rev 22)
IOMMU group 1 00:01.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 [8086:3408] (rev 22)
IOMMU group 20 00:1a.0 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4 [8086:3a37]
IOMMU group 20 00:1a.1 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5 [8086:3a38]
IOMMU group 20 00:1a.2 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 [8086:3a39]
IOMMU group 20 00:1a.7 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2 [8086:3a3c]
IOMMU group 21 00:1b.0 Audio device [0403]: Intel Corporation 82801JI (ICH10 Family) HD Audio Controller [8086:3a3e]
IOMMU group 22 00:1c.0 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 [8086:3a40]
IOMMU group 22 00:1c.4 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5 [8086:3a48]
IOMMU group 22 00:1c.5 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6 [8086:3a4a]
IOMMU group 22 08:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) [102b:0522] (rev 02)
IOMMU group 22 09:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02)
IOMMU group 23 00:1d.0 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 [8086:3a34]
IOMMU group 23 00:1d.1 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 [8086:3a35]
IOMMU group 23 00:1d.2 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 [8086:3a36]
IOMMU group 23 00:1d.7 USB controller [0c03]: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 [8086:3a3a]
IOMMU group 24 00:1e.0 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev 90)
IOMMU group 24 0a:01.0 FireWire (IEEE 1394) [0c00]: Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx] [104c:8023]
IOMMU group 25 00:1f.0 ISA bridge [0601]: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller [8086:3a16]
IOMMU group 25 00:1f.2 SATA controller [0106]: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller [8086:3a22]
IOMMU group 25 00:1f.3 SMBus [0c05]: Intel Corporation 82801JI (ICH10 Family) SMBus Controller [8086:3a30]
IOMMU group 26 01:00.0 Ethernet controller [0200]: Intel Corporation 82575EB Gigabit Network Connection [8086:10a7] (rev 02)
IOMMU group 27 01:00.1 Ethernet controller [0200]: Intel Corporation 82575EB Gigabit Network Connection [8086:10a7] (rev 02)
IOMMU group 28 03:00.0 PCI bridge [0604]: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch [111d:8018] (rev 0e)
IOMMU group 29 04:02.0 PCI bridge [0604]: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch [111d:8018] (rev 0e)
IOMMU group 29 05:00.0 Ethernet controller [0200]: Intel Corporation 82575GB Gigabit Network Connection [8086:10d6] (rev 02)
IOMMU group 29 05:00.1 Ethernet controller [0200]: Intel Corporation 82575GB Gigabit Network Connection [8086:10d6] (rev 02)
IOMMU group 2 00:03.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 [8086:340a] (rev 22)
IOMMU group 30 04:04.0 PCI bridge [0604]: Microsemi / PMC / IDT PES12N3A 12-lane 3-Port PCI Express Switch [111d:8018] (rev 0e)
IOMMU group 30 06:00.0 Ethernet controller [0200]: Intel Corporation 82575GB Gigabit Network Connection [8086:10d6] (rev 02)
IOMMU group 30 06:00.1 Ethernet controller [0200]: Intel Corporation 82575GB Gigabit Network Connection [8086:10d6] (rev 02)
IOMMU group 31 ff:00.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers [8086:2c70] (rev 02)
IOMMU group 31 ff:00.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder [8086:2d81] (rev 02)
IOMMU group 32 ff:02.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Link 0 [8086:2d90] (rev 02)
IOMMU group 32 ff:02.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Physical 0 [8086:2d91] (rev 02)
IOMMU group 32 ff:02.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Mirror Port Link 0 [8086:2d92] (rev 02)
IOMMU group 32 ff:02.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Mirror Port Link 1 [8086:2d93] (rev 02)
IOMMU group 32 ff:02.4 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Link 1 [8086:2d94] (rev 02)
IOMMU group 32 ff:02.5 Host bridge [0600]: Intel Corporation Xeon 5600 Series QPI Physical 1 [8086:2d95] (rev 02)
IOMMU group 33 ff:03.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers [8086:2d98] (rev 02)
IOMMU group 33 ff:03.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder [8086:2d99] (rev 02)
IOMMU group 33 ff:03.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers [8086:2d9a] (rev 02)
IOMMU group 33 ff:03.4 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers [8086:2d9c] (rev 02)
IOMMU group 34 ff:04.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control [8086:2da0] (rev 02)
IOMMU group 34 ff:04.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address [8086:2da1] (rev 02)
IOMMU group 34 ff:04.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank [8086:2da2] (rev 02)
IOMMU group 34 ff:04.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control [8086:2da3] (rev 02)
IOMMU group 35 ff:05.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control [8086:2da8] (rev 02)
IOMMU group 35 ff:05.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address [8086:2da9] (rev 02)
IOMMU group 35 ff:05.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank [8086:2daa] (rev 02)
IOMMU group 35 ff:05.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control [8086:2dab] (rev 02)
IOMMU group 36 ff:06.0 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control [8086:2db0] (rev 02)
IOMMU group 36 ff:06.1 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address [8086:2db1] (rev 02)
IOMMU group 36 ff:06.2 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank [8086:2db2] (rev 02)
IOMMU group 36 ff:06.3 Host bridge [0600]: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control [8086:2db3] (rev 02)
IOMMU group 3 00:07.0 PCI bridge [0604]: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 [8086:340e] (rev 22)
IOMMU group 4 00:10.0 PIC [0800]: Intel Corporation 7500/5520/5500/X58 Physical and Link Layer Registers Port 0 [8086:3425] (rev 22)
IOMMU group 4 00:10.1 PIC [0800]: Intel Corporation 7500/5520/5500/X58 Routing and Protocol Layer Registers Port 0 [8086:3426] (rev 22)
IOMMU group 5 00:11.0 PIC [0800]: Intel Corporation 7500/5520/5500 Physical and Link Layer Registers Port 1 [8086:3427] (rev 22)
IOMMU group 5 00:11.1 PIC [0800]: Intel Corporation 7500/5520/5500 Routing & Protocol Layer Register Port 1 [8086:3428] (rev 22)
IOMMU group 6 00:13.0 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller [8086:342d] (rev 22)
IOMMU group 7 00:14.0 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers [8086:342e] (rev 22)
IOMMU group 8 00:14.1 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers [8086:3422] (rev 22)
IOMMU group 9 00:14.2 PIC [0800]: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers [8086:3423] (rev 22)

EDIT: The igfx_off didn't work.

 

[leesteken]

IOMMU group 22 00:1c.0 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 [8086:3a40]
IOMMU group 22 00:1c.4 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 5 [8086:3a48]
IOMMU group 22 00:1c.5 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6 [8086:3a4a]
IOMMU group 22 08:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. MGA G200e [Pilot] ServerEngines (SEP1) [102b:0522] (rev 02)
IOMMU group 22 09:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller [1912:0015] (rev 02)

EDIT: The igfx_off didn't work.

I assume you did not use the unsecure pcie_acs_override (check with cat /proc/cmdline)?
Make sure not to passthrough device 09:00.0 (or any of the PCIe root ports) as the Proxmox host will lose all devices in group 22. If that's not it, I have no clue why your CPU and motherboard behave this way.
You cannot use an integrated graphics from the Intel CPU(s) as an alternative? I'm not familiar with the MGA G200e, sorry.
Maybe check journalctl (scroll with the arrow keys) around the time that the Proxmox host console dissapears for messages about device 08:00.0, 09:00.0 or maybe something about vfio or matrox?

EDIT: There are some threads about Matrox G200 here but there mostly about installation issues.

 

[NaokiS28]

No I have not done anything more than turn on IOMMU. I also cant use the iGPU as the VGA port is connected to the IPMI BMC which has the G200 and theres no other provisions for GPU other than a PCIe card.

Ill check the journal when I can.

EDIT: Just seen that the uPD720202 is in group 22, which I pass through to the VM. Is that just going to be the case of moving it to a different PCIe slot then?

 

[leesteken]

EDIT: Just seen that the uPD720202 is in group 22, which I pass through to the VM. Is that just going to be the case of moving it to a different PCIe slot then?

You cannot share devices from the same IOMMU group between VMs and/or the Proxmox host. Yes, please do not passthrough that device while it is in the same IOMMU group. That would explain your issue.
Since the CPU and motherboard determine the IOMMU groups, it's possible that moving a PCIe add-in card to another slot will also move it to another group (but some slots might be in the same group).