Proxmox not working correctly after trying to start a new VM

[avinashsah]

https://imgur.com/a/QANQZbn

I'm not able to access web dashboard not able to use shell after I tried to create and start a VM for OPNsense.

Have already re-installed proxmox several times trying to solve this issue, but mo luck.

This is my first time using proxmox

 

[Dunuin]

Do you PCI passthrough any NICs into the OPNsense VM?

 

[avinashsah]

Do you PCI passthrough any NICs into the OPNsense VM?

Yes, I have a dual port Intel 82576 PCIE NIC that I'm adding to the VM as Hardware -> PCIE device.

I'm adding both ports of this NIC to the VM as it's a requirement for OPNsense

 

[Dunuin]

Did you check the IOMMU groups? If other parts of your mainboard share the same IOMMU group you will pass them through as well and your host is then missing these. If your onboard NIC would be in the same group this would explain why you can't reach your PVE host after starting the VM.

What is for d in /sys/kernel/iommu_groups/*/devices/*; do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done; returning?

 

[avinashsah]

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 10 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 11 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
IOMMU Group 11 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 12 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
IOMMU Group 12 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
IOMMU Group 12 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
IOMMU Group 12 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
IOMMU Group 12 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
IOMMU Group 12 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
IOMMU Group 12 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
IOMMU Group 12 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU Group 13 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01)
IOMMU Group 13 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
IOMMU Group 13 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
IOMMU Group 13 20:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 13 20:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 13 20:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 13 21:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
IOMMU Group 13 21:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01)
IOMMU Group 13 22:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 14 26:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] [10de:1e84] (rev a1)
IOMMU Group 14 26:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller [10de:10f8] (rev a1)
IOMMU Group 14 26:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller [10de:1ad8] (rev a1)
IOMMU Group 14 26:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller [10de:1ad9] (rev a1)
IOMMU Group 15 27:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 16 28:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 17 28:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 18 28:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 19 28:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU Group 1 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 3 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 5 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 6 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 9 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]

This is the output

 

[avinashsah]

I think I need to follow this: https://pve.proxmox.com/wiki/PCI_Passthrough

 

[avinashsah]

Added amd_iommu=on, no luck

 

[leesteken]

IOMMU Group 13 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01) IOMMU Group 13 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01) IOMMU Group 13 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01) IOMMU Group 13 20:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01) IOMMU Group 13 20:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01) IOMMU Group 13 20:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01) IOMMU Group 13 21:00.0 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01) IOMMU Group 13 21:00.1 Ethernet controller [0200]: Intel Corporation 82576 Gigabit Network Connection [8086:10c9] (rev 01) IOMMU Group 13 22:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)

This is exacly like Dunuin predicted:

If other parts of your mainboard share the same IOMMU group you will pass them through as well and your host is then missing these. If your onboard NIC would be in the same group this would explain why you can't reach your PVE host after starting the VM.

Added amd_iommu=on, no luck

That does nothing because it is always on by default.

Your Ryzen motherboard (which make and model?) does not have a X570 chipset and your onboard SATA, USB, network controller are in the same group as the network controller that you are trying to passthrough. Probably only the first x16 PCIe slot and the first M.2 slot are connected to the CPU. Only devices connected to the CPU are in separate groups.

 

[avinashsah]

This is exacly like Dunuin predicted:


That does nothing because it is always on by default.

Your Ryzen motherboard (which make and model?) does not have a X570 chipset and your onboard SATA, USB, network controller are in the same group as the network controller that you are trying to passthrough. Probably only the first x16 PCIe slot and the first M.2 slot are connected to the CPU. Only devices connected to the CPU are in separate groups.

So that means i need to put my NIC in the 1st x16 slot?

I have a B450 motherboard btw. MSI B450 Tomahawk MAX II ATX Gaming Motherboard.

If you could lay down the step by step instructions for how to make it work then it would really help me a lot.

Thank you

 

[leesteken]

So that means i need to put my NIC in the 1st x16 slot?

I have a B450 motherboard btw. MSI B450 Tomahawk MAX II ATX Gaming Motherboard.

Yes.Looking at the specifications of that motherboard, only add-in PCIe cards in the first x16 slot will be in a separate IOMMU group. The other x16 slot (which is only x4 or even x2) and all the x1 slots will be part of the big "chipset group".
You can lie about your IOMMU groups with the pcie_acs_override but that breaks security isolation. If your OPNsense VM get hacked then they can read (and write) all of the system memory including other VMs and the Proxmox host.

If you could lay down the step by step instructions for how to make it work then it would really help me a lot.

Take out the PCIe card from the current PCIe slot and put it in the first x16 slot (closest to the CPU). Note that PCI IDs for the device and possibly other devices in your system (including the network controller name) can change because of this. If you lose network because of this please search the forum as this has happens often and make sure you can login to the Proxmox host using a physical display and keyboard.

 

[avinashsah]

Understood, I was planning to have a windows VM with a GPU passthrough, I guess that's not possible anymore? Or maybe I can add it to the last slot and fake like group like you mentioned?

 

[leesteken]

Understood, I was planning to have a windows VM with a GPU passthrough, I guess that's not possible anymore? Or maybe I can add it to the last slot and fake like group like you mentioned?

You can lie about your IOMMU groups with the pcie_acs_override but that breaks security isolation. If your OPNsense VM get hacked then they can read (and write) all of the system memory including other VMs and the Proxmox host.