Vm not reachable and unable to ssh after attaching to a PCIe graphics card via GPU passthrough

[Sujith Arangan]

Hello everyone,

I attached a PCIe nvidia graphics card to a Ubuntu VM via gpu passthrough. After booting the VM there is no console access(Which says the graphic card is attached) but I am unable to ssh to the VM or even ping it. It's totally unreachable

 

[Sujith Arangan]

I think the VM is not even booting. I tried changing the machine type to q35 and no luck.
I have configured the /etc/default/grub file with
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"
updated and rebooted.
proxmox-boot-tool refresh
nano /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo "options kvm ignore_msrs=1" > /etc/modprobe.d/kvm.conf

echo "blacklist nouveau" >> /etc/modprobe.d/blacklist.conf
echo "blacklist nvidia" >> /etc/modprobe.d/blacklist.conf

echo "options vfio-pci ids=####.####,####.#### disable_vga=1"> /etc/modprobe.d/vfio.conf


-Apply all changes-

update-initramfs -u -k all

--REBOOT--

I can see the Nvidia A100 PCIe graphics card in the pcie devices list. I created a VM in UEFI mode, Attached the PCIe device and booted it. Now there is no console which means the graphics card is attached. But the VM is not pinging and unable to ssh to it.

This is the error I get when I launch the console



Request for early help.

 

[leesteken]

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on"

amd_iommu=on is invalid and ignored because it is on by default.

vfio_virqfd

This module no longer exists with newer kernels. Are you on an old version of Proxmox?

I can see the Nvidia A100 PCIe graphics card in the pcie devices list. I created a VM in UEFI mode, Attached the PCIe device and booted it. Now there is no console which means the graphics card is attached.

Not necessarily but probably yes.

But the VM is not pinging and unable to ssh to it.

Make first configure your Ubuntu VM for a serial console (inside the VM) and add a virtual serial port to the VM and set Display to serial0. Then you can maybe see inside what the VM is doing when it appear to not boot with the GPU passed through.

EDIT: Also, look into the Proxmox Syslog (or journalctl -b 0) when starting the VM to see if there are messages that contain a clue about the problem.

 

[Sujith Arangan]

amd_iommu=on is invalid and ignored because it is on by default.

This module no longer exists with newer kernels. Are you on an old version of Proxmox?

Not necessarily but probably yes.

Make first configure your Ubuntu VM for a serial console (inside the VM) and add a virtual serial port to the VM and set Display to serial0. Then you can maybe see inside what the VM is doing when it appear to not boot with the GPU passed through.

EDIT: Also, look into the Proxmox Syslog (or journalctl -b 0) when starting the VM to see if there are messages that contain a clue about the problem.

I meant to mention GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on". Because it's an intel processor. It is working fine. I copy pasted from my notes and made a mistake. S0 you meant to say to create a serial port inside the VM before attaching the PCIe device?

 

[leesteken]

S0 you meant to say to create a serial port inside the VM before attaching the PCIe device?

Yes. Add a virtual serial port (serial0), then configure GRUB inside the VM to use the serial terminal and to make Ubuntu (also) use the serial terminal. And then set Display of the VM to serial0. If that works, then add the GPU passthrough and you might be able to see with xterm.js what is happening inside the VM.

 

[Sujith Arangan]

Yes. Add a virtual serial port (serial0), then configure GRUB inside the VM to use the serial terminal and to make Ubuntu (also) use the serial terminal. And then set Display of the VM to serial0. If that works, then add the GPU passthrough and you might be able to see with xterm.js what is happening inside the VM.

After configuring, I get a message as starting serial terminal on serial 0. And the terminal is not going forward. I have also configured the grub in the VMs terminal by adding the below lines.

GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"

 

[leesteken]

After configuring, I get a message as starting serial terminal on serial 0. And the terminal is not going forward. I have also configured the grub in the VMs terminal by adding the below lines.

GRUB_TERMINAL="console serial"
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"

Sorry, I did not realize that you did not know how to do this. I think you only need to do:
GRUB_CMDLINE_LINUX="console=ttyS0" or Linux won't use the serial for a console. Maybe you already have other kernel parameters in the VM, then just add console=ttyS0 (within the quotes).
GRUB_TERMINAL=serial without console and no need to set the GRUB_SERIAL_COMMAND (maybe don't set it at all, because it works fine without it)

 

[Sujith Arangan]

Sorry, I did not realize that you did not know how to do this. I think you only need to do:
GRUB_CMDLINE_LINUX="console=ttyS0" or Linux won't use the serial for a console. Maybe you already have other kernel parameters in the VM, then just add console=ttyS0 (within the quotes).
GRUB_TERMINAL=serial without console and no need to set the GRUB_SERIAL_COMMAND (maybe don't set it at all, because it works fine without it)

Thank you very much. The situation is the same. The VM is stuck at "starting terminal at interface 0" and doesn't move further. But there is output and the VM is reachable once I remove the Graphics card PCI device.

 

[leesteken]

Thank you very much. The situation is the same. The VM is stuck at "starting terminal at interface 0" and doesn't move further. But there is output and the VM is reachable once I remove the Graphics card PCI device.
View attachment 68434

There might be informative messages in the Proxmox Syslog (or journalctl) about problems with resetting the GPU.