No more output to console mid-boot if iGPU is passed through

[imjuzcy]

I have an Ubuntu Server VM installed and configured. I have also set up that the iGPU on the host is passed through with iGVT-G. However, with this PCI passthrough, the output to console in PVE web UI is stopped when the iGPU is detected by the guest OS (after GRUB, but boots successfully, able to connect via SSH).

I have read some post, that using VirtIO-GPU for display type solved it for them, but for me, the VM it would not boot (cannot connect via SSH).

With VirtIO-GPU as display:


With default display:


Boot parameters from GRUB:


qm config 101:

agent: 1
balloon: 4096
bios: ovmf
boot: order=scsi0;ide2;net0
cores: 4
efidisk0: local-lvm:vm-101-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:00:02.0,mdev=i915-GVTg_V5_8,pcie=1
ide2: local:iso/ubuntu-22.04-live-server-amd64.iso,media=cdrom,size=1432338K
machine: q35
memory: 8196
meta: creation-qemu=6.2.0,ctime=1656331866
name: ubuntu-server
net0: virtio=6A:D9:85:EC:E1:BC,bridge=vmbr0,firewall=1
numa: 1
onboot: 1
ostype: l26
scsi0: local-lvm:vm-101-disk-1,discard=on,size=125G,ssd=1
scsi1: lightning:vm-101-disk-0,discard=on,size=850G,ssd=1
scsi2: thunder:vm-101-disk-0,discard=on,size=3200G
scsihw: virtio-scsi-pci
smbios1: uuid=78097fe9-c6d6-448d-8028-28cb450a740f
sockets: 1
tpmstate0: local-lvm:vm-101-disk-2,size=4M,version=v2.0
usb0: host=0a12:0001,usb3=1
vmgenid: ca6e94e9-e6ae-49ef-a2a1-7055017362d4

 

[imjuzcy]

Reply to add that both DisplayPort outputs of host is showing host's console

 

[leesteken]

As far as I understand it: either the whole device is passed to a single VM (GVT-d) and only that VM can output to the connected physical displays, or the device is mediated (GVT-g) and none of the VMs using it can output to the connected physical displays. I'm not sure whether the host can still output when GVT-g is active.

 

[imjuzcy]

It's setup as GVT-g and the host is outputting to the connected displays. The passed through PCIe GPU in the VM works too.

The problem is the VM does not output to the PVE web UI's console after the above screenshots.

 

[leesteken]

It's setup as GVT-g and the host is outputting to the connected displays. The passed through PCIe GPU in the VM works too.
The problem is the VM does not output to the PVE web UI's console after the above screenshots.

Right, the VM probably only outputs to the passed through GPU and not longer uses the virtual display/console from Proxmox. This is normal behavior for a Ubuntu VM to only display on one device. Use the following steps to configure a virtual serial console for Ubuntu/Debian:

In the Proxmox web GUI: add a virtual serial device to the VM and set the VM display to serial0. In the VM configuration file this will look like serial0: socket and vga: serial0

Inside the VM: change GRUB_TERMINAL=console to GRUB_TERMINAL=serial and add the following kernel parameters console=tty0 console=ttyS0. Run update-grubinside the VM to apply the changes. After a reboot, the VM should not output console message to both the GPU (tty0, before the graphical login screen) and the virtual serial console (ttyS0), which you can access via the Console button (xterm.js) in the Proxmox web GUI.

Is this what you want?

 

[imjuzcy]

Right, the VM probably only outputs to the passed through GPU and not longer uses the virtual display/console from Proxmox. This is normal behavior for a Ubuntu VM to only display on one device. Use the following steps to configure a virtual serial console for Ubuntu/Debian:

In the Proxmox web GUI: add a virtual serial device to the VM and set the VM display to serial0. In the VM configuration file this will look like serial0: socket and vga: serial0

Inside the VM: change GRUB_TERMINAL=console to GRUB_TERMINAL=serial and add the following kernel parameters console=tty0 console=ttyS0. Run update-grubinside the VM to apply the changes. After a reboot, the VM should not output console message to both the GPU (tty0, before the graphical login screen) and the virtual serial console (ttyS0), which you can access via the Console button (xterm.js) in the Proxmox web GUI.

Is this what you want?

Yes! Thank you very much!

 

[skymkmk]

Right, the VM probably only outputs to the passed through GPU and not longer uses the virtual display/console from Proxmox. This is normal behavior for a Ubuntu VM to only display on one device. Use the following steps to configure a virtual serial console for Ubuntu/Debian:

In the Proxmox web GUI: add a virtual serial device to the VM and set the VM display to serial0. In the VM configuration file this will look like serial0: socket and vga: serial0

Inside the VM: change GRUB_TERMINAL=console to GRUB_TERMINAL=serial and add the following kernel parameters console=tty0 console=ttyS0. Run update-grubinside the VM to apply the changes. After a reboot, the VM should not output console message to both the GPU (tty0, before the graphical login screen) and the virtual serial console (ttyS0), which you can access via the Console button (xterm.js) in the Proxmox web GUI.

Is this what you want?

Sorry to bother you. Directing the output to the serial causes my host to restart. My GPU is Intel Arc 380. Is there any other way to direct the output of Debian VM to the web UI of PVE?

 

[leesteken]

Sorry to bother you. Directing the output to the serial causes my host to restart.

That's really weird and should not happen just because of a virtual serial console.

My GPU is Intel Arc 380. Is there any other way to direct the output of Debian VM to the web UI of PVE?

Is that the host GPU or do you use GPU passthrough? In the latter case, does it also happen when not doing passthrough but using SPICE for example? I don't see how the GPU should be relevant for a virtual serial console.
Please share the VM configuration file (qm showconfig).

 

[skymkmk]

That's really weird and should not happen just because of a virtual serial console.

Is that the host GPU or do you use GPU passthrough? In the latter case, does it also happen when not doing passthrough but using SPICE for example? I don't see how the GPU should be relevant for a virtual serial console.
Please share the VM configuration file (qm showconfig).

This only occurs when the graphics card is passed through to the virtual machine. Even if the console=tty1 parameter of the virtual machine is deleted, as long as the graphics card is passed through to the virtual machine, using serial will cause the host machine to restart. I find it strange too. Logically, this shouldn't happen, but it did.

bios: ovmf
boot: order=scsi0;ide2;net0
cores: 10
cpu: host
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:03:00,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 30720
meta: creation-qemu=8.0.2,ctime=1690220774
name: debian
net0: virtio=FE:16:22:01:7E:58,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-100-disk-1,iothread=1,size=300G
scsi1: /dev/disk/by-id/ata-WDC_WD22EJRX-89BEMY0_WD-WX52DC1JSRXS,size=1953514584K
scsi2: /dev/disk/by-id/ata-HGST_HUH728080ALE600_2EG81S4R,size=7814026584K
scsihw: virtio-scsi-single
smbios1: uuid=c0998b3c-a65b-4e2d-bff6-5a9ae73f8c32
sockets: 1
vmgenid: 21d4a1bf-f9fa-43a8-be04-67e3e674fb1e

 

[leesteken]

This only occurs when the graphics card is passed through to the virtual machine. Even if the console=tty1 parameter of the virtual machine is deleted, as long as the graphics card is passed through to the virtual machine, using serial will cause the host machine to restart. I find it strange too. Logically, this shouldn't happen, but it did.

bios: ovmf
boot: order=scsi0;ide2;net0
cores: 10
cpu: host
efidisk0: local-lvm:vm-100-disk-0,efitype=4m,pre-enrolled-keys=1,size=4M
hostpci0: 0000:03:00,pcie=1
ide2: none,media=cdrom
machine: q35
memory: 30720
meta: creation-qemu=8.0.2,ctime=1690220774
name: debian
net0: virtio=FE:16:22:01:7E:58,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: local-lvm:vm-100-disk-1,iothread=1,size=300G
scsi1: /dev/disk/by-id/ata-WDC_WD22EJRX-89BEMY0_WD-WX52DC1JSRXS,size=1953514584K
scsi2: /dev/disk/by-id/ata-HGST_HUH728080ALE600_2EG81S4R,size=7814026584K
scsihw: virtio-scsi-single
smbios1: uuid=c0998b3c-a65b-4e2d-bff6-5a9ae73f8c32
sockets: 1
vmgenid: 21d4a1bf-f9fa-43a8-be04-67e3e674fb1e

The serial0: socket is missing from the VM configuration (there is no vrtual serial console at all) and you should use console=ttyS0 (console=tty0 is not necessary) on the kernel command line. Where did your console=tty1 come from?. What does the /etc/default/grub look like inside your VM?

 

[skymkmk]

The serial0: socket is missing from the VM configuration (there is no vrtual serial console at all) and you should use console=ttyS0 (console=tty0 is not necessary) on the kernel command line. Where did your console=tty1 come from?. What does the /etc/default/grub look like inside your VM?

Sorry, it's my current config, the previous configuration file was strictly done according to your instructions: adding serial, modifying display to serial, and modifying the VM grub file (also update it). However, I seem to have found where the problem is: When I was configuring according to your instructions, I also raised the virtual machine memory from 14GiB to 30GiB (because I just wanted to compile aosp xD) (the host physical memory is 32GB), this caused the host to restart after that, and I only removed the GPU passthrough, and the serial output was normal. But when I restored the configuration file to the one you saw (including restoring the VM grub), it caused the host to restart again. Only when I restored the virtual machine memory to 16G, I could start normally.
This is really strange, what does memory size have to do with the GPU? I will try your guidance again under the condition of 16G memory later, hoping that it can works.

 

[skymkmk]

The serial0: socket is missing from the VM configuration (there is no vrtual serial console at all) and you should use console=ttyS0 (console=tty0 is not necessary) on the kernel command line. Where did your console=tty1 come from?. What does the /etc/default/grub look like inside your VM?

Confirmed! The problem was caused by me upgrading the VM memory to 30GiB. Your instruction was totally worked. I really appreciate you and your contributions to the community! Good day!

 

[leesteken]

Sorry, it's my current config, the previous configuration file was strictly done according to your instructions: adding serial, modifying display to serial, and modifying the VM grub file (also update it). However, I seem to have found where the problem is: When I was configuring according to your instructions, I also raised the virtual machine memory from 14GiB to 30GiB (because I just wanted to compile aosp xD) (the host physical memory is 32GB), this caused the host to restart after that, and I only removed the GPU passthrough, and the serial output was normal. But when I restored the configuration file to the one you saw (including restoring the VM grub), it caused the host to restart again. Only when I restored the virtual machine memory to 16G, I could start normally.
This is really strange, what does memory size have to do with the GPU? I will try your guidance again under the condition of 16G memory later, hoping that it can works.

When using PCIe passthrough, all VM memory must be pinned into actual host RAM (because of possible DMA at any time), and ballooning and other dynamic memory does not work.