Weird color distortions on GPU PCIe Passthrough

mimesot

Well-Known Member
Aug 17, 2017
36
4
48
Hi!
I hope someone else already encountered that problem, as I have no clue hat to do about it.

I usually work with Linux (Debian 9 with Cinnamon Desktop) in a dual screen setup, but in occasionally I want to be able to do some heavy 3D stuff. In order to use the power of my Radeon 580 in an Windows environment I virtualize Windows 10 (using Proxmox VE 5) and perform a PCIe passthrough (according to this). When I switch to Windows I use a KVM Switch to change the connection of the second screen from a port on the mainboard (IGP: Intel P630) to the HDMI Port on the Radeon card. This works so far.

But when I do this, the remaining (primary) Linux monitor, which stays attached to the mainboard (Intel IGP), changes colors in a weird fashion. Pixels, which appear to be selected depending on the colors displayed at that point, have a random color. Probably tot totally random - it looks as it it were colors with no saturation depth or like in 256 color mode or something like that. This especially seems to be true for colors with alpha values (like blow the start menu). Some areas have a swirly chaotic look and some rather plain wrong colors.

In order to remove this effect (until I stop and restart the Windows VM again), I go into the display settings of the desktop environment and change something. It doesn't matter what - resolution, rotation - everything does the job.

The second monitor, which has its gpu passed through to the VM, does not show any abnormal behavior, but sometimes the VM tries to start and appears to die after the boot screen. Its screen goes black, the VM does not stay it stopped, but it is not running any more either.

The color distortion effect is not dependent on the link type. HDMI DVI and VGA all produce the same effect. It is not dependent on the desktop environment either. I tried Cinnamon, Cinnamon in Software rendering mode, Debians generic Gnome and XFCE.

My hardware:
  • CPU: Intel Xeon E3-1245 v6 (Quad core, Hyperthreading, 3,7 GHz, VD-x, VT-d, ECC)
  • RAM: 16GB ECC Crucial 2133MHz DDR4
  • Mainboard: MSI C236a
  • System SSD: Crucial MX300 525GB
What uses PCIe-Lanes (lspci excerpt):
  • 01:00 VGA AMD/ATI Ellesmere (Radeon 580)
  • 03:00 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller
    (successfully passed through to Win 10 VM)
  • 04:00 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 (on Mainboard, not in use)
  • 05:00 SATA controller: ASMedia Technology Inc. ASM1062 (not in use)
  • 06:00 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411
  • 07:00 Non-Volatile memory controller: Samsung Electronics
Operating Systems:
  • Debian 9.4 with Proxmox 5.1-52, Kernel 4.15.17-1-pve
  • Windows 10 Pro 64bit (with a passed through AMD/ATI-Card, USB-3-card)
This is my config of the Windows 10 VM (vm number 1121):
Code:
balloon: 4096
bootdisk: sata0
cores: 6
hostpci0: 01:00,pcie=1,x-vga=on
hostpci1: 03:00,pcie=1
ide2: local:iso/Win10_1703_German_x64.iso,media=cdrom
keyboard: de
machine: q35
memory: 12288
name: zirkon-vm-win10
net0: e1000=32:3F:E5:84:E6:DC,bridge=vmbr0
numa: 0
ostype: win10
sata0: local:1121/vm-1121-disk-1.qcow2,cache=writeback,size=100G
scsihw: virtio-scsi-pci
smbios1: uuid=f140ea6e-a5b2-4b2e-be80-e1b2fcb0733f
sockets: 1

This is my IOMMU group output:
Code:
find /sys/kernel/iommu_groups/ -type l | sort
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.2
/sys/kernel/iommu_groups/11/devices/0000:00:1f.3
/sys/kernel/iommu_groups/11/devices/0000:00:1f.4
/sys/kernel/iommu_groups/12/devices/0000:00:1f.6
/sys/kernel/iommu_groups/13/devices/0000:03:00.0
/sys/kernel/iommu_groups/14/devices/0000:04:00.0
/sys/kernel/iommu_groups/15/devices/0000:05:00.0
/sys/kernel/iommu_groups/16/devices/0000:06:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/1/devices/0000:02:00.0
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/3/devices/0000:00:08.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.2
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/6/devices/0000:00:17.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.2
/sys/kernel/iommu_groups/9/devices/0000:00:1c.4

What additional information shall I provide?
Thanks in advance for your help.
Greetings
mimesot
 
This is a "screenshot" of the display which shows the color issue after the gpu passthrough. The screen (right) is the one which is not passed through and remains with the debian-proxmox. The one that is passed through does not show any malfunctions. Screenshots taken from within debian-proxmox using the print key do not show any color distortions, which means that the source of the problem has to originate from something happening after the image is rendered.

Something else I tested: I did remove the KVM switch. The issue also occurs, when one screen is exclusively connected to the IGP and one exclusively connected to the Radeon GPU.

The problem also occurs in a non graphical mode. When I open a console using Ctrl+Alt+F3 and enter qm start 1121 (the specific windows VM) i get a green ribbon on top of the screen.

Interestingly, when I attach a keyboard to (an USB port on a USB controller card which is passed through to) the Windows 10 VM starts normally, but when no keyboard is connected the Windows VM dies off. The proxmox boot screen appears, then the windows logo and then it shuts down. Why is that?
 

Attachments

  • Proxmox gpu pcie passthrough color problem.jpg
    Proxmox gpu pcie passthrough color problem.jpg
    696.7 KB · Views: 26
Last edited:
did you try with omvf as bios (instead of seabios)?
i had similar problems with my workstation when x was already started when i started the vm (only with seabios and intel igp)
 
  • Like
Reactions: mimesot
Thanks for the hint! I am currently pursuing a different approach with different problems (see here), but I will try it with ovmf next.
 
did you try with omvf as bios (instead of seabios)?
i had similar problems with my workstation when x was already started when i started the vm (only with seabios and intel igp)

Hi!
I did try the OVMF approach. This indeed solced my graphics issues. Thanks a lot. But now I do not have any access to my keyboard and cannot launch the installation medium. Both, USB-passrthrough and PCIe-controller-card passthrough fail. Do you have any ideas for me?

Kind regards!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!