[SOLVED] UNCLAIMED Device - EVGA GTS 450 - VGA Arbitration - SeaBIOS - Ubuntu 16.04

[leoj3n]

Hardware:

• Motherboard: X10SRH-CLN4F
• CPU: E5-2648L v3 (ES: Engineering Sample)
• RAM: DDR4 2133MHZ 1.2V LRDIMM
• GPU: EVGA GeForce GTS 450 FPB (Free Performance Boost)
• Images of hardware: http://imgur.com/a/49omH

Software:

• Proxmox Virtual Environment 4.2-2/725d76f0 (pve-manager/4.2-2/725d76f0)
  
• Linux 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016
  
• GPU ROM: 70.06.1F.00.50 (2010-09-29 00:00:00) (no UEFI support)
• Ubuntu 16.04 Desktop (64-bit) (updated via Software Updater gui)

<vmid>.conf:

bootdisk: virtio0
cores: 4
memory: 2048
name: UBUNTU
hostpci0: 03:00,x-vga=on
net0: bridge=vmbr0,virtio=32:63:36:32:33:62
numa: 0
onboot: 1
ostype: l26
scsihw: virtio-scsi-pci
smbios1: uuid=c5157a1d-39c0-4298-83cc-1f7acafc58b0
sockets: 1
tablet: 0
virtio0: dat:vm-100-disk-1,size=32G

Baremetal Quirks:

To get the GTS 450 card working with default nouveau driver on Ubuntu installed natively on baremetal, I had to change to boot using "offboard" VGA in Supermicro BIOS, versus default "onboard".

Otherwise, Ubuntu would use the onboard graphics and ast driver, ignoring the GTS 450. Blacklisting the ast driver caused Ubuntu graphics to error unless mobo BIOS set to "offboard" before boot.

I did not expect to HAVE to specify "offboard" in mobo BIOS, and probably has implications regarding failure to see graphics when virtualizing with x-vga.

VGA Arbitration Off:

When the <vmid>.conf contains "hostpci0: 03:00", the VM boots into Ubuntu graphically but is using virtual VGA with cirrus driver and the GTS 450 is "UNCLAIMED" without a driver:

leoj@ubuntu:~$ sudo lshw -c video
  *-display:0
       description: VGA compatible controller
       product: GD 5446
       vendor: Cirrus Logic
       physical id: 2
       bus info: pci@0000:00:02.0
       version: 00
       width: 32 bits
       clock: 33MHz
       capabilities: vga_controller rom
       configuration: driver=cirrus latency=0
       resources: irq:0 memory:f8000000-f9ffffff memory:fe4d0000-fe4d0fff memory:fe4c0000-fe4cffff
  *-display:1 UNCLAIMED
       description: VGA compatible controller
       product: GF106 [GeForce GTS 450]
       vendor: NVIDIA Corporation
       physical id: 10
       bus info: pci@0000:00:10.0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller cap_list
       configuration: latency=0
       resources: memory:fd000000-fdffffff memory:f0000000-f7ffffff memory:fa000000-fbffffff ioport:e000(size=128) memory:fe400000-fe47ffff

VGA Arbitration On:

When the <vmid>.conf contains "hostpci0: 03:00,x-vga=on", the VM boots into Ubuntu, but the virtualized VGA device has disappeared and there are no graphics displayed over VNC:

leoj@local:~$ ssh leoj@10.0.1.36
leoj@ubuntu:~$ sudo lshw -c video
  *-display UNCLAIMED
       description: VGA compatible controller
       product: GF106 [GeForce GTS 450]
       vendor: NVIDIA Corporation
       physical id: 10
       bus info: pci@0000:00:10.0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller cap_list
       configuration: latency=0
       resources: memory:fd000000-fdffffff memory:f0000000-f7ffffff memory:f8000000-f9ffffff ioport:e000(size=128) memory:fe400000-fe47ffff

Ubuntu is running and I can SSH into it, but it seems to have no graphics adapter available for display!

Edit: I think this was the case because I needed to restart the entire Proxmox node for various reasons before the card could be picked up again after updating the <vmid>.conf; See "Final note" in reply below.

I enabled SSH on the VM like so:

sudo apt-get install openssh-server
sudo ufw allow 22
# then note IP address of Ubuntu VM in System Settings > Network

An alternative to connecting by SSH would be to use serial terminal, by first enabling it on the host:

qm set 100 -serial0 socket
# this adds a line to the <vmid>.conf

Then on the VM:

sudo vi /etc/default/grub
GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0"
sudo update-grub
# stop and then re-start the VM

And finally, back on the host:

qm terminal 100

Regarding Devices on Proxmox HOST:

I don't think the Proxmox VE host node is using the GTS 450, because I successfully followed the PCI Passthrough guide for compatability with SeaBIOS (not OMVF/UEFI because the GTS 450 has no support).

Result of lshw on the host (after installing lshw to the host via apt-get):

root@proxmox:~# lshw -c video
  *-display
       description: VGA compatible controller
       product: GF106 [GeForce GTS 450]
       vendor: NVIDIA Corporation
       physical id: 0
       bus info: pci@0000:03:00.0
       version: a1
       width: 64 bits
       clock: 33MHz
       capabilities: pm msi pciexpress vga_controller cap_list rom
       configuration: driver=vfio-pci latency=0
       resources: irq:32 memory:f6000000-f6ffffff memory:e8000000-efffffff memory:f0000000-f1ffffff ioport:d000(size=128) memory:f8000000-f807ffff
  *-display
       description: VGA compatible controller
       product: ASPEED Graphics Family
       vendor: ASPEED Technology, Inc.
       physical id: 0
       bus info: pci@0000:09:00.0
       version: 30
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi vga_controller cap_list
       configuration: driver=ast latency=0
       resources: irq:18 memory:f9000000-f9ffffff memory:fa000000-fa01ffff ioport:b000(size=128)

Conclusion:

How can I get VGA arbitration to work with my GTS 450 in an Ubuntu VM?

Please consider that I was only able to get the GTS 450 card working on baremetal Ubuntu by setting "offboard" in mobo BIOS settings, and that the device is UNCLAIMED in the Ubuntu VM when x-vga=on.

(Edit: Probably because I needed to restart the entire Proxmox node, see final note in reply below...)

Please help me with your suggestions as to what I should try next! How can I get the GTS 450 card out of UNCLAIMED state on the VM? Ubuntu is running and the device is there, but Why doesn't it use it!?

Let me know any log info you want to see from the VM.

Related Threads:

• https://forum.proxmox.com/threads/nvidia-gpu-passthrough-assistance.27237/

Other Info:

Some have suggested EVGA may provide UEFI BIOS ROM for flashing to the graphics card, but this is the reply I got from them:

Unfortunately, none of the cards that are Fermi-based (the 400 and 500 series) support UEFI. We don't have a BIOS flash that will add that functionality. Only cards in the 600 series or higher (Kepler, Maxwell, or Pascal) will support UEFI.

 

[leoj3n]

It turns out I had a fundamental misunderstanding that, once realized, enabled me to see passthrough and arbitration was working with my graphics card!

What it meant when I could not see anything via noVNC, yet the Ubuntu VM was running with "x-vga=on" added to the <vmid>.conf, was that the screen connected to the server was receiving all display and noVNC had no ability to capture that through software.

Once I finally thought to go to the server room and check the display connected to the server, I could see that Ubuntu was running graphically on that physical monitor! I had already figured out how to SSH into the VM (above), so the next step was to figure out how to install a VNC server on the Ubuntu VM so I could use the "Screen Sharing.app" on my mac in order to connect to the X session running in Ubuntu.

So, here's what I did to remote connect to the Ubuntu desktop...

SSH into the VM:

ssh leoj@10.0.1.36

Install x11vnc:

sudo apt-get install -y x11vnc

Create VNC password:

x11vnc -storepasswd

Launch VNC server:

sudo x11vnc -xkb -noxrecord -forever -noxfixes -noxdamage -display :0 -auth /var/run/lightdm/root/:0 -usepw

Which allowed me to connect via Screen Sharing.app using "vnc://10.0.1.36:5901" but your port may be different like "5900"; the output of the x11vnc command will tell you what port (more info: http://askubuntu.com/a/757398).

One other glitch was that with both nouveau driver and nvidia driver, I had to use the CCSM tool to configure Unity (the window manager) to work with the new OpenGL graphics drivers...

Install compizconfig-settings-manager:

sudo apt-get install compizconfig-settings-manager

Run ccsm:

DISPLAY=:0 ccsm

Search for Unity > Enable Plugin > Agree to enable for all prompts, then Unity starts right up and windows can be dragged around and tabbed through like normal!

However, while the nouveau driver seemed very stable, I did have trouble with the nvidia driver giving segmentation faults when testing it with glmark2, see here for more information: https://bugs.launchpad.net/ubuntu/+source/glmark2/+bug/1475902

(Update: No segmentation faults when running glmark2 on Ubuntu MATE)

So, mostly solved! Hope to eventually have the nvidia drivers working for the 3D acceleration support.

Final note: It is necessary to restart the entire node when the Ubuntu VM with VGA passthrough needs to be restarted, as otherwise the host reports the GPU has invalid ROM contents, and the VM hangs.