Code 43 NVIDIA driver error within Windows 10 VM, Geforce 750 Ti Passthrough

Yaay!! it works! many thanks!
It took me 3 weeks of struggle!
my hw is
gigabyte 650ti
i5-3470
mb asus P8Q77M2
config
Code:
bios: ovmf
boot: c
bootdisk: virtio0
cores: 2
hostpci0: 01:00,pcie=1,x-vga=on
ide0: iso1:iso/virtio-win-0.1.112.iso,media=cdrom,size=55592K
machine: q35
memory: 4096
name: win1
net0: e1000=32:33:62:38:63:35,bridge=vmbr0
numa: 0
ostype: win8
scsihw: virtio-scsi-pci
smbios1: uuid=6b380506-55b3-4c7e-a4cb-caacded045fe
sockets: 1
usb0: host=3-1
usb1: host=1-4
virtio0: local:100/vm-100-disk-1.qcow2,size=32G
also, i`ve tried it on some old and cheap ati5450 - no luck.
And if I reboot the wm - no signal on the monitor, I have to reboot the host. any way to fix this?
also, is there any way I can put my internal hd4000 to another linux wm(I want some machine with kodi mediacenter)?
 
Yaay!! it works! many thanks!
It took me 3 weeks of struggle!
my hw is
gigabyte 650ti
i5-3470
mb asus P8Q77M2
config
Thanks for sharing the conf !

also, i`ve tried it on some old and cheap ati5450 - no luck.
And if I reboot the wm - no signal on the monitor, I have to reboot the host. any way to fix this?
Can you send the vm config ?

But this blog post said it's working for 5450.
http://vfio.blogspot.be/2014/08/dual-vga-assignment-geforce-radeon.html

also, is there any way I can put my internal hd4000 to another linux wm(I want some machine with kodi mediacenter)?
for intel, I have seen some improvements in last kernel 4.4. So it's better to wait for proxmox 4.2 in some month to test it again.
 
YESSS !

can you send your
- vmid.conf
- #pveversion -v
- maybe other tuning done in grub

I'll update the wiki


my email can be public : aderumier@odiso.com ;)

No need to paypal, I'm glad to help the proxmox community.
And thank you to have take time to test, because my geforce 970 is not able to passthrough currently, so I'm blind for testing currently ;)

All I did was make sure to PCI Stub the card. I followed a debian(or maybe ubuntu) tutorial for that I cant remember. I do remember you have to check after that your card is running PCI stub driver and not getting claimed by nvidia driver. with lspci -nnk.
Then I did the prerelease repo you recommended. The rest was just like the proxmox wiki.

Here is my conf
Code:
bootdisk: sata0
cores: 2
hostpci0: 01:00,pcie=1,x-vga=on
ide2: local:iso/virtio-win.iso,media=cdrom
machine: q35
memory: 6000
name: Win7
net0: e1000=36:31:62:36:30:37
numa: 0
ostype: win7
sata0: Tank:vm-100-disk-1,cache=writeback,size=32G
smbios1: uuid=cc892184-b29d-4193-8f8f-378ba36c538a
sockets: 1
usb0: host=1-10.1
usb1: host=1-10.2
usb2: host=1-10.3
usb3: host=1-4

Thanks again buddy you rock :)
 
Last edited:
Thanks for sharing the conf !


Can you send the vm config ?

But this blog post said it's working for 5450.
http://vfio.blogspot.be/2014/08/dual-vga-assignment-geforce-radeon.html


for intel, I have seen some improvements in last kernel 4.4. So it's better to wait for proxmox 4.2 in some month to test it again.
sure
Code:
cat /etc/pve/qemu-server/100.conf
agent: 1
bios: ovmf
boot: c
bootdisk: virtio0
cores: 4
hostpci0: 01:00,pcie=1,x-vga=on
machine: q35
memory: 6144
name: win1
net0: e1000=32:33:62:38:63:35,bridge=vmbr0
numa: 0
onboot: 1
ostype: win8
scsihw: virtio-scsi-pci
smbios1: uuid=6b380506-55b3-4c7e-a4cb-caacded045fe
sockets: 1
usb0: host=3-1.6
usb1: host=1-4
virtio0: local:100/vm-100-disk-1.qcow2,size=32G
this is my vm config for nvidia 650ti with freshly installed windows guest
As for ati card - i mistyped hwid in vfio.conf, and I gave it back to my friend, so once I see him again I will try it again.
 
sure
this is my vm config for nvidia 650ti with freshly installed windows guest
As for ati card - i mistyped hwid in vfio.conf, and I gave it back to my friend, so once I see him again I will try it again.

Is it the same config for ati card ? (because I don't think it's ovmf/uefi compatible, card is too old)
 
Is it the same config for ati card ? (because I don't think it's ovmf/uefi compatible, card is too old)
yes, I changed only hwid in vfio.conf. So, I get it, why ati card did not start at all. anyway, I took it as a test. Do you still need test results with it?
Any solution with rebooting wm with nvidia card and not showing anything (It works to reboot the host)?
Also I would greatly appreciate any advice on fine-tuning windows wm (like pushing the whole sata controller inside or similiar information)
 
mmm, it's known that some cards have difficulties for resetting.
Do you reboot inside windows ?
or stop/start vm from proxmox gui directly also fail ?
Yes, if I restart windows guest inside, like, I`ve installed some updates - it fails to start video after restart.
If I restart guest from my host (from web interface of proxmox) -it gives me errors like timeout(I can show them later, when I get to my machine)
 
Somehow the guest restart issue gone away. I just reinstalled guest win8.1 to the ssd disk (which is placed in config like virtio0: /dev/disk/by-id/my-ssd-serial). so guest restarts normally, display shows my desktop.
 
Hey Sprit after doing apt-get dist-upgrade this week my card stopped working again. got any ideas whats up? I did not make any changes to the windows OS. just the proxmox host update.
 
nothing has change in proxmox code, maybe is it related to pve-qemu-kvm package upgrade , or pve-kernel upgrade (vfio framework).

maybe can you try to downgrade them manually ?

(you should have previous version in /var/cache/apt/archives/*.deb)
 
nothing has change in proxmox code, maybe is it related to pve-qemu-kvm package upgrade , or pve-kernel upgrade (vfio framework).

maybe can you try to downgrade them manually ?

(you should have previous version in /var/cache/apt/archives/*.deb)
Hey Sprit yes you are right. I actually moved the card into a different PCI slot and thats why the issue happened. I had forgotten that I moved it. Right now I need to load the vbios manually to get my 16x slot to work. Otherwise I have to move it to my 4x slot. I can do it with tis line.

args: -device vfio-pci,host=01:00.0,bus=pcie.0,addr=09.0,romfile=/root/bios/vbios.rom -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia43FIX

But it seems to not perform so well.

I noticed that the code proxomx generates is
bus=ich9-pcie-port-1 and I'm wondering if that makes a performance difference?



I tried to let proxmox generate the code itself with
hostpci0: 01:00.0,pcie=1,x-vga=on,romfile=/root/bios/vbios.rom

But that does not work
unable to parse value of 'hostpci0' - format error
romfile: property is not defined in schema and the schema does not allow additional properties


I also tried
args: -device 'vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on,romfile=/root/bios/vbios.rom' -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia43FIX

but that results in a kvm error of Bus 'ich9-pcie-port-1' not found

So I'm kinda poking around in the dark. Do you have any ideas how I can passthrough the card using the default proxmox settings but add a bios rom? thanks buddy :)
 
Last edited:
Ok making some progress. If I manually start the VM from the terminal with this command.

/usr/bin/kvm -id 120 -chardev 'socket,id=qmp,path=/var/run/qemu-server/120.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/120.pid -daemonize -smbios 'type=1,uuid=8bf2ecf9-95b7-4d75-8871-cf1c793874e2' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/kvm/OVMF_CODE-pure-efi.fd' -drive 'if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/dev/zvol/tank/vm/vm-120-disk-2' -name Win10 -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,enforce,kvm=off' -m 6000 -object 'memory-backend-ram,id=ram-node0,size=6000M' -numa 'node,nodeid=0,cpus=0-1,memdev=ram-node0' -k en-us -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=01:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,romfile=/root/bios/vbios.rom' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:88a649aa3bbd' -drive 'file=/dev/zvol/tank/vm/vm-120-disk-1,if=none,id=drive-ide0,cache=writethrough,format=raw,aio=threads,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' -drive 'file=/dev/zvol/tank/vm/vm-110-heroes,if=none,id=drive-sata0,cache=writethrough,format=raw,aio=threads,detect-zeroes=on' -device 'ide-drive,bus=ahci0.0,drive=drive-sata0,id=sata0' -netdev 'type=tap,id=net0,ifname=tap120i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=8A:69:7C:79:A3:B6,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'



everything works correctly and performance is great! now if only I could get it to launch with the bios from the web interface....
 
Hey Sprit yes you are right. I actually moved the card into a different PCI slot and thats why the issue happened. I had forgotten that I moved it. Right now I need to load the vbios manually to get my 16x slot to work. Otherwise I have to move it to my 4x slot. I can do it with tis line.

args: -device vfio-pci,host=01:00.0,bus=pcie.0,addr=09.0,romfile=/root/bios/vbios.rom -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia43FIX

But it seems to not perform so well.

I noticed that the code proxomx generates is
bus=ich9-pcie-port-1 and I'm wondering if that makes a performance difference?



I tried to let proxmox generate the code itself with
hostpci0: 01:00.0,pcie=1,x-vga=on,romfile=/root/bios/vbios.rom

But that does not work
unable to parse value of 'hostpci0' - format error
romfile: property is not defined in schema and the schema does not allow additional properties


I also tried
args: -device 'vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on,romfile=/root/bios/vbios.rom' -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia43FIX

but that results in a kvm error of Bus 'ich9-pcie-port-1' not found

So I'm kinda poking around in the dark. Do you have any ideas how I can passthrough the card using the default proxmox settings but add a bios rom? thanks buddy :)

Hi,
I'll make a patch to add romfile option to hostpci.

I'll try to send a package tomorrow
 
can you try this deb ?
http://odisoweb1.odiso.net/qemu-server_4.0-103_amd64.deb

it's add romfile option to hostpci

here the patch, if you want to apply it to your current proxmox

Code:
From 37af51077fb1cb42823fc2bbd69c3f32c53e70a5 Mon Sep 17 00:00:00 2001
From: Alexandre Derumier <aderumier@odiso.com>
Date: Fri, 6 Jan 2017 16:57:03 +0100
Subject: [PATCH] add romfile option to hostpci

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/QemuServer.pm | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 3c9d30a..654a985 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -999,6 +999,13 @@ EODESCR
        optional => 1,
        default => 1,
     },
+    romfile => {
+        type => 'string',
+        pattern => '\S+',
+        format_description => 'string',
+        description => "Custom romfile.",
+        optional => 1,
+    },
     pcie => {
        type => 'boolean',
         description =>  "Choose the PCI-express bus (needs the 'q35' machine model).",
@@ -2908,6 +2915,8 @@ sub config_to_command {
        }

        my $rombar = defined($d->{rombar}) && !$d->{rombar} ? ',rombar=0' : '';
+       my $romfile = $d->{romfile} if $d->{romfile};
+
        my $xvga = '';
        if ($d->{'x-vga'}) {
            $xvga = ',x-vga=on';
@@ -2934,6 +2943,7 @@ sub config_to_command {
            if($j == 0){
                $devicestr .= "$rombar$xvga";
                $devicestr .= ",multifunction=on" if $multifunction;
+               $devicestr .= ",romfile=$romfile" if $romfile;
            }

            push @$devices, '-device', $devicestr;
--
 
I installed the deb. Working great! thanks so much!! :):D:D:)

Also for anyone who is in the same situation as me Xeon cpu (no integrated graphics). They may need to dump the bios and use this flag to get the first PCI-e slot working.

I found the instructions on how to dump the bios here incase it helps anyone else.
http://lime-technology.com/forum/index.php?topic=43644.msg482110#msg482110

My machine is a Lenovo TS-140.

can you try this deb ?
http://odisoweb1.odiso.net/qemu-server_4.0-103_amd64.deb

it's add romfile option to hostpci

here the patch, if you want to apply it to your current proxmox

Code:
From 37af51077fb1cb42823fc2bbd69c3f32c53e70a5 Mon Sep 17 00:00:00 2001
From: Alexandre Derumier <aderumier@odiso.com>
Date: Fri, 6 Jan 2017 16:57:03 +0100
Subject: [PATCH] add romfile option to hostpci

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
PVE/QemuServer.pm | 10 ++++++++++
1 file changed, 10 insertions(+)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 3c9d30a..654a985 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -999,6 +999,13 @@ EODESCR
        optional => 1,
        default => 1,
     },
+    romfile => {
+        type => 'string',
+        pattern => '\S+',
+        format_description => 'string',
+        description => "Custom romfile.",
+        optional => 1,
+    },
     pcie => {
        type => 'boolean',
         description =>  "Choose the PCI-express bus (needs the 'q35' machine model).",
@@ -2908,6 +2915,8 @@ sub config_to_command {
        }

        my $rombar = defined($d->{rombar}) && !$d->{rombar} ? ',rombar=0' : '';
+       my $romfile = $d->{romfile} if $d->{romfile};
+
        my $xvga = '';
        if ($d->{'x-vga'}) {
            $xvga = ',x-vga=on';
@@ -2934,6 +2943,7 @@ sub config_to_command {
            if($j == 0){
                $devicestr .= "$rombar$xvga";
                $devicestr .= ",multifunction=on" if $multifunction;
+               $devicestr .= ",romfile=$romfile" if $romfile;
            }

            push @$devices, '-device', $devicestr;
--
 
I installed the deb. Working great! thanks so much!! :):D:D:)

Also for anyone who is in the same situation as me Xeon cpu (no integrated graphics). They may need to dump the bios and use this flag to get the first PCI-e slot working.

I found the instructions on how to dump the bios here incase it helps anyone else.
http://lime-technology.com/forum/index.php?topic=43644.msg482110#msg482110

My machine is a Lenovo TS-140.

ok, thanks for testing.
I'll send the patch to proxmox dev to have it upstream.

I'll add a note in the wiki for the dump of the bios
 
Hi everybody, I've just regged to this forum. First of all, thank you all, browsing this forum was a tremendous help so far. Also, kudos to the dev team, Proxmox came a long way in every aspect since I've last played with it around version 2.x. My setup is working on the level I can live with it, but that being said, there is still some things I want to ask you to help me solve.

My setup: HP Z620 with Xeon E5-2620 (VTx,VTd,IOMMU,IOMMU isolation,IOMMU interrupt remapping, so basically all the bells and whistles needed for PCI passthrough), with 2 VGA (MSI GT630 and NV Quadro FX 580). The host boots from the Quadro card and the GT630 is passed to a Win10 VM via the Seabios PCI method.
The system is fully updated from the no-subscription and the pvetest repos and also has spirit's patched qemu-server from this thread for the romfile support.

1st issue:
This setup works, but only after a clean boot/reboot of the host. If I reboot the VM, the card doesn't work. In dmesg the following message can be seen:
Code:
vfio-pci 0000:05:00.0: Invalid ROM contents
This thread contains the exact same issue: https://forum.proxmox.com/threads/gpu-passthrough-invalid-rom-contents-after-vm-restart.27583/
Passing the VGA's BIOS with the romfile option suppresses the error in the dmesg, but the VGA still doesn't want to work. :(

2nd issue:
In the Win10 VM there is a HUGE i/o issue. In task manager the disk graph is always around 95-100% with around 1.5-2MB/s io activity. The storage is VirtIO SCSI device using the latest drivers from the Fedora project. :-/ I don't think it a bottleneck of the actual, physical HDD as none of the tools I've tried (top,iotop) shows any significant iowait. (spikes are around max 25-30%). Also, before I've jumped to KVM/Proxmox, this host was running ESXi and running multiple Win10 VMs never were a problem, everything was snappy.

3rd issue:
As I said earlier, the Quadro card is used to boot the host. Now, I've configured it for passthrough, but can't seem to make it work. During boot, the screen of the host goes blank, I guess that when the vfio-pci driver grabs the card, but when I try to boot a VM with it, dmesg shows this error:
Code:
vfio-pci 0000:04:00.0: BAR 3: can't reserve [mem 0xca000000-0xcbffffff 64bit]
The screen doesn't even turn on. I guess some parts of the card's memory is not released by the host?
When I try to boot the VM with the card's romfile, the screen turns on, shows the last screen from the host's boot, just before it was grabbed by vfio-pci and a ton of artifacts.

(just an interesting thing, if the host boots with the GT630, the Quadro card host no problem with VM reboots :-/)

Sorry for the long post, but I'm too new to this and my google-fu bringed me so far...
I'm open to any idea, suggestion, willing to try beta packages, etc.

Thank you all!
 
1st issue:
This setup works, but only after a clean boot/reboot of the host. If I reboot the VM, the card doesn't work. In dmesg the following message can be seen:
Code:
vfio-pci 0000:05:00.0: Invalid ROM contents
This thread contains the exact same issue: https://forum.proxmox.com/threads/gpu-passthrough-invalid-rom-contents-after-vm-restart.27583/
Passing the VGA's BIOS with the romfile option suppresses the error in the dmesg, but the VGA still doesn't want to work. :(

all info I have found on the internet, is to use to romfile option for this bug.
also, Maybe try to change pci slot ?

2nd issue:
In the Win10 VM there is a HUGE i/o issue. In task manager the disk graph is always around 95-100% with around 1.5-2MB/s io activity. The storage is VirtIO SCSI device using the latest drivers from the Fedora project. :-/ I don't think it a bottleneck of the actual, physical HDD as none of the tools I've tried (top,iotop) shows any significant iowait. (spikes are around max 25-30%). Also, before I've jumped to KVM/Proxmox, this host was running ESXi and running multiple Win10 VMs never were a problem, everything was snappy.
can you post your vm config file ?

3rd issue:
As I said earlier, the Quadro card is used to boot the host. Now, I've configured it for passthrough, but can't seem to make it work. During boot, the screen of the host goes blank, I guess that when the vfio-pci driver grabs the card, but when I try to boot a VM with it, dmesg shows this error:
Code:
vfio-pci 0000:04:00.0: BAR 3: can't reserve [mem 0xca000000-0xcbffffff 64bit]

I have found some post saying to pass to grub kernel options:
Code:
"video=efifb:off"
 
  • Like
Reactions: Hipurnism

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!