Code 43 NVIDIA driver error within Windows 10 VM, Geforce 750 Ti Passthrough

Yaay!! it works! many thanks!
It took me 3 weeks of struggle!
my hw is
gigabyte 650ti
i5-3470
mb asus P8Q77M2
config
Code: 
bios: ovmf
boot: c
bootdisk: virtio0
cores: 2
hostpci0: 01:00,pcie=1,x-vga=on
ide0: iso1:iso/virtio-win-0.1.112.iso,media=cdrom,size=55592K
machine: q35
memory: 4096
name: win1
net0: e1000=32:33:62:38:63:35,bridge=vmbr0
numa: 0
ostype: win8
scsihw: virtio-scsi-pci
smbios1: uuid=6b380506-55b3-4c7e-a4cb-caacded045fe
sockets: 1
usb0: host=3-1
usb1: host=1-4
virtio0: local:100/vm-100-disk-1.qcow2,size=32G
also, i`ve tried it on some old and cheap ati5450 - no luck.
And if I reboot the wm - no signal on the monitor, I have to reboot the host. any way to fix this?
also, is there any way I can put my internal hd4000 to another linux wm(I want some machine with kodi mediacenter)?
 
lexa500 said:
Yaay!! it works! many thanks!
It took me 3 weeks of struggle!
my hw is
gigabyte 650ti
i5-3470
mb asus P8Q77M2
config
Click to expand...
Thanks for sharing the conf !

also, i`ve tried it on some old and cheap ati5450 - no luck.
And if I reboot the wm - no signal on the monitor, I have to reboot the host. any way to fix this?
Click to expand...
Can you send the vm config ?

But this blog post said it's working for 5450.
http://vfio.blogspot.be/2014/08/dual-vga-assignment-geforce-radeon.html

also, is there any way I can put my internal hd4000 to another linux wm(I want some machine with kodi mediacenter)?
Click to expand...
for intel, I have seen some improvements in last kernel 4.4. So it's better to wait for proxmox 4.2 in some month to test it again.
 
spirit said:
YESSS !

can you send your
- vmid.conf
- #pveversion -v
- maybe other tuning done in grub

I'll update the wiki


my email can be public : aderumier@odiso.com ;)

No need to paypal, I'm glad to help the proxmox community.
And thank you to have take time to test, because my geforce 970 is not able to passthrough currently, so I'm blind for testing currently ;)
Click to expand...

All I did was make sure to PCI Stub the card. I followed a debian(or maybe ubuntu) tutorial for that I cant remember. I do remember you have to check after that your card is running PCI stub driver and not getting claimed by nvidia driver. with lspci -nnk.
Then I did the prerelease repo you recommended. The rest was just like the proxmox wiki.

Here is my conf
Code: 
bootdisk: sata0
cores: 2
hostpci0: 01:00,pcie=1,x-vga=on
ide2: local:iso/virtio-win.iso,media=cdrom
machine: q35
memory: 6000
name: Win7
net0: e1000=36:31:62:36:30:37
numa: 0
ostype: win7
sata0: Tank:vm-100-disk-1,cache=writeback,size=32G
smbios1: uuid=cc892184-b29d-4193-8f8f-378ba36c538a
sockets: 1
usb0: host=1-10.1
usb1: host=1-10.2
usb2: host=1-10.3
usb3: host=1-4

Thanks again buddy you rock :)
 
Last edited:
spirit said:
Thanks for sharing the conf !


Can you send the vm config ?

But this blog post said it's working for 5450.
http://vfio.blogspot.be/2014/08/dual-vga-assignment-geforce-radeon.html


for intel, I have seen some improvements in last kernel 4.4. So it's better to wait for proxmox 4.2 in some month to test it again.
Click to expand...
sure
Code: 
cat /etc/pve/qemu-server/100.conf
agent: 1
bios: ovmf
boot: c
bootdisk: virtio0
cores: 4
hostpci0: 01:00,pcie=1,x-vga=on
machine: q35
memory: 6144
name: win1
net0: e1000=32:33:62:38:63:35,bridge=vmbr0
numa: 0
onboot: 1
ostype: win8
scsihw: virtio-scsi-pci
smbios1: uuid=6b380506-55b3-4c7e-a4cb-caacded045fe
sockets: 1
usb0: host=3-1.6
usb1: host=1-4
virtio0: local:100/vm-100-disk-1.qcow2,size=32G
this is my vm config for nvidia 650ti with freshly installed windows guest
As for ati card - i mistyped hwid in vfio.conf, and I gave it back to my friend, so once I see him again I will try it again.
 
lexa500 said:
sure
this is my vm config for nvidia 650ti with freshly installed windows guest
As for ati card - i mistyped hwid in vfio.conf, and I gave it back to my friend, so once I see him again I will try it again.
Click to expand...

Is it the same config for ati card ? (because I don't think it's ovmf/uefi compatible, card is too old)
 
spirit said:
Is it the same config for ati card ? (because I don't think it's ovmf/uefi compatible, card is too old)
Click to expand...
yes, I changed only hwid in vfio.conf. So, I get it, why ati card did not start at all. anyway, I took it as a test. Do you still need test results with it?
Any solution with rebooting wm with nvidia card and not showing anything (It works to reboot the host)?
Also I would greatly appreciate any advice on fine-tuning windows wm (like pushing the whole sata controller inside or similiar information)
 
spirit said:
mmm, it's known that some cards have difficulties for resetting.
Do you reboot inside windows ?
or stop/start vm from proxmox gui directly also fail ?
Click to expand...
Yes, if I restart windows guest inside, like, I`ve installed some updates - it fails to start video after restart.
If I restart guest from my host (from web interface of proxmox) -it gives me errors like timeout(I can show them later, when I get to my machine)
 
Somehow the guest restart issue gone away. I just reinstalled guest win8.1 to the ssd disk (which is placed in config like virtio0: /dev/disk/by-id/my-ssd-serial). so guest restarts normally, display shows my desktop.
 
Hey Sprit after doing apt-get dist-upgrade this week my card stopped working again. got any ideas whats up? I did not make any changes to the windows OS. just the proxmox host update.
 
nothing has change in proxmox code, maybe is it related to pve-qemu-kvm package upgrade , or pve-kernel upgrade (vfio framework).

maybe can you try to downgrade them manually ?

(you should have previous version in /var/cache/apt/archives/*.deb)
 
spirit said:
nothing has change in proxmox code, maybe is it related to pve-qemu-kvm package upgrade , or pve-kernel upgrade (vfio framework).

maybe can you try to downgrade them manually ?

(you should have previous version in /var/cache/apt/archives/*.deb)
Click to expand...
Hey Sprit yes you are right. I actually moved the card into a different PCI slot and thats why the issue happened. I had forgotten that I moved it. Right now I need to load the vbios manually to get my 16x slot to work. Otherwise I have to move it to my 4x slot. I can do it with tis line.

args: -device vfio-pci,host=01:00.0,bus=pcie.0,addr=09.0,romfile=/root/bios/vbios.rom -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia43FIX

But it seems to not perform so well.

I noticed that the code proxomx generates is
bus=ich9-pcie-port-1 and I'm wondering if that makes a performance difference?



I tried to let proxmox generate the code itself with
hostpci0: 01:00.0,pcie=1,x-vga=on,romfile=/root/bios/vbios.rom

But that does not work
unable to parse value of 'hostpci0' - format error
romfile: property is not defined in schema and the schema does not allow additional properties

I also tried
args: -device 'vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on,romfile=/root/bios/vbios.rom' -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia43FIX

but that results in a kvm error of Bus 'ich9-pcie-port-1' not found

So I'm kinda poking around in the dark. Do you have any ideas how I can passthrough the card using the default proxmox settings but add a bios rom? thanks buddy :)
 
Last edited:
Ok making some progress. If I manually start the VM from the terminal with this command.

/usr/bin/kvm -id 120 -chardev 'socket,id=qmp,path=/var/run/qemu-server/120.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/120.pid -daemonize -smbios 'type=1,uuid=8bf2ecf9-95b7-4d75-8871-cf1c793874e2' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/kvm/OVMF_CODE-pure-efi.fd' -drive 'if=pflash,unit=1,id=drive-efidisk0,format=raw,file=/dev/zvol/tank/vm/vm-120-disk-2' -name Win10 -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,enforce,kvm=off' -m 6000 -object 'memory-backend-ram,id=ram-node0,size=6000M' -numa 'node,nodeid=0,cpus=0-1,memdev=ram-node0' -k en-us -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=01:00.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,romfile=/root/bios/vbios.rom' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:88a649aa3bbd' -drive 'file=/dev/zvol/tank/vm/vm-120-disk-1,if=none,id=drive-ide0,cache=writethrough,format=raw,aio=threads,detect-zeroes=on' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -device 'ahci,id=ahci0,multifunction=on,bus=pci.0,addr=0x7' -drive 'file=/dev/zvol/tank/vm/vm-110-heroes,if=none,id=drive-sata0,cache=writethrough,format=raw,aio=threads,detect-zeroes=on' -device 'ide-drive,bus=ahci0.0,drive=drive-sata0,id=sata0' -netdev 'type=tap,id=net0,ifname=tap120i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown' -device 'e1000,mac=8A:69:7C:79:A3:B6,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'



everything works correctly and performance is great! now if only I could get it to launch with the bios from the web interface....
 
SpaceCowBoy said:
Hey Sprit yes you are right. I actually moved the card into a different PCI slot and thats why the issue happened. I had forgotten that I moved it. Right now I need to load the vbios manually to get my 16x slot to work. Otherwise I have to move it to my 4x slot. I can do it with tis line.

args: -device vfio-pci,host=01:00.0,bus=pcie.0,addr=09.0,romfile=/root/bios/vbios.rom -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia43FIX

But it seems to not perform so well.

I noticed that the code proxomx generates is
bus=ich9-pcie-port-1 and I'm wondering if that makes a performance difference?



I tried to let proxmox generate the code itself with
hostpci0: 01:00.0,pcie=1,x-vga=on,romfile=/root/bios/vbios.rom

But that does not work
unable to parse value of 'hostpci0' - format error
romfile: property is not defined in schema and the schema does not allow additional properties

I also tried
args: -device 'vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-vga=on,multifunction=on,romfile=/root/bios/vbios.rom' -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia43FIX

but that results in a kvm error of Bus 'ich9-pcie-port-1' not found

So I'm kinda poking around in the dark. Do you have any ideas how I can passthrough the card using the default proxmox settings but add a bios rom? thanks buddy :)
Click to expand...

Hi,
I'll make a patch to add romfile option to hostpci.

I'll try to send a package tomorrow
 
can you try this deb ?
http://odisoweb1.odiso.net/qemu-server_4.0-103_amd64.deb

it's add romfile option to hostpci

here the patch, if you want to apply it to your current proxmox

Code: 
From 37af51077fb1cb42823fc2bbd69c3f32c53e70a5 Mon Sep 17 00:00:00 2001
From: Alexandre Derumier <aderumier@odiso.com>
Date: Fri, 6 Jan 2017 16:57:03 +0100
Subject: [PATCH] add romfile option to hostpci

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
 PVE/QemuServer.pm | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 3c9d30a..654a985 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -999,6 +999,13 @@ EODESCR
        optional => 1,
        default => 1,
     },
+    romfile => {
+        type => 'string',
+        pattern => '\S+',
+        format_description => 'string',
+        description => "Custom romfile.",
+        optional => 1,
+    },
     pcie => {
        type => 'boolean',
         description =>  "Choose the PCI-express bus (needs the 'q35' machine model).",
@@ -2908,6 +2915,8 @@ sub config_to_command {
        }

        my $rombar = defined($d->{rombar}) && !$d->{rombar} ? ',rombar=0' : '';
+       my $romfile = $d->{romfile} if $d->{romfile};
+
        my $xvga = '';
        if ($d->{'x-vga'}) {
            $xvga = ',x-vga=on';
@@ -2934,6 +2943,7 @@ sub config_to_command {
            if($j == 0){
                $devicestr .= "$rombar$xvga";
                $devicestr .= ",multifunction=on" if $multifunction;
+               $devicestr .= ",romfile=$romfile" if $romfile;
            }

            push @$devices, '-device', $devicestr;
--
 
I installed the deb. Working great! thanks so much!! :):D:D:)

Also for anyone who is in the same situation as me Xeon cpu (no integrated graphics). They may need to dump the bios and use this flag to get the first PCI-e slot working.

I found the instructions on how to dump the bios here incase it helps anyone else.
http://lime-technology.com/forum/index.php?topic=43644.msg482110#msg482110

My machine is a Lenovo TS-140.

spirit said:
can you try this deb ?
http://odisoweb1.odiso.net/qemu-server_4.0-103_amd64.deb

it's add romfile option to hostpci

here the patch, if you want to apply it to your current proxmox

Code: 
From 37af51077fb1cb42823fc2bbd69c3f32c53e70a5 Mon Sep 17 00:00:00 2001
From: Alexandre Derumier <aderumier@odiso.com>
Date: Fri, 6 Jan 2017 16:57:03 +0100
Subject: [PATCH] add romfile option to hostpci

Signed-off-by: Alexandre Derumier <aderumier@odiso.com>
---
PVE/QemuServer.pm | 10 ++++++++++
1 file changed, 10 insertions(+)

diff --git a/PVE/QemuServer.pm b/PVE/QemuServer.pm
index 3c9d30a..654a985 100644
--- a/PVE/QemuServer.pm
+++ b/PVE/QemuServer.pm
@@ -999,6 +999,13 @@ EODESCR
        optional => 1,
        default => 1,
     },
+    romfile => {
+        type => 'string',
+        pattern => '\S+',
+        format_description => 'string',
+        description => "Custom romfile.",
+        optional => 1,
+    },
     pcie => {
        type => 'boolean',
         description =>  "Choose the PCI-express bus (needs the 'q35' machine model).",
@@ -2908,6 +2915,8 @@ sub config_to_command {
        }

        my $rombar = defined($d->{rombar}) && !$d->{rombar} ? ',rombar=0' : '';
+       my $romfile = $d->{romfile} if $d->{romfile};
+
        my $xvga = '';
        if ($d->{'x-vga'}) {
            $xvga = ',x-vga=on';
@@ -2934,6 +2943,7 @@ sub config_to_command {
            if($j == 0){
                $devicestr .= "$rombar$xvga";
                $devicestr .= ",multifunction=on" if $multifunction;
+               $devicestr .= ",romfile=$romfile" if $romfile;
            }

            push @$devices, '-device', $devicestr;
--
Click to expand...
 
SpaceCowBoy said:
I installed the deb. Working great! thanks so much!! :):D:D:)

Also for anyone who is in the same situation as me Xeon cpu (no integrated graphics). They may need to dump the bios and use this flag to get the first PCI-e slot working.

I found the instructions on how to dump the bios here incase it helps anyone else.
http://lime-technology.com/forum/index.php?topic=43644.msg482110#msg482110

My machine is a Lenovo TS-140.
Click to expand...

ok, thanks for testing.
I'll send the patch to proxmox dev to have it upstream.

I'll add a note in the wiki for the dump of the bios
 
Hi everybody, I've just regged to this forum. First of all, thank you all, browsing this forum was a tremendous help so far. Also, kudos to the dev team, Proxmox came a long way in every aspect since I've last played with it around version 2.x. My setup is working on the level I can live with it, but that being said, there is still some things I want to ask you to help me solve.

My setup: HP Z620 with Xeon E5-2620 (VTx,VTd,IOMMU,IOMMU isolation,IOMMU interrupt remapping, so basically all the bells and whistles needed for PCI passthrough), with 2 VGA (MSI GT630 and NV Quadro FX 580). The host boots from the Quadro card and the GT630 is passed to a Win10 VM via the Seabios PCI method.
The system is fully updated from the no-subscription and the pvetest repos and also has spirit's patched qemu-server from this thread for the romfile support.

1st issue:
This setup works, but only after a clean boot/reboot of the host. If I reboot the VM, the card doesn't work. In dmesg the following message can be seen:
Code: 
vfio-pci 0000:05:00.0: Invalid ROM contents
This thread contains the exact same issue: https://forum.proxmox.com/threads/gpu-passthrough-invalid-rom-contents-after-vm-restart.27583/
Passing the VGA's BIOS with the romfile option suppresses the error in the dmesg, but the VGA still doesn't want to work. :(

2nd issue:
In the Win10 VM there is a HUGE i/o issue. In task manager the disk graph is always around 95-100% with around 1.5-2MB/s io activity. The storage is VirtIO SCSI device using the latest drivers from the Fedora project. :-/ I don't think it a bottleneck of the actual, physical HDD as none of the tools I've tried (top,iotop) shows any significant iowait. (spikes are around max 25-30%). Also, before I've jumped to KVM/Proxmox, this host was running ESXi and running multiple Win10 VMs never were a problem, everything was snappy.

3rd issue:
As I said earlier, the Quadro card is used to boot the host. Now, I've configured it for passthrough, but can't seem to make it work. During boot, the screen of the host goes blank, I guess that when the vfio-pci driver grabs the card, but when I try to boot a VM with it, dmesg shows this error:
Code: 
vfio-pci 0000:04:00.0: BAR 3: can't reserve [mem 0xca000000-0xcbffffff 64bit]
The screen doesn't even turn on. I guess some parts of the card's memory is not released by the host?
When I try to boot the VM with the card's romfile, the screen turns on, shows the last screen from the host's boot, just before it was grabbed by vfio-pci and a ton of artifacts.

(just an interesting thing, if the host boots with the GT630, the Quadro card host no problem with VM reboots :-/)

Sorry for the long post, but I'm too new to this and my google-fu bringed me so far...
I'm open to any idea, suggestion, willing to try beta packages, etc.

Thank you all!
 
Hipurnism said:
1st issue:
This setup works, but only after a clean boot/reboot of the host. If I reboot the VM, the card doesn't work. In dmesg the following message can be seen:
Code: 
vfio-pci 0000:05:00.0: Invalid ROM contents
This thread contains the exact same issue: https://forum.proxmox.com/threads/gpu-passthrough-invalid-rom-contents-after-vm-restart.27583/
Passing the VGA's BIOS with the romfile option suppresses the error in the dmesg, but the VGA still doesn't want to work. :(
Click to expand...

all info I have found on the internet, is to use to romfile option for this bug.
also, Maybe try to change pci slot ?

2nd issue:
In the Win10 VM there is a HUGE i/o issue. In task manager the disk graph is always around 95-100% with around 1.5-2MB/s io activity. The storage is VirtIO SCSI device using the latest drivers from the Fedora project. :-/ I don't think it a bottleneck of the actual, physical HDD as none of the tools I've tried (top,iotop) shows any significant iowait. (spikes are around max 25-30%). Also, before I've jumped to KVM/Proxmox, this host was running ESXi and running multiple Win10 VMs never were a problem, everything was snappy.
Click to expand...
can you post your vm config file ?

3rd issue:
As I said earlier, the Quadro card is used to boot the host. Now, I've configured it for passthrough, but can't seem to make it work. During boot, the screen of the host goes blank, I guess that when the vfio-pci driver grabs the card, but when I try to boot a VM with it, dmesg shows this error:
Code: 
vfio-pci 0000:04:00.0: BAR 3: can't reserve [mem 0xca000000-0xcbffffff 64bit]
Click to expand...

I have found some post saying to pass to grub kernel options:
Code: 
"video=efifb:off"
 
  • Like
Reactions: Hipurnism
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back