Is anybody who succeeded GPU passthrough (NVIDIA) with Windows 7 ?

Discussion in 'Proxmox VE: Installation and configuration' started by Stargazer, Jan 25, 2018.

  1. Stargazer

    Stargazer New Member

    Joined:
    Jan 25, 2018
    Messages:
    6
    Likes Received:
    0
    Thank you for enduring my bad English, first. ^ _^ ;

    I used KVM with libvirt (virt-manager) and 'migrated' to Proxmox recently.

    I had succeeded GPU passthrough in libvirt with win7 win8.1 win10 but now I just do it with win10 with Proxmox.
    (win 10 seems very easy as it requires 'almost' only GPU rom)

    Win 8.1 seems to be easy but it isn't because it looks succeeded but the screen display isn't
    although it shows menu "Nvidia Control Panel" and 3d accelated Chrome doesn't show up (looks white blank screen.

    Win 7 is stuck on boot ( I suppose with cpu usage because I can't look at without VNC)
    My settings is


    ....
    machine: q35
    hostpci0: 01:00.0;01:00.1, pcie=1, x-vga=on, romfile=1.rom
    .....

    - pre-required works are done (iommu, vfio, acs patch, rom file ......)
    - installed Win 7 as UEFI with OVMF
    - boot option edited with boot up screen and set only "PCI-E Device" boot
    - manually updated kernel and qemu-kvm
    - added efi disk

    I set almost similar circumtances with similar options as libvirt with virt-manager,
    but just stuck on boot in Proxmox
    Is there any hints?

    Thanks in advance.
     
    #1 Stargazer, Jan 25, 2018
    Last edited: Jan 27, 2018
  2. frickelpit

    frickelpit New Member

    Joined:
    Oct 29, 2017
    Messages:
    15
    Likes Received:
    0
    I have the same problem. Win7 stucks on the bootscreen logo. With RDP connect you can see, the grafic card becomes a code 12 error in the device manager. No matter if UEFI or seabios. The result is the same. I think, it's actually not possible because nobody has a running config with gpu passthrough and proxmox 5.2.
     
    #2 frickelpit, Jan 26, 2018
    Last edited: Jan 26, 2018
  3. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    2,622
    Likes Received:
    240
    mhmm, could you post the output of 'qm showcmd ID' ?
    maybe some cpu/hv flags are not good for windows 7
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. frickelpit

    frickelpit New Member

    Joined:
    Oct 29, 2017
    Messages:
    15
    Likes Received:
    0
    No problem...

    Code:
    /usr/bin/kvm -id 205 -chardev 'socket,id=qmp,path=/var/run/qemu-server/205.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/205.pid -daemonize -smbios 'type=1,uuid=d4c04550-2da2-447e-8d4c-a2b38f192ed0' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/kvm/OVMF_CODE-pure-efi.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-205-disk-2' -name WS5 -smp '2,sockets=1,cores=2,maxcpus=2' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -cpu 'kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,kvm=off' -m 4096 -k de -object 'iothread,id=iothread-virtio1' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=01:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'usb-host,vendorid=0x05ac,productid=0x0221,id=usb0' -device 'usb-host,vendorid=0x045e,productid=0x00cb,id=usb1' -device 'usb-host,vendorid=0x04b8,productid=0x011f,id=usb2' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:17d28020a41f' -drive 'file=/pool1/TEMPLATES/template/iso/virtio-win-0.1.141.iso,if=none,id=drive-ide2,media=cdrom,aio=threads' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2' -drive 'file=/dev/pve/vm-205-disk-1,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -drive 'file=/dev/zvol/pool1/vm-205-disk-1,if=none,id=drive-virtio1,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb,iothread=iothread-virtio1' -netdev 'type=tap,id=net0,ifname=tap205i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=B2:6E:2F:28:3D:8A,netdev=net0,bus=pci.0,addr=0x12,id=net0' -machine 'type=q35'
     
  5. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    2,622
    Likes Received:
    240
    can you try to set the windows version to >= win8 (even though its running win7)?
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. frickelpit

    frickelpit New Member

    Joined:
    Oct 29, 2017
    Messages:
    15
    Likes Received:
    0
    With "win8" in config win7 crashes on boot logo screen.
     
  7. Stargazer

    Stargazer New Member

    Joined:
    Jan 25, 2018
    Messages:
    6
    Likes Received:
    0
    In libvirt/virt-manager, Win7 was successful because I can directly assign PCI-E device to boot device.

    I finally know it's unable in Proxmox because I can't assign GPU to boot.

    Windows 8.1, as I said above, it seems work normally but it doesn't show 3D part of screen.
    (as I know till now)
     
    #7 Stargazer, Jan 26, 2018
    Last edited: Jan 26, 2018
  8. JustAMan

    JustAMan New Member

    Joined:
    Jul 31, 2018
    Messages:
    8
    Likes Received:
    3
    Hello there!

    I've found the relevant bug against qemu-kvm,. google for "Red Hat Bugzilla – Bug 1273172" (I cannot post direct link to this bug as I'm a new user and not allowed to post external links so far).

    Following the ideas in that bug I was able to bring up a Win7 VM with NVidia GTX 1080 Ti passed through (last night Unigine Heaven on 4K output with everything on max yielded a score of about 970, while I see native results ranging from 980 to 1010, so not that bad).

    The problem haunting my configuration (error code 12 - not enough resources) has something to do with the fact that by default host PCI devices are passed through as though they are behind a PCI bridge (which works just fine on Win10 and MacOS High Sierra, BTW, on the same hardware).

    Below is the configuration file for my VM. I have passed through VGA, HDMI audio and a USB controller.

    Code:
    vga: none
    agent: 1
    args: -cpu host,hv_time,kvm=off,hv_vendor_id=TheNeverhood,-hypervisor -device vfio-pci,host=02:00.0,id=hostpci0.0,bus=pcie.0,addr=0x10.0,multifunction=on -device vfio-pci,host=02:00.1,id=hostpci0.1,bus=pcie.0,addr=0x10.1
    balloon: 0
    bios: ovmf
    boot: dcn
    bootdisk: scsi1
    cores: 4
    cpu: host
    efidisk0: local:101/vm-101-disk-2.qcow2,size=128K
    hostpci1: 00:14.0,pcie=1
    ide2: none,media=cdrom
    machine: q35
    memory: 8192
    name: win7main
    net0: virtio=E2:68:03:36:43:6A,bridge=vmbr0
    numa: 0
    ostype: win7
    scsi1: local:101/vm-101-disk-1.qcow2,discard=on,iothread=1,size=120G
    scsihw: virtio-scsi-pci
    smbios1: uuid=4c2c2891-99b8-4287-b057-9f2aeaad7692
    sockets: 1
    
    Hope that helps other people interested in such a thing as Windows 7 with passed through video card... :)
     
    frickelpit and Tahsin like this.
  9. frickelpit

    frickelpit New Member

    Joined:
    Oct 29, 2017
    Messages:
    15
    Likes Received:
    0
    Best hint ever! I have tried so many hours to run this shitty W7 with passed through video card. I always run into error 12. Now i can use my old CAD soft virtualised. GREAT!!!
     
  10. Proxmox India

    Proxmox India New Member
    Proxmox VE Subscriber

    Joined:
    Oct 16, 2017
    Messages:
    20
    Likes Received:
    0
    did this finally work ?
     
  11. frickelpit

    frickelpit New Member

    Joined:
    Oct 29, 2017
    Messages:
    15
    Likes Received:
    0
    Yes. It's not a joke! :)
     
  12. frickelpit

    frickelpit New Member

    Joined:
    Oct 29, 2017
    Messages:
    15
    Likes Received:
    0
    After testing a while i have to say unfortunately it works only one time. After shutting down the win7 vm, it is not possible to start again or to start another vm for example win10 with gpu passthrough. The host system freeze shortly after vm start :(

    I think that the proxmox mechanism to release the grafic card isn't made for that use case.... ?
     
  13. dcsapak

    dcsapak Proxmox Staff Member
    Staff Member

    Joined:
    Feb 1, 2016
    Messages:
    2,622
    Likes Received:
    240
    unfortunately this is more related to the hardware and the guest os (which is responsible to release the devices)
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  14. JustAMan

    JustAMan New Member

    Joined:
    Jul 31, 2018
    Messages:
    8
    Likes Received:
    3
    My own setup (ASUS Z170 PRO motherboard + NVidia 1080 Ti video) works fine so far.
    I did not give it any significant load, though, just ran Unigine Heaven a couple of times.

    What video card do you have?
     
  15. frickelpit

    frickelpit New Member

    Joined:
    Oct 29, 2017
    Messages:
    15
    Likes Received:
    0
    I use a Fujitsu DS D3417-B2 Mobo with a XEON E3-1245 v6 and 64GB ECC-RAM. Grafic card is a NVidia Quadro P2000 with driver v391.89 WHQL for Win7x64.
     
  16. JustAMan

    JustAMan New Member

    Joined:
    Jul 31, 2018
    Messages:
    8
    Likes Received:
    3
    I have tinkered my own version of qemu-server that is able to control how to pass the card (as it seems it does a few more things when you put "x-vga=on" in the hostpci line compared to my initial config I posted above).
    Let me get back to home and I can try and show you qemu command line so we can try to compare mine and yours and see if there's some difference.

    Then you can try to construct a commandline yourself and see if patched commandline fixes your issues.
     
  17. JustAMan

    JustAMan New Member

    Joined:
    Jul 31, 2018
    Messages:
    8
    Likes Received:
    3
    Here is the result of qm showcmd 101 (101 is the id of my Win7 VM):


    Try comparing with yours to see if there's any difference.
     
    frickelpit likes this.
  18. frickelpit

    frickelpit New Member

    Joined:
    Oct 29, 2017
    Messages:
    15
    Likes Received:
    0
    Thx. I have tested it, but nothing helps. Now my vm.cfg looks like yours. But with the difference that my second card which i passthrough to the vm is a pci (USB3-Card) and not a pcie card. I have tried to add the device-line from the USB3-Card to the args-line in cfg, but Proxmox find this not usefull :(

    If i run this command at console proxmox can't find "bus=pci.0", if i start same cfg with webgui the vm starts without a problem?!
    Code:
    /usr/bin/kvm -id 203 -name CAD7 -chardev 'socket,id=qmp,path=/var/run/qemu-server/203.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/203.pid -daemonize -smbios 'type=1,uuid=a7d9be4d-014c-476f-b42f-ab7bc1738471' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/dev/pve/vm-203-disk-2' -smp '8,sockets=1,cores=8,maxcpus=8' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'Skylake-Client,+kvm_pv_unhalt,+kvm_pv_eoi,enforce,kvm=off,vendor=GenuineIntel' -m 8192 -object 'iothread,id=iothread-virtio1' -readconfig /usr/share/qemu-server/pve-q35.cfg  -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=01:00.0,id=hostpci0.0,bus=pcie.0,addr=0x10.0,multifunction=on' -device 'vfio-pci,host=01:00.1,id=hostpci0.1,bus=pcie.0,addr=0x10.1' -device 'vfio-pci,host=02:00.0,id=hostpci1,bus=pci.0,addr=0x11' -chardev 'socket,path=/var/run/qemu-server/203.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:17d28020a41f' -drive 'file=/dev/pve/vm-203-disk-1,if=none,id=drive-virtio0,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -drive 'file=/dev/zvol/pool1/vm-203-disk-1,if=none,id=drive-virtio1,cache=writeback,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb,iothread=iothread-virtio1' -netdev 'type=tap,id=net0,ifname=tap203i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=B2:6E:2F:28:3D:8A,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'

    btw... the device manager in win7 is free of errors or warnings. The problem comes after the shutdown from win7. Start another vm and the host will be frozen.
     
    #18 frickelpit, Aug 7, 2018
    Last edited: Aug 13, 2018
  19. JustAMan

    JustAMan New Member

    Joined:
    Jul 31, 2018
    Messages:
    8
    Likes Received:
    3
    What happens if you start this VM again after shutting it down, does it work? I can not test other VMs now as I don't have them yet.
    Also, what is the output when you double-click "Start" task in task log? And is there anything interesting in dmesg after turning problematic VM on or off?
     
  20. frickelpit

    frickelpit New Member

    Joined:
    Oct 29, 2017
    Messages:
    15
    Likes Received:
    0
    Problem solved!
    The reason was the USB3 PCI-Card which i want to passthrough to the Win-VMs. At host start the card becomes the kernel driver "xhci_hcd", same as the host use for his USB3-chip. WIN10 has no problems with this, differently to WIN7. I haven't found a way to prevent the card from loading the driver because it's a kernel driver and not a modul.

    So i passthrough the usb ports and not the hole pci card. Not nice but it works :)
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice