[SOLVED] 2 GPU 1050ti , 2 NVME passthrough for 2VM with 1GPU,1NVME/VM

C

ChonLaWan

Active Member
Oct 6, 2018
3
0
41
46
I'm follow step by.
https://pve.proxmox.com/wiki/Pci_passthrough for 1050Ti
https://pve.proxmox.com/wiki/Physical_disk_to_kvm for NVME

Hardware
ROG MAXIMUS X CODE
Intel i7 8700
NVME SAMSUNG 970 EVO x2
GTX 1050Ti x2

Now I can boot and install windows for 2 VM 101,102
Config is

/etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1c82,10de:0fb9,144d:a808 disable_vga=1

/etc/modprobe.d/iommu_unsafe_interrupts.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1 I try remove yet it still have a problem.

/etc/default/grub
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=efifb:off"
I try add "pcie_acs_override=downstream" yet.

/etc/modules
vfio_pci
vfio
vfio_iommu_type1
vfio_virqfd

101
agent: 1
balloon: 0
bios: ovmf
boot: cd
bootdisk: virtio0
cores: 6
cpu: host
hostpci0: 01:00.0;01:00.1,pcie=1,x-vga=on
machine: q35
memory: 14336
name: GAME01
net0: virtio=16:EC:0D:0A:3D:52,bridge=vmbr0
numa: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=5b949b6b-6f91-4e4e-8bec-438992033497
sockets: 1
usb0: host=1-5.1,usb3=1
usb1: host=1-5.2,usb3=1
usb2: host=1-5.3,usb3=1
usb3: host=1-5.4,usb3=1
virtio0: /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_250GB_S465NB0K578902Z,cache=writethrough,size=244198584K
Click to expand...

102
agent: 1
balloon: 0
bios: ovmf
boot: cd
bootdisk: virtio1
cores: 6
cpu: host
hostpci1: 02:00.0;02:00.1,pcie=1,x-vga=on
machine: q35
memory: 14336
name: GAME02
net1: virtio=DE:98:9C:9B:6C:1C,bridge=vmbr0
numa: 1
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=004c9002-601b-45a7-b26c-b81f6e92bdbc
sockets: 1
usb0: host=1-6.1,usb3=1
usb1: host=1-6.2,usb3=1
usb2: host=1-6.3,usb3=1
usb3: host=1-6.4,usb3=1
virtio1: /dev/disk/by-id/nvme-Samsung_SSD_970_EVO_250GB_S465NB0K592220V,cache=writethrough,size=244198584K
Click to expand...


Problem is it can boot only 1 VM
If I start 101 yet when start 102 it show error
Virtual Environment 5.2-9

Search
Virtual Machine 102 (GAME02) on node 'pve'

Server View
Logs
()
no efidisk configured! Using temporary efivars disk.
kvm: -device vfio-pci,host=02:00.0,id=hostpci1.0,bus=ich9-pcie-port-2,addr=0x0.0,multifunction=on: vfio error: 0000:02:00.0: failed to open /dev/vfio/1: Device or resource busy
TASK ERROR: start failed: command '/usr/bin/kvm -id 102 -name GAME02 -chardev 'socket,id=qmp,path=/var/run/qemu-server/102.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/102.pid -daemonize -smbios 'type=1,uuid=004c9002-601b-45a7-b26c-b81f6e92bdbc' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/tmp/102-ovmf.fd' -smp '6,sockets=1,cores=6,maxcpus=6' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,kvm=off' -m 14336 -object 'memory-backend-ram,id=ram-node0,size=14336M' -numa 'node,nodeid=0,cpus=0-5,memdev=ram-node0' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'nec-usb-xhci,id=xhci,bus=pci.1,addr=0x1b' -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=02:00.0,id=hostpci1.0,bus=ich9-pcie-port-2,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=02:00.1,id=hostpci1.1,bus=ich9-pcie-port-2,addr=0x0.1' -device 'usb-host,bus=xhci.0,hostbus=1,hostport=6.1,id=usb0' -device 'usb-host,bus=xhci.0,hostbus=1,hostport=6.2,id=usb1' -device 'usb-host,bus=xhci.0,hostbus=1,hostport=6.3,id=usb2' -device 'usb-host,bus=xhci.0,hostbus=1,hostport=6.4,id=usb3' -chardev 'socket,path=/var/run/qemu-server/102.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:525f42b94819' -drive 'file=/dev/disk/by-id/nvme-Samsung_SSD_970_EVO_250GB_S465NB0K592220V,if=none,id=drive-virtio1,cache=writethrough,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio1,id=virtio1,bus=pci.0,addr=0xb,bootindex=100' -netdev 'type=tap,id=net1,ifname=tap102i1,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=DE:98:9C:9B:6C:1C,netdev=net1,bus=pci.0,addr=0x13,id=net1' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
Click to expand...

But if start 102 ready when start 101 it will show same error.
Virtual Environment 5.2-9

Search
Virtual Machine 101 (GAME01) on node 'pve'

Server View
Logs
()
no efidisk configured! Using temporary efivars disk.
kvm: -device vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on: vfio error: 0000:01:00.0: failed to open /dev/vfio/1: Device or resource busy
TASK ERROR: start failed: command '/usr/bin/kvm -id 101 -name GAME01 -chardev 'socket,id=qmp,path=/var/run/qemu-server/101.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -pidfile /var/run/qemu-server/101.pid -daemonize -smbios 'type=1,uuid=5b949b6b-6f91-4e4e-8bec-438992033497' -drive 'if=pflash,unit=0,format=raw,readonly,file=/usr/share/pve-edk2-firmware//OVMF_CODE.fd' -drive 'if=pflash,unit=1,format=raw,id=drive-efidisk0,file=/tmp/101-ovmf.fd' -smp '6,sockets=1,cores=6,maxcpus=6' -nodefaults -boot 'menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg' -vga none -nographic -no-hpet -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=proxmox,hv_spinlocks=0x1fff,hv_vapic,hv_time,hv_reset,hv_vpindex,hv_runtime,hv_relaxed,kvm=off' -m 14336 -object 'memory-backend-ram,id=ram-node0,size=14336M' -numa 'node,nodeid=0,cpus=0-5,memdev=ram-node0' -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'nec-usb-xhci,id=xhci,bus=pci.1,addr=0x1b' -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=01:00.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on' -device 'vfio-pci,host=01:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1' -device 'usb-host,bus=xhci.0,hostbus=1,hostport=5.1,id=usb0' -device 'usb-host,bus=xhci.0,hostbus=1,hostport=5.2,id=usb1' -device 'usb-host,bus=xhci.0,hostbus=1,hostport=5.3,id=usb2' -device 'usb-host,bus=xhci.0,hostbus=1,hostport=5.4,id=usb3' -chardev 'socket,path=/var/run/qemu-server/101.qga,server,nowait,id=qga0' -device 'virtio-serial,id=qga0,bus=pci.0,addr=0x8' -device 'virtserialport,chardev=qga0,name=org.qemu.guest_agent.0' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:525f42b94819' -drive 'file=/dev/disk/by-id/nvme-Samsung_SSD_970_EVO_250GB_S465NB0K578902Z,if=none,id=drive-virtio0,cache=writethrough,format=raw,aio=threads,detect-zeroes=on' -device 'virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap101i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=16:EC:0D:0A:3D:52,netdev=net0,bus=pci.0,addr=0x12,id=net0' -rtc 'driftfix=slew,base=localtime' -machine 'type=q35' -global 'kvm-pit.lost_tick_policy=discard'' failed: exit code 1
Click to expand...

2 GPU have same ids

root@pve:/# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/15/devices/0000:06:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:17.0
/sys/kernel/iommu_groups/13/devices/0000:03:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/11/devices/0000:00:1f.4
/sys/kernel/iommu_groups/11/devices/0000:00:1f.2
/sys/kernel/iommu_groups/11/devices/0000:00:1f.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/1/devices/0000:02:00.1
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:02:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.4
/sys/kernel/iommu_groups/16/devices/0000:07:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:1b.0
/sys/kernel/iommu_groups/14/devices/0000:05:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:16.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.6
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.6



root@pve:/# lspci -n -s 01:00
01:00.0 0300: 10de:1c82 (rev a1)
01:00.1 0403: 10de:0fb9 (rev a1)
root@pve:/# lspci -n -s 02:00
02:00.0 0300: 10de:1c82 (rev a1)
02:00.1 0403: 10de:0fb9 (rev a1)

root@pve:~# lspci
00:00.0 Host bridge: Intel Corporation Device 3ec2 (rev 07)
00:01.0 PCI bridge: Intel Corporation Skylake PCIe Controller (x16) (rev 07)
00:01.1 PCI bridge: Intel Corporation Skylake PCIe Controller (x8) (rev 07)
00:02.0 VGA compatible controller: Intel Corporation Device 3e92
00:14.0 USB controller: Intel Corporation Device a2af
00:16.0 Communication controller: Intel Corporation Device a2ba
00:17.0 SATA controller: Intel Corporation Device a282
00:1b.0 PCI bridge: Intel Corporation Device a2e7 (rev f0)
00:1c.0 PCI bridge: Intel Corporation Device a290 (rev f0)
00:1c.4 PCI bridge: Intel Corporation Device a294 (rev f0)
00:1c.6 PCI bridge: Intel Corporation Device a296 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Device a298 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Device a2c9
00:1f.2 Memory controller: Intel Corporation Device a2a1
00:1f.4 SMBus: Intel Corporation Device a2a3
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V
01:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1)
01:00.1 Audio device: NVIDIA Corporation Device 0fb9 (rev a1)
02:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1)
02:00.1 Audio device: NVIDIA Corporation Device 0fb9 (rev a1)
03:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a808
05:00.0 USB controller: ASMedia Technology Inc. Device 2142
06:00.0 USB controller: ASMedia Technology Inc. Device 2142
07:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd Device a808



How I can fix this.
Thanks
 
You can't passthrough devices in the same IOMMU group to different VM's at the same time
"
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/1/devices/0000:02:00.1
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:02:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
"

apply the PCI quirk and see if the devices are in their own group.
 
MikeWebb said:
You can't passthrough devices in the same IOMMU group to different VM's at the same time
"
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/1/devices/0000:02:00.1
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:02:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
"

apply the PCI quirk and see if the devices are in their own group.
Click to expand...
Yes. I see this problem. How I can fix it? change slot for GPU or can change,move device from another iommu group?
 
All done I'm forgot update-grub !!!!!
Code: 
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt video=efifb:off pcie_acs_override=downstream,multifunction"
root@pve:~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:02:00.1
/sys/kernel/iommu_groups/7/devices/0000:00:1b.0
/sys/kernel/iommu_groups/15/devices/0000:01:00.1
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/13/devices/0000:00:1f.6
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/21/devices/0000:07:00.0
/sys/kernel/iommu_groups/11/devices/0000:00:1d.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/18/devices/0000:03:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
/sys/kernel/iommu_groups/16/devices/0000:02:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:17.0
/sys/kernel/iommu_groups/14/devices/0000:01:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.4
/sys/kernel/iommu_groups/12/devices/0000:00:1f.2
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/2/devices/0000:00:01.1
/sys/kernel/iommu_groups/20/devices/0000:06:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:1c.6
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/19/devices/0000:05:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.4
Click to expand...
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back