I trying to get PCI passthrough work with my gpu pve servers:
UEFI boot with less than 8 Nvidia RTX 3090, win10 and ubuntu 22.04 is boot fine without any issue; but won't boot for 8 cards, not booting from disk(can not find bootable disk), both win10 and ubuntu.
I tried:
I assume RTX 3090 is support UEFI, because 7 cards or less working fine.
I tested seabios with 8 cards, both win10 and ubuntu working fine.
I have been struggling with this issue for a long time without finding any further clues. Below is my setup, could someone please offer some advice?
Host machine:
1. boot with uefi mode
2. Enabled Intel VT for directed I/O, ASC control, Interrupt remapping.
UEFI boot with less than 8 Nvidia RTX 3090, win10 and ubuntu 22.04 is boot fine without any issue; but won't boot for 8 cards, not booting from disk(can not find bootable disk), both win10 and ubuntu.
I tried:
- Turn off security boot from for guest vm during starting vm
- Switch Display mode: Standard VGA, SPICE, VirtIO-GPU
I assume RTX 3090 is support UEFI, because 7 cards or less working fine.
I tested seabios with 8 cards, both win10 and ubuntu working fine.
I have been struggling with this issue for a long time without finding any further clues. Below is my setup, could someone please offer some advice?
Code:
pveversion -v
proxmox-ve: 8.2.0 (running kernel: 6.8.4-2-pve)
pve-manager: 8.2.2 (running version: 8.2.2/9355359cd7afbae4)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.4-2
proxmox-kernel-6.8.4-2-pve-signed: 6.8.4-2
ceph-fuse: 17.2.7-pve3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown2: 3.2.0-1+pmx8
ksm-control-daemon: 1.5-1
libjs-extjs: 7.0.0-4
libknet1: 1.28-pve1
libproxmox-acme-perl: 1.5.0
libproxmox-backup-qemu0: 1.4.1
libproxmox-rs-perl: 0.3.3
libpve-access-control: 8.1.4
libpve-apiclient-perl: 3.3.2
libpve-cluster-api-perl: 8.0.6
libpve-cluster-perl: 8.0.6
libpve-common-perl: 8.2.1
libpve-guest-common-perl: 5.1.1
libpve-http-server-perl: 5.1.0
libpve-network-perl: 0.9.8
libpve-rs-perl: 0.8.8
libpve-storage-perl: 8.2.1
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 6.0.0-1
lxcfs: 6.0.0-pve2
novnc-pve: 1.4.0-3
proxmox-backup-client: 3.2.0-1
proxmox-backup-file-restore: 3.2.0-1
proxmox-kernel-helper: 8.1.0
proxmox-mail-forward: 0.2.3
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.6
proxmox-widget-toolkit: 4.2.1
pve-cluster: 8.0.6
pve-container: 5.0.10
pve-docs: 8.2.1
pve-edk2-firmware: 4.2023.08-4
pve-esxi-import-tools: 0.7.0
pve-firewall: 5.0.5
pve-firmware: 3.11-1
pve-ha-manager: 4.0.4
pve-i18n: 3.2.2
pve-qemu-kvm: 8.1.5-5
pve-xtermjs: 5.3.0-3
qemu-server: 8.2.1
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.2.3-pve2
Code:
root@node8:~# qm config 102
agent: 1
bios: ovmf
boot: order=scsi0;net0
cores: 8
cpu: host
efidisk0: vm1:102/vm-102-disk-1.qcow2,efitype=4m,pre-enrolled-keys=1,size=528K
hostpci0: mapping=RTX3090,pcie=1,x-vga=1
hostpci1: mapping=RTX3090,pcie=1
hostpci2: mapping=RTX3090,pcie=1
hostpci3: mapping=RTX3090,pcie=1
hostpci4: mapping=RTX3090,pcie=1
hostpci5: mapping=RTX3090,pcie=1
hostpci6: mapping=RTX3090,pcie=1
hostpci7: mapping=RTX3090,pcie=1
machine: pc-q35-8.1
memory: 32768
meta: creation-qemu=8.1.5,ctime=1716211018
name: windows-10-nvidia-base
net0: virtio=BC:24:11:9C:91:37,bridge=vmbr0,firewall=1
numa: 0
ostype: win11
scsi0: vm1:102/vm-102-disk-0.qcow2,cache=writeback,discard=on,iothread=1,size=50G
scsihw: virtio-scsi-single
smbios1: uuid=621e908a-41d2-472f-b775-a94ebacbf5ba
sockets: 1
tablet: 1
tpmstate0: vm1:102/vm-102-disk-0.raw,size=4M,version=v2.0
vga: std
vmgenid: bf6a964b-21d5-49a9-b2b1-3830b84553ae
Code:
## /etc/default/grub:
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt"
## /etc/modules:
vfio
vfio_iommu_type1
vfio_pci
## /etc/modprobe.d/kvm.conf
options kvm ignore_msrs=1 report_ignored_msrs=0
## /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:2204,10de:1aef
softdep nouveau pre: vfio-pci
softdep nvidia pre: vfio-pci
softdep nvidiafb pre: vfio-pci
softdep nvidia_drm pre: vfio-pci
softdep drm pre: vfio-pciHost machine:
1. boot with uefi mode
2. Enabled Intel VT for directed I/O, ASC control, Interrupt remapping.
Last edited:
