PCI Passthrough with NVIDIA DGX A100 80GB: 4 VMs, GPU only works on one

[ 6.155600] systemd[1]: Starting Load Kernel Module pstore_zone...
[ 6.156357] systemd[1]: Starting Load Kernel Module ramoops...
[ 6.156906] systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped.
[ 6.156931] systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
[ 6.158193] systemd[1]: Starting Load Kernel Modules...
[ 6.164629] systemd[1]: Starting Remount Root and Kernel File Systems...
[ 6.169878] systemd[1]: Starting udev Coldplug all Devices...
[ 6.184508] systemd[1]: Starting Uncomplicated firewall...
[ 6.189060] EXT4-fs (sda2): re-mounted. Opts: errors=remount-ro. Quota mode: none.
[ 6.199384] systemd[1]: Mounted Huge Pages File System.
[ 6.199484] systemd[1]: Mounted POSIX Message Queue File System.
[ 6.200561] systemd[1]: Mounted Kernel Debug File System.
[ 6.200621] systemd[1]: Mounted Kernel Trace File System.
[ 6.206066] systemd[1]: Finished Create list of static device nodes for the current kernel.
[ 6.207138] usb 9-5.2: new high-speed USB device number 4 using xhci_hcd
[ 6.207249] systemd[1]: modprobe@chromeos_pstore.service: Succeeded.
[ 6.210464] systemd[1]: Finished Load Kernel Module chromeos_pstore.
[ 6.211644] systemd[1]: modprobe@pstore_blk.service: Succeeded.
[ 6.213889] systemd[1]: Finished Load Kernel Module pstore_blk.
[ 6.217120] systemd[1]: modprobe@pstore_zone.service: Succeeded.
[ 6.222579] systemd[1]: Finished Load Kernel Module pstore_zone.
[ 6.223762] systemd[1]: modprobe@ramoops.service: Succeeded.
[ 6.225960] systemd[1]: Finished Load Kernel Module ramoops.
[ 6.227150] pstore: Using crash dump compression: deflate
[ 6.227161] pstore: Registered efi as persistent store backend
[ 6.227367] systemd[1]: Finished Remount Root and Kernel File Systems.
[ 6.228026] systemd[1]: modprobe@drm.service: Succeeded.
[ 6.228260] systemd[1]: Finished Load Kernel Module drm.
[ 6.228420] systemd[1]: modprobe@efi_pstore.service: Succeeded.
[ 6.228609] systemd[1]: Finished Load Kernel Module efi_pstore.
[ 6.228897] systemd[1]: Finished Uncomplicated firewall.
[ 6.229665] systemd[1]: Activating swap /swapfile...
[ 6.232141] systemd[1]: Condition check resulted in Rebuild Hardware Database being skipped.
[ 6.233214] systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped.
[ 6.235206] lp: driver loaded but no devices found
[ 6.235326] systemd[1]: Starting Load/Save Random Seed...
[ 6.236167] systemd[1]: Starting Create System Users...
[ 6.238110] ppdev: user-space parallel port driver
[ 6.238116] Adding 2097148k swap on /swapfile. Priority:-2 extents:6 across:2260988k FS
[ 6.238192] systemd[1]: Activated swap /swapfile.
[ 6.238232] systemd[1]: Reached target Swap.
[ 6.254906] systemd[1]: Finished Load/Save Random Seed.
[ 6.260754] systemd[1]: Finished Create System Users.
[ 6.261925] systemd[1]: Starting Create Static Device Nodes in /dev...
[ 6.273368] systemd[1]: Finished Set the console keyboard layout.
[ 6.276292] systemd[1]: Finished Create Static Device Nodes in /dev.
[ 6.276359] systemd[1]: Reached target Local File Systems (Pre).
[ 6.278692] systemd[1]: Mounting Mount unit for bare, revision 5...
[ 6.280110] systemd[1]: Mounting Mount unit for core20, revision 1828...
[ 6.281254] loop0: detected capacity change from 0 to 8
[ 6.281666] systemd[1]: Mounting Mount unit for core20, revision 2105...
[ 6.282931] systemd[1]: Mounting Mount unit for core22, revision 1033...
[ 6.284271] systemd[1]: Mounting Mount unit for gnome-3-38-2004, revision 119...
[ 6.285701] systemd[1]: Mounting Mount unit for gnome-3-38-2004, revision 143...
[ 6.286846] systemd[1]: Mounting Mount unit for gtk-common-themes, revision 1535...
[ 6.288033] systemd[1]: Mounting Mount unit for snap-store, revision 638...
[ 6.289256] systemd[1]: Mounting Mount unit for snap-store, revision 959...
[ 6.290434] systemd[1]: Mounting Mount unit for snapd, revision 18357...
[ 6.292165] systemd[1]: Starting udev Kernel Device Manager...
[ 6.292955] systemd[1]: Finished Load Kernel Modules.
[ 6.294130] systemd[1]: Mounting FUSE Control File System...
[ 6.295194] systemd[1]: Mounting Kernel Configuration File System...
[ 6.296542] systemd[1]: Starting Apply Kernel Variables...
[ 6.297666] systemd[1]: Mounted FUSE Control File System.
[ 6.297748] systemd[1]: Mounted Kernel Configuration File System.
[ 6.315194] loop1: detected capacity change from 0 to 129608
[ 6.315237] loop2: detected capacity change from 0 to 716176
[ 6.316759] usb 9-5.2: New USB device found, idVendor=0b1f, idProduct=03ee, bcdDevice= 5.04
[ 6.316763] usb 9-5.2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 6.316765] usb 9-5.2: Product: RNDIS/Ethernet Gadget
[ 6.316766] usb 9-5.2: Manufacturer: Linux 5.4.62 with aspeed_vhub
[ 6.317834] systemd[1]: Mounted Mount unit for bare, revision 5.
[ 6.323647] systemd[1]: Started Journal Service.
[ 6.331897] systemd-journald[955]: Received client request to flush runtime journal.
[ 6.333981] systemd-journald[955]: File /var/log/journal/d7a9e34e2082447ab8cdc20b2989aba5/system.journal corrupted or uncleanly shut down, renaming and replacing.
[ 6.338238] loop3: detected capacity change from 0 to 130888
[ 6.338283] loop4: detected capacity change from 0 to 94064
[ 6.376237] loop5: detected capacity change from 0 to 151784
[ 6.410367] loop6: detected capacity change from 0 to 102072
[ 6.461320] loop7: detected capacity change from 0 to 709280
[ 6.470723] cryptd: max_cpu_qlen set to 1000
[ 6.507193] AVX2 version of gcm_enc/dec engaged.
[ 6.507926] loop9: detected capacity change from 0 to 25240
[ 6.510713] AES CTR mode by8 optimization enabled
[ 6.531268] loop8: detected capacity change from 0 to 187776
[ 6.826548] usb-storage 10-3:1.0: USB Mass Storage device detected
[ 6.826961] scsi host7: usb-storage 10-3:1.0
[ 6.827110] usbcore: registered new interface driver usb-storage
[ 6.832299] usbcore: registered new interface driver uas
[ 7.029451] nvidia: loading out-of-tree module taints kernel.
[ 7.029462] nvidia: module license 'NVIDIA' taints kernel.
[ 7.029463] Disabling lock debugging due to kernel taint
[ 7.084745] nvidia-nvlink: Nvlink Core is being initialized, major device number 510
[ 7.084751] NVRM: The NVIDIA GPU 0000:01:00.0 (PCI ID: 10de:2331)
NVRM: installed in this system is not supported by the
NVRM: NVIDIA 535.154.05 driver release.
NVRM: Please see 'Appendix A - Supported NVIDIA GPU Products'
NVRM: in this release's README, available on the operating system
NVRM: specific graphics driver download page at www.nvidia.com.
[ 7.089244] nvidia: probe of 0000:01:00.0 failed with error -1
[ 7.089272] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 7.089274] NVRM: None of the NVIDIA devices were initialized.
[ 7.089808] nvidia-nvlink: Unregistered Nvlink Core, major device number 510
[ 7.699781] nvidia-nvlink: Nvlink Core is being initialized, major device number 510
[ 7.699788] NVRM: The NVIDIA GPU 0000:01:00.0 (PCI ID: 10de:2331)
NVRM: installed in this system is not supported by the
NVRM: NVIDIA 535.154.05 driver release.
NVRM: Please see 'Appendix A - Supported NVIDIA GPU Products'
NVRM: in this release's README, available on the operating system
NVRM: specific graphics driver download page at www.nvidia.com.
[ 7.704677] nvidia: probe of 0000:01:00.0 failed with error -1
[ 7.704702] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 7.704703] NVRM: None of the NVIDIA devices were initialized.
[ 7.705021] nvidia-nvlink: Unregistered Nvlink Core, major device number 510
[ 7.843494] scsi 7:0:0:0: Direct-Access USB SanDisk 3.2Gen1 1.00 PQ: 0 ANSI: 6
[ 7.843805] sd 7:0:0:0: Attached scsi generic sg1 type 0
[ 7.844155] sd 7:0:0:0: [sdb] 60088320 512-byte logical blocks: (30.8 GB/28.7 GiB)
[ 7.844698] sd 7:0:0:0: [sdb] Write Protect is off
[ 7.844699] sd 7:0:0:0: [sdb] Mode Sense: 43 00 00 00
[ 7.845205] sd 7:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 7.851317] sdb: sdb1 sdb2
[ 7.852929] sd 7:0:0:0: [sdb] Attached SCSI removable disk
[ 8.019807] audit: type=1400 audit(1707226262.616:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="ippusbxd" pid=1392 comm="apparmor_parser"
[ 8.019839] audit: type=1400 audit(1707226262.616:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=1396 comm="apparmor_parser"
[ 8.020423] audit: type=1400 audit(1707226262.616:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=1398 comm="apparmor_parser"
[ 8.020429] audit: type=1400 audit(1707226262.616:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=1398 comm="apparmor_parser"
[ 8.020433] audit: type=1400 audit(1707226262.616:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=1398 comm="apparmor_parser"
[ 8.020487] audit: type=1400 audit(1707226262.616:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=1390 comm="apparmor_parser"
[ 8.020490] audit: type=1400 audit(1707226262.616:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=1390 comm="apparmor_parser"
[ 8.021377] audit: type=1400 audit(1707226262.616:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/sbin/cups-browsed" pid=1395 comm="apparmor_parser"
[ 8.027127] audit: type=1400 audit(1707226262.624:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=1393 comm="apparmor_parser"
[ 8.027132] audit: type=1400 audit(1707226262.624:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=1393 comm="apparmor_parser"
[ 8.380896] nvidia-nvlink: Nvlink Core is being initialized, major device number 510
[ 8.380903] NVRM: The NVIDIA GPU 0000:01:00.0 (PCI ID: 10de:2331)
NVRM: installed in this system is not supported by the
NVRM: NVIDIA 535.154.05 driver release.
NVRM: Please see 'Appendix A - Supported NVIDIA GPU Products'
NVRM: in this release's README, available on the operating system
NVRM: specific graphics driver download page at www.nvidia.com.
[ 28.883682] nvidia: probe of 0000:01:00.0 failed with error -1
[ 28.883710] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 28.883711] NVRM: None of the NVIDIA devices were initialized.
[ 28.884100] nvidia-nvlink: Unregistered Nvlink Core, major device number 510
[ 29.164677] systemd-journald[955]: File /var/log/journal/d7a9e34e2082447ab8cdc20b2989aba5/user-1000.journal corrupted or uncleanly shut down, renaming and replacing.
[ 29.248726] nvidia-nvlink: Nvlink Core is being initialized, major device number 510
[ 29.248735] NVRM: The NVIDIA GPU 0000:01:00.0 (PCI ID: 10de:2331)
NVRM: installed in this system is not supported by the
NVRM: NVIDIA 535.154.05 driver release.
NVRM: Please see 'Appendix A - Supported NVIDIA GPU Products'
NVRM: in this release's README, available on the operating system
NVRM: specific graphics driver download page at www.nvidia.com.
[ 29.253228] nvidia: probe of 0000:01:00.0 failed with error -1
[ 29.253261] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 29.253262] NVRM: None of the NVIDIA devices were initialized.
[ 29.253628] nvidia-nvlink: Unregistered Nvlink Core, major device number 510
[ 29.652560] nvidia-nvlink: Nvlink Core is being initialized, major device number 510
[ 29.652566] NVRM: The NVIDIA GPU 0000:01:00.0 (PCI ID: 10de:2331)
NVRM: installed in this system is not supported by the
NVRM: NVIDIA 535.154.05 driver release.
NVRM: Please see 'Appendix A - Supported NVIDIA GPU Products'
NVRM: in this release's README, available on the operating system
NVRM: specific graphics driver download page at www.nvidia.com.
[ 29.657317] nvidia: probe of 0000:01:00.0 failed with error -1
[ 29.657343] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 29.657344] NVRM: None of the NVIDIA devices were initialized.
[ 29.657673] nvidia-nvlink: Unregistered Nvlink Core, major device number 510
[ 30.034685] nvidia-nvlink: Nvlink Core is being initialized, major device number 510
[ 30.034691] NVRM: The NVIDIA GPU 0000:01:00.0 (PCI ID: 10de:2331)
[ 115.805276] nvidia: probe of 0000:01:00.0 failed with error -1
[ 115.805300] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 115.805301] NVRM: None of the NVIDIA devices were initialized.
[ 115.805673] nvidia-nvlink: Unregistered Nvlink Core, major device number 510
[ 116.121170] nvidia-nvlink: Nvlink Core is being initialized, major device number 510
[ 116.121177] NVRM: The NVIDIA GPU 0000:01:00.0 (PCI ID: 10de:2331)
NVRM: installed in this system is not supported by the
NVRM: NVIDIA 535.154.05 driver release.
NVRM: Please see 'Appendix A - Supported NVIDIA GPU Products'
NVRM: in this release's README, available on the operating system
NVRM: specific graphics driver download page at www.nvidia.com.
[ 116.124631] nvidia: probe of 0000:01:00.0 failed with error -1
[ 116.124655] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 116.124656] NVRM: None of the NVIDIA devices were initialized.
[ 116.125069] nvidia-nvlink: Unregistered Nvlink Core, major device number 510
[ 116.461445] nvidia-nvlink: Nvlink Core is being initialized, major device number 510


[ 116.479482] nvidia 0000:01:00.0: refused to change power state from D3hot to D0
[ 116.479797] NVRM: The NVIDIA GPU 0000:01:00.0
NVRM: (PCI ID: 10de:2331) installed in this system has
NVRM: fallen off the bus and is not responding to commands.
[ 116.481285] nvidia: probe of 0000:01:00.0 failed with error -1
[ 116.481311] NVRM: The NVIDIA probe routine failed for 1 device(s).
[ 116.481312] NVRM: None of the NVIDIA devices were initialized.
[ 116.481592] nvidia-nvlink: Unregistered Nvlink Core, major device number 510
[ 116.799665] nvidia-nvlink: Nvlink Core is being initialized, major device number 510


[ 116.847368] nvidia 0000:01:00.0: enabling device (0000 -> 0002)
[ 116.861796] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 535.154.05 Thu Dec 28 15:37:48 UTC 2023
[ 116.878663] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 535.154.05 Thu Dec 28 15:51:29 UTC 2023
[ 116.909660] [drm] [nvidia-drm] [GPU ID 0x00000100] Loading driver
[ 117.019438] ACPI Warning: \_SB.PCI0.SE0.S00._DSM: Argument #4 type mismatch - Found [Buffer], ACPI requires [Package] (20210730/nsarguments-61)
[ 121.810531] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 0
[ 122.131370] nvidia_uvm: module uses symbols from proprietary module nvidia, inheriting taint.
[ 122.136928] nvidia-uvm: Loaded the UVM driver, major device number 508.
[ 122.146144] nvidia-uvm: Unloaded the UVM driver.
[ 122.419818] nvidia_uvm: module uses symbols from proprietary module nvidia, inheriting taint.
[ 122.425098] nvidia-uvm: Loaded the UVM driver, major device number 508.
 
mhm.. the driver in the guest seemingly seems to think that it's not supported (even if the drivers readme says it is)
you could ask nvidia whats the problem here, since from our side it seems to work, and we cannot really debug proprietary driver issues ...

can you post the vm config, maybe i see something, but no promises... you could also play around with some options (seabios/ovmf, i440fx/q35, secure boot on/off (in the vm), etc...)
 
JasonFeng said:
View attachment 62661Feb 06 20:39:41 H100 kernel: vfio-pci 0000:98:00.0: vfio_bar_restore: reset recovery - restoring BARs
this log appear when VM freeze and host reboot,which 0000:98:00.0 is the H100 gpu's pcie bus
Click to expand...

Hi!
What exactly do you do ?

LXC or VM ?

VM with IOMMU will work.
Make sure IOMMU is enabled, all the GPU drivers are blacklisted on Proxmox.

Code: 
/etc/modprobe.d/nvidia.conf
blacklist nvidiafb
blacklist nouveau

$> update-initramfs -c -d -u

I do not recommend to use the "pcie_acs_override=" kernel parameter, if you have "server hardware" you don't need it.
( Symptom: If the "pcie_acs_override" option is used, when you start the VM, the entire server will reboot/freeze. )

Also, add GPU driver blacklist to the VM too -> "/etc/modprobe.d/nvidia.conf"

First add the "nvidia" driver ( https://www.nvidia.com/download/index.aspx ) to the VM, after enable GPU pass-through.
 
Last edited:
dcsapak said:
mhm.. the driver in the guest seemingly seems to think that it's not supported (even if the drivers readme says it is)
you could ask nvidia whats the problem here, since from our side it seems to work, and we cannot really debug proprietary driver issues ...

can you post the vm config, maybe i see something, but no promises... you could also play around with some options (seabios/ovmf, i440fx/q35, secure boot on/off (in the vm), etc...)
Click to expand...

Ubuntu-20.04
1707276110296.png

Windows111707276146516.png
 
emunt6 said:
Hi!
What exactly do you do ?

LXC or VM ?

VM with IOMMU will work.
Make sure IOMMU is enabled, all the GPU drivers are blacklisted on Proxmox.

Code: 
/etc/modprobe.d/nvidia.conf
blacklist nvidiafb
blacklist nouveau

$> update-initramfs -c -d -u

I do not recommend to use the "pcie_acs_override=" kernel parameter, if you have "server hardware" you don't need it.
( Symptom: If the "pcie_acs_override" option is used, when you start the VM, the entire server will reboot/freeze. )

Also, add GPU driver blacklist to the VM too -> "/etc/modprobe.d/nvidia.conf"

First add the "nvidia" driver ( https://www.nvidia.com/download/index.aspx ) to the VM, after enable GPU pass-through.
Click to expand...
VM,we also own a A100 server runs LXCs,it works well,but we need usb passthrough to test our sensors computing,so on H100 server it need to be VM used.I'm sure that IOMMU Worked because USB controller could be passthroughed and work well,also blacklist has edited,here's the output1707276347988.png
 
could you maybe try to add the following to the 'args' part of the config:

Code: 
-fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536

you could do this with
Code: 
qm set ID --args '-fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536'
 
dcsapak said:
could you maybe try to add the following to the 'args' part of the config:

Code: 
-fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536

you could do this with
Code: 
qm set ID --args '-fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=65536'
Click to expand...
No, it doesn't work well1707319467842.png1707319485888.png
 
just to be sure: you stopped(shutdown) and startet the vm again after setting this?
 
mhmm theres not much left to try .. did you try in a windows vm with the windows driver? maybe there's something wrong with the driver after all?
in any case i'd probably ask nvidia for help, as from pve side it seems to work, but the driver seems to have issues and we can't really debug the proprietary nvidia driver

EDIT: oh did you also try a lower kernel version in the vm? e.g. the 6.2 or 5.15 kernel from ubuntu? maybe there's some problem?
 
dcsapak said:
mhmm theres not much left to try .. did you try in a windows vm with the windows driver? maybe there's something wrong with the driver after all?
in any case i'd probably ask nvidia for help, as from pve side it seems to work, but the driver seems to have issues and we can't really debug the proprietary nvidia driver

EDIT: oh did you also try a lower kernel version in the vm? e.g. the 6.2 or 5.15 kernel from ubuntu? maybe there's some problem?
Click to expand...
On windows VM,host may immediatly crash to reboot if you install Nvidia Windows Driver,but it did recognized the correct model of GPU instead of "3D controller NVIDIA" on Linux VMs. I've tried both Debian12 with 6.5kernel,Ubuntu20.04 with 5.15 kernel and Ubuntu22.04 with 6.1 kernel, none of them worked well
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom