PCI Passthrough - Dell XC730xd-24

[sl3786]

Lets see how much I can post ahead of questions. Feels like the interrupt (IOMMU) is only half enabling?

System is a Dell XC730xd-24
2x Intel(R) Xeon(R) CPU E5-2660 v4 @ 2.00GHz
Trying to get a NVidia P40 to pass through.

https://imgur.com/a/LF2wIML
lots of output of various commands I see referenced in other threads here or elsewhere hopefully cutting down on the 'did you run X'.

https://enterprise-support.nvidia.c...eredge-r730-bios-parameters-to-support-sr-iov is what I followed for enabling it at the bios level.

For reference for any future searches who may find this and the imgur link is invalid due to age, from my perspective things are giving appropriate output EXCEPT for 'cat /proc/cmdline' but I am unsure. Adding a pci device, the dialogue box indicates its not functional, and VMs with the device (available via 'raw device' selection) do not boot.

 

[sl3786]

Of course the one I forget...

 

[VinnyG]

p40 has no SR-IOV
refer to this guide for iommu/vfio passthrough

https://pve.proxmox.com/wiki/PCI(e)_Passthrough

 

[sl3786]

Your quick reply while appreciated doesn't seem to provide actual guidance.

The url you link references kernel modules, and setting the 'intel_iommu' flag on the kernel command line, which I did (screenshot 2), but the system doesn't seem to be processing these correctly at a guess.

So unless you are saying 'disable SR-IOV in bios because its (likely) causing a conflict' I am not sure what guidance you are offering.

 

[leesteken]

The url you link references kernel modules, and setting the 'intel_iommu' flag on the kernel command line, which I did (screenshot 2), but the system doesn't seem to be processing these correctly at a guess.

What are the output of cat /proc/cmdline , cat /etc/default/grub and cat /etc/kernel/cmdline (in CODE tags please)?

 

[sl3786]

I am unsure why you want to check /etc/default/grub when the system is EFI, but:

root@pve:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro quiet
root@pve:~# cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""

# If your computer has multiple operating systems installed, then you
# probably want to run os-prober. However, if your computer is a host
# for guest OSes installed via LVM or raw disk devices, running
# os-prober can cause damage to those guest OSes as it mounts
# filesystems to look for things.
#GRUB_DISABLE_OS_PROBER=false

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
root@pve:~# cat /etc/kernel/cmdline
intel_iommu=on

 

[leesteken]

I am unsure why you want to check /etc/default/grub when the system is EFI, but:

You are not booting from ZFS, so your Proxmox is using GRUB: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysboot

root@pve:~# cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro quiet

Your /proc/cmdline also looks a lot more like your GRUB than your systemd-boot.

root@pve:~# cat /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="quiet"

root@pve:~# cat /etc/kernel/cmdline

intel_iommu=on

Looks like you did not follow the exact steps for enabling IOMMU: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_configuration_14 (which links to different steps for systemd-boot and GRUB, and you are definitely using GRUB). Alteratively, you could ugdate to Proxmox 8.2 and kernel 6.8 and IOMMU is enabled by default.

 

[sl3786]

That could explain it thank you for your patience - will report back with results.

 

[sl3786]

Kudos to you leesteken - apparently I was misled with some of the commands into thinking it was not grub. Soon as I put the command in for a grub install, worked like magic.