[SOLVED] IOMMU GRUB config not working.

ThatNaysayer` · Apr 4, 2023

Trying to get IOMMU working on a second node (HP DL380p Gen8) that I just acquired, had no issues following the IOMMU documentation on the first server and it worked amazingly. However with this one I cannot seem to get the intel_iommu=on flag to actually pass to the kernel on boot. I have edited the /etc/default/grub ran a update-grub, no luck cat /proc/cmdline returned:

BOOT_IMAGE=/vmlinuz-5.15.74-1-pve root=ZFS=rpool/ROOT/pve-1 ro root=ZFS=rpool/ROOT/pve-1 boot=zfs quiet

Found another post mentioning that I may need to edit /etc/kernel/cmdline and add the same flag on that, so I did and ran proxmox-boot-tool refresh. Still no luck. I feel like I am missing something very obvious. I have my boot partition on a separate drive which was the easiest way to get the server to boot, not sure if that is where my issue may be coming from.

leesteken · Apr 4, 2023

It does look like your system uses sytemd-boot. If you followed the steps in the manual, it should work. Maybe you forgot proxmox-boot-tool refresh and reboot?

ThatNaysayer` · Apr 4, 2023

Just re-ran proxmox-boot-tool refresh to make sure and gave it another reboot, and sadly nothing. I am pretty sure I am using grub, unless systemd-boot would then route to grub for some reason. I am able to edit the boot options through the grub menu and add the flag sucessfully. I could do this on boot each time, however I would very much like to avoid that.

leesteken · Apr 4, 2023

Yes, that is GRUB. What was the conclusion of the check from the manual?
You are booting from a ZFS pool in legacy mode (instead of UEFI mode)? Unfortunately, I have no experience with that.
What Proxmox VE version are you using? Did you run update-grub and what was/is the output of that comand?

ThatNaysayer` · Apr 5, 2023

I am on 7.4-3 on both nodes, here is the output of both of the checks in the manual:

root@server:~# efibootmgr -v
EFI variables are not supported on this system.
root@server:~# proxmox-boot-tool status
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios

Could this have anything to do with having the boot partition on the internal SD card slot of the server? Maybe some other GRUB conf file getting in the way or just being the main one to being with?

_gabriel · Apr 5, 2023

post your /etc/default/grub in a "insert code" tag please

ThatNaysayer` · Apr 5, 2023

Here it is as it stands now.

Code:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

_gabriel · Apr 5, 2023

thanks, output of "fstab" and "mount"

ThatNaysayer` · Apr 5, 2023

here is my fstab:

Code:

# <file system> <mount point> <type> <options> <dump> <pass>
proc /proc proc defaults 0 0

and the output of mount:

Code:

sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=32918488k,nr_inodes=8229622,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=6590396k,mode=755,inode64)
rpool/ROOT/pve-1 on / type zfs (rw,relatime,xattr,noacl)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=31800)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
ssdpool on /ssdpool type zfs (rw,xattr,noacl)
rpool on /rpool type zfs (rw,relatime,xattr,noacl)
rpool/data on /rpool/data type zfs (rw,relatime,xattr,noacl)
rpool/ROOT on /rpool/ROOT type zfs (rw,relatime,xattr,noacl)
vmpool02 on /vmpool02 type zfs (rw,xattr,noacl)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
/dev/fuse on /etc/pve type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=6590392k,nr_inodes=1647598,mode=700,inode64)

Thanks for the help btw!

_gabriel · Apr 5, 2023

edited: missing output of your already "proxmox-boot-tool status"
so it seems the "boot part" isn't mounted so upgrade-grub cannot update its /boot/grub/grub.cfg.
/boot is missing isn't it ?
output "lsblk"

edit: remove wrong efi part terms

ThatNaysayer` · Apr 5, 2023

/boot is present here are the contents if needed:

Code:

config-5.15.102-1-pve  initrd.img-5.15.102-1-pve  pve                        vmlinuz-5.15.74-1-pve
config-5.15.74-1-pve   initrd.img-5.15.74-1-pve   System.map-5.15.102-1-pve
efi                    memtest86+.bin             System.map-5.15.74-1-pve
grub                   memtest86+_multiboot.bin   vmlinuz-5.15.102-1-pve

here's lsblk:

Code:

NAME     MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda        8:0    0 119.2G  0 disk
└─sda3     8:3    0 118.7G  0 part
sdb        8:16   0 558.9G  0 disk
├─sdb1     8:17   0 558.9G  0 part
└─sdb9     8:25   0     8M  0 part
sdc        8:32   0 558.9G  0 disk
├─sdc1     8:33   0 558.9G  0 part
└─sdc9     8:41   0     8M  0 part
sdd        8:48   0 119.2G  0 disk
├─sdd1     8:49   0 119.2G  0 part
└─sdd9     8:57   0     8M  0 part
sde        8:64   0 558.9G  0 disk
├─sde1     8:65   0 558.9G  0 part
└─sde9     8:73   0     8M  0 part
sdf        8:80   0 558.9G  0 disk
├─sdf1     8:81   0 558.9G  0 part
└─sdf9     8:89   0     8M  0 part
sdg        8:96   0 558.9G  0 disk
├─sdg1     8:97   0 558.9G  0 part
└─sdg9     8:105  0     8M  0 part
sdh        8:112  0 558.9G  0 disk
├─sdh1     8:113  0 558.9G  0 part
└─sdh9     8:121  0     8M  0 part
sdi        8:128  0  29.8G  0 disk
├─sdi1     8:129  0  1007K  0 part
└─sdi2     8:130  0   512M  0 part
sdj        8:144  0   256M  1 disk
└─sdj1     8:145  0   251M  1 part
zd0      230:0    0    50G  0 disk
├─zd0p1  230:1    0   100M  0 part
├─zd0p2  230:2    0    16M  0 part
├─zd0p3  230:3    0  49.4G  0 part
└─zd0p4  230:4    0   530M  0 part
zd16     230:16   0    32G  0 disk
├─zd16p1 230:17   0   300M  0 part
└─zd16p2 230:18   0  31.7G  0 part
zd32     230:32   0     1M  0 disk
zd48     230:48   0    10G  0 disk
zd64     230:64   0     1M  0 disk

and the output of proxmox-boot-tool:

Code:

Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
System currently booted with legacy bios

edit: Just noticed that the /boot/grub/grub.cfg has four menuentry sections, while the grub menu I see on boot only has two.

_gabriel · Apr 5, 2023

sdi is your boot drive, correct ?
i think sdi2 is vfat boot partition that grub use to load kernel, mount it to check its boot/grub.cfg

ThatNaysayer` · Apr 6, 2023

That was it! I could have sworn I tried to mount it before but apparently not correctly. Ended up copying over the contents of the boot folder on sda to the one on sdi - is there a better way to sync the boot folder on sdi with the one on sda for future kernel updates and such? Or is rsyncing after an update them the best option. Otherwise that fixed my problem, thanks so much!

_gabriel · Apr 6, 2023

I bet on sdi is just an usb key , missing to remove ? temporary test ? install key ?
try to boot directly from sda , removing sdi drive

Search

Search

[SOLVED] IOMMU GRUB config not working.

ThatNaysayer`

New Member

leesteken

Distinguished Member

ThatNaysayer`

New Member

leesteken

Distinguished Member

ThatNaysayer`

New Member

_gabriel

Renowned Member

ThatNaysayer`

New Member

_gabriel

Renowned Member

ThatNaysayer`

New Member

_gabriel

Renowned Member

ThatNaysayer`

New Member

_gabriel

Renowned Member

ThatNaysayer`

New Member

_gabriel

Renowned Member