[SOLVED] NVME-TCP module on Kernel 5.15.0-1069-kvm

[FabioLeal]

Hello everyone!

I'm trying to install the nvme-tcp package on an Ubuntu VM with kernel 5.15.0-1069-kvm.

$ uname -a
Linux dev-node-X 5.15.0-1069-kvm #74-Ubuntu SMP Fri Oct 18 14:22:54 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

The Ubuntu version is 22.04 minimal and the use of these VM is for a kubernetes cluster. I'm trying to install OpenEBS Mayastor storage and one of the prerequisites is to have nvme-tcp installed.

The installation went as expected, but when I use the command sudo modpbobe nvme-tcp the error modprobe: FATAL: Module nvme-tcp not found in directory /lib/modules/5.15.0-1069-kvm appears.

Digging on the internet, I only found one post, from this LINK, regarding the problem with this kernel 5.15.0-10XX-kvm, but it didn't work, because the suggestion was to install linux-image-extras, which returns the error E: Unable to locate package linux-image-extras.

Some stupid things I've tried:
- Changing the existing kernel 5.15.0-1069-kvm in GRUB to version 5.15.0-126-generic, but as a noob in this matter I almost destroyed the VM (LOL)
- Copying the folder /lib/modules/5.15.0-126-generic/kernel/drivers/nvme to the folder /lib/modules/5.15.0-1069-kvm/kernel/drivers/nvme
- Changing the minimal version to the "Full" version with the command sudo unminimize, which probably installed a bunch of stuff I probably don't need or will use...

Anyway, nothing worked.

Does anyone have any useful information that I missed?

 

[FabioLeal]

Can anyone help? Is there any light at the end of the tunnel?

If that's not possible, is there a tutorial on how to change from kernel 5.15.0-1069-kvm to kernel 5.15.0-126-generic? Of course, without breaking the VM, as happened when I tried to make the change...

 

[AveryFreeman]

Can anyone help? Is there any light at the end of the tunnel?

If that's not possible, is there a tutorial on how to change from kernel 5.15.0-1069-kvm to kernel 5.15.0-126-generic? Of course, without breaking the VM, as happened when I tried to make the change...

Hi, just randomly came across your post looking for something else and thought I'd help

TL;DR
I actually dug this down pretty far, and I'm guessing you'll want to install `linux-hwe-$(uname -r)`, if I'm reading this right: https://launchpad.net/ubuntu/focal/+package/linux-modules-extra-5.15.0-52-generic

Did you make sure `kernel 5.15.0-1069-kvm` is compiled with the necessary flags? I didn't see anything about the kvm-variant being compiled with `nvme-tcp` support. The `linux-meta-kvm` (package that groups usual `linux-kvm` packages) doesn't include a modules package (at bottom): https://launchpad.net/ubuntu/+source/linux-meta-kvm/5.15.0.1069.65

Usually if a `.ko` isn't in `/usr/lib/modules/$(uname -r)`, it's because the `=m` flag isn't set at compilation. OR, it was compiled with `=y`, meaning it's built-in (always available - `modprobe` not necessary), but conversely doesn't show up in `lsmod`. If your kernel release was compiled (`.ko` file), or if it's even possible to install, that flag will need to be in your /boot/config-$(uname -r) file.

To check, do a `grep /boot/config-$(uname -r)`. Here's an example on Fedora 41 (`.config`'s in a different place, but you get the idea...)

grep NVME_TCP /usr/lib/modules/$(uname -r)/config
CONFIG_NVME_TCP=m
CONFIG_NVME_TCP_TLS=y

This is tangential to your situation above, but `find` can be satisfying for looking for already available `.ko` files...:

find /usr/lib/modules/$(uname -r) -name 'nvme*'
...
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/common/nvme-auth.ko.xz
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/common/nvme-keyring.ko.xz
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/host/nvme-core.ko.xz
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/host/nvme-fabrics.ko.xz
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/host/nvme-fc.ko.xz
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/host/nvme-tcp.ko.xz
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/host/nvme.ko.xz
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/host/nvme-rdma.ko.xz
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/target/nvme-fcloop.ko.xz
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/target/nvme-loop.ko.xz
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/target/nvmet-fc.ko.xz
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/target/nvmet-tcp.ko.xz
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/target/nvmet.ko.xz
/usr/lib/modules/6.11.10-300.fc41.x86_64/kernel/drivers/nvme/target/nvmet-rdma.ko.xz

Good luck!

 

[FabioLeal]

Hey, @AveryFreeman. Thank you for your comment.

Did you make sure `kernel 5.15.0-1069-kvm` is compiled with the necessary flags? I didn't see anything about the kvm-variant being compiled with `nvme-tcp` support.

At first I didn't even know that nvme-tcp wasn't precompiled. During my research to solve the problem, I found out about it. However, I don't have enough knowledge to compile a kernel with the nvme-tcp flags enabled.
Everywhere I ask I always ask for a tutorial to help me with this.

To check, do a `grep /boot/config-$(uname -r)`.

Here's the grep command:

$ grep NVME /boot/config-5.15.0-1069-kvm
# NVME Support
# CONFIG_BLK_DEV_NVME is not set
# CONFIG_NVME_FC is not set
# CONFIG_NVME_TCP is not set
# CONFIG_NVME_TARGET is not set
# end of NVME Support
CONFIG_NVMEM=y
CONFIG_NVMEM_SYSFS=y
# CONFIG_NVMEM_RMEM is not set

I downloaded the generic kernel 5.15.0-126-generic, which has nvme-tcp compiled in, but I can't change the kernel from xxx-1069-kvm to xxx-126-generic. When the VM starts got into a boot loop, and I had to use a Live CD to go back to the old kernel.

At this point, I would like to know if:
Case 1 - Is there a tutorial for beginners (or intermediate beginners or advanced beginners hehehe) to compile the kernel the way I need, that is, with nvme-tcp?
Case 2 - Change the kernel from the xxx-1069-kvm used today to xxx-126-generic without the VM starting up in a loop?

 

[bbgeek17]

Hi @FabioLeal ,

You will want to install HWE Kernel: https://askubuntu.com/questions/1442208/how-to-enable-hwe-on-ubuntu-22-04


VERSION_ID=$( 'source /etc/os-release;echo $VERSION_ID')
apt-get update -y 2>&1
apt -y install linux-generic-hwe-$VERSION_ID
reboot

Then load nvme-tcp module, if necessary:
apt-get install -y linux-modules-extra-$(uname -r)
depmod
modprobe nvme-tcp && echo nvme-tcp | sudo tee -a /etc/modules-load.d/nvme-tcp.conf

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[FabioLeal]

@bbgeek17 THANK YOU, dude!
Using this HWE Kernel solve my problems with nvme-tcp