Container creation stopped allowing Ceph RBD storage

[GorgonzolaPrimavera]

I have made several LXCs in my PVE 9.1.1 cluster over the past few months but when I tried to create a new one today, I was unable to choose my `vms` RBD store. The only choices I got were either the two NFS shares or the `local` store. I thought maybe I had been dreaming that I had created them there but I looked at them all and they were all assigned to the Ceph store. My store is RBD, [obviously], enabled, with "Disk Image" and "Container" content types. Ceph version per the Datacenter UI is 19.2.4.
I tried creating the container on different nodes in the cluster but they all gave me the same behavior.
If I create a new VM, that does allow me to choose the RBD.

What else should I investigate?

  cluster:
    id:     big-uuid-here
    health: HEALTH_WARN
            There are daemons running an older version of ceph

  services:
    mon: 3 daemons, quorum host1,host2,host3 (age 3w)
    mgr: host1(active, since 3w), standbys: host3
    osd: 25 osds: 24 up (since 3d), 24 in (since 3d)

  data:
    pools:   2 pools, 129 pgs
    objects: 1.17M objects, 4.4 TiB
    usage:   20 TiB used, 59 TiB / 79 TiB avail
    pgs:     129 active+clean

  io:
    client:   723 KiB/s rd, 3.0 MiB/s wr, 33 op/s rd, 143 op/s wr

 

[GorgonzolaPrimavera]

Odd. I created a new Ceph Pool that I will use just for my containers but now I see both of them. I must have been looking at the wrong page or something.

 

[aaron]

My first guess, does a storage configuration (Datacenter -> Storage) exist, and what content types are enabled?

 

[powersupport]

The issue is almost certainly that KRBD (Kernel RBD) is disabled on your vms storage configuration.

Why VMs work, but LXCs don't​

• VMs access Ceph directly in userspace via QEMU (librbd).
• LXCs share the host kernel and need the host to map the block device directly into /dev/rbdX. For LXC creation, PVE requires KRBD to be enabled on the storage.

How to Fix​

Option 1: Via Web UI​

1. Go to Datacenter → Storage.
2. Select your vms storage and click Edit.
3. Check the KRBD box and save.

Option 2: Via CLI (​

Ensure your storage block includes krbd 1 and rootdir:

rbd: vms
        pool vms
        content rootdir,images
        krbd 1

Quick Ceph Notes​

• 1 Down OSD: Your status shows 24 up / 25 osds. Run ceph osd tree to locate and bring up the missing OSD.
• Version Mismatch: Run ceph versions to identify which daemons still need a restart following updates.

 

[aaron]

The issue is almost certainly that KRBD (Kernel RBD) is disabled on your vms storage configuration.

The KRBD setting in the storage config only affects how VMs access their virtual disks on RBD. For CTs it will always be the kernel variant where rbdmap is run, to map an RBD image to a local block dev.

 

[GorgonzolaPrimavera]

The KRBD setting in the storage config only affects how VMs access their virtual disks on RBD. For CTs it will always be the kernel variant where rbdmap is run, to map an RBD image to a local block dev.

So if the `krbd` option is there to force use of the kernel module, that implies that there is a userland interface to the RBD. Is that correct?

I'm also assuming that this forces the *host* to use the kernel module for a VM's block requests since there's not even an awareness of the RBD's existence in the VM.

 

[GorgonzolaPrimavera]

My first guess, does a storage configuration (Datacenter -> Storage) exist, and what content types are enabled?

Oh, yes, for sure. Lots of storages, in fact. I have lots of VMs and CTs using the RBD. It's the default config with the two types, Disk image and Container. The new one I just created is just Container and I easily moved the images of my existing containers to the new one.

I'm just braindead. I was not seeing the storage from the list in the templates, I'm pretty sure now. You don't get the *CT's* storage until you move along in the definition of the CT.

 

[GorgonzolaPrimavera]

The issue is almost certainly that KRBD (Kernel RBD) is disabled on your vms storage configuration.

Why VMs work, but LXCs don't​

• VMs access Ceph directly in userspace via QEMU (librbd).
• LXCs share the host kernel and need the host to map the block device directly into /dev/rbdX. For LXC creation, PVE requires KRBD to be enabled on the storage.

How to Fix​

Option 1: Via Web UI​

1. Go to Datacenter → Storage.
2. Select your vms storage and click Edit.
3. Check the KRBD box and save.

Option 2: Via CLI (​

Ensure your storage block includes krbd 1 and rootdir:

rbd: vms
        pool vms
        content rootdir,images
        krbd 1

Quick Ceph Notes​

• 1 Down OSD: Your status shows 24 up / 25 osds. Run ceph osd tree to locate and bring up the missing OSD.
• Version Mismatch: Run ceph versions to identify which daemons still need a restart following updates.

Yeah, yet again, one of my decade-old 4T drives has died. They are rapidly aging away. I'm not sure if it's the controller eating them or they really are just giving up after 90k hours.