pveceph - unable to get device info for '/dev/nvme2n1p1'?

[victorhooi]

I'm trying to setup a new 4-node Proxmox/Ceph cluster.

Each node has 6 x NVMe SSDs, as well as an Intel Optane drive (used for WAL/DB).

I have partitioned each NVMe SSD like so:

parted /dev/nvme6n1 mklabel gpt
parted -a optimal /dev/nvme6n1 mkpart primary 0% 25%
parted -a optimal /dev/nvme6n1 mkpart primary 25% 50%
parted -a optimal /dev/nvme6n1 mkpart primary 50% 75%
parted -a optimal /dev/nvme6n1 mkpart primary 75% 100

Rationale - I'm setting up 4 OSDs per NVMe drive, in order to better utilise resources.

I then try to create a new OSD using pveceph, using the Optane drive (/dev/nvme0n1) as the WAL/DB drive. However, it fails, saying it can't get device info for my partition:

# pveceph osd create /dev/nvme2n1p1 -db_Dev /dev/nvme0n1 -db_size 35
unable to get device info for '/dev/nvme2n1p1'

Is there a reason that pveceph doesn't recognise /dev/nvme2n1p1? Is this the correct way to pass a disk partition to pveceph?

 

[Rainerle]

You could achieve this easier. See https://forum.proxmox.com/threads/r...multiple-osds-per-nvme-disk.52252/post-242117

When you look at the result it creates a LVM device with 4 volumes. You should be able to use the volumes with pveceph.

 

[victorhooi]

Right - but using that command (ceph-volume) only lets you create multiple OSDs on a single device.

It doesn't let you assign the WAL/DB location to a different device as far as I can tell?

That's why I'm manually creating the partitions myself, then using pveceph osd create - since that does let you assign the WAL/DB to a difference device, and handles automatically partitioning that.

 

[Rainerle]

What I tried to say was:
- Use pvcreate on your NVMe to create a LVM device
- Use lvcreate on your LVM to create 4 logical volumes
- Use these LVs in pveceph plus your WAL device

 

[Rainerle]

Another Idea:
- Use ceph-volume first to create the OSDs
- Then out, stop and purge them
- Then recreate them with pveceph using the already existing LVs...