pveceph - unable to get device info for '/dev/nvme2n1p1'?

victorhooi

Member
Apr 3, 2018
249
18
23
36
I'm trying to setup a new 4-node Proxmox/Ceph cluster.

Each node has 6 x NVMe SSDs, as well as an Intel Optane drive (used for WAL/DB).

I have partitioned each NVMe SSD like so:
Code:
parted /dev/nvme6n1 mklabel gpt
parted -a optimal /dev/nvme6n1 mkpart primary 0% 25%
parted -a optimal /dev/nvme6n1 mkpart primary 25% 50%
parted -a optimal /dev/nvme6n1 mkpart primary 50% 75%
parted -a optimal /dev/nvme6n1 mkpart primary 75% 100

Rationale - I'm setting up 4 OSDs per NVMe drive, in order to better utilise resources.

I then try to create a new OSD using pveceph, using the Optane drive (/dev/nvme0n1) as the WAL/DB drive. However, it fails, saying it can't get device info for my partition:

Code:
# pveceph osd create /dev/nvme2n1p1 -db_Dev /dev/nvme0n1 -db_size 35
unable to get device info for '/dev/nvme2n1p1'

Is there a reason that pveceph doesn't recognise /dev/nvme2n1p1? Is this the correct way to pass a disk partition to pveceph?
 
Last edited:

victorhooi

Member
Apr 3, 2018
249
18
23
36
Right - but using that command (ceph-volume) only lets you create multiple OSDs on a single device.

It doesn't let you assign the WAL/DB location to a different device as far as I can tell?

That's why I'm manually creating the partitions myself, then using pveceph osd create - since that does let you assign the WAL/DB to a difference device, and handles automatically partitioning that.
 
Jan 29, 2019
119
31
33
What I tried to say was:
- Use pvcreate on your NVMe to create a LVM device
- Use lvcreate on your LVM to create 4 logical volumes
- Use these LVs in pveceph plus your WAL device
 
Jan 29, 2019
119
31
33
Another Idea:
- Use ceph-volume first to create the OSDs
- Then out, stop and purge them
- Then recreate them with pveceph using the already existing LVs...
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!