Ceph issue

C

Canadian Pinguin

Member
Dec 6, 2019
11
1
23
40
Hello,
Yesterday I updated my proxmox from 5.4 to 6.
Than i update my ceph.
Now when i try to add a OSD I get this error :

Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
stderr: 2019-12-05 20:47:59.421 7f2dd282c700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2019-12-05 20:47:59.421 7f2dd282c700 -1 AuthRegistry(0x7f2dcc080208) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
stderr: purged osd.0
--> RuntimeError: unable to use device

my /etc/pve/ceph.conf:

[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 192.168.2.0/24
fsid = e24ab9b2-6d63-4c87-bc2c-4a1c0a7fe3f6
mon_allow_pool_delete = true
osd_journal_size = 2000
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = 192.168.2.0/24

[client]
keyring = /etc/pve/priv/$cluster.$name.keyring

[mds]
keyring = /var/lib/ceph/mds/ceph-$id/keyring

[mds.home3]
host = home3
mds standby for name = pve

[mds.home1]
host = home1
mds_standby_for_name = pve

[mon.home2]
host = home2
mon_addr = 192.168.2.61:6789

[mon.home1]
host = home1
mon_addr = 192.168.2.60:6789

[mon.home3]
host = home3
mon_addr = 192.168.2.62:6789
I'm not sure what I'm missing.
 
Hi Alwin,
I did restart all the ceph services. I even restarted all the nodes.
I fallowed the guide, and modified the ceph.conf. What part did I miss?

Should I also remove that part :
[mds]
keyring = /var/lib/ceph/mds/ceph-$id/keyring
?
Thanks for the answer
 
For each host that has been upgraded, you should update your /etc/pve/ceph.conf file so that it either specifies no monitor port (if you are running the monitors on the default ports) or references both the v2 and v1 addresses and ports explicitly. Things will still work if only the v1 IP and port are listed, but each CLI instantiation or daemon will need to reconnect after learning the monitors also speak the v2 protocol, slowing things down a bit and preventing a full transition to the v2 protocol.
Click to expand...
As I have seen the port numbers, I figured the upgrade might not have been completed.

Canadian Pinguin said:
/var/lib/ceph/bootstrap-osd/ceph.keyring
Click to expand...
Is there a file at that path?
 
As I have seen the port numbers, I figured the upgrade might not have been completed.
Click to expand...
Seems like I missed that part. I remove the ports.
Is there a file at that path?
Click to expand...
Yes there is :
root@home1:~# ls /var/lib/ceph/bootstrap-osd/
ceph.keyring

After those changes I have a different issue :
Code: 
create OSD on /dev/sdc (bluestore)
creating block.db on '/dev/sdb'
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
Use of uninitialized value $part_or_lv in concatenation (.) or string at /usr/share/perl5/PVE/API2/Ceph/OSD.pm line 439.
using '' for block.db
wipe disk/partition: /dev/sdc
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 1.23717 s, 170 MB/s
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 9af62b91-bf3e-4d26-afca-c8efff6220e2
Running command: /sbin/vgcreate -s 1G --force --yes ceph-dc75fac6-7237-4f4b-b670-ba4cb5e705a2 /dev/sdc
stdout: Physical volume "/dev/sdc" successfully created.
stdout: Volume group "ceph-dc75fac6-7237-4f4b-b670-ba4cb5e705a2" successfully created
Running command: /sbin/lvcreate --yes -l 100%FREE -n osd-block-9af62b91-bf3e-4d26-afca-c8efff6220e2 ceph-dc75fac6-7237-4f4b-b670-ba4cb5e705a2
stdout: Logical volume "osd-block-9af62b91-bf3e-4d26-afca-c8efff6220e2" created.
--> blkid could not detect a PARTUUID for device:
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
stderr: 2019-12-12 09:37:46.616 7f17d928f700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2019-12-12 09:37:46.616 7f17d928f700 -1 AuthRegistry(0x7f17d4080208) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
stderr: purged osd.0
-->  RuntimeError: unable to use device
TASK ERROR: command 'ceph-volume lvm create --cluster-fsid e24ab9b2-6d63-4c87-bc2c-4a1c0a7fe3f6 --block.db '' --data /dev/sdc' failed: exit code 1
In response to that warning :
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
I ran partprobe, Tried again same result (with same warning).
I rebooted, Tried again same result (with same warning).
 
Last edited:
Canadian Pinguin said:
Use of uninitialized value $part_or_lv in concatenation (.) or string at /usr/share/perl5/PVE/API2/Ceph/OSD.pm line 439. using '' for block.db
Click to expand...
What is the disk layout of the OSD? And could you please try to create the OSD on the CLI, pveceph osd create?
 
What is the disk layout of the OSD? And could you please try to create the OSD on the CLI, pveceph osd create?
Click to expand...
Not sure what you mean by disk layout. Its a disk with no partition on it. I use ceph-volume lvm zap /dev/sdc --destroy before trying to add it.
Samething if i use cli :
Code: 
root@home1:~# pveceph osd create /dev/sdc -db_dev /dev/sdb -db_size 30
create OSD on /dev/sdc (bluestore)
creating block.db on '/dev/sdb'
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
Use of uninitialized value $part_or_lv in concatenation (.) or string at /usr/share/perl5/PVE/API2/Ceph/OSD.pm line 439.
using '' for block.db
wipe disk/partition: /dev/sdc
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 1.24088 s, 169 MB/s
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new e346307a-d47c-4e24-92b0-d9c36372f33a
Running command: /sbin/vgcreate -s 1G --force --yes ceph-54ae46e8-cfd5-452c-95c2-71a4e1a5243f /dev/sdc
 stdout: Physical volume "/dev/sdc" successfully created.
 stdout: Volume group "ceph-54ae46e8-cfd5-452c-95c2-71a4e1a5243f" successfully created
Running command: /sbin/lvcreate --yes -l 100%FREE -n osd-block-e346307a-d47c-4e24-92b0-d9c36372f33a ceph-54ae46e8-cfd5-452c-95c2-71a4e1a5243f
 stdout: Logical volume "osd-block-e346307a-d47c-4e24-92b0-d9c36372f33a" created.
--> blkid could not detect a PARTUUID for device:
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
 stderr: 2019-12-12 11:36:52.653 7f10bede4700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2019-12-12 11:36:52.653 7f10bede4700 -1 AuthRegistry(0x7f10b8080208) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
 stderr: purged osd.0
-->  RuntimeError: unable to use device
command 'ceph-volume lvm create --cluster-fsid e24ab9b2-6d63-4c87-bc2c-4a1c0a7fe3f6 --block.db '' --data /dev/sdc' failed: exit code 1
 
Canadian Pinguin said:
Not sure what you mean by disk layout. Its a disk with no partition on it. I use ceph-volume lvm zap /dev/sdc --destroy before trying to add it.
Click to expand...
Yes, but do you specify a separate DB / WAL or both?

Canadian Pinguin said:
command 'ceph-volume lvm create --cluster-fsid e24ab9b2-6d63-4c87-bc2c-4a1c0a7fe3f6 --block.db '' --data /dev/sdc' failed: exit code 1
Click to expand...
It fails because the block.db is specified as empty. There should be a path or VG/LV.
 
You mean it fails because there is other DB partition on the disk? (/dev/sdb)
So it wont work using proxmox GUI? I need to do it in cli and specified the partition (/dev/sdb6) ?
That wasnt an issue before.
 
From Luminous to Nautilus the tooling changed. ceph-disk was replaced by ceph-volume. The OSDs are now setup on LVs. And the DB+WAL will life on the same disk as the data, if you don't specify otherwise. Also, a too-small DB partition will only create an overflow back to the data partition. The sizes should be somewhere bigger than in 3, 30, 300 GiB.
 
This is why im specifying that the DB should be created on /dev/sdb (this is a ssd). And should be 30 GB. I know the rule of thumb is supposed to be 10% but this is a home setup. And I figured that 30GB on a ssd is still better than having the database on a mechanical drive no? Even if that means some spillover.
The OSD i'm trying to add is /dev/sdc and I want to DB to be on /dev/sdb. This is why i ended up with the command :
pveceph osd create /dev/sdc -db_dev /dev/sdb -db_size 30
In the end it does create a 30GB partiation on my ssd. It aslo does format the drive I want as an OSD. But it fails to add it.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom