Multiple osd for device fail

P

petruzzo

Active Member
Proxmox Subscriber
Jun 23, 2022
43
1
28
Hi I try to create multiple osd with

pveceph os create DEV --osds-per-device 2
but the script give me an error and dont see The osd on web interface
proxmox-ve: 9.2.0 (running kernel: 7.0.14-4-pve)pve-manager: 9.2.4 (running version: 9.2.4/5e5ae681198514d4)proxmox-kernel-helper: 9.2.0proxmox-kernel-7.0.14-4-pve-signed: 7.0.14-4proxmox-kernel-7.0: 7.0.14-4proxmox-kernel-7.0.2-6-pve-signed: 7.0.2-6ceph: 20.2.2-pve1ceph-fuse: 20.2.2-pve1corosync: 3.1.10-pve2criu: 4.1.1-1frr-pythontools: 10.6.1-1+pve2ifupdown2: 3.3.0-1+pmx12intel-microcode: 3.20251111.1~deb13u1ksm-control-daemon: 1.5-1libjs-extjs: 7.0.0-5libproxmox-acme-perl: 1.7.1libproxmox-backup-qemu0: 2.0.2libproxmox-rs-perl: 0.4.1libpve-access-control: 9.1.1libpve-apiclient-perl: 3.4.2libpve-cluster-api-perl: 9.1.6libpve-cluster-perl: 9.1.6libpve-common-perl: 9.1.16libpve-guest-common-perl: 6.0.4libpve-http-server-perl: 6.0.5libpve-network-perl: 1.6.6libpve-notify-perl: 9.1.6libpve-rs-perl: 0.15.3libpve-storage-perl: 9.1.6libspice-server1: 0.15.2-1+b1lvm2: 2.03.31-2+pmx1lxc-pve: 7.0.0-2lxcfs: 7.0.0-pve1novnc-pve: 1.7.0-2openvswitch-switch: 3.5.0-1+b1proxmox-backup-client: 4.2.2-1proxmox-backup-file-restore: 4.2.2-1proxmox-backup-restore-image: 1.0.0proxmox-firewall: 1.2.3proxmox-kernel-helper: 9.2.0proxmox-mail-forward: 1.0.3proxmox-mini-journalreader: 1.7proxmox-offline-mirror-helper: 0.7.4proxmox-widget-toolkit: 5.2.6pve-cluster: 9.1.6pve-container: 6.1.10pve-docs: 9.2.3pve-edk2-firmware: 4.2025.05-2pve-esxi-import-tools: 1.0.1pve-firewall: 6.0.4pve-firmware: 3.18-4pve-ha-manager: 5.2.4pve-i18n: 3.9.0pve-qemu-kvm: 11.0.2-1pve-xtermjs: 6.0.0-2qemu-server: 9.1.18smartmontools: 7.5-pve2spiceterm: 3.4.2swtpm: 0.8.0+pve3vncterm: 1.9.2zfsutils-linux: 2.4.3-pve1

create OSD on /dev/nvme0n1 (bluestore)
wiping block device /dev/nvme0n1
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.176457 s, 1.2 GB/s
--> passed data devices: 1 physical, 0 LVM
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 06b46a68-90dd-46e9-acbf-7960753b5add
Running command: /sbin/nvme format /dev/nvme0n1 --force
stdout: Success formatting namespace:1
--> nvme format completed for /dev/nvme0n1
Running command: vgcreate --force --yes ceph-3dab9b5b-bc4a-4969-868f-da4af08f84f8 /dev/nvme0n1
stdout: Physical volume "/dev/nvme0n1" successfully created.
stdout: Volume group "ceph-3dab9b5b-bc4a-4969-868f-da4af08f84f8" successfully created
Running command: lvcreate --yes -l 915706 -n osd-block-06b46a68-90dd-46e9-acbf-7960753b5add ceph-3dab9b5b-bc4a-4969-868f-da4af08f84f8
stdout: Logical volume "osd-block-06b46a68-90dd-46e9-acbf-7960753b5add" created.
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
--> Executable selinuxenabled not in PATH: /sbin:/bin:/usr/sbin:/usr/bin
Running command: /bin/chown -h ceph:ceph /dev/ceph-3dab9b5b-bc4a-4969-868f-da4af08f84f8/osd-block-06b46a68-90dd-46e9-acbf-7960753b5add
Running command: /bin/chown -R ceph:ceph /dev/dm-5
Running command: /bin/ln -s /dev/ceph-3dab9b5b-bc4a-4969-868f-da4af08f84f8/osd-block-06b46a68-90dd-46e9-acbf-7960753b5add /var/lib/ceph/osd/ceph-0/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
stderr: 2026-07-30T14:30:31.577+0200 7b8a04e9a6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2026-07-30T14:30:31.577+0200 7b8a04e9a6c0 -1 AuthRegistry(0x7b8a000502f0) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
stderr: got monmap epoch 6
--> Creating keyring file for osd.0
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --bdev-enable-discard false --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 06b46a68-90dd-46e9-acbf-7960753b5add --setuser ceph --setgroup ceph
stderr: 2026-07-30T14:30:31.931+0200 74d1dfe61840 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) No valid bdev label found
stderr: 2026-07-30T14:30:31.963+0200 74d1dfe61840 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
--> ceph-volume lvm prepare successful for: /dev/nvme0n1
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-3dab9b5b-bc4a-4969-868f-da4af08f84f8/osd-block-06b46a68-90dd-46e9-acbf-7960753b5add --path /var/lib/ceph/osd/ceph-0 --no-mon-config
Running command: /bin/ln -snf /dev/ceph-3dab9b5b-bc4a-4969-868f-da4af08f84f8/osd-block-06b46a68-90dd-46e9-acbf-7960753b5add /var/lib/ceph/osd/ceph-0/block
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
Running command: /bin/chown -R ceph:ceph /dev/dm-5
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: /bin/systemctl enable ceph-volume@lvm-0-06b46a68-90dd-46e9-acbf-7960753b5add
stderr: Created symlink '/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-06b46a68-90dd-46e9-acbf-7960753b5add.service' -> '/usr/lib/systemd/system/ceph-volume@.service'.
Running command: /bin/systemctl enable --runtime ceph-osd@0
stderr: Created symlink '/run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service' -> '/usr/lib/systemd/system/ceph-osd@.service'.
Running command: /bin/systemctl start ceph-osd@0
--> ceph-volume lvm activate successful for osd ID: 0
--> ceph-volume lvm create successful for: /dev/nvme0n1
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 5c213446-330e-411e-9b0a-ca5d513efb5e
Running command: /sbin/nvme format /dev/nvme0n1 --force
stdout: Success formatting namespace:1
--> nvme format completed for /dev/nvme0n1
Running command: vgcreate --force --yes ceph-3dc513f3-07c3-473e-a5c9-09b36664ca35 /dev/nvme0n1
stderr: Can't open /dev/nvme0n1 exclusively. Mounted filesystem?
Can't open /dev/nvme0n1 exclusively. Mounted filesystem?
Error opening device /dev/nvme0n1 for reading at 0 length 512.
Can't open /dev/nvme0n1 exclusively. Mounted filesystem?
Error opening device /dev/nvme0n1 for reading at 0 length 4096.
stderr: Cannot use /dev/nvme0n1: device has a signature
--> Was unable to complete a new OSD, will rollback changes
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it
stderr: 2026-07-30T14:30:35.766+0200 7d768929a6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2026-07-30T14:30:35.767+0200 7d768929a6c0 -1 AuthRegistry(0x7d76840502f0) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
stderr: purged osd.1
--> No OSD identified by "1" was found among LVM-based OSDs.
--> Proceeding to check RAW-based OSDs.
No OSD were found.
command 'ceph-volume lvm batch --crush-device-class NVMe --osds-per-device 2 --yes --no-auto -- /dev/nvme0n1' failed: exit code 1


Anyone had an idea how to fix?
the list of the osd give me

ceph osd ls
0
 
Hi @petruzzo

thanks for posting on the forum!

As a test could you try to use pvesh instead of pveceph directly?
Your command would look something like this
pvesh create /nodes/<node>/ceph/osd --dev <DEV> --node <node> --osds-per-device 2
Ref [1]

Also to cleanup the device you could use something like this
ceph-volume lvm zap /dev/sd[X] --destroy
Ref [2]

Best regards
Jonas

[1] https://pve.proxmox.com/pve-docs/api-viewer/index.html#/nodes/{node}/ceph/osd
[2] https://pve.proxmox.com/pve-docs/pveceph.1.html
 
hi I try with pvesh but I have the same resoult
command 'ceph-volume lvm batch --crush-device-class NVMe --osds-per-device 2 --yes --no-auto -- /dev/nvme0n1' failed: exit code 1
create OSD on /dev/nvme0n1 (bluestore)wiping block device /dev/nvme0n1200+0 records in200+0 records out209715200 bytes (210 MB, 200 MiB) copied, 0.153041 s, 1.4 GB/s--> passed data devices: 1 physical, 0 LVMRunning command: /bin/ceph-authtool --gen-print-keyRunning command: /bin/ceph-authtool --gen-print-keyRunning command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new ca24370e-1c7d-4f75-8176-c96ecd3ced8fRunning command: /sbin/nvme format /dev/nvme0n1 --force stdout: Success formatting namespace:1--> nvme format completed for /dev/nvme0n1Running command: vgcreate --force --yes ceph-1bde92d3-7eb0-4f2f-8311-39d15d0a8109 /dev/nvme0n1 stdout: Physical volume "/dev/nvme0n1" successfully created. stdout: Volume group "ceph-1bde92d3-7eb0-4f2f-8311-39d15d0a8109" successfully createdRunning command: lvcreate --yes -l 915706 -n osd-block-ca24370e-1c7d-4f75-8176-c96ecd3ced8f ceph-1bde92d3-7eb0-4f2f-8311-39d15d0a8109 stdout: Logical volume "osd-block-ca24370e-1c7d-4f75-8176-c96ecd3ced8f" created.Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-2--> Executable selinuxenabled not in PATH: /sbin:/bin:/usr/sbin:/usr/binRunning command: /bin/chown -h ceph:ceph /dev/ceph-1bde92d3-7eb0-4f2f-8311-39d15d0a8109/osd-block-ca24370e-1c7d-4f75-8176-c96ecd3ced8fRunning command: /bin/chown -R ceph:ceph /dev/dm-5Running command: /bin/ln -s /dev/ceph-1bde92d3-7eb0-4f2f-8311-39d15d0a8109/osd-block-ca24370e-1c7d-4f75-8176-c96ecd3ced8f /var/lib/ceph/osd/ceph-2/blockRunning command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-2/activate.monmap stderr: 2026-07-30T18:57:31.264+0200 72270709a6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory2026-07-30T18:57:31.264+0200 72270709a6c0 -1 AuthRegistry(0x7227000502f0) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx stderr: got monmap epoch 3--> Creating keyring file for osd.2Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/keyringRunning command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2/Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 2 --monmap /var/lib/ceph/osd/ceph-2/activate.monmap --bdev-enable-discard false --keyfile - --osd-data /var/lib/ceph/osd/ceph-2/ --osd-uuid ca24370e-1c7d-4f75-8176-c96ecd3ced8f --setuser ceph --setgroup ceph stderr: 2026-07-30T18:57:31.543+0200 7e3fc7643840 -1 bluestore(/var/lib/ceph/osd/ceph-2//block) No valid bdev label found stderr: 2026-07-30T18:57:31.577+0200 7e3fc7643840 -1 bluestore(/var/lib/ceph/osd/ceph-2/) _read_fsid unparsable uuid--> ceph-volume lvm prepare successful for: /dev/nvme0n1Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-1bde92d3-7eb0-4f2f-8311-39d15d0a8109/osd-block-ca24370e-1c7d-4f75-8176-c96ecd3ced8f --path /var/lib/ceph/osd/ceph-2 --no-mon-configRunning command: /bin/ln -snf /dev/ceph-1bde92d3-7eb0-4f2f-8311-39d15d0a8109/osd-block-ca24370e-1c7d-4f75-8176-c96ecd3ced8f /var/lib/ceph/osd/ceph-2/blockRunning command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/blockRunning command: /bin/chown -R ceph:ceph /dev/dm-5Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2Running command: /bin/systemctl enable ceph-volume@lvm-2-ca24370e-1c7d-4f75-8176-c96ecd3ced8f stderr: Created symlink '/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-2-ca24370e-1c7d-4f75-8176-c96ecd3ced8f.service' -> '/usr/lib/systemd/system/ceph-volume@.service'.Running command: /bin/systemctl enable --runtime ceph-osd@2 stderr: Created symlink '/run/systemd/system/ceph-osd.target.wants/ceph-osd@2.service' -> '/usr/lib/systemd/system/ceph-osd@.service'.Running command: /bin/systemctl start ceph-osd@2--> ceph-volume lvm activate successful for osd ID: 2--> ceph-volume lvm create successful for: /dev/nvme0n1Running command: /bin/ceph-authtool --gen-print-keyRunning command: /bin/ceph-authtool --gen-print-keyRunning command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new a7eb7629-80af-4bfe-b99e-f9f43bfedb0cRunning command: /sbin/nvme format /dev/nvme0n1 --force stdout: Success formatting namespace:1--> nvme format completed for /dev/nvme0n1Running command: vgcreate --force --yes ceph-a00cb3d0-085e-4679-98d9-e586a4f19e83 /dev/nvme0n1 stderr: Can't open /dev/nvme0n1 exclusively. Mounted filesystem? Can't open /dev/nvme0n1 exclusively. Mounted filesystem? Error opening device /dev/nvme0n1 for reading at 0 length 512. Can't open /dev/nvme0n1 exclusively. Mounted filesystem? Error opening device /dev/nvme0n1 for reading at 0 length 4096. Cannot use /dev/nvme0n1: device has a signature--> Was unable to complete a new OSD, will rollback changesRunning command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.3 --yes-i-really-mean-it stderr: 2026-07-30T18:57:35.365+0200 7035cd69a6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory2026-07-30T18:57:35.365+0200 7035cd69a6c0 -1 AuthRegistry(0x7035c80502f0) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx stderr: purged osd.3--> No OSD identified by "3" was found among LVM-based OSDs.--> Proceeding to check RAW-based OSDs.No OSD were found."UPID:cl8kvm3:00003BAD:0002FA0F:6A6B8278:cephcreateosd:nvme0n1:root@pam:"
if I create only one osd for device the job is ok
 
SteveITS said:
As I understood it there is no reason to use multiple OSD per disk anymore...?
Click to expand...
Yes, for HARD DRIVES. here is a link for a more recent version of that document: https://docs.ceph.com/en/tentacle/start/hardware-recommendations/

note that OP is using NVMEs. --edit worth discussing when its actually desireable/advisable to do this- It is entirely dependent on your usecase/payload. more OSDs per nvme will scale performance wider (more available iops for more guests) BUT will necessarily push you to a higher PG count- which increases all OSD Processing overhead. remember that you should plan on a cpu core + 4GB of ram for EACH OSD, not each drive.

as to ops problem: note the actual error that is breaking your creation:
stderr: 2026-07-30T14:30:31.577+0200 7b8a04e9a6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory

Its possible (probable) that the key being sought after is different in the different manners you're calling the scripts. start by double checking that the above file exists on the node you're running the command on. If it doesnt exist, create or copy it from another node.
 
Last edited:
I had to create one OSD per disk because it was the only way to provision them. However, the keyring file is still missing from the priv directory, even with the single OSD created
The script worked in version 9.1, but it no longer does
the log of the creation give the same error but the osds was created
create OSD on /dev/nvme0n1 (bluestore)
wiping block device /dev/nvme0n1
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.139238 s, 1.5 GB/s
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new cb0b6a08-5118-451b-ad22-d246f69a8125
Running command: /sbin/nvme format /dev/nvme0n1 --force
stdout: Success formatting namespace:1
--> nvme format completed for /dev/nvme0n1
Running command: vgcreate --force --yes ceph-064c4f3c-74cf-4145-8d1a-a4e4d0566ec5 /dev/nvme0n1
stdout: Physical volume "/dev/nvme0n1" successfully created.
stdout: Volume group "ceph-064c4f3c-74cf-4145-8d1a-a4e4d0566ec5" successfully created
Running command: lvcreate --yes -l 1831412 -n osd-block-cb0b6a08-5118-451b-ad22-d246f69a8125 ceph-064c4f3c-74cf-4145-8d1a-a4e4d0566ec5
stdout: Logical volume "osd-block-cb0b6a08-5118-451b-ad22-d246f69a8125" created.
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-13
--> Executable selinuxenabled not in PATH: /sbin:/bin:/usr/sbin:/usr/bin
Running command: /bin/chown -h ceph:ceph /dev/ceph-064c4f3c-74cf-4145-8d1a-a4e4d0566ec5/osd-block-cb0b6a08-5118-451b-ad22-d246f69a8125
Running command: /bin/chown -R ceph:ceph /dev/dm-5
Running command: /bin/ln -s /dev/ceph-064c4f3c-74cf-4145-8d1a-a4e4d0566ec5/osd-block-cb0b6a08-5118-451b-ad22-d246f69a8125 /var/lib/ceph/osd/ceph-13/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-13/activate.monmap
stderr: 2026-07-30T19:39:08.944+0200 72ea3fa9a6c0 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2026-07-30T19:39:08.944+0200 72ea3fa9a6c0 -1 AuthRegistry(0x72ea380502f0) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
stderr: got monmap epoch 6
--> Creating keyring file for osd.13
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-13/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-13/
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 13 --monmap /var/lib/ceph/osd/ceph-13/activate.monmap --bdev-enable-discard false --keyfile - --osd-data /var/lib/ceph/osd/ceph-13/ --osd-uuid cb0b6a08-5118-451b-ad22-d246f69a8125 --setuser ceph --setgroup ceph
stderr: 2026-07-30T19:39:09.223+0200 73c0b3a61840 -1 bluestore(/var/lib/ceph/osd/ceph-13//block) No valid bdev label found
stderr: 2026-07-30T19:39:09.270+0200 73c0b3a61840 -1 bluestore(/var/lib/ceph/osd/ceph-13/) _read_fsid unparsable uuid
--> ceph-volume lvm prepare successful for: /dev/nvme0n1
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-13
Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-064c4f3c-74cf-4145-8d1a-a4e4d0566ec5/osd-block-cb0b6a08-5118-451b-ad22-d246f69a8125 --path /var/lib/ceph/osd/ceph-13 --no-mon-config
Running command: /bin/ln -snf /dev/ceph-064c4f3c-74cf-4145-8d1a-a4e4d0566ec5/osd-block-cb0b6a08-5118-451b-ad22-d246f69a8125 /var/lib/ceph/osd/ceph-13/block
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-13/block
Running command: /bin/chown -R ceph:ceph /dev/dm-5
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-13
Running command: /bin/systemctl enable ceph-volume@lvm-13-cb0b6a08-5118-451b-ad22-d246f69a8125
stderr: Created symlink '/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-13-cb0b6a08-5118-451b-ad22-d246f69a8125.service' -> '/usr/lib/systemd/system/ceph-volume@.service'.
Running command: /bin/systemctl enable --runtime ceph-osd@13
stderr: Created symlink '/run/systemd/system/ceph-osd.target.wants/ceph-osd@13.service' -> '/usr/lib/systemd/system/ceph-osd@.service'.
Running command: /bin/systemctl start ceph-osd@13
--> ceph-volume lvm activate successful for osd ID: 13
--> ceph-volume lvm create successful for: /dev/nvme0n1
edit
The command to clean the disk give error
--> Zapping: /dev/nvme0n1
--> --destroy was not specified, but zapping a whole device will remove the partition table
stderr: wipefs: error: /dev/nvme0n1: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again (1/9) to workaround probable race condition
stderr: wipefs: error: /dev/nvme0n1: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again (2/9) to workaround probable race condition
stderr: wipefs: error: /dev/nvme0n1: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again (3/9) to workaround probable race condition
stderr: wipefs: error: /dev/nvme0n1: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again (4/9) to workaround probable race condition
stderr: wipefs: error: /dev/nvme0n1: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again (5/9) to workaround probable race condition
stderr: wipefs: error: /dev/nvme0n1: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again (6/9) to workaround probable race condition
stderr: wipefs: error: /dev/nvme0n1: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again (7/9) to workaround probable race condition
stderr: wipefs: error: /dev/nvme0n1: probing initialization failed: Device or resource busy
--> failed to wipefs device, will try again (8/9) to workaround probable race condition
--> RuntimeError: could not complete wipefs on device: /dev/nvme0n1
 
Last edited:
Here is the uncomfortable truth for people wanting to deploy ceph: Its not a black box. you WILL need to understand what it (and by extension, you) are doing.

If a key file in the preconfigured location is missing, you need to either create/obtain it and place it in its expected location, OR specify the correct location at runtime. time to start reading documentation.

https://docs.ceph.com/en/latest/rados/configuration/auth-config-ref/
 
I had resolved the key issue but the problem was not it
this is the new log without key error
pveceph osd create /dev/nvme0n1 --osds-per-device 2 --crush-device-class NVMe
create OSD on /dev/nvme0n1 (bluestore)
wiping block device /dev/nvme0n1
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.137392 s, 1.5 GB/s
--> passed data devices: 1 physical, 0 LVM
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 957b5383-580c-49e6-86d8-8e1e19900bd5
Running command: /sbin/nvme format /dev/nvme0n1 --force
stdout: Success formatting namespace:1
--> nvme format completed for /dev/nvme0n1
Running command: vgcreate --force --yes ceph-2cd77ef5-a01b-4af8-be0f-f53aeff985cc /dev/nvme0n1
stdout: Physical volume "/dev/nvme0n1" successfully created.
stdout: Volume group "ceph-2cd77ef5-a01b-4af8-be0f-f53aeff985cc" successfully created
Running command: lvcreate --yes -l 915706 -n osd-block-957b5383-580c-49e6-86d8-8e1e19900bd5 ceph-2cd77ef5-a01b-4af8-be0f-f53aeff985cc
stdout: Logical volume "osd-block-957b5383-580c-49e6-86d8-8e1e19900bd5" created.
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
--> Executable selinuxenabled not in PATH: /sbin:/bin:/usr/sbin:/usr/bin
Running command: /bin/chown -h ceph:ceph /dev/ceph-2cd77ef5-a01b-4af8-be0f-f53aeff985cc/osd-block-957b5383-580c-49e6-86d8-8e1e19900bd5
Running command: /bin/chown -R ceph:ceph /dev/dm-5
Running command: /bin/ln -s /dev/ceph-2cd77ef5-a01b-4af8-be0f-f53aeff985cc/osd-block-957b5383-580c-49e6-86d8-8e1e19900bd5 /var/lib/ceph/osd/ceph-0/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
stderr: got monmap epoch 6
--> Creating keyring file for osd.0
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --bdev-enable-discard false --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 957b5383-580c-49e6-86d8-8e1e19900bd5 --setuser ceph --setgroup ceph
stderr: 2026-07-30T22:30:30.078+0200 71125dc6a840 -1 bluestore(/var/lib/ceph/osd/ceph-0//block) No valid bdev label found
stderr: 2026-07-30T22:30:30.104+0200 71125dc6a840 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
--> ceph-volume lvm prepare successful for: /dev/nvme0n1
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-2cd77ef5-a01b-4af8-be0f-f53aeff985cc/osd-block-957b5383-580c-49e6-86d8-8e1e19900bd5 --path /var/lib/ceph/osd/ceph-0 --no-mon-config
Running command: /bin/ln -snf /dev/ceph-2cd77ef5-a01b-4af8-be0f-f53aeff985cc/osd-block-957b5383-580c-49e6-86d8-8e1e19900bd5 /var/lib/ceph/osd/ceph-0/block
Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
Running command: /bin/chown -R ceph:ceph /dev/dm-5
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
Running command: /bin/systemctl enable ceph-volume@lvm-0-957b5383-580c-49e6-86d8-8e1e19900bd5
stderr: Created symlink '/etc/systemd/system/multi-user.target.wants/ceph-volume@lvm-0-957b5383-580c-49e6-86d8-8e1e19900bd5.service' -> '/usr/lib/systemd/system/ceph-volume@.service'.
Running command: /bin/systemctl enable --runtime ceph-osd@0
stderr: Created symlink '/run/systemd/system/ceph-osd.target.wants/ceph-osd@0.service' -> '/usr/lib/systemd/system/ceph-osd@.service'.
Running command: /bin/systemctl start ceph-osd@0
--> ceph-volume lvm activate successful for osd ID: 0
--> ceph-volume lvm create successful for: /dev/nvme0n1
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2f210c64-61eb-4b9c-a8ff-450962d76bcb
Running command: /sbin/nvme format /dev/nvme0n1 --force
stdout: Success formatting namespace:1
--> nvme format completed for /dev/nvme0n1
Running command: vgcreate --force --yes ceph-3b40414a-c135-4809-b704-42d24cf5efe8 /dev/nvme0n1
stderr: Can't open /dev/nvme0n1 exclusively. Mounted filesystem?
Can't open /dev/nvme0n1 exclusively. Mounted filesystem?
Error opening device /dev/nvme0n1 for reading at 0 length 512.
Can't open /dev/nvme0n1 exclusively. Mounted filesystem?
Error opening device /dev/nvme0n1 for reading at 0 length 4096.
Cannot use /dev/nvme0n1: device has a signature
--> Was unable to complete a new OSD, will rollback changes
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it
stderr: purged osd.1
--> No OSD identified by "1" was found among LVM-based OSDs.
--> Proceeding to check RAW-based OSDs.
No OSD were found.
command 'ceph-volume lvm batch --crush-device-class NVMe --osds-per-device 2 --yes --no-auto -- /dev/nvme0n1' failed: exit code 1
 
You must log in or register to reply here.
Top Bottom
Back