When using the following command (in a bash script):
pveceph osd create $DEVICE --crush-device-class $DEVICECLASS --encrypted 1
about 5% of the devices result in the following error (see end of my message). The raw data at the end is truncated since there are quite a few lines.
The devices are namespaces on Micron 9300's. This occurs on multiple drives, so it is definitely not a media issue.
After cleaning up the mess (deleting the LV, VG, and PV in LVM - after dealing with direct mapping left-overs), I can run the command again and it runs fine. So, i'm assuming it is a race condition somewhere.
Any ideas? I was going to address this the Ceph mailing list, but thought I'd check here first.
Thanks!
Eric
create OSD on /dev/nvme0n7 (bluestore)
wipe disk/partition: /dev/nvme0n7
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.15525 s, 1.4 GB/s
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new cd44fcb0-8d75-4e4a-bf2e-e7ba6fa9f190
Running command: /sbin/vgcreate --force --yes ceph-b450a1a7-4760-43e3-96d8-ed1db7cbcb85 /dev/nvme0n7
stdout: Physical volume "/dev/nvme0n7" successfully created.
stdout: Volume group "ceph-b450a1a7-4760-43e3-96d8-ed1db7cbcb85" successfully created
Running command: /sbin/lvcreate --yes -l 455679 -n osd-block-cd44fcb0-8d75-4e4a-bf2e-e7ba6fa9f190 ceph-b450a1a7-4760-43e3-96d8-ed1db7cbcb85
stdout: Logical volume "osd-block-cd44fcb0-8d75-4e4a-bf2e-e7ba6fa9f190" created.
Running command: /bin/ceph-authtool --gen-print-key
Running command: /sbin/cryptsetup --batch-mode --key-file - luksFormat /dev/ceph-b450a1a7-4760-43e3-96d8-ed1db7cbcb85/osd-block-cd44fcb0-8d75-4e4a-bf2e-e7ba6fa9f190
Running command: /sbin/cryptsetup --key-file - --allow-discards luksOpen /dev/ceph-b450a1a7-4760-43e3-96d8-ed1db7cbcb85/osd-block-cd44fcb0-8d75-4e4a-bf2e-e7ba6fa9f190 q22DOn-Tnep-BPmR-CA3a-XAjg-iIQc-6p0HvO
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-6
--> Executable selinuxenabled not in PATH: /sbin:/bin:/usr/sbin:/usr/bin
Running command: /bin/chown -h ceph:ceph /dev/mapper/q22DOn-Tnep-BPmR-CA3a-XAjg-iIQc-6p0HvO
Running command: /bin/chown -R ceph:ceph /dev/dm-18
Running command: /bin/ln -s /dev/mapper/q22DOn-Tnep-BPmR-CA3a-XAjg-iIQc-6p0HvO /var/lib/ceph/osd/ceph-6/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-6/activate.monmap
stderr: 2021-08-03T13:58:21.960-0500 7f6614651700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2021-08-03T13:58:21.960-0500 7f6614651700 -1 AuthRegistry(0x7f660c05aed8) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
stderr: got monmap epoch 3
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-6/keyring --create-keyring --name osd.6 --add-key AQDBkQlhNO+6BhAA+sSHQu131C1bicD4MHbAHg==
stdout: creating /var/lib/ceph/osd/ceph-6/keyring
added entity osd.6 auth(key=AQDBkQlhNO+6BhAA+sSHQu131C1bicD4MHbAHg==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6/
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 6 --monmap /var/lib/ceph/osd/ceph-6/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-6/ --osd-uuid cd44fcb0-8d75-4e4a-bf2e-e7ba6fa9f190 --setuser ceph --setgroup ceph
stderr: 2021-08-03T13:58:22.172-0500 7f4469e92f00 -1 bluestore(/var/lib/ceph/osd/ceph-6/) _read_fsid unparsable uuid
stderr: 2021-08-03T13:58:22.176-0500 7f4469e92f00 -1 bluefs _replay 0x0: stop: uuid e9379130-c294-0933-b3bb-55bf459d4216 != super.uuid 7037cd3c-d089-4aef-a422-91efd94c9a0f, block dump:
stderr: 00000000 54 5d f4 25 4a 1e e9 37 91 30 c2 94 09 33 b3 bb |T].%J..7.0...3..|
stderr: 00000010 55 bf 45 9d 42 16 00 17 95 e8 6d e1 ef 38 37 70 |U.E.B.....m..87p|
stderr: 00000020 03 00 cd b0 af e8 3f 65 8c cb 9d 30 20 30 66 2a |......?e...0 0f*|
stderr: 00000030 a5 1c ad 9f fd 2b 03 3d bc 5f 75 54 bc 07 fa da |.....+.=._uT....|
pveceph osd create $DEVICE --crush-device-class $DEVICECLASS --encrypted 1
about 5% of the devices result in the following error (see end of my message). The raw data at the end is truncated since there are quite a few lines.
The devices are namespaces on Micron 9300's. This occurs on multiple drives, so it is definitely not a media issue.
After cleaning up the mess (deleting the LV, VG, and PV in LVM - after dealing with direct mapping left-overs), I can run the command again and it runs fine. So, i'm assuming it is a race condition somewhere.
Any ideas? I was going to address this the Ceph mailing list, but thought I'd check here first.
Thanks!
Eric
create OSD on /dev/nvme0n7 (bluestore)
wipe disk/partition: /dev/nvme0n7
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 0.15525 s, 1.4 GB/s
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph-authtool --gen-print-key
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new cd44fcb0-8d75-4e4a-bf2e-e7ba6fa9f190
Running command: /sbin/vgcreate --force --yes ceph-b450a1a7-4760-43e3-96d8-ed1db7cbcb85 /dev/nvme0n7
stdout: Physical volume "/dev/nvme0n7" successfully created.
stdout: Volume group "ceph-b450a1a7-4760-43e3-96d8-ed1db7cbcb85" successfully created
Running command: /sbin/lvcreate --yes -l 455679 -n osd-block-cd44fcb0-8d75-4e4a-bf2e-e7ba6fa9f190 ceph-b450a1a7-4760-43e3-96d8-ed1db7cbcb85
stdout: Logical volume "osd-block-cd44fcb0-8d75-4e4a-bf2e-e7ba6fa9f190" created.
Running command: /bin/ceph-authtool --gen-print-key
Running command: /sbin/cryptsetup --batch-mode --key-file - luksFormat /dev/ceph-b450a1a7-4760-43e3-96d8-ed1db7cbcb85/osd-block-cd44fcb0-8d75-4e4a-bf2e-e7ba6fa9f190
Running command: /sbin/cryptsetup --key-file - --allow-discards luksOpen /dev/ceph-b450a1a7-4760-43e3-96d8-ed1db7cbcb85/osd-block-cd44fcb0-8d75-4e4a-bf2e-e7ba6fa9f190 q22DOn-Tnep-BPmR-CA3a-XAjg-iIQc-6p0HvO
Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-6
--> Executable selinuxenabled not in PATH: /sbin:/bin:/usr/sbin:/usr/bin
Running command: /bin/chown -h ceph:ceph /dev/mapper/q22DOn-Tnep-BPmR-CA3a-XAjg-iIQc-6p0HvO
Running command: /bin/chown -R ceph:ceph /dev/dm-18
Running command: /bin/ln -s /dev/mapper/q22DOn-Tnep-BPmR-CA3a-XAjg-iIQc-6p0HvO /var/lib/ceph/osd/ceph-6/block
Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-6/activate.monmap
stderr: 2021-08-03T13:58:21.960-0500 7f6614651700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2021-08-03T13:58:21.960-0500 7f6614651700 -1 AuthRegistry(0x7f660c05aed8) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
stderr: got monmap epoch 3
Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-6/keyring --create-keyring --name osd.6 --add-key AQDBkQlhNO+6BhAA+sSHQu131C1bicD4MHbAHg==
stdout: creating /var/lib/ceph/osd/ceph-6/keyring
added entity osd.6 auth(key=AQDBkQlhNO+6BhAA+sSHQu131C1bicD4MHbAHg==)
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6/keyring
Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6/
Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 6 --monmap /var/lib/ceph/osd/ceph-6/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-6/ --osd-uuid cd44fcb0-8d75-4e4a-bf2e-e7ba6fa9f190 --setuser ceph --setgroup ceph
stderr: 2021-08-03T13:58:22.172-0500 7f4469e92f00 -1 bluestore(/var/lib/ceph/osd/ceph-6/) _read_fsid unparsable uuid
stderr: 2021-08-03T13:58:22.176-0500 7f4469e92f00 -1 bluefs _replay 0x0: stop: uuid e9379130-c294-0933-b3bb-55bf459d4216 != super.uuid 7037cd3c-d089-4aef-a422-91efd94c9a0f, block dump:
stderr: 00000000 54 5d f4 25 4a 1e e9 37 91 30 c2 94 09 33 b3 bb |T].%J..7.0...3..|
stderr: 00000010 55 bf 45 9d 42 16 00 17 95 e8 6d e1 ef 38 37 70 |U.E.B.....m..87p|
stderr: 00000020 03 00 cd b0 af e8 3f 65 8c cb 9d 30 20 30 66 2a |......?e...0 0f*|
stderr: 00000030 a5 1c ad 9f fd 2b 03 3d bc 5f 75 54 bc 07 fa da |.....+.=._uT....|