[pve6] ceph lumi to nautilus guide problem: Required devices (block and data) not present for bluest

grin

grin

Renowned Member
Proxmox Subscriber
Dec 8, 2008
174
22
83
Hungary
grin.hu
# ceph-volume simple scan
stderr: lsblk: /var/lib/ceph/osd/ceph-2: not a block device
stderr: Bad argument "/var/lib/ceph/osd/ceph-2", expected an absolute path in /dev/ or /sys or a unit name: Invalid argument
Running command: /sbin/cryptsetup status /dev/sdd1
--> OSD 2 got scanned and metadata persisted to file: /etc/ceph/osd/2-d49cdeea-391e-41c1-ada8-fef3be0b6bc2.json
--> To take over management of this scanned OSD, and disable ceph-disk and udev, run:
--> ceph-volume simple activate 2 d49cdeea-391e-41c1-ada8-fef3be0b6bc2

etc...
and no surprise that activation isn't happy either:
# ceph-volume simple activate --all
--> activating OSD specified in /etc/ceph/osd/4-a95f413e-8f07-41ba-96e1-3a9ba73b8beb.json
--> Required devices (block and data) not present for bluestore
--> bluestore devices found: [u'data']
--> RuntimeError: Unable to activate bluestore OSD due to missing devices
....etc...
 
grin said:
/etc/ceph/osd/2-d49cdeea-391e-41c1-ada8-fef3be0b6bc2.json
Click to expand...
What is the content of the json file?

grin said:
stderr: Bad argument "/var/lib/ceph/osd/ceph-2", expected an absolute path in /dev/ or /sys or a unit name: Invalid argument
Click to expand...
Usually it finds the corresponding device, try the command with the specific device in question.
Code: 
ceph-volume simple scan /dev/sdX1
 
hi,

I have the exact same problem

# ceph-volume simple scan

Code: 
stderr: lsblk: /var/lib/ceph/osd/ceph-5: not a block device
 stderr: Bad argument "/var/lib/ceph/osd/ceph-5", expected an absolute path in /dev/ or /sys or a unit name: Invalid argument
Running command: /sbin/cryptsetup status /dev/sdh1
--> OSD 5 got scanned and metadata persisted to file: /etc/ceph/osd/5-acd2a716-4504-4163-b315-fee6b5a4ac42.json
--> To take over management of this scanned OSD, and disable ceph-disk and udev, run:
-->     ceph-volume simple activate 5 acd2a716-4504-4163-b315-fee6b5a4ac42
 stderr: lsblk: /var/lib/ceph/osd/ceph-36: not a block device
 stderr: Bad argument "/var/lib/ceph/osd/ceph-36", expected an absolute path in /dev/ or /sys or a unit name: Invalid argument
Running command: /sbin/cryptsetup status tmpfs
 stderr: blkid: error: tmpfs: No such file or directory
 stderr: lsblk: tmpfs: not a block device
--> Ignoring /var/lib/ceph/osd/ceph-36 because it's not a ceph-disk created osd.
 stderr: lsblk: /var/lib/ceph/osd/ceph-1: not a block device
 stderr: Bad argument "/var/lib/ceph/osd/ceph-1", expected an absolute path in /dev/ or /sys or a unit name: Invalid argument
Running command: /sbin/cryptsetup status /dev/sdc1
--> OSD 1 got scanned and metadata persisted to file: /etc/ceph/osd/1-ada95d3b-2271-417f-8957-092dbad83b62.json
--> To take over management of this scanned OSD, and disable ceph-disk and udev, run:
-->     ceph-volume simple activate 1 ada95d3b-2271-417f-8957-092dbad83b62
 stderr: lsblk: /var/lib/ceph/osd/ceph-2: not a block device
 stderr: Bad argument "/var/lib/ceph/osd/ceph-2", expected an absolute path in /dev/ or /sys or a unit name: Invalid argument
Running command: /sbin/cryptsetup status /dev/sde1
--> OSD 2 got scanned and metadata persisted to file: /etc/ceph/osd/2-f9a8609d-437b-4e11-93f7-6edc5e9fdbb4.json
--> To take over management of this scanned OSD, and disable ceph-disk and udev, run:
-->     ceph-volume simple activate 2 f9a8609d-437b-4e11-93f7-6edc5e9fdbb4
 stderr: lsblk: /var/lib/ceph/osd/ceph-0: not a block device
 stderr: Bad argument "/var/lib/ceph/osd/ceph-0", expected an absolute path in /dev/ or /sys or a unit name: Invalid argument
Running command: /sbin/cryptsetup status /dev/sdb1
--> OSD 0 got scanned and metadata persisted to file: /etc/ceph/osd/0-c905557d-3619-4d8b-9da5-cff66fa6340d.json
--> To take over management of this scanned OSD, and disable ceph-disk and udev, run:
-->     ceph-volume simple activate 0 c905557d-3619-4d8b-9da5-cff66fa6340d
 stderr: lsblk: /var/lib/ceph/osd/ceph-3: not a block device
 stderr: Bad argument "/var/lib/ceph/osd/ceph-3", expected an absolute path in /dev/ or /sys or a unit name: Invalid argument
Running command: /sbin/cryptsetup status /dev/sdf1
--> OSD 3 got scanned and metadata persisted to file: /etc/ceph/osd/3-0ed815ee-5cd5-4cfc-b2be-7e42b084ae6b.json
--> To take over management of this scanned OSD, and disable ceph-disk and udev, run:
-->     ceph-volume simple activate 3 0ed815ee-5cd5-4cfc-b2be-7e42b084ae6b
 stderr: lsblk: /var/lib/ceph/osd/ceph-4: not a block device
 stderr: Bad argument "/var/lib/ceph/osd/ceph-4", expected an absolute path in /dev/ or /sys or a unit name: Invalid argument
Running command: /sbin/cryptsetup status /dev/sdg1
--> OSD 4 got scanned and metadata persisted to file: /etc/ceph/osd/4-e5f9a7ec-0613-4a0d-b340-93219597cd46.json
--> To take over management of this scanned OSD, and disable ceph-disk and udev, run:
-->     ceph-volume simple activate 4 e5f9a7ec-0613-4a0d-b340-93219597cd46

Example from osd-0

Code: 
# cat /etc/ceph/osd/0-c905557d-3619-4d8b-9da5-cff66fa6340d.json
{
    "active": "ok",
    "block": {
        "path": "/dev/disk/by-partuuid/020eda5b-a959-4bfd-87e8-3dee5ad20564",
        "uuid": "020eda5b-a959-4bfd-87e8-3dee5ad20564"
    },
    "block_uuid": "020eda5b-a959-4bfd-87e8-3dee5ad20564",
    "bluefs": 1,
    "ceph_fsid": "e27fcfc8-4390-46e7-9f7e-4ada0c965785",
    "cluster_name": "ceph",
    "data": {
        "path": "/dev/sdb1",
        "uuid": "c905557d-3619-4d8b-9da5-cff66fa6340d"
    },
    "fsid": "c905557d-3619-4d8b-9da5-cff66fa6340d",
    "keyring": "AQCwo/BeVwY6FBAAuRBLU6UlMQoM3ztx6BVCRw==",
    "kv_backend": "rocksdb",
    "magic": "ceph osd volume v026",
    "mkfs_done": "yes",
    "ready": "ready",
    "require_osd_release": 12,
    "systemd": "",
    "type": "bluestore",
    "whoami": 0
}

If I try by hand:

Code: 
# ceph-volume simple scan /dev/sdb1

Running command: /sbin/cryptsetup status /dev/sdb1
-->  RuntimeError: --force was not used and OSD metadata file exists: /etc/ceph/osd/0-c905557d-3619-4d8b-9da5-cff66fa6340d.json


Code: 
# mount |grep osd
/
dev/sdc1 on /var/lib/ceph/osd/ceph-1 type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdb1 on /var/lib/ceph/osd/ceph-0 type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdf1 on /var/lib/ceph/osd/ceph-3 type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sde1 on /var/lib/ceph/osd/ceph-2 type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdg1 on /var/lib/ceph/osd/ceph-4 type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdh1 on /var/lib/ceph/osd/ceph-5 type xfs (rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
tmpfs on /var/lib/ceph/osd/ceph-36 type tmpfs (rw,relatime)
 
Last edited:
hi,

I tried it with one node and "ignored" the messages:

Code: 
# ceph-volume simple activate --all
--> activating OSD specified in /etc/ceph/osd/0-c905557d-3619-4d8b-9da5-cff66fa6340d.json
Running command: /bin/ln -snf /dev/sdb2 /var/lib/ceph/osd/ceph-0/block
Running command: /bin/chown -R ceph:ceph /dev/sdb2
Running command: /bin/systemctl enable ceph-volume@simple-0-c905557d-3619-4d8b-9da5-cff66fa6340d
 stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@simple-0-c905557d-3619-4d8b-9da5-cff66fa6340d.service → /lib/systemd/system/ceph-volume@.service.
Running command: /bin/ln -sf /dev/null /etc/systemd/system/ceph-disk@.service
--> All ceph-disk systemd units have been disabled to prevent OSDs getting triggered by UDEV events
Running command: /bin/systemctl enable --runtime ceph-osd@0
Running command: /bin/systemctl start ceph-osd@0
--> Successfully activated OSD 0 with FSID c905557d-3619-4d8b-9da5-cff66fa6340d
--> activating OSD specified in /etc/ceph/osd/2-f9a8609d-437b-4e11-93f7-6edc5e9fdbb4.json
Running command: /bin/ln -snf /dev/sde2 /var/lib/ceph/osd/ceph-2/block
Running command: /bin/chown -R ceph:ceph /dev/sde2
Running command: /bin/systemctl enable ceph-volume@simple-2-f9a8609d-437b-4e11-93f7-6edc5e9fdbb4
 stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@simple-2-f9a8609d-437b-4e11-93f7-6edc5e9fdbb4.service → /lib/systemd/system/ceph-volume@.service.
Running command: /bin/ln -sf /dev/null /etc/systemd/system/ceph-disk@.service
--> All ceph-disk systemd units have been disabled to prevent OSDs getting triggered by UDEV events
Running command: /bin/systemctl enable --runtime ceph-osd@2
Running command: /bin/systemctl start ceph-osd@2
--> Successfully activated OSD 2 with FSID f9a8609d-437b-4e11-93f7-6edc5e9fdbb4
--> activating OSD specified in /etc/ceph/osd/1-ada95d3b-2271-417f-8957-092dbad83b62.json
Running command: /bin/ln -snf /dev/sdc2 /var/lib/ceph/osd/ceph-1/block
Running command: /bin/chown -R ceph:ceph /dev/sdc2
Running command: /bin/systemctl enable ceph-volume@simple-1-ada95d3b-2271-417f-8957-092dbad83b62
 stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@simple-1-ada95d3b-2271-417f-8957-092dbad83b62.service → /lib/systemd/system/ceph-volume@.service.
Running command: /bin/ln -sf /dev/null /etc/systemd/system/ceph-disk@.service
--> All ceph-disk systemd units have been disabled to prevent OSDs getting triggered by UDEV events
Running command: /bin/systemctl enable --runtime ceph-osd@1
Running command: /bin/systemctl start ceph-osd@1
--> Successfully activated OSD 1 with FSID ada95d3b-2271-417f-8957-092dbad83b62
--> activating OSD specified in /etc/ceph/osd/5-acd2a716-4504-4163-b315-fee6b5a4ac42.json
Running command: /bin/ln -snf /dev/sdh2 /var/lib/ceph/osd/ceph-5/block
Running command: /bin/chown -R ceph:ceph /dev/sdh2
Running command: /bin/systemctl enable ceph-volume@simple-5-acd2a716-4504-4163-b315-fee6b5a4ac42
 stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@simple-5-acd2a716-4504-4163-b315-fee6b5a4ac42.service → /lib/systemd/system/ceph-volume@.service.
Running command: /bin/ln -sf /dev/null /etc/systemd/system/ceph-disk@.service
--> All ceph-disk systemd units have been disabled to prevent OSDs getting triggered by UDEV events
Running command: /bin/systemctl enable --runtime ceph-osd@5
Running command: /bin/systemctl start ceph-osd@5
--> Successfully activated OSD 5 with FSID acd2a716-4504-4163-b315-fee6b5a4ac42
--> activating OSD specified in /etc/ceph/osd/4-e5f9a7ec-0613-4a0d-b340-93219597cd46.json
Running command: /bin/ln -snf /dev/sdg2 /var/lib/ceph/osd/ceph-4/block
Running command: /bin/chown -R ceph:ceph /dev/sdg2
Running command: /bin/systemctl enable ceph-volume@simple-4-e5f9a7ec-0613-4a0d-b340-93219597cd46
 stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@simple-4-e5f9a7ec-0613-4a0d-b340-93219597cd46.service → /lib/systemd/system/ceph-volume@.service.
Running command: /bin/ln -sf /dev/null /etc/systemd/system/ceph-disk@.service
--> All ceph-disk systemd units have been disabled to prevent OSDs getting triggered by UDEV events
Running command: /bin/systemctl enable --runtime ceph-osd@4
Running command: /bin/systemctl start ceph-osd@4
--> Successfully activated OSD 4 with FSID e5f9a7ec-0613-4a0d-b340-93219597cd46
--> activating OSD specified in /etc/ceph/osd/3-0ed815ee-5cd5-4cfc-b2be-7e42b084ae6b.json
Running command: /bin/ln -snf /dev/sdf2 /var/lib/ceph/osd/ceph-3/block
Running command: /bin/chown -R ceph:ceph /dev/sdf2
Running command: /bin/systemctl enable ceph-volume@simple-3-0ed815ee-5cd5-4cfc-b2be-7e42b084ae6b
 stderr: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-volume@simple-3-0ed815ee-5cd5-4cfc-b2be-7e42b084ae6b.service → /lib/systemd/system/ceph-volume@.service.
Running command: /bin/ln -sf /dev/null /etc/systemd/system/ceph-disk@.service
--> All ceph-disk systemd units have been disabled to prevent OSDs getting triggered by UDEV events
Running command: /bin/systemctl enable --runtime ceph-osd@3
Running command: /bin/systemctl start ceph-osd@3
--> Successfully activated OSD 3 with FSID 0ed815ee-5cd5-4cfc-b2be-7e42b084ae6b

and rebooted the node ... it seems, it worked.
 
You must log in or register to reply here.
Top Bottom