Upgrading to 6.x

[Piotr K]

Hi.
I am currently upgrading my cluster to 6.x by the official guide. I am stuck in the ceph upgtade by this guide https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus at the topic of "Restart the OSD daemon on all nodes".

The restart goes fine, but the command "ceph-volume simple scan" gives

root@pve02:~# ceph-volume simple scan
 stderr: lsblk: /var/lib/ceph/osd/ceph-5: not a block device
 stderr: Bad argument "/var/lib/ceph/osd/ceph-5", expected an absolute path in /dev/ or /sys or a unit name: Invalid argument
Running command: /sbin/cryptsetup status /dev/sdb1
--> OSD 5 got scanned and metadata persisted to file: /etc/ceph/osd/5-e172bf8a-4840-4447-aa0a-396238d2ba51.json
--> To take over management of this scanned OSD, and disable ceph-disk and udev, run:
-->     ceph-volume simple activate 5 e172bf8a-4840-4447-aa0a-396238d2ba51
 stderr: lsblk: /var/lib/ceph/osd/ceph-6: not a block device
 stderr: Bad argument "/var/lib/ceph/osd/ceph-6", expected an absolute path in /dev/ or /sys or a unit name: Invalid argument
Running command: /sbin/cryptsetup status tmpfs
 stderr: blkid: error: tmpfs: No such file or directory
 stderr: lsblk: tmpfs: not a block device
--> Ignoring /var/lib/ceph/osd/ceph-6 because it's not a ceph-disk created osd.
root@pve02:~#

So it found the /dev/sdb1 which is a HDD, but it's missing the nvme drive at /dev/nvme0n1.

If i try to give the drive it failes too:

# ceph-volume simple scan /dev/nvme0n1
usage: ceph-volume simple scan [-h] [-f] [--stdout] [OSD_PATH]
ceph-volume simple scan: error: Argument is not a directory or device which is required to scan

Any idea how to proceed?

 

[Piotr K]

Anyone?

 

[dzajc]

I have same problem.

 

[Piotr K]

Mine is still unsolved too. Seems the server has no problem to find the OSD's at boot, but have not tried all.

 

[dzajc]

I thing this scan is for old "Luminous" disc type.

If you create new OSD with command "ceph-volume lvm create --data /dev/sdc" your disc is different.

Old have partition and new not.

Old:
df -h

Filesystem                   Size  Used     Avail  Use% Mounted on
/dev/sdb1                  97M   5.5M    92M   6%      /var/lib/ceph/osd/ceph-0

fdisk -l

Disk /dev/sdc: 1.1 TiB, 1200210141184 bytes, 2344160432 sectors
Disk model: LOGICAL VOLUME
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 8F95F1AE-5F39-45C1-AA48-60EE9C5F7781

Device      Start       End   Sectors   Size Type
/dev/sdc1    2048      206847      204800     100M Ceph OSD
/dev/sdc2    206848    2344160398    2343953551   1.1T unknown

New:
df -h

Filesystem                   Size  Used     Avail  Use% Mounted on
tmpfs                           16G   52K      16G   1%      /var/lib/ceph/osd/ceph-0

fdisk -l

Disk /dev/sdc: 1.1 TiB, 1200210141184 bytes, 2344160432 sectors
Disk model: LOGICAL VOLUME
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

 

[JohnJames]

Hi.
I am currently upgrading my cluster to 6.x by the official guide. I am stuck in the ceph upgtade by this guide https://pve.proxmox.com/wiki/Ceph_Luminous_to_Nautilus at the topic of "Restart the OSD daemon on all nodes".

The restart goes fine, but the command "ceph-volume simple scan" gives

root@pve02:~# ceph-volume simple scan
stderr: lsblk: /var/lib/ceph/osd/ceph-5: not a block device
stderr: Bad argument "/var/lib/ceph/osd/ceph-5", expected an absolute path in /dev/ or /sys or a unit name: Invalid argument
Running command: /sbin/cryptsetup status /dev/sdb1
--> OSD 5 got scanned and metadata persisted to file: /etc/ceph/osd/5-e172bf8a-4840-4447-aa0a-396238d2ba51.json
--> To take over management of this scanned OSD, and disable ceph-disk and udev, run:
-->     ceph-volume simple activate 5 e172bf8a-4840-4447-aa0a-396238d2ba51
stderr: lsblk: /var/lib/ceph/osd/ceph-6: not a block device
stderr: Bad argument "/var/lib/ceph/osd/ceph-6", expected an absolute path in /dev/ or /sys or a unit name: Invalid argument
Running command: /sbin/cryptsetup status tmpfs
stderr: blkid: error: tmpfs: No such file or directory
stderr: lsblk: tmpfs: not a block device
--> Ignoring /var/lib/ceph/osd/ceph-6 because it's not a ceph-disk created osd.
root@pve02:~#

So it found the /dev/sdb1 which is a HDD, but it's missing the nvme drive at /dev/nvme0n1.

If i try to give the drive it failes too:

# ceph-volume simple scan /dev/nvme0n1
usage: ceph-volume simple scan [-h] [-f] [--stdout] [OSD_PATH]
ceph-volume simple scan: error: Argument is not a directory or device which is required to scan

Any idea how to proceed?

Did you ever reach a solution to your issue? I'm having the same issue.

 

[Piotr K]

I think dzajc is right,
"
I thing this scan is for old "Luminous" disc type.

If you create new OSD with command "ceph-volume lvm create --data /dev/sdc" your disc is different.
"

 

[JohnJames]

I think dzajc is right,
"
I thing this scan is for old "Luminous" disc type.

If you create new OSD with command "ceph-volume lvm create --data /dev/sdc" your disc is different.
"

Got it. Correct, disks were setup using ceph-volume and not ceph-disk.