Hi there,
running on proxmox 6 i have a problem adding another osd (bought a few new disks) to the system.
system has been installed on version 5.x and upgraded to latest.
adding a fresh disk (/dev/sdh with journal on /dev/sdb) to the system ends up with the following error:
after that, lsblk shows uplike this
which is interesting, because the old osd disks have been parted to a 100mb and another 1.1 tb instead of one big lvm PV (with vg+lv beneath it) and the journal on sdb was 1G (the new one is 112gb)
all actions where done via the web-gui, only action taken on cli was stop each osd, run "ceph-bluestore-tool repair" on the disks to clear the "legacy bluestore blah" warning after the upgrade to PVE 6
any suggestions what i did wrong/how to do it properly?
running on proxmox 6 i have a problem adding another osd (bought a few new disks) to the system.
system has been installed on version 5.x and upgraded to latest.
adding a fresh disk (/dev/sdh with journal on /dev/sdb) to the system ends up with the following error:
Code:
create OSD on /dev/sdh (bluestore)
creating block.db on '/dev/sdb'
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
Use of uninitialized value $part_or_lv in concatenation (.) or string at /usr/share/perl5/PVE/API2/Ceph/OSD.pm line 439.
using '' for block.db
wipe disk/partition: /dev/sdh
200+0 records in
200+0 records out
209715200 bytes (210 MB, 200 MiB) copied, 1.10591 s, 190 MB/s
--> RuntimeError: unable to use device
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 8320d1c2-aa50-48b6-99a4-f390f1df07c1
Running command: /sbin/vgcreate -s 1G --force --yes ceph-613a905c-cc64-41ce-bf0d-9d173fc3af8d /dev/sdh
stdout: Physical volume "/dev/sdh" successfully created.
stdout: Volume group "ceph-613a905c-cc64-41ce-bf0d-9d173fc3af8d" successfully created
Running command: /sbin/lvcreate --yes -l 100%FREE -n osd-block-8320d1c2-aa50-48b6-99a4-f390f1df07c1 ceph-613a905c-cc64-41ce-bf0d-9d173fc3af8d
stdout: Logical volume "osd-block-8320d1c2-aa50-48b6-99a4-f390f1df07c1" created.
--> blkid could not detect a PARTUUID for device:
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.16 --yes-i-really-mean-it
stderr: 2019-10-02 13:18:48.767 7fe437115700 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.bootstrap-osd.keyring: (2) No such file or directory
2019-10-02 13:18:48.767 7fe437115700 -1 AuthRegistry(0x7fe43007f818) no keyring found at /etc/pve/priv/ceph.client.bootstrap-osd.keyring, disabling cephx
stderr: purged osd.16
TASK ERROR: command 'ceph-volume lvm create --cluster-fsid a362929e-63ab-4361-97ca-f152656dcab1 --block.db '' --data /dev/sdh' failed: exit code 1after that, lsblk shows uplike this
Code:
root@lxc-prox1:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 278.5G 0 disk
├─sda1 8:1 0 1007K 0 part
├─sda2 8:2 0 512M 0 part
└─sda3 8:3 0 278G 0 part
├─pve-root 253:0 0 69.3G 0 lvm /
├─pve-swap 253:2 0 8G 0 lvm [SWAP]
├─pve-data_tmeta 253:3 0 1.9G 0 lvm
│ └─pve-data-tpool 253:5 0 181G 0 lvm
│ ├─pve-data 253:6 0 181G 0 lvm
│ └─pve-vm--100--disk--0 253:7 0 8G 0 lvm
└─pve-data_tdata 253:4 0 181G 0 lvm
└─pve-data-tpool 253:5 0 181G 0 lvm
├─pve-data 253:6 0 181G 0 lvm
└─pve-vm--100--disk--0 253:7 0 8G 0 lvm
sdb 8:16 0 745.2G 0 disk
├─sdb1 8:17 0 1G 0 part
├─sdb2 8:18 0 1G 0 part
└─sdb3 8:19 0 111.8G 0 part
sdc 8:32 0 745.2G 0 disk
├─sdc1 8:33 0 1G 0 part
└─sdc2 8:34 0 1G 0 part
sdd 8:48 0 1.1T 0 disk
├─sdd1 8:49 0 100M 0 part /var/lib/ceph/osd/ceph-0
└─sdd2 8:50 0 1.1T 0 part
sde 8:64 0 1.1T 0 disk
├─sde1 8:65 0 100M 0 part /var/lib/ceph/osd/ceph-1
└─sde2 8:66 0 1.1T 0 part
sdf 8:80 0 1.1T 0 disk
├─sdf1 8:81 0 100M 0 part /var/lib/ceph/osd/ceph-2
└─sdf2 8:82 0 1.1T 0 part
sdg 8:96 0 1.1T 0 disk
├─sdg1 8:97 0 100M 0 part /var/lib/ceph/osd/ceph-3
└─sdg2 8:98 0 1.1T 0 part
sdh 8:112 0 1.1T 0 disk
└─ceph--613a905c--cc64--41ce--bf0d--9d173fc3af8d-osd--block--8320d1c2--aa50--48b6--99a4--f390f1df07c1 253:1 0 1.1T 0 lvm
sdi 8:128 0 1.1T 0 disk
sdj 8:144 0 1.1T 0 disk
sdk 8:160 0 1.1T 0 disk
sr0 11:0 1 1024M 0 rom
rbd0 252:0 0 8G 0 disk
rbd1 252:16 0 20G 0 diskwhich is interesting, because the old osd disks have been parted to a 100mb and another 1.1 tb instead of one big lvm PV (with vg+lv beneath it) and the journal on sdb was 1G (the new one is 112gb)
all actions where done via the web-gui, only action taken on cli was stop each osd, run "ceph-bluestore-tool repair" on the disks to clear the "legacy bluestore blah" warning after the upgrade to PVE 6
any suggestions what i did wrong/how to do it properly?
