proxmox 5 ceph osd create

[Kevin Myers]

HI there,

I had been running the beta 5 and the deb http://download.proxmox.com/debian/pve stretch pvetest repo, last night on this node I added some new OSD's however when using


pveceph createosd /dev/sdx -journal_dev /dev/sdxx

it appears that the osd that was created was bluestore type ( all other osd's in cluster are filestore), and the capacity of the ceph cluster did NOT increase with the osd being added.

Any clues as to why it defaulted to bluestore and and why the cluster capacity did not increase

 

[Kevin Myers]

pveceph createosd /dev/sdf -bluestore=0 -fstype=xfs -journal_dev /dev/sda4

does still appears to create a bluestore OSD !! , please advise.

 

[Kevin Myers]

pveceph createosd /dev/sdf -bluestore 0 -fstype xfs -journal_dev /dev/sda4
The ZFS modules are not loaded.
Try running '/sbin/modprobe zfs' as root to load them.
command '/sbin/zpool list -HPLv' failed: exit code 1

create OSD on /dev/sdf (xfs)
using device '/dev/sda4' for journal
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
The operation has completed successfully.
Setting name!
partNum is 0
REALLY setting name!
The operation has completed successfully.
Setting name!
partNum is 1
REALLY setting name!
The operation has completed successfully.
The operation has completed successfully.
meta-data=/dev/sdf1 isize=2048 agcount=4, agsize=6400 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0
data = bsize=4096 blocks=25600, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=864, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
root@storage3:/etc/apt# df -h
Filesystem Size Used Avail Use% Mounted on
udev 95G 0 95G 0% /dev
tmpfs 19G 42M 19G 1% /run
/dev/mapper/pve-root 20G 2.7G 16G 15% /
tmpfs 95G 45M 95G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 95G 0 95G 0% /sys/fs/cgroup
tmpfs 19G 0 19G 0% /run/user/0
/dev/fuse 30M 100K 30M 1% /etc/pve
/dev/sdf1 97M 5.5M 92M 6% /var/lib/ceph/tmp/mnt.kokOjC

 

[Dan Nicolae]

Did you know,

https://pve.proxmox.com/wiki/Ceph_Server#Ceph_on_Proxmox_VE_5.0

If you need to roll out now please install Proxmox VE 4.4 with Ceph Jewel and make a inplace upgrade to Proxmox VE 5.0 later, when Ceph Luminous is stable.

In Proxmox VE 5.0 the only available Ceph version is Luminous. So when you install the ceph packages use --version luminous instead of jewel.

 

[Kevin Myers]

im already running proxmox 5 on all nodes, and filestore with luminous on all these node. There appears to however now be a bug where no matter what I do with pveceph createosd , a bluestore type is created

 

[Kevin Myers]

I 'guess' its caused by this :

http://ceph.com/releases/v12-1-0-luminous-rc-released/

"
Notable Changes

• bluestore: ceph-disk: add –filestore argument, default to –bluestore (pr#15437, Loic Dachary, Sage Weil)"

so --filestore now needs to be passed else bluestore will be created

 

[Kevin Myers]

hard adding

--filestore to

API2/Ceph.pm

eg


my $cmd = ['ceph-disk', 'prepare', '--zap-disk', '--filestore' , '--fs-type', $fstype,

does create the the osd with the filestore type.

still have the other issue where new osd's added backfill, but the reported capacity the monitor see's does NOT increase with the osd's coming online..

 

[Kevin Myers]

answering my own question for any lost sole's that see the same problem, if you have upgraded from jewel to proxmox 5 stable (with ceph luminious) , then ensure you have a ceph mngr installed and working.

 

[arnaudd]

not working for me can you tell me more about your configuration or setup step ?