proxmox 5 ceph osd create

Kevin Myers

Member
Jul 14, 2017
20
0
6
46
HI there,

I had been running the beta 5 and the deb http://download.proxmox.com/debian/pve stretch pvetest repo, last night on this node I added some new OSD's however when using


pveceph createosd /dev/sdx -journal_dev /dev/sdxx

it appears that the osd that was created was bluestore type ( all other osd's in cluster are filestore), and the capacity of the ceph cluster did NOT increase with the osd being added.

Any clues as to why it defaulted to bluestore and and why the cluster capacity did not increase
 

Kevin Myers

Member
Jul 14, 2017
20
0
6
46
pveceph createosd /dev/sdf -bluestore=0 -fstype=xfs -journal_dev /dev/sda4

does still appears to create a bluestore OSD !! , please advise.
 

Kevin Myers

Member
Jul 14, 2017
20
0
6
46
pveceph createosd /dev/sdf -bluestore 0 -fstype xfs -journal_dev /dev/sda4
The ZFS modules are not loaded.
Try running '/sbin/modprobe zfs' as root to load them.
command '/sbin/zpool list -HPLv' failed: exit code 1

create OSD on /dev/sdf (xfs)
using device '/dev/sda4' for journal
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
The operation has completed successfully.
Setting name!
partNum is 0
REALLY setting name!
The operation has completed successfully.
Setting name!
partNum is 1
REALLY setting name!
The operation has completed successfully.
The operation has completed successfully.
meta-data=/dev/sdf1 isize=2048 agcount=4, agsize=6400 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0
data = bsize=4096 blocks=25600, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=864, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
root@storage3:/etc/apt# df -h
Filesystem Size Used Avail Use% Mounted on
udev 95G 0 95G 0% /dev
tmpfs 19G 42M 19G 1% /run
/dev/mapper/pve-root 20G 2.7G 16G 15% /
tmpfs 95G 45M 95G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 95G 0 95G 0% /sys/fs/cgroup
tmpfs 19G 0 19G 0% /run/user/0
/dev/fuse 30M 100K 30M 1% /etc/pve
/dev/sdf1 97M 5.5M 92M 6% /var/lib/ceph/tmp/mnt.kokOjC
 

Kevin Myers

Member
Jul 14, 2017
20
0
6
46
im already running proxmox 5 on all nodes, and filestore with luminous on all these node. There appears to however now be a bug where no matter what I do with pveceph createosd , a bluestore type is created
 

Kevin Myers

Member
Jul 14, 2017
20
0
6
46
hard adding

--filestore to

API2/Ceph.pm

eg


my $cmd = ['ceph-disk', 'prepare', '--zap-disk', '--filestore' , '--fs-type', $fstype,

does create the the osd with the filestore type.

still have the other issue where new osd's added backfill, but the reported capacity the monitor see's does NOT increase with the osd's coming online..
 

Kevin Myers

Member
Jul 14, 2017
20
0
6
46
answering my own question for any lost sole's that see the same problem, if you have upgraded from jewel to proxmox 5 stable (with ceph luminious) , then ensure you have a ceph mngr installed and working.
 

arnaudd

New Member
Aug 4, 2017
11
0
1
47
not working for me can you tell me more about your configuration or setup step ?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!