osd issue after adding node to ceph

RobFantini · Jun 3, 2014

symmcom said:
Were you in #/etc/pve on Proxmox node while running the CLI commands? If not then thats why it cannot find the ceph.conf file.

ok from /etc/pve on both nodes

Code:

ceph6-ib  /etc/pve # ceph-deploy disk list ceph6-ib
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.3): /usr/bin/ceph-deploy disk list ceph6-ib
[ceph6-ib][DEBUG ] connected to host: ceph6-ib
[ceph6-ib][DEBUG ] detect platform information from remote host
[ceph6-ib][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: debian 7.5 wheezy
[ceph_deploy.osd][DEBUG ] Listing disks on ceph6-ib...
[ceph6-ib][INFO  ] Running command: ceph-disk list
[ceph6-ib][WARNIN] Traceback (most recent call last):
[ceph6-ib][WARNIN]   File "/usr/sbin/ceph-disk", line 2579, in <module>
[ceph6-ib][WARNIN]     main()
[ceph6-ib][WARNIN]   File "/usr/sbin/ceph-disk", line 2557, in main
[ceph6-ib][WARNIN]     args.func(args)
[ceph6-ib][WARNIN]   File "/usr/sbin/ceph-disk", line 2197, in main_list
[ceph6-ib][WARNIN]     tpath = mount(dev=dev, fstype=fs_type, options='')
[ceph6-ib][WARNIN]   File "/usr/sbin/ceph-disk", line 812, in mount
[ceph6-ib][WARNIN]     path,
[ceph6-ib][WARNIN]   File "/usr/sbin/ceph-disk", line 303, in command_check_call
[ceph6-ib][WARNIN]     return subprocess.check_call(arguments)
[ceph6-ib][WARNIN]   File "/usr/lib/python2.7/subprocess.py", line 506, in check_call
[ceph6-ib][WARNIN]     retcode = call(*popenargs, **kwargs)
[ceph6-ib][WARNIN]   File "/usr/lib/python2.7/subprocess.py", line 493, in call
[ceph6-ib][WARNIN]     return Popen(*popenargs, **kwargs).wait()
[ceph6-ib][WARNIN]   File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
[ceph6-ib][WARNIN]     errread, errwrite)
[ceph6-ib][WARNIN]   File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
[ceph6-ib][WARNIN]     raise child_exception
[ceph6-ib][WARNIN] TypeError: execv() arg 2 must contain only strings
[ceph6-ib][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: ceph-disk list

Code:

ceph4-ib  /etc/pve # ceph-deploy disk list ceph4-ib
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.3): /usr/bin/ceph-deploy disk list ceph4-ib
[ceph4-ib][DEBUG ] connected to host: ceph4-ib
[ceph4-ib][DEBUG ] detect platform information from remote host
[ceph4-ib][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: debian 7.5 wheezy
[ceph_deploy.osd][DEBUG ] Listing disks on ceph4-ib...
[ceph4-ib][INFO  ] Running command: ceph-disk list
[ceph4-ib][DEBUG ] /dev/sda :
[ceph4-ib][DEBUG ]  /dev/sda1 other, primary
[ceph4-ib][DEBUG ]  /dev/sda2 other, ext3, mounted on /boot
[ceph4-ib][DEBUG ]  /dev/sda3 other, LVM2_member
[ceph4-ib][DEBUG ] /dev/sdb :
[ceph4-ib][DEBUG ]  /dev/sdb1 ceph data, active, cluster ceph, osd.0, journal /dev/sdd1
[ceph4-ib][DEBUG ] /dev/sdc :
[ceph4-ib][DEBUG ]  /dev/sdc1 ceph data, active, cluster ceph, osd.1, journal /dev/sdd2
[ceph4-ib][DEBUG ] /dev/sdd :
[ceph4-ib][DEBUG ]  /dev/sdd1 ceph journal, for /dev/sdb1
[ceph4-ib][DEBUG ]  /dev/sdd2 ceph journal, for /dev/sdc1
[ceph4-ib][DEBUG ] /dev/sde :
[ceph4-ib][DEBUG ]  /dev/sde1 ceph data, active, cluster ceph, osd.2, journal /dev/sdg1
[ceph4-ib][DEBUG ] /dev/sdf :
[ceph4-ib][DEBUG ]  /dev/sdf1 ceph data, active, cluster ceph, osd.3, journal /dev/sdg2
[ceph4-ib][DEBUG ] /dev/sdg :
[ceph4-ib][DEBUG ]  /dev/sdg1 ceph journal, for /dev/sde1
[ceph4-ib][DEBUG ]  /dev/sdg2 ceph journal, for /dev/sdf1
[ceph4-ib][DEBUG ] /dev/sr0 other, unknown

RobFantini · Jun 3, 2014

also:

Code:

ceph6-ib  /etc/pve # parted -l|grep Disk
Disk /dev/sda: 100GB
Disk /dev/sdb: 250GB
Disk /dev/sdc: 250GB
Disk /dev/sdd: 1000GB
Disk /dev/sde: 1000GB
Disk /dev/sdf: 1000GB
Disk /dev/mapper/pve-data: 49.6GB
Disk /dev/mapper/pve-swap: 12.5GB
Disk /dev/mapper/pve-root: 25.0GB
Disk /dev/sdh: 1000GB
Disk /dev/sdg: 1000GB
Disk /dev/sdj: 1000GB
Disk /dev/sdi: 1000GB
Disk /dev/sdk: 1000GB

wahmed · Jun 3, 2014

Ok. Run the disk list from ceph4 to ceph6.
#ceph4-ib /etc/pve# ceph-deploy disk list ceph6-ib

also run from #ceph -v on both nodes

RobFantini · Jun 3, 2014

symmcom said:
Ok. Run the disk list from ceph4 to ceph6.
#ceph4-ib /etc/pve# ceph-deploy disk list ceph6-ib

Code:

ceph4-ib  /etc/pve # ceph-deploy disk list ceph6-ib
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.3): /usr/bin/ceph-deploy disk list ceph6-ib
[ceph6-ib][DEBUG ] connected to host: ceph6-ib
[ceph6-ib][DEBUG ] detect platform information from remote host
[ceph6-ib][DEBUG ] detect machine type
[ceph_deploy.osd][INFO  ] Distro info: debian 7.5 wheezy
[ceph_deploy.osd][DEBUG ] Listing disks on ceph6-ib...
[ceph6-ib][INFO  ] Running command: ceph-disk list
[ceph6-ib][WARNIN] Traceback (most recent call last):
[ceph6-ib][WARNIN]   File "/usr/sbin/ceph-disk", line 2579, in <module>
[ceph6-ib][WARNIN]     main()
[ceph6-ib][WARNIN]   File "/usr/sbin/ceph-disk", line 2557, in main
[ceph6-ib][WARNIN]     args.func(args)
[ceph6-ib][WARNIN]   File "/usr/sbin/ceph-disk", line 2197, in main_list
[ceph6-ib][WARNIN]     tpath = mount(dev=dev, fstype=fs_type, options='')
[ceph6-ib][WARNIN]   File "/usr/sbin/ceph-disk", line 812, in mount
[ceph6-ib][WARNIN]     path,
[ceph6-ib][WARNIN]   File "/usr/sbin/ceph-disk", line 303, in command_check_call
[ceph6-ib][WARNIN]     return subprocess.check_call(arguments)
[ceph6-ib][WARNIN]   File "/usr/lib/python2.7/subprocess.py", line 506, in check_call
[ceph6-ib][WARNIN]     retcode = call(*popenargs, **kwargs)
[ceph6-ib][WARNIN]   File "/usr/lib/python2.7/subprocess.py", line 493, in call
[ceph6-ib][WARNIN]     return Popen(*popenargs, **kwargs).wait()
[ceph6-ib][WARNIN]   File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
[ceph6-ib][WARNIN]     errread, errwrite)
[ceph6-ib][WARNIN]   File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
[ceph6-ib][WARNIN]     raise child_exception
[ceph6-ib][WARNIN] TypeError: execv() arg 2 must contain only strings
[ceph6-ib][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: ceph-disk list

>also run from #ceph -v on both nodes

Code:

ceph4-ib  /etc/pve # ceph -v
ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)

ceph6-ib  /etc/pve # ceph -v
ceph version 0.80.1 (a38fe1169b6d2ac98b427334c12d7cf81f809b74)

wahmed · Jun 3, 2014

This is an odd issue indeed. Lets reinstall Ceph on the ceph6-ib node.
First we have to remove Ceph monitor from ceph6-ib:
ceph6-ib /etc/pve# pveceph destroymon <mon_id>

Then remove ceph itself:
ceph6-ib /etc/pve# pveceph purge

Reinstall Ceph
ceph6-ib # pveceph install -version firefly

Now try again to create OSD.

Hopefully this will work this time.

RobFantini · Jun 3, 2014

symmcom said:
This is an odd issue indeed. Lets reinstall Ceph on the ceph6-ib node.
First we have to remove Ceph monitor from ceph6-ib:
ceph6-ib /etc/pve# pveceph destroymon <mon_id>

Then remove ceph itself:
ceph6-ib /etc/pve# pveceph purge

Reinstall Ceph
ceph6-ib # pveceph install -version firefly

Now try again to create OSD.

Hopefully this will work this time.

note `pveceph purge` not working:

Code:

ceph6-ib  /etc/pve # pveceph destroymon  3
=== mon.3 ===
Stopping Ceph mon.3 on ceph6-ib...done

ceph6-ib  /etc/pve # pveceph purge
detected running ceph services- unable to purge data

ceph6-ib  /etc/pve # /etc/init.d/ceph stop

ceph6-ib  /etc/pve # pveceph purge
detected running ceph services- unable to purge data

note there are no ceph processes running per

Code:

ceph6-ib  /etc/pve # ps -ef|grep ceph |  grep -v ceph

wahmed · Jun 3, 2014

Reboot the node then try to purge again

Sent from my SGH-I747M using Tapatalk

RobFantini · Jun 3, 2014

still no luck:

Code:

ceph6-ib  ~ #  pveceph purge
detected running ceph services- unable to purge data
ceph6-ib  ~ # w
 16:31:16 up 2 min,  1 user,  load average: 0.01, 0.02, 0.00

wahmed · Jun 3, 2014

Lets see if we can reinstall on top of it. From ceph6-ib run: #pveceph install -version firefly

RobFantini · Jun 3, 2014

symmcom said:
Lets see if we can reinstall on top of it. From ceph6-ib run: #pveceph install -version firefly

Code:

ceph6-ib  ~ # pveceph install -version firefly
download and import ceph reqpository keys
update available package list
Reading package lists...
Building dependency tree...
Reading state information...
gdisk is already the newest version.
ceph is already the newest version.
ceph-common is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

wahmed · Jun 4, 2014

Only thing i can suggest now is to detach the node ceph6 from proxmox cluster and do a clean reinstall again from the beginning.

RobFantini · Jun 5, 2014

symmcom said:
Only thing i can suggest now is to detach the node ceph6 from proxmox cluster and do a clean reinstall again from the beginning.

symmcom: I had already started over last week... I think the issue is that
1- i did not delete the mon before removing the node
2- the i/p address for the delete node is in a config file or data base
3- when the node was recreated , i used the same i/p ....

here is some more info.. and i think it'd be good to solve this... i'm not in a horry this is a test 4-th node...

A: on the ceph cluster I restored a backup this morning. note pve/ceph thinks the 4-th mon stil exists:

Code:

map 'drive-virtio0' to 'rbd:rbd/vm-7100-disk-2:mon_host=10.11.12.41 10.11.12.42 10.11.12.43 10.11.12.44:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/ceph-storage.keyring' (write zeros = 0)
new volume ID is 'ceph-storage:vm-7100-disk-3'
map 'drive-virtio1' to 'rbd:rbd/vm-7100-disk-3:mon_host=10.11.12.41 10.11.12.42 10.11.12.43 10.11.12.44:id=admin:auth_supported=cephx:keyring=/etc/pve/priv/ceph/ceph-storage.keyring' (write zeros = 0)
progress 1% (read 848297984 bytes, duration 65 sec)

here is a screenshot of the pve ceph mon page showing only 3 nodes:

So if i need to reinstall a new IP address will be used, along with a new hostname.

However this would be good to solve.... i think it can e reeproduced by

1- add a node to ceph along with mon for that node

2- delete the node without removing the mon 1-st

3- add a node using the IP address from step 1.

udo · Jun 5, 2014

RobFantini said:
note there are no ceph processes running per

Code:

ceph6-ib /etc/pve # ps -ef|grep ceph | grep -v ceph

Hi,
but with this command you don't see any ceph-command (-v is without ceph)

Udo

RobFantini · Jun 5, 2014

udo said:
Hi,
but with this command you don't see any ceph-command (-v is without ceph)

Udo

DUH my bad... i meant to type ' grep -v grep '

actually i use a script:

Code:

 # cat /fbc/bin/psgrep
ps -ef|grep  $1    |grep -v grep

so
psgrep ceph showed no ceph process at the time.

RobFantini · Jun 8, 2014

Re: osd issue after adding node to ceph ** Start over **

Hello
I just did a fresh install to the old ceph6-ib . it's new hostname = ceph1-ib

It now has a new hostname and ip address.

So at ceph1-ib after install:

1- update debs
2- set up /etc/hosts for dns to other nodes. set up Infiniband network.
3- join cluster
4- join ceph
5- add mon
6- zap partitions on a couple of disks that had partitions
All OK until:

6- add osd .

the osd's do not show up in pve gui.

Notes: I'll post some debug oiutput and config files soon.

RobFantini · Jun 8, 2014

Re: osd issue after adding node to ceph ** Start over **

at storage.confif the IP of new node does not show up:

Code:

# cat /etc/pve/storage.cfg
dir: local
        path /var/lib/vz
        content iso,vztmpl,rootdir
        maxfiles 0


rbd: ceph-storage
        monhost 10.11.12.41 10.11.12.42 10.11.12.43
        pool rbd
        content images
        username admin


nfs: nfs-pve
        path /mnt/pve/nfs-pve
        server 10.1.10.241
        export /pve-nfs-shared
        options vers=3
        content iso,vztmpl,backup
        maxfiles 2

and tree is:

Code:

# ceph osd tree
# id    weight  type name       up/down reweight
-1      20.02   root default
-2      7.28            host ceph4-ib
0       1.82                    osd.0   up      1
1       1.82                    osd.1   up      1
2       1.82                    osd.2   up      1
3       1.82                    osd.3   up      1
-3      7.28            host ceph3-ib
4       1.82                    osd.4   up      1
5       1.82                    osd.5   up      1
6       1.82                    osd.6   up      1
7       1.82                    osd.7   up      1
-4      5.46            host ceph2-ib
8       1.82                    osd.8   up      1
9       1.82                    osd.9   up      1
10      1.82                    osd.10  up      1

RobFantini · Jun 8, 2014

Re: osd issue after adding node to ceph ** Start over **

the above shows the IP of new node 10.11.12.46 is a mon .

and pve log from https:

Code:

[COLOR=#000000][FONT=tahoma]ceph-mon: set fsid to 4267b4fe-78bb-4670-86e5-60807f39e6c1[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]ceph-mon: created monfs at /var/lib/ceph/mon/ceph-3 for mon.3[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]=== mon.3 === [/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Starting Ceph mon.3 on ceph1-ib...[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Starting ceph-create-keys on ceph1-ib...[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]TASK OK
[/FONT][/COLOR]

RobFantini · Jun 8, 2014

Re: osd issue after adding node to ceph ** Start over **

osd pic:

and log:

Code:

[COLOR=#000000][FONT=tahoma]create OSD on /dev/sdd (xfs)[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]using device '/dev/sdb' for journal[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Caution: invalid backup GPT header, but valid main header; regenerating[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]backup header from main header.[/FONT][/COLOR]

[COLOR=#000000][FONT=tahoma]****************************************************************************[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]verification and recovery are STRONGLY recommended.[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]****************************************************************************[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]GPT data structures destroyed! You may now partition the disk using fdisk or[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]other utilities.[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]The operation has completed successfully.[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]WARNING:ceph-disk:OSD will not be hot-swappable if journal is not the same device as the osd data[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Information: Moved requested sector from 34 to 2048 in[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]order to align on 2048-sector boundaries.[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]The operation has completed successfully.[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]Information: Moved requested sector from 34 to 2048 in[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]order to align on 2048-sector boundaries.[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]The operation has completed successfully.[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]meta-data=/dev/sdd1              isize=2048   agcount=4, agsize=61034431 blks[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]         =                       sectsz=512   attr=2, projid32bit=0[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]data     =                       bsize=4096   blocks=244137723, imaxpct=25[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]         =                       sunit=0      swidth=0 blks[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]naming   =version 2              bsize=4096   ascii-ci=0[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]log      =internal log           bsize=4096   blocks=119207, version=2[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]         =                       sectsz=512   sunit=0 blks, lazy-count=1[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]realtime =none                   extsz=4096   blocks=0, rtextents=0[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]The operation has completed successfully.[/FONT][/COLOR]
[COLOR=#000000][FONT=tahoma]TASK OK

[/FONT][/COLOR]

RobFantini · Jun 15, 2014

Re: osd issue after adding node to ceph ** Start over **

the issue is caused by using disks that used to be part of raid 10 on a 3ware controller card.

I'll post solution to wiki. It is same thing that I had to do in order to use same for zfs.

in addition to zapping the disk i use this script:

Code:

DISK=$1

if [ "$1" = "" ]; then
 echo "Need to supply a  dev name like  sdg . exiting"
 exit 1
fi


echo " make sure this is the correct disk "
echo $DISK

echo " you will end up with NO partition table when this procedes .  example:

Disk /dev/$1  doesn't contain a valid partition table

Press enter to procede , or ctl-c to exit "

read x

dd if=/dev/zero of=/dev/$DISK bs=512 count=50000

mir · Jun 15, 2014

Re: osd issue after adding node to ceph ** Start over **

To remove partition table and boot sector the following should be sufficient:
dd if=/dev/zero of=/dev/$DISK bs=1024 count=1

osd issue after adding node to ceph

Famous Member

Famous Member

Famous Member

Famous Member

Famous Member

Famous Member

Famous Member

Famous Member

Famous Member

Famous Member

Famous Member

Famous Member

Distinguished Member

Famous Member

Famous Member

Famous Member

Famous Member

Famous Member

Famous Member

Famous Member

We value your privacy