ceph - added pool, can not move kvm disk to it

R

RobFantini

Famous Member
May 24, 2012
2,043
111
133
Boston,Mass
Hello

so I added some osd's knowing that page groups would have to be added.

ceph -s shows:
Code: 
# ceph -s
  cluster 63efaa45-7507-428f-9443-82a0a546b70d
  health HEALTH_WARN
  too many PGs per OSD (369 > max 300)
  pool ceph-kvm has many more objects per pg than average (too few pgs?)
  monmap e5: 5 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0,3=10.2.2.6:6789/0,4=10.2.2.65:6789/0}
  election epoch 94, quorum 0,1,2,3,4 3,1,0,4,2
  osdmap e413: 9 osds: 9 up, 9 in
  flags sortbitwise,require_jewel_osds
  pgmap v1579531: 1152 pgs, 4 pools, 446 GB data, 112 kobjects
  893 GB used, 3083 GB / 3977 GB avail
  1152 active+clean
  client io 32044 B/s wr, 0 op/s rd, 7 op/s wr

I tried to add page groups from pve gui by :
create new pool with these settings : size/min: 3/1 , pg_num 512

Then I figured to use move disks from the orig. pool to the new one.

the move did not work, the messages was:
Code: 
create full clone of drive scsi0 (ceph-kvm:vm-118-disk-1)
TASK ERROR: storage migration failed: rbd error: rbd: couldn't connect to the cluster!

Is that supposed to work?


PS: If there is a bug to report let me know, will do so.
 
and I tried to restore a backup to the new pool:
Code: 
restore vma archive: lzop -d -c /mnt/pve/bkup-longterm/dump/vzdump-qemu-101-2017_02_03-10_13_24.vma.lzo|vma extract -v -r /var/tmp/vzdumptmp8801.fifo - /var/tmp/vzdumptmp8801
CFG: size: 307 name: qemu-server.conf
DEV: dev_id=1 size: 4294967296 devname: drive-scsi0
CTIME: Fri Feb 3 10:13:25 2017
TASK ERROR: command 'lzop -d -c /mnt/pve/bkup-longterm/dump/vzdump-qemu-101-2017_02_03-10_13_24.vma.lzo|vma extract -v -r /var/tmp/vzdumptmp8801.fifo - /var/tmp/vzdumptmp8801' failed: rbd error: rbd: couldn't connect to the cluster!
 
Hi,
does it works if you disable the warning? But be aware - the warning make sense - but if you later remove an pool you will get less PGs per osd...

Put this in ceph.conf
Code: 
mon_pg_warn_max_per_osd = 0
and restarts the Mons (one by one). After that your cluster should be healthy.

Udo
 
1- I did the mon restart command can be done from one node. does the restart need to be done at each node? the disk move still does not work

2- I put this ' mon_pg_warn_max_per_osd = 0 ' in the [global] of ceph.conf
yet the warning is still there. note I do not mind the warning asit should be dealt with.

I mention the warning is still there as the restarts are not working or something else is up.
 
there are 5 mons. 4 restarted OK. really fast.
on one node there is an issue:
Code: 
sys6  ~ # service ceph-mon@sys6 restart  
Job for ceph-mon@sys6.service failed. See 'systemctl status ceph-mon@sys6.service' and 'journalctl -xn' for details.

Code: 
sys6  ~ # systemctl status ceph-mon@sys6.service
● ceph-mon@sys6.service - Ceph cluster monitor daemon
  Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled)
  Drop-In: /lib/systemd/system/ceph-mon@.service.d
  └─ceph-after-pve-cluster.conf
  Active: failed (Result: start-limit) since Fri 2017-02-03 10:50:05 EST; 24min ago
  Process: 23889 ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
 Main PID: 23889 (code=exited, status=1/FAILURE)

Feb 03 10:50:05 sys6 systemd[1]: Unit ceph-mon@sys6.service entered failed state.
Feb 03 10:50:05 sys6 systemd[1]: ceph-mon@sys6.service holdoff time over, scheduling restart.
Feb 03 10:50:05 sys6 systemd[1]: Stopping Ceph cluster monitor daemon...
Feb 03 10:50:05 sys6 systemd[1]: Starting Ceph cluster monitor daemon...
Feb 03 10:50:05 sys6 systemd[1]: ceph-mon@sys6.service start request repeated too quickly, refusing to start.
Feb 03 10:50:05 sys6 systemd[1]: Failed to start Ceph cluster monitor daemon.
Feb 03 10:50:05 sys6 systemd[1]: Unit ceph-mon@sys6.service entered failed state.
Feb 03 11:14:10 sys6 systemd[1]: Starting Ceph cluster monitor daemon...
Feb 03 11:14:10 sys6 systemd[1]: ceph-mon@sys6.service start request repeated too quickly, refusing to start.
Feb 03 11:14:10 sys6 systemd[1]: Failed to start Ceph cluster monitor daemon.
 
RobFantini said:
there are 5 mons. 4 restarted OK. really fast.
Click to expand...
Hi,
5 mons but 9 OSDs only?? BTW. 3 mons is enough for normal sized ceph-cluster (like 10 OSD-Nodes with > 100 OSDs).

on one node there is an issue:
Code: 
sys6  ~ # systemctl status ceph-mon@sys6.service
● ceph-mon@sys6.service - Ceph cluster monitor daemon
  Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled)
  Drop-In: /lib/systemd/system/ceph-mon@.service.d
  └─ceph-after-pve-cluster.conf
  Active: failed (Result: start-limit) since Fri 2017-02-03 10:50:05 EST; 24min ago
  Process: 23889 ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
Main PID: 23889 (code=exited, status=1/FAILURE)

Feb 03 10:50:05 sys6 systemd[1]: Unit ceph-mon@sys6.service entered failed state.
Feb 03 10:50:05 sys6 systemd[1]: ceph-mon@sys6.service holdoff time over, scheduling restart.
Feb 03 10:50:05 sys6 systemd[1]: Stopping Ceph cluster monitor daemon...
Feb 03 10:50:05 sys6 systemd[1]: Starting Ceph cluster monitor daemon...
Feb 03 10:50:05 sys6 systemd[1]: ceph-mon@sys6.service start request repeated too quickly, refusing to start.
Feb 03 10:50:05 sys6 systemd[1]: Failed to start Ceph cluster monitor daemon.
Feb 03 10:50:05 sys6 systemd[1]: Unit ceph-mon@sys6.service entered failed state.
Feb 03 11:14:10 sys6 systemd[1]: Starting Ceph cluster monitor daemon...
Feb 03 11:14:10 sys6 systemd[1]: ceph-mon@sys6.service start request repeated too quickly, refusing to start.
Feb 03 11:14:10 sys6 systemd[1]: Failed to start Ceph cluster monitor daemon.
Click to expand...
Perhaps stop the service instead restart and look if the process died.
If yes, try to start again.

Udo
 
1- just 3 mons now.

2- still can not move disk. same error msg as before.

3- I do not think the mon restarts are working. I tried 'sys5 ~ # service ceph-mon@sys5 stop' pve > ceph shows the mon still up.:
Code: 
sys5  ~ # service ceph-mon@sys5 stop
sys5  ~ # ceph -s
  cluster 63efaa45-7507-428f-9443-82a0a546b70d
  health HEALTH_WARN
  too many PGs per OSD (369 > max 300)
  pool ceph-kvm has many more objects per pg than average (too few pgs?)
  monmap e7: 3 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0}
  election epoch 102, quorum 0,1,2 1,0,2
  osdmap e414: 9 osds: 9 up, 9 in
  flags sortbitwise,require_jewel_osds
  pgmap v1584466: 1152 pgs, 4 pools, 258 GB data, 66522 objects
  517 GB used, 3459 GB / 3977 GB avail
  1152 active+clean
  client io 19775 B/s wr, 0 op/s rd, 1 op/s wr
sys5  ~ # ip a|grep 10.2.2
  inet 10.2.2.21/24 brd 10.2.2.255 scope global eth2
 
RobFantini said:
Udo -

Is it OK to add a ceph pool that has different Size/min and pg's then the existing pools?
Click to expand...
Hi,
yes - you can create pools (for less data) with 64pgs and pools with more pgs like 512pgs for more data.

Use the pgcalc tool on the ceph-website to calculate the right ammount of pgs. (you can change the PGs to an higher value but not to an smaller one).

About the mons - if you restart an mon, the process (ps aux | grep ceph-mon) has an new start-time.

Udo
 
yep the mon is not restarting:
Code: 
sys1  /etc/pve # ps aux | grep ceph-mon
ceph  3900  0.6  0.0 449804 65244 ?  Ssl  07:00  2:19 /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph

sys1  /etc/pve # service ceph-mon@sys1 restart

sys1  /etc/pve # ps aux | grep ceph-mon
ceph  3900  0.6  0.0 448696 65092 ?  Ssl  07:00  2:19 /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph

stop does same:
Code: 
# service ceph-mon@sys1  stop
sys1  /etc/pve # ps aux | grep ceph-mon
ceph  3900  0.6  0.0 448600 65136 ?  Ssl  07:00  2:20 /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
 
then start has issue after that stop:
Code: 
# service ceph-mon@sys1  start
Job for ceph-mon@sys1.service failed. See 'systemctl status ceph-mon@sys1.service' and 'journalctl -xn' for details.
sys1  /etc/pve # systemctl status ceph-mon@sys1.service
● ceph-mon@sys1.service - Ceph cluster monitor daemon
  Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled)
  Drop-In: /lib/systemd/system/ceph-mon@.service.d
  └─ceph-after-pve-cluster.conf
  Active: failed (Result: start-limit) since Fri 2017-02-03 13:08:58 EST; 2min 51s ago
  Process: 12087 ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
 Main PID: 12087 (code=exited, status=1/FAILURE)

Feb 03 13:08:58 sys1 systemd[1]: ceph-mon@sys1.service start request repeated too quickly, refusing to start.
Feb 03 13:08:58 sys1 systemd[1]: Failed to start Ceph cluster monitor daemon.
Feb 03 13:08:58 sys1 systemd[1]: Unit ceph-mon@sys1.service entered failed state.
Feb 03 13:10:45 sys1 systemd[1]: Stopped Ceph cluster monitor daemon.
Feb 03 13:11:30 sys1 systemd[1]: Starting Ceph cluster monitor daemon...
Feb 03 13:11:30 sys1 systemd[1]: ceph-mon@sys1.service start request repeated too quickly, refusing to start.
Feb 03 13:11:30 sys1 systemd[1]: Failed to start Ceph cluster monitor daemon.
Feb 03 13:11:40 sys1 systemd[1]: Starting Ceph cluster monitor daemon...
Feb 03 13:11:40 sys1 systemd[1]: ceph-mon@sys1.service start request repeated too quickly, refusing to start.
Feb 03 13:11:40 sys1 systemd[1]: Failed to start Ceph cluster monitor daemon.
 
RobFantini said:
yep the mon is not restarting:
Code: 
sys1  /etc/pve # ps aux | grep ceph-mon
ceph  3900  0.6  0.0 449804 65244 ?  Ssl  07:00  2:19 /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph

sys1  /etc/pve # service ceph-mon@sys1 restart

sys1  /etc/pve # ps aux | grep ceph-mon
ceph  3900  0.6  0.0 448696 65092 ?  Ssl  07:00  2:19 /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph

stop does same:
Code: 
# service ceph-mon@sys1  stop
sys1  /etc/pve # ps aux | grep ceph-mon
ceph  3900  0.6  0.0 448600 65136 ?  Ssl  07:00  2:20 /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
Click to expand...
Hi,
does this help? (on sys1)
Code: 
service ceph-mon@sys1 stop
kill 3900
sleep 20
service ceph-mon@sys1 start
ps aux | grep ceph-mon
Udo
 
Code: 
# service ceph-mon@sys1 start
Job for ceph-mon@sys1.service failed. See 'systemctl status ceph-mon@sys1.service' and 'journalctl -xn' for details.

 # service ceph-mon@sys1 start
Job for ceph-mon@sys1.service failed. See 'systemctl status ceph-mon@sys1.service' and 'journalctl -xn' for details.
sys1  /etc/pve # ps aux | grep ceph-mon
root  21891  0.0  0.0  12728  1920 pts/0  S+  13:26  0:00 grep --color=auto --binary-files=without-match --directories=skip ceph-mon
sys1  /etc/pve # ps aux | grep ceph-mon|grep -v grep
 
Does the mon start (as try in foreground) if you start manually:
Code: 
/usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
if yes, Control-C and start again without foreground:
Code: 
/usr/bin/ceph-mon --cluster ceph --id 1 --setuser ceph --setgroup ceph
Udo
 
before the test:
Code: 
  monmap e7: 3 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0}
  election epoch 156, quorum 1,2 0,2
I do not know if that means ok or not. i assume ok.

tests:
Code: 
# /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
starting mon.1 rank 0 at 10.2.2.10:6789/0 mon_data /var/lib/ceph/mon/ceph-1 fsid 63efaa45-7507-428f-9443-82a0a546b70d
^C2017-02-03 13:41:28.838601 7f3cb47c7700 -1 mon.1@0(leader) e7 *** Got Signal Interrupt ***
sys1  ~ # /usr/bin/ceph-mon --cluster ceph --id 1 --setuser ceph --setgroup ceph

ceph -s
Code: 
..
  monmap e7: 3 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0}
  election epoch 160, quorum 0,1,2 1,0,2
..
 
RobFantini said:
before the test:
Code: 
  monmap e7: 3 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0}
  election epoch 156, quorum 1,2 0,2
I do not know if that means ok or not. i assume ok.

tests:
Code: 
# /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
starting mon.1 rank 0 at 10.2.2.10:6789/0 mon_data /var/lib/ceph/mon/ceph-1 fsid 63efaa45-7507-428f-9443-82a0a546b70d
^C2017-02-03 13:41:28.838601 7f3cb47c7700 -1 mon.1@0(leader) e7 *** Got Signal Interrupt ***
sys1  ~ # /usr/bin/ceph-mon --cluster ceph --id 1 --setuser ceph --setgroup ceph

ceph -s
Code: 
..
  monmap e7: 3 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0}
  election epoch 160, quorum 0,1,2 1,0,2
..
Click to expand...
How looks ceph -s now? You should have "quorum 0,1,2".
Something with your start/stop script is wrong. Which ceph-version do you have? 10.2.3?

Udo
 
You must log in or register to reply here.
Top Bottom