ceph - added pool, can not move kvm disk to it

R

RobFantini

Famous Member
May 24, 2012
2,023
107
133
Boston,Mass
Hello

so I added some osd's knowing that page groups would have to be added.

ceph -s shows:
Code: 
# ceph -s
  cluster 63efaa45-7507-428f-9443-82a0a546b70d
  health HEALTH_WARN
  too many PGs per OSD (369 > max 300)
  pool ceph-kvm has many more objects per pg than average (too few pgs?)
  monmap e5: 5 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0,3=10.2.2.6:6789/0,4=10.2.2.65:6789/0}
  election epoch 94, quorum 0,1,2,3,4 3,1,0,4,2
  osdmap e413: 9 osds: 9 up, 9 in
  flags sortbitwise,require_jewel_osds
  pgmap v1579531: 1152 pgs, 4 pools, 446 GB data, 112 kobjects
  893 GB used, 3083 GB / 3977 GB avail
  1152 active+clean
  client io 32044 B/s wr, 0 op/s rd, 7 op/s wr

I tried to add page groups from pve gui by :
create new pool with these settings : size/min: 3/1 , pg_num 512

Then I figured to use move disks from the orig. pool to the new one.

the move did not work, the messages was:
Code: 
create full clone of drive scsi0 (ceph-kvm:vm-118-disk-1)
TASK ERROR: storage migration failed: rbd error: rbd: couldn't connect to the cluster!

Is that supposed to work?


PS: If there is a bug to report let me know, will do so.
 
and I tried to restore a backup to the new pool:
Code: 
restore vma archive: lzop -d -c /mnt/pve/bkup-longterm/dump/vzdump-qemu-101-2017_02_03-10_13_24.vma.lzo|vma extract -v -r /var/tmp/vzdumptmp8801.fifo - /var/tmp/vzdumptmp8801
CFG: size: 307 name: qemu-server.conf
DEV: dev_id=1 size: 4294967296 devname: drive-scsi0
CTIME: Fri Feb 3 10:13:25 2017
TASK ERROR: command 'lzop -d -c /mnt/pve/bkup-longterm/dump/vzdump-qemu-101-2017_02_03-10_13_24.vma.lzo|vma extract -v -r /var/tmp/vzdumptmp8801.fifo - /var/tmp/vzdumptmp8801' failed: rbd error: rbd: couldn't connect to the cluster!
 
Hi,
does it works if you disable the warning? But be aware - the warning make sense - but if you later remove an pool you will get less PGs per osd...

Put this in ceph.conf
Code: 
mon_pg_warn_max_per_osd = 0
and restarts the Mons (one by one). After that your cluster should be healthy.

Udo
 
1- I did the mon restart command can be done from one node. does the restart need to be done at each node? the disk move still does not work

2- I put this ' mon_pg_warn_max_per_osd = 0 ' in the [global] of ceph.conf
yet the warning is still there. note I do not mind the warning asit should be dealt with.

I mention the warning is still there as the restarts are not working or something else is up.
 
there are 5 mons. 4 restarted OK. really fast.
on one node there is an issue:
Code: 
sys6  ~ # service ceph-mon@sys6 restart  
Job for ceph-mon@sys6.service failed. See 'systemctl status ceph-mon@sys6.service' and 'journalctl -xn' for details.

Code: 
sys6  ~ # systemctl status ceph-mon@sys6.service
● ceph-mon@sys6.service - Ceph cluster monitor daemon
  Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled)
  Drop-In: /lib/systemd/system/ceph-mon@.service.d
  └─ceph-after-pve-cluster.conf
  Active: failed (Result: start-limit) since Fri 2017-02-03 10:50:05 EST; 24min ago
  Process: 23889 ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
 Main PID: 23889 (code=exited, status=1/FAILURE)

Feb 03 10:50:05 sys6 systemd[1]: Unit ceph-mon@sys6.service entered failed state.
Feb 03 10:50:05 sys6 systemd[1]: ceph-mon@sys6.service holdoff time over, scheduling restart.
Feb 03 10:50:05 sys6 systemd[1]: Stopping Ceph cluster monitor daemon...
Feb 03 10:50:05 sys6 systemd[1]: Starting Ceph cluster monitor daemon...
Feb 03 10:50:05 sys6 systemd[1]: ceph-mon@sys6.service start request repeated too quickly, refusing to start.
Feb 03 10:50:05 sys6 systemd[1]: Failed to start Ceph cluster monitor daemon.
Feb 03 10:50:05 sys6 systemd[1]: Unit ceph-mon@sys6.service entered failed state.
Feb 03 11:14:10 sys6 systemd[1]: Starting Ceph cluster monitor daemon...
Feb 03 11:14:10 sys6 systemd[1]: ceph-mon@sys6.service start request repeated too quickly, refusing to start.
Feb 03 11:14:10 sys6 systemd[1]: Failed to start Ceph cluster monitor daemon.
 
RobFantini said:
there are 5 mons. 4 restarted OK. really fast.
Click to expand...
Hi,
5 mons but 9 OSDs only?? BTW. 3 mons is enough for normal sized ceph-cluster (like 10 OSD-Nodes with > 100 OSDs).

on one node there is an issue:
Code: 
sys6  ~ # systemctl status ceph-mon@sys6.service
● ceph-mon@sys6.service - Ceph cluster monitor daemon
  Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled)
  Drop-In: /lib/systemd/system/ceph-mon@.service.d
  └─ceph-after-pve-cluster.conf
  Active: failed (Result: start-limit) since Fri 2017-02-03 10:50:05 EST; 24min ago
  Process: 23889 ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
Main PID: 23889 (code=exited, status=1/FAILURE)

Feb 03 10:50:05 sys6 systemd[1]: Unit ceph-mon@sys6.service entered failed state.
Feb 03 10:50:05 sys6 systemd[1]: ceph-mon@sys6.service holdoff time over, scheduling restart.
Feb 03 10:50:05 sys6 systemd[1]: Stopping Ceph cluster monitor daemon...
Feb 03 10:50:05 sys6 systemd[1]: Starting Ceph cluster monitor daemon...
Feb 03 10:50:05 sys6 systemd[1]: ceph-mon@sys6.service start request repeated too quickly, refusing to start.
Feb 03 10:50:05 sys6 systemd[1]: Failed to start Ceph cluster monitor daemon.
Feb 03 10:50:05 sys6 systemd[1]: Unit ceph-mon@sys6.service entered failed state.
Feb 03 11:14:10 sys6 systemd[1]: Starting Ceph cluster monitor daemon...
Feb 03 11:14:10 sys6 systemd[1]: ceph-mon@sys6.service start request repeated too quickly, refusing to start.
Feb 03 11:14:10 sys6 systemd[1]: Failed to start Ceph cluster monitor daemon.
Click to expand...
Perhaps stop the service instead restart and look if the process died.
If yes, try to start again.

Udo
 
1- just 3 mons now.

2- still can not move disk. same error msg as before.

3- I do not think the mon restarts are working. I tried 'sys5 ~ # service ceph-mon@sys5 stop' pve > ceph shows the mon still up.:
Code: 
sys5  ~ # service ceph-mon@sys5 stop
sys5  ~ # ceph -s
  cluster 63efaa45-7507-428f-9443-82a0a546b70d
  health HEALTH_WARN
  too many PGs per OSD (369 > max 300)
  pool ceph-kvm has many more objects per pg than average (too few pgs?)
  monmap e7: 3 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0}
  election epoch 102, quorum 0,1,2 1,0,2
  osdmap e414: 9 osds: 9 up, 9 in
  flags sortbitwise,require_jewel_osds
  pgmap v1584466: 1152 pgs, 4 pools, 258 GB data, 66522 objects
  517 GB used, 3459 GB / 3977 GB avail
  1152 active+clean
  client io 19775 B/s wr, 0 op/s rd, 1 op/s wr
sys5  ~ # ip a|grep 10.2.2
  inet 10.2.2.21/24 brd 10.2.2.255 scope global eth2
 
RobFantini said:
Udo -

Is it OK to add a ceph pool that has different Size/min and pg's then the existing pools?
Click to expand...
Hi,
yes - you can create pools (for less data) with 64pgs and pools with more pgs like 512pgs for more data.

Use the pgcalc tool on the ceph-website to calculate the right ammount of pgs. (you can change the PGs to an higher value but not to an smaller one).

About the mons - if you restart an mon, the process (ps aux | grep ceph-mon) has an new start-time.

Udo
 
yep the mon is not restarting:
Code: 
sys1  /etc/pve # ps aux | grep ceph-mon
ceph  3900  0.6  0.0 449804 65244 ?  Ssl  07:00  2:19 /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph

sys1  /etc/pve # service ceph-mon@sys1 restart

sys1  /etc/pve # ps aux | grep ceph-mon
ceph  3900  0.6  0.0 448696 65092 ?  Ssl  07:00  2:19 /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph

stop does same:
Code: 
# service ceph-mon@sys1  stop
sys1  /etc/pve # ps aux | grep ceph-mon
ceph  3900  0.6  0.0 448600 65136 ?  Ssl  07:00  2:20 /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
 
then start has issue after that stop:
Code: 
# service ceph-mon@sys1  start
Job for ceph-mon@sys1.service failed. See 'systemctl status ceph-mon@sys1.service' and 'journalctl -xn' for details.
sys1  /etc/pve # systemctl status ceph-mon@sys1.service
● ceph-mon@sys1.service - Ceph cluster monitor daemon
  Loaded: loaded (/lib/systemd/system/ceph-mon@.service; enabled)
  Drop-In: /lib/systemd/system/ceph-mon@.service.d
  └─ceph-after-pve-cluster.conf
  Active: failed (Result: start-limit) since Fri 2017-02-03 13:08:58 EST; 2min 51s ago
  Process: 12087 ExecStart=/usr/bin/ceph-mon -f --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
 Main PID: 12087 (code=exited, status=1/FAILURE)

Feb 03 13:08:58 sys1 systemd[1]: ceph-mon@sys1.service start request repeated too quickly, refusing to start.
Feb 03 13:08:58 sys1 systemd[1]: Failed to start Ceph cluster monitor daemon.
Feb 03 13:08:58 sys1 systemd[1]: Unit ceph-mon@sys1.service entered failed state.
Feb 03 13:10:45 sys1 systemd[1]: Stopped Ceph cluster monitor daemon.
Feb 03 13:11:30 sys1 systemd[1]: Starting Ceph cluster monitor daemon...
Feb 03 13:11:30 sys1 systemd[1]: ceph-mon@sys1.service start request repeated too quickly, refusing to start.
Feb 03 13:11:30 sys1 systemd[1]: Failed to start Ceph cluster monitor daemon.
Feb 03 13:11:40 sys1 systemd[1]: Starting Ceph cluster monitor daemon...
Feb 03 13:11:40 sys1 systemd[1]: ceph-mon@sys1.service start request repeated too quickly, refusing to start.
Feb 03 13:11:40 sys1 systemd[1]: Failed to start Ceph cluster monitor daemon.
 
RobFantini said:
yep the mon is not restarting:
Code: 
sys1  /etc/pve # ps aux | grep ceph-mon
ceph  3900  0.6  0.0 449804 65244 ?  Ssl  07:00  2:19 /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph

sys1  /etc/pve # service ceph-mon@sys1 restart

sys1  /etc/pve # ps aux | grep ceph-mon
ceph  3900  0.6  0.0 448696 65092 ?  Ssl  07:00  2:19 /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph

stop does same:
Code: 
# service ceph-mon@sys1  stop
sys1  /etc/pve # ps aux | grep ceph-mon
ceph  3900  0.6  0.0 448600 65136 ?  Ssl  07:00  2:20 /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
Click to expand...
Hi,
does this help? (on sys1)
Code: 
service ceph-mon@sys1 stop
kill 3900
sleep 20
service ceph-mon@sys1 start
ps aux | grep ceph-mon
Udo
 
Code: 
# service ceph-mon@sys1 start
Job for ceph-mon@sys1.service failed. See 'systemctl status ceph-mon@sys1.service' and 'journalctl -xn' for details.

 # service ceph-mon@sys1 start
Job for ceph-mon@sys1.service failed. See 'systemctl status ceph-mon@sys1.service' and 'journalctl -xn' for details.
sys1  /etc/pve # ps aux | grep ceph-mon
root  21891  0.0  0.0  12728  1920 pts/0  S+  13:26  0:00 grep --color=auto --binary-files=without-match --directories=skip ceph-mon
sys1  /etc/pve # ps aux | grep ceph-mon|grep -v grep
 
Does the mon start (as try in foreground) if you start manually:
Code: 
/usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
if yes, Control-C and start again without foreground:
Code: 
/usr/bin/ceph-mon --cluster ceph --id 1 --setuser ceph --setgroup ceph
Udo
 
before the test:
Code: 
  monmap e7: 3 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0}
  election epoch 156, quorum 1,2 0,2
I do not know if that means ok or not. i assume ok.

tests:
Code: 
# /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
starting mon.1 rank 0 at 10.2.2.10:6789/0 mon_data /var/lib/ceph/mon/ceph-1 fsid 63efaa45-7507-428f-9443-82a0a546b70d
^C2017-02-03 13:41:28.838601 7f3cb47c7700 -1 mon.1@0(leader) e7 *** Got Signal Interrupt ***
sys1  ~ # /usr/bin/ceph-mon --cluster ceph --id 1 --setuser ceph --setgroup ceph

ceph -s
Code: 
..
  monmap e7: 3 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0}
  election epoch 160, quorum 0,1,2 1,0,2
..
 
RobFantini said:
before the test:
Code: 
  monmap e7: 3 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0}
  election epoch 156, quorum 1,2 0,2
I do not know if that means ok or not. i assume ok.

tests:
Code: 
# /usr/bin/ceph-mon -f --cluster ceph --id 1 --setuser ceph --setgroup ceph
starting mon.1 rank 0 at 10.2.2.10:6789/0 mon_data /var/lib/ceph/mon/ceph-1 fsid 63efaa45-7507-428f-9443-82a0a546b70d
^C2017-02-03 13:41:28.838601 7f3cb47c7700 -1 mon.1@0(leader) e7 *** Got Signal Interrupt ***
sys1  ~ # /usr/bin/ceph-mon --cluster ceph --id 1 --setuser ceph --setgroup ceph

ceph -s
Code: 
..
  monmap e7: 3 mons at {0=10.2.2.21:6789/0,1=10.2.2.10:6789/0,2=10.2.2.67:6789/0}
  election epoch 160, quorum 0,1,2 1,0,2
..
Click to expand...
How looks ceph -s now? You should have "quorum 0,1,2".
Something with your start/stop script is wrong. Which ceph-version do you have? 10.2.3?

Udo
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back