[SOLVED] Ceph Luminous to Nautilus double check

[RobFantini]

so I've read the docs a few times and ran pve5to6 and as expected got this:

INFO: checking Ceph config..
WARN: No 'mon_host' entry found in ceph config.
  It's recommended to add mon_host with all monitor addresses (without ports) to the global section.                        
PASS: 'ms_bind_ipv6' not enabled
WARN: [global] config section contains 'keyring' option, which will prevent services from starting with Nautilus.
 Move 'keyring' option to [client] section instead.

our important vm's are running on ceph storage .

My questions:

Is it OK to make the two changes to ceph.conf while still running Luminous ? [ I am 99% sure that the changes can be made.]

Should I shut down all virtual machines which are using ceph before the upgrade?

 

[RobFantini]

well i completed the upgrade.

we left vm's running and edited the config file in advance.

 

[dmulk]

JOOC, how long did you leave it running with these changes before doing the upgrade? I'm working through some of the pre-flight checks and it could be a week or two before I can actually get to the upgrade. Wondering if I can set these and run for a while or if it's something you did node by node?

Thanks!
<D>

 

[RobFantini]

Hello. i see jocc = just out of curiosity..
anyway we ended up just following the instructions and all worked out.

also - while reading ceph release notes we saw there were health monitoring settings that can be turned on. that let to being exposed to a bad bug which caused all our vms to stop working. vm's had to be restored from backup , data restored from their recent backups [ imap, order entry etc ] it was a very bad Sunday.

my advise is to just go with the default settings and be very careful about any adjustments.

 

[RobFantini]

i see that i did not answer your question. AFAIR we just made the changes shortly before the upgrade.

 

[dmulk]

Thank you very much for the quick response, Rob!

Just to confirm, you didn't have a "mon =" line in the global config of your ceph.conf and you added something like this to it?:

mon_host = 10.10.10.132 10.10.10.133 10.10.10.134

It sounds like a very bad sunday indeed. I've been running CEPH for years. I'm now curious about what this health monitoring setting / bug was? Was it enabled "while someone was inside the ceph.conf adding the other info above? (prior to upgrading the cluster?) Curious how things fell over...


Thanks for the warning. I think I'll simple add only the lines we need to complete the upgrade. Everything else has been running quite smoothly so far.

<D>

 

[RobFantini]

Thank you very much for the quick response, Rob!

Just to confirm, you didn't have a "mon =" line in the global config of your ceph.conf and you added something like this to it?:

mon_host = 10.10.10.132 10.10.10.133 10.10.10.134

It sounds like a very bad sunday indeed. I've been running CEPH for years. I'm now curious about what this health monitoring setting / bug was? Was it enabled "while someone was inside the ceph.conf adding the other info above? (prior to upgrading the cluster?) Curious how things fell over...


Thanks for the warning. I think I'll simple add only the lines we need to complete the upgrade. Everything else has been running quite smoothly so far.

<D>

mon_host - yes we just have a line like that in the global section.

the bug: I'll dig up notes on that and send some links in a bit.

 

[RobFantini]

some info on the bug here https://bugzilla.proxmox.com/show_bug.cgi?id=2375 and check links i put at the bug.

 

[dmulk]

@RobFantini It's really wonderful that you actually took the time to come back here to update this thread. Too often people just abandon the thread and never update it (of course we've all probably done that!). Thank you for doing it in this case. It really helped. If you are ever in the San Diego area, PM me and I'll buy you a beer. Cheers!
Dan

 

[RobFantini]

you are welcome. i've received a lot of help on this forum, and as i work most weekends do help out while the devs take a hike. literally a lot of them love to hike.

anyway overall ceph is amazing.

 

[dmulk]

Agreed. I've been running it for about 4 years. It's pretty great.