[SOLVED] Small issue with Proxmox/Ceph cluster after re-imaging failed node

[jslanier]

Hey everyone,
I recently upgraded the HBA in a Dell R720 with one that supports IT mode. This host was in a 3-node cluster with Ceph. When I booted after the HBA replacement, the rpool could not be imported, and I was taken to the initramfs prompt. Instead of just importing the rpool and then following this thread https://www.reddit.com/r/Proxmox/comments/fbdgos/cannot_import_rpool_automatically/ to just change the sleep values for ZFS_INITRD_PRE_MOUNTROOT_SLEEP and ZFS_INITRD_POST_MODPROBE_SLEEP in /etc/default/zfs, I decided to just reinstall Proxmox on the host. After reinstalling Proxmox on the host, I ran into the same issue at boot but was able to fix the issue by changing those sleep values. After configuring network settings and rejoining the cluster and following the instructions from this thread https://forum.proxmox.com/threads/reinstall-pve-node-with-ceph.59665/, I was able to get everything back up. Here is what I did:

1. Install Proxmox again
2. Recover networking settings (can be retrieved from another working node).
3. Make sure /etc/resolv.conf /etc/apt/sources /etc/hosts are all correct.
4. Join node to cluster (sometimes you will have to remove the failed node first. ie: pvecm rm nodename)
5. Install Ceph on the new node
6. Remove old failed node monitor.
- ceph mon rm nodename
- remove IP from mon_host in /etc/ceph/ceph.conf
7. Create monitor for new node - resource: https://docs.ceph.com/en/nautilus/rados/operations/add-or-rm-mons/
- follow steps for adding a monitor manually
- make sure that the /var/lib/ceph/mon/nodename directory is owned by the user ceph (because that is the user that the ceph monitor service runs as)
- monitor keyring can be retrieved from another working host at /etc/pve/priv/ceph.mon.keyring
- monitor map is retrieved by running: ceph mon getmap > monmap.file on a working host in the cluster.
- port number is not necessary on the last step: ceph-mon -i nodename --public-addr 172.16.1.xx
8. Run: ceph-volume lvm activate --all
9. In the Gui, go to the node->Ceph->OSD and click the OSD for the new node and then click the start button.
10. Add the IP back to mon_host in /etc/ceph/ceph.conf

Now that this has all been done, my Ceph cluster is healthy and everything looks okay except for one thing. When I go to my re-imaged host and click Ceph->OSD, sometimes no OSD info comes up at all. If I hit reload, I get a spinning wheel for a few seconds and sometimes the OSDs show up and other times it still displays nothing. I'll attach a screenshot of this particular screen from the GUI. I have a feeling that there is some configuration that is still not quite right, but I am not sure which config that could be. Any ideas?

Thanks,
Stan

 

[jslanier]

NVM, I rebooted the affected host, and when it came back up the VMs could not migrate back to this host because of an SSH key issue. I removed the offending keys from the other hosts and then migration was successful and my original issue is also gone.

 

[jslanier]

Still an issue. The mon service did not start successfully on the node. After manually stopping it and starting it again, the issue is back. The reason it did not start is because the service was disabled. I re-enabled it.

 

[jslanier]

Feb 15 13:53:02 prox1 ceph-mon[2441]: -24> 2021-02-15 13:53:02.529 7f01fa79a700 -1 received signal: Terminated from /sbin/init (PID: 1) UID: 0 Feb 15 13:53:02 prox1 ceph-mon[2441]: -23> 2021-02-15 13:53:02.529 7f01fa79a700 -1 mon.prox1@2(peon) e5 *** Got Signal Terminated *** Feb 15 13:53:02 prox1 ceph-mon[2441]: -1> 2021-02-15 13:53:02.765 7f01fc476400 -1 /build/ceph/ceph-14.2.16/src/common/TrackedOp.cc: In function 'OpTracker::~OpTracker()' thread 7f01fc476400 time 2021-02-15 13:53:02.765869 Feb 15 13:53:02 prox1 ceph-mon[2441]: /build/ceph/ceph-14.2.16/src/common/TrackedOp.cc: 163: FAILED ceph_assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty()) Feb 15 13:53:02 prox1 ceph-mon[2441]: ceph version 14.2.16 (5d5ae817209e503a412040d46b3374855b7efe04) nautilus (stable) Feb 15 13:53:02 prox1 ceph-mon[2441]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x152) [0x7f01fdfebcf2] Feb 15 13:53:02 prox1 ceph-mon[2441]: 2: (()+0x277eca) [0x7f01fdfebeca] Feb 15 13:53:02 prox1 ceph-mon[2441]: 3: (OpTracker::~OpTracker()+0x35) [0x7f01fe0c81a5] Feb 15 13:53:02 prox1 ceph-mon[2441]: 4: (Monitor::~Monitor()+0x2bb) [0x56011227581b] Feb 15 13:53:02 prox1 ceph-mon[2441]: 5: (Monitor::~Monitor()+0x9) [0x560112275d89] Feb 15 13:53:02 prox1 ceph-mon[2441]: 6: (main()+0x2875) [0x560112209325] Feb 15 13:53:02 prox1 ceph-mon[2441]: 7: (__libc_start_main()+0xeb) [0x7f01fc98c09b] Feb 15 13:53:02 prox1 ceph-mon[2441]: 8: (_start()+0x2a) [0x56011223858a] Feb 15 13:53:02 prox1 ceph-mon[2441]: 0> 2021-02-15 13:53:02.769 7f01fc476400 -1 *** Caught signal (Aborted) ** Feb 15 13:53:02 prox1 ceph-mon[2441]: in thread 7f01fc476400 thread_name:ceph-mon Feb 15 13:53:02 prox1 ceph-mon[2441]: ceph version 14.2.16 (5d5ae817209e503a412040d46b3374855b7efe04) nautilus (stable) Feb 15 13:53:02 prox1 ceph-mon[2441]: 1: (()+0x12730) [0x7f01fcebc730] Feb 15 13:53:02 prox1 ceph-mon[2441]: 2: (gsignal()+0x10b) [0x7f01fc99f7bb] Feb 15 13:53:02 prox1 ceph-mon[2441]: 3: (abort()+0x121) [0x7f01fc98a535] Feb 15 13:53:02 prox1 ceph-mon[2441]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a3) [0x7f01fdfebd43] Feb 15 13:53:02 prox1 ceph-mon[2441]: 5: (()+0x277eca) [0x7f01fdfebeca] Feb 15 13:53:02 prox1 ceph-mon[2441]: 6: (OpTracker::~OpTracker()+0x35) [0x7f01fe0c81a5] Feb 15 13:53:02 prox1 ceph-mon[2441]: 7: (Monitor::~Monitor()+0x2bb) [0x56011227581b] Feb 15 13:53:02 prox1 ceph-mon[2441]: 8: (Monitor::~Monitor()+0x9) [0x560112275d89] Feb 15 13:53:02 prox1 ceph-mon[2441]: 9: (main()+0x2875) [0x560112209325] Feb 15 13:53:02 prox1 ceph-mon[2441]: 10: (__libc_start_main()+0xeb) [0x7f01fc98c09b] Feb 15 13:53:02 prox1 ceph-mon[2441]: 11: (_start()+0x2a) [0x56011223858a] Feb 15 13:53:02 prox1 ceph-mon[2441]: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this. Feb 15 13:53:02 prox1 systemd[1]: ceph-mon@prox1.service: Main process exited, code=killed, status=6/ABRT Feb 15 13:53:02 prox1 systemd[1]: ceph-mon@prox1.service: Failed with result 'signal'.

 

[Alwin Antreich]

4. Join node to cluster (sometimes you will have to remove the failed node first. ie: pvecm rm nodename)

You always have to remove the failed node first. And that could be where your other problems with that node come from.

 

[jslanier]

You always have to remove the failed node first. And that could be where your other problems with that node come from.

And I did. I got an error when I tried joining the cluster because I had not removed the old node. So I removed the old node and added this one.

 

[Alwin Antreich]

Are you using the same PVE versions on the cluster? And maybe you will need to update the certificates on the failed node. But that is regardless of the ceph error.

 

[jslanier]

Are you using the same PVE versions on the cluster? And maybe you will need to update the certificates on the failed node. But that is regardless of the ceph error.

Definitely running the same version on all nodes. I just checked the certs and the look fine.

 

[Alwin Antreich]

If the PVE state is fine, then it may be more on the Ceph side. Do you see anything suspicious to why the MON failed, in the logs?

 

[jslanier]

If the PVE state is fine, then it may be more on the Ceph side. Do you see anything suspicious to why the MON failed, in the logs?

You know what is weird? My little issue went away after rebooting host2. Since it has come back up, the little quirky issue is gone.