Okay, so it's working again.
I don't know what exactly did the trick and it might not be a solution to apply in similar situations, I'm afraid...
So what did I do?
Well, I have been waiting to switch from 10gbe ethernet to Infiniband for a while (I opened another thread in that respect). My issues was that I did not know how to replace the networking hardware in a running cluster one by one and I did not have enough free PCIe slots to put in the new Infiniband card next to the ethernet card. So I had waited.
But with the one node separated from the cluster anyway, I thought, I would use the opportunity and experiment with that one node. I removed another PCIe card that I could do without for a while and put in the Infiniband card. This triggered a renaming of all my 10gbe ports (which are used for Corosync and Ceph). So I had to update the networking (just copy the entries from the old port name entries over to the new port name entries). After restarting the networking, my cluster was whole again.
Come to think of it - maybe it was a networking issue after all. I had tried to ping all of the hosts from one another and this did work. And I had tried to ssh into the other nodes from each of the nodes and this did work. But what I had not tried, was to ping over the dedicated Corosync and Ceph networks, just over the normal admin network. So maybe the Corosync and/or Ceph networks were affected by some weird renaming problem caused by the PVE upgrade (maybe the one I described above and that I attributed to adding the Infiniband card)?
When I muster the courage to upgrade the next node, I will check whether the Corosync and Ceph networks are renamed and then I will update this thread.
