remount ceph disk on diff machine

[tad]

Hi,

I am wondering how I can remount a cephdisk if proxmox server is crashed and you want to startup the VM on a different proxmox host.

Reconstructing the VM is easy, but I cannot find how to reassign the disk.

Is there anyone who can help me with this?

 

[Q-wulf]

Ceph osd's you can take from one Cluster-member and stick em into another Cluster-member without doing anything.
That is because all those nodes on that ceph cluster run with the same cluster/keys.

Now AFAIK what you'd need to do to reconstruct a Ceph-Cluster from a Proxmox server that has crashed is to have backups of the following files:

• /etc/pve/ceph.conf <-- or replicate it based on your fsid
• /etc/pve/storage.cfg <-- or replicate it
• /etc/ceph/ceph.client.admin.keyring

you'd then recreate your ceph-cluster, plug the osd in and thats about it.


ps.: I have never done this, since all i ever do is restore a backup or plug the OSD into a working node of the same cluster to resolve such an issue.

Or consider the data not restore-worthy (no backup of node OS, no other cluster members)

 

[tad]

@Q-wulf, that is not really what I mean I think.

We have a solid ceph cluster (not managed by proxmox) of 22 nodes with 2 osd disks each. 145TB redundant storage.

My question is, what happens when a proxmox server crashed (totally, not recoverable)?

Normally with an iscsi target, you can remount iscsi and see all the disks/files (same for NFS). But you cannot browse ceph RBD as far as I know. And even if you can, how can I mount a existing (ceph) disk to a new VM and rebuild the proxmox server manually?

 

[Q-wulf]

oh i see.

well, if you keep a copy of the VMID.conf, you can just plug the appropriate ceph pool back into a new proxmox-node, then copy the vmID.conf over, then bring up the the VM.

you'd need to make sure to name the pool the same you have named it in the first proxmoxnode. (compare /etc/pve/storage.cfg)


you migth also wan't to read this:
http://docs.ceph.com/docs/hammer/man/8/rados/


ps.: i'd just keep a copy/backup of the working proxmox-node and/or have a working 3-node Proxmox HA-Cluster (so the necessary Configs are kept in sync) in case you loose a proxmox node. with 22 Ceph nodes, i doubt that would be too much of a burden , then you'd just follow the proxmox wiki, by wiping the broke node, installing a new node and then joining to the cluster. (then move the Vm back from the node where proxmox-HA has restarted it. )

 

[tad]

@Q-wulf, Thanks for your answer! If I understand you correctly, the proxmox server will re-map the rbd automatically when it has been disconnected?

We will setup a cluster But it is also good to understand how it works before we run into problems

 

[Q-wulf]

Afaik yes.

As with all my answers :
"no warranty" and test it extensively if you rely on it for production.

 

[tad]

@Q-wulf, offcourse, I'll test every advise thanks for your help!