Ceph node crashes

[abarakat]

Hello, 4 node cluster running ceph 18.2.2. all 4 as monitors, all 4 setup for manager, all 4 have osd. Pool size 2 with min 2. After one of the nodes had 1 drive fail and 1 drive go offline it fully crashed. I am not using HA. Is it possible to move the VM's to other working nodes?

New to Ceph but my understanding is that it keeps copies on other node osd so theoratically I should be able to recover but how?

thanks

 

[aaron]

The VM configs are stored in the pmxcfs. If the disk images (and other resources) are available on the other nodes, you could manually move them to another node:

mv /etc/pve/nodes/{crashed node}/qemu-server/* /etc/pve/nodes/{target node}/qemu-server/

 

[abarakat]

The VM configs are stored in the pmxcfs. If the disk images (and other resources) are available on the other nodes, you could manually move them to another node:

mv /etc/pve/nodes/{crashed node}/qemu-server/* /etc/pve/nodes/{target node}/qemu-server/

Thank you this worked effortlessly

 

[LnxBil]

I am not using HA.

Why not? Would have solved your problem in the first place.

 

[abarakat]

Why not? Would have solved your problem in the first place.

 

[abarakat]

We have automation that configures network ports dynamically and is bound to speciific hosts.

 

[LnxBil]

You may want to look into open vSwitch, which could handle this.

 

[abarakat]

You may want to look into open vSwitch, which could handle this.

Yes on my list of things to try but it's an engineering dev/test envirenment and dedicated ports are connected to dedicated devices. they don't want anyting in the middle. Alas word on the net is that openvs is not so stable is it???

 

[LnxBil]

Alas word on the net is that openvs is not so stable is it???

I have no idea, I've never tried it myself, yet I heard good things about it and that it can be used to have defined ingress and egress routes over specific ports for the whole cluster.

 

[abarakat]

I have no idea, I've never tried it myself, yet I heard good things about it and that it can be used to have defined ingress and egress routes over specific ports for the whole cluster.

Defenitly something to play with when time permits. Thanks for your input.