Hi Guys,
So, I'm in a situation where I want to change the ceph public network,
I've read up on it quite a bit, and before "testing" it in production ;-) I built a virtual test environment,
which amazed me, it was actually quite easy and I want to fact check it here...
The virtual cluster consisted of:
3x pve 7.3 host, 2 networks on 2 nic's, ceph octopus, 3 OSD's per node.
ceph cluster network 10.10.10.1/24
ceph public network 10.0.0.1/24
monitors are on the 10.0.0.1/24 this way initally.
I made 1 VM after clustering and put it on RBD.
Within the VM I ran a while loop to see if disk IO was possible during the process, as I want to do it in a live environment.
within ssh session 1: while true; do echo $RANDOM >> test; sleep 2; done
within ssh session 2: tail -f test
During the whole process writing was possible, and the SSH session stayed alive as I migrated the VM between the nodes.
Steps I did are:
- edit the ceph public network to 10.10.10.1/24 in /etc/pve/ceph.conf, save the file;
- destroy / recreate all monitors one by one via the GUI, they will come up with a new ip address in the new network, check the ceph and VM health in between;
- destroy / recreate all managers one by one via the GUI, they will come up with a new ip address in the new network, check the ceph and VM health in between;
- migrate VM to other node;
- restart one node;
- move VM back;
- reboot other nodes;
- done.
I was so amazed, I also seperated the ceph public network again after merging it to see if I could, and it works perfectly.
In between I also checked the RBD watchers with rbd ls, find a VM disk, and check it with rbd status vm-100-disk-0 for example,
you will see after live migrating that the listener IP changes to the new ip address, too.
With netstat -tulpn | grep ceph you also see that the old ip's will be gone after rebooting the nodes.
Am I missing something, or is it really this easy? It looks like it is...
Kind regards,
David
So, I'm in a situation where I want to change the ceph public network,
I've read up on it quite a bit, and before "testing" it in production ;-) I built a virtual test environment,
which amazed me, it was actually quite easy and I want to fact check it here...
The virtual cluster consisted of:
3x pve 7.3 host, 2 networks on 2 nic's, ceph octopus, 3 OSD's per node.
ceph cluster network 10.10.10.1/24
ceph public network 10.0.0.1/24
monitors are on the 10.0.0.1/24 this way initally.
I made 1 VM after clustering and put it on RBD.
Within the VM I ran a while loop to see if disk IO was possible during the process, as I want to do it in a live environment.
within ssh session 1: while true; do echo $RANDOM >> test; sleep 2; done
within ssh session 2: tail -f test
During the whole process writing was possible, and the SSH session stayed alive as I migrated the VM between the nodes.
Steps I did are:
- edit the ceph public network to 10.10.10.1/24 in /etc/pve/ceph.conf, save the file;
- destroy / recreate all monitors one by one via the GUI, they will come up with a new ip address in the new network, check the ceph and VM health in between;
- destroy / recreate all managers one by one via the GUI, they will come up with a new ip address in the new network, check the ceph and VM health in between;
- migrate VM to other node;
- restart one node;
- move VM back;
- reboot other nodes;
- done.
I was so amazed, I also seperated the ceph public network again after merging it to see if I could, and it works perfectly.
In between I also checked the RBD watchers with rbd ls, find a VM disk, and check it with rbd status vm-100-disk-0 for example,
you will see after live migrating that the listener IP changes to the new ip address, too.
With netstat -tulpn | grep ceph you also see that the old ip's will be gone after rebooting the nodes.
Am I missing something, or is it really this easy? It looks like it is...
Kind regards,
David