Has anyone run dual stack ceph? (IPv4 + IPv6)

[scyto]

I am sure this falls into the buck of 'because the docs say you can' when someone asks 'but why?'

Has anyone else been silly enough to try dual stack ceph?
Any horror stories, or all been fine?

This was really just a step on my way to doing ceph over IPv6 but i started on IPv4 and wanted to 'safely' migrate.

To be clear it is currently working and 100% green - though the UI got a bit wonky and there were a couple of systemctl restart cepg,target issued to get to this...

 

[gurubert]

Has anyone else been silly enough to try dual stack ceph?

Ceph is not able to run dual stack. You either have the OSD with IPv4 or IPv6 addresses registered in the cluster map.

 

[scyto]

Ceph is not able to run dual stack. You either have the OSD with IPv4 or IPv6 addresses registered in the cluster map.

https://docs.ceph.com/en/latest/rados/configuration/network-config-ref/#ipv4-ipv6-dual-stack-mode

yes a manager only binds to either ipv4 or ipv6 but apparently a fully working3 node cluster can have say 2 managers on ipv6 and 1 on ipv4 and replicate between all 3 just fine.

 

[gurubert]

https://docs.ceph.com/en/latest/rados/configuration/msgr2/#address-formats has a note at the end that dual stack is being worked on but still not supported.

 

[gurubert]

2 managers on ipv6 and 1 on ipv4 and replicate between all 3 just fine

How would that work?

 

[scyto]

How would that work?

Dunno, but it patently does, guess you didn't look at the image. I also verified with the native ceph tools, ss -l etc. Guess i just got lucky ;-) as I implied this was just a fun hill to go climb for no other reason than 'it was there'.

 

[scyto]

but still not supported.

nothing about my cluster is supported at this time as I am running a custom linux kernel... this is just a homelab