Problem with RBD on Ceph IPv6 external cluster

[blaz]

Hi all,

So I have two external Ceph clusters - one is using IPv4 and the other is using IPv6. When I'm using IPv4 cluster, the dashborad shows status and content of RBD and I have no problem creating new images for VMs. But when I try to configure storage information about IPv6 cluster, I get Not Available tag on usage. For both clusters I make the same procedure for assigning keyrings and configuration settings for Proxmox.

Ceph clusters have installed Mimic version and the Proxmox cluster has version Luminous. min_compat_client is set to Luminous version and the Crush tunnables are set on optimal on both clusters.

Does anyone have any clue what could be wrong, or how to debug this issue?

Thanks.

 

[Alwin]

The Proxmox cluster can reach the Ceph cluster over IPv6?

 

[blaz]

The Proxmox cluster can reach the Ceph cluster over IPv6?

Yes. I can even manually list or create image on pool with command:
"rbd create --size 1024 libvirt-pool/temp_img -m [2001:1470:8000:403::xxx:xxx]:6789 ..."

 

[Alwin]

Yes. I can even manually list or create image on pool with command:
"rbd create --size 1024 libvirt-pool/temp_img -m [2001:1470:8000:403::xxx:xxx]:6789 ..."

You are creating the rbd images on Ceph from the proxmox cluster? If you didn't specify an alternative ceph.conf, then you need to add the IPs in '/etc/ceph/ceph.conf' too (may be a symlink to /etc/pve/ceph.conf).

 

[blaz]

You are creating the rbd images on Ceph from the proxmox cluster? If you didn't specify an alternative ceph.conf, then you need to add the IPs in '/etc/ceph/ceph.conf' too (may be a symlink to /etc/pve/ceph.conf).

Yes, from the Proxmox cluster, I just wanted to saw if Proxmox can access the pool. I tried to make ceph.conf file with public_network, ms_bind_ipv6, mon_initial_members and mon_host, but no luck. Am I doing something wrong? Should I specify IPs in different way?

 

[Alwin]

The pve-storage code uses the IPs set in the storage.cfg, but for the ceph tools you need to set the IPs in the ceph.conf. You can create one in /etc/pve/ceph.conf and symlink it on the hosts, so only one update is needed for the cluster.