Error Connecting Drive using NBD CephCluster

[hcorneliussen]

Hello All,

I have an issue with my Ceph Network and a Backup Solution i would like to implement.


If i try to connect a Drive to NBD without specifying the CEPH Cluster It goes wrong with an error: “unable to get monitor info from DNS SRV with service name: ceph-mon
but if I mount it specifying the Ceph Cluster it works flawless:


we have the follwing Infrastructure:
3 Proxmox Nodes configured as a storage cluster:
Ceph Network: connected in a ring using Fabrics
Public Network: 25GB Switch

Additionally 2 Proxmox Nodes for Computer also connected to the 25GB Switch so they can access the Ceph Cluster Storage.
That works without an issue for the virtual machines and everything is fine.
But mapping the Drives as an NBD is not possible and the Backup Solution Provider is telling us, that we need to ask Proxmox why. Does anyone have an idea?
Input is highly appreciated!

Greetings
Harald Corneliussen

 

[Francois Baillargeon]

Do you have the URI of what your vendor passes?

It's probably just not passing the keyring correctly

 

[DamienDye]

Building on what Francois asked -- that error "unable to get monitor info from DNS SRV with service name: ceph-mon" is a DNS resolution issue. When you don't specify the cluster or monitors explicitly, the Ceph client library tries to auto-discover monitors by looking up DNS SRV records for _ceph-mon._tcp in your domain. That lookup is failing on your compute nodes.

Few things to check.

Does your DNS actually have SRV records for the Ceph monitors, or does it only have SRV records for the FQDN? Run dig SRV _ceph-mon._tcp.yourdomain from one of the compute nodes and post the output. That will show whether the records exist and what they point to.

If SRV records do exist, do the hostnames they return actually resolve to the correct monitor IPs on the Ceph public network? Check with dig or nslookup against each hostname in the SRV response. If your monitors only resolve by short hostname and the SRV records point to FQDNs, the lookup will fail even though the records are technically there. I have seen this catch folk out more than once.

Also worth checking /etc/hosts on the compute nodes -- are your Ceph monitor hostnames listed there with the correct public network IPs? On the three storage nodes Proxmox will have sorted this out, but the two compute nodes may have nowt in there for the monitors.

Quickest way to prove it -- add the monitor IP addresses to /etc/hosts on the compute node where the backup solution runs, mapping the monitor FQDNs to their Ceph public network addresses. If the NBD mount then works without specifying the cluster, you know it is DNS and nowt to do with Ceph or Proxmox.

The reason it works when you specify the cluster explicitly is that you are bypassing the DNS SRV lookup entirely and telling the client exactly where the monitors are. That is the giveaway.