How do nodes resolve other nodes hostname?

[Onyx]

In my old 4-node cluster I can just ping/ssh from one node to an other using it's hostname.

Now I'm moving to a new, fresh installed cluster and this doesn't work (3.4-13 for now, still need OpenVZ)

root@vh02:/# ping vh04
ping: unknown host vh04

This just works on my old cluster but not on the new one and I can't figure out why...
Everything else works just fine on that new cluster.

I would expect this is configured in /etc/hosts but it isn't. I'm sure that I could configure it there manually on each node but somehow "vh04" resolves via some other mechanism on my old cluster.

So, how do nodes resolve other nodes hostname?

This is the content of /etc/hosts. It's identical on my old- and new cluster nodes:

root@vh02:/# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
10.100.0.102 vh02.mydomain.com vh02 pvelocalhost

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Thanks.

 

[wosp]

Normally it's done by /etc/hosts as you expected. Maybe you have added a custom DNS-zone to the old nodes resolver(s)? Check /etc/resolv.conf for your nameservers and run "host -t a vh04 IP.OF.FIRST.DNS" to see if it answer correct.

 

[Onyx]

Nope. I looked at this already and I'm using Google's nameservers 8.8.8.8 and 8.8.4.4 on my old cluster.

 

[Onyx]

Found it! (after more then a day being braindead...)

Actually, on my old cluster nodes are named like "vh002" and on my new one like "vh02".

I do have dns records for "vh002.mydomain.com" and since "search mydomain.com" is in my /etc/resolv.conf "vh002" resolves on my old cluster.

Making dns records for my new clusternames like "vh02.mydomain.com" solved this mystery.

Thx wosp for rebooting my brain! ;-)

 

[wosp]

Found it! (after more then a day being braindead...)

Actually, on my old cluster nodes are named like "vh002" and on my new one like "vh02".

I do have dns records for "vh002.mydomain.com" and since "search mydomain.com" is in my /etc/resolv.conf "vh002" resolves on my old cluster.

Making dns records for my new clusternames like "vh02.mydomain.com" solved this mystery.

Thx wosp for rebooting my brain! ;-)

Your welcome!

BTW, I would recommend to also add the hostnames/IP's to your hostfile, because your cluster core communications depends on it (if you don't have WAN, for example when your ISP have a problem, you still don't want your cluster to screw up). Besides that, a lookup from your local hostfile is faster than external DNS server lookups.

 

[Onyx]

Are you sure of that? Everything worked fine on that new cluster except that I couldn't use "ssh vh02" as I always did on the old cluster.
So I believe internal workings of the cluster don't need external resolution of the cluster hostnames to function properly.

If it was that vital, I suspect /etc/hosts would be automatically configured like that by the Proxmox setup scripts.

 

[wosp]

Are you sure of that? Everything worked fine on that new cluster except that I couldn't use "ssh vh02" as I always did on the old cluster.
So I believe internal workings of the cluster don't need external resolution of the cluster hostnames to function properly.

If it was that vital, I suspect /etc/hosts would be automatically configured like that by the Proxmox setup scripts.

Yup, I'm sure. As far as I know this is done already automaticly (I can't remember I've added it manually). No idea why it isn't added in your setup automaticly. For example when you install PVE on a clean Debian Jessie install, Proxmox also recommand to add your hostnames/IP's to /etc/hosts: https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie#Add_an_.2Fetc.2Fhosts_entry_for_your_IP_address

If your nodes connected to each other succesfully before your resolvers are unreachable, your cluster won't break directly because of DNS cache. But when it will take some longer (when the TTL is expired), you will have issues. A lookup of /etc/hosts is always faster than a external DNS lookup and because it doesn't require WAN connectivity, it's much more reliable.

 

[Onyx]

That link only explains to configure it's own hostname in /etc/hosts, not all hostnames of all clustermembers.

The /etc/hosts file I posted in my first post is how each clusternode is automatically configured by a fresh Proxmox installation. This does not change after configuring the cluster. Your link explains that you have to do the same manually if you install Proxmox on top on a pre-installed Debian box.

Hostname/IP relation for cluster internals is configured in /etc/pve/.

There was never a dns cache involved here. As I explained, everything worked fine before I ever configured the dns records for the nodes.