Cannot query node api from other node

[arogarth]

Hello,

one of our proxmox cluster has a problem to query api status of other nodes:

node01
pvesh get /nodes/node01/status - works
pvesh get /nodes/node02/status - works

node02
pvesh get /nodes/node01/status - not working
pvesh get /nodes/node02/status - works

Can someone explain, why? I tried to restart some services, but this did not fixed it.

Best regards,
aro

 

[VictorSTS]

Please define "not working" in detail, messages, logs, etc.

 

[arogarth]

Hi,

sorry... It results in a timeout. Hostname ist corrent, /etc/hosts is managed and fine. nc -zv <ip> 8006 is also ok, both are in same subnet.

Same situation on WebUI on checking cluster states (also disks etc..)

 

[VictorSTS]

Maybe firewall or network related?
Can you ssh without password from node 02 to node 01?

pvesh uses SSH to reach other nodes, but webUI uses api on port 8006.

Make really sure that /etc/hosts file in every node has an entry for the node with the correct IP. PVE does not need an entry in hosts file for every node in the cluster, only one entry for the host itself, like:

127.0.0.1 localhost.localdomain localhost
172.22.2.193 meshpve03-of.lab.local meshpve03-of

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

 

[arogarth]

I found the issue. During the installation I left the default IP because we have vlan and I have to change the bridge and vlan settings after setup.

The Problem was more critical on migrating vms - I was able to move vms from node1 to node2 but not back. There I got the error that ssh key not matched.

Now I found that /etc/ssh/ssh_known_hosts contained a entry with the old ip from installation. pvecm show the right ip but /etc/pve/.members has the old one on node1 - this was the error.
The solution was to restart pve-cluster service and the required ip was now in .members.

ha-manager and pvecm did show the correct ips - this was confusing...

Now, Migation to node1 works and also pvesh get works fine

Thanks for Help!