TASK ERROR: 500 Can't connect to X.X.X.X:8006

[Gomjabs82]

Hello!
I encountered a problem (in the attachment) when trying to create a cluster of two servers. Shows error 500 without additional information (in some threads there were similar problems, after the address there was still some description of the error), but mine is empty.
I did everything according to the instructions, filled out the hosts file, the servers ping each other by full name (with domain), also telnet works fine, tried to do everything without using the GUI and got the same error.
Servers have ip's:
10.100.200.212 - Cluster (1node)
10.100.200.213 - (2node)
At first I tried to do this through the management interface, but encountered the same error and thought that this interface was not suitable for creating a cluster.
mgmt ip
10.100.200.112 node 1
10.100.200.113 node 2
journalctl -f also gives 0 info, just "end task" and "500 Can't Connect to 10.100.200.212:8006"

PVE 8.2.2
Servers are blades dell640

I would be grateful for any advice!


_________________________________________________________________________________
Problem solved with using MTU 1400 on switch and PVE network settings!

 

[bbgeek17]

You have two interfaces on the same subnet, likely creating asymmetric routing/replies.
Avoid such configuration unless you have good reason for it and good understanding of the consequences.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Gomjabs82]

You have two interfaces on the same subnet, likely creating asymmetric routing/replies.
Avoid such configuration unless you have good reason for it and good understanding of the consequences.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Thanks for the answer!
I'm afraid this is not the problem, I tried to make a different network for communication between nodes, even in a different vlan. Did not help.Perhaps there is a way to look at the connection logs in more detail besides journalctl, because there is no useful information in it at all.

 

[bbgeek17]

Perhaps there is a way to look at the connection logs in more detail

You can start tcpdump on each side and filter by host pair. I'd also shutdown the mgmt interfaces to simplify the configuration.
Pay attention to Requests/Replies on each side.

The "cant connect" is likely at a TCP layer and really means that it cant connect. There is not much more it can log if its an L3 problem.

Good luck

PS: can you "curl -k https://the_other_node_ip:8006" from each node?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Gomjabs82]

You can start tcpdump on each side and filter by host pair. I'd also shutdown the mgmt interfaces to simplify the configuration.
Pay attention to Requests/Replies on each side.

The "cant connect" is likely at a TCP layer and really means that it cant connect. There is not much more it can log if its an L3 problem.

Good luck

PS: can you "curl -k https://the_other_node_ip:8006" from each node?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Seems curl get 35 error, at least can google that, thanks again!
Telnet and ping still works fine. (turned ip back to different network)
Also find thread about MTU, maybe that is also my case, need to find some switch config too.
my is 1500 as default, some people get 1400 works.

 

[bbgeek17]

Now try "curl -s https://ip_of_thenode:8006" on the same node as the IP. I.e. local connection.
When you speak of "telnet" do you mean ssh?

Check MTU on each side and external firewall, if any. Default PVE installation does not enable any firewall, so its something external in your network.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[alexskysilk]

stupid question, but is the api service running properly on your target server?

systemctl status pveproxy
systemctl status pvestatd

 

[Gomjabs82]

Now try "curl -s https://ip_of_thenode:8006" on the same node as the IP. I.e. local connection.
When you speak of "telnet" do you mean ssh?

Check MTU on each side and external firewall, if any. Default PVE installation does not enable any firewall, so its something external in your network.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Curl -s just continue to command line with no text, if i go local connection.
curl -sk get me pages:

<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
<title>proxmox-lab-02 - Proxmox Virtual Environment</title>

and so on seems it works fine


Telnet mean checking port 8006 with node address (second pic) so there is no firewall between.
(both blades connect with each other with one switch, in one vlan, so no firewall used 'till they in one subnet)
so MTU is only one option seems for me, gonna try change on switch.

stupid question, but is the api service running properly on your target server?

systemctl status pveproxy
systemctl status pvestatd

Both servers both services seems have Active (running) and enabled/loaded.
But pvestatd have errors 3rd pic.




So guys dont want to send before check it all but using MTU 1400 is work for me (saw that in one thread before) default MTU settings is not working. Kinda strange but ill keep it, thanks you all for assistance!

 

[bbgeek17]

You mentioned VLANs previously. The MTU has to fit everything in it, including VLAN headers. It sounds like something is adding those headers, overflowing your packet size.

You should review all your settings again and increase the MTU at all points to accommodate VLANs if you use them. Or remove the VLAN config if you dont need it.

"curl -s" means that the output will be silent, ie no error (man curl):

Spoiler: curl -s, --silent

-s, --silent
Silent or quiet mode. Don't show progress meter or error messages. Makes Curl mute. It will still output the data you ask for, potentially even to the terminal/stdout unless you redi‐
rect it.

Use -S, --show-error in addition to this option to disable progress meter but still show error messages.

See also -v, --verbose and --stderr.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox