[SOLVED] proxmox cluster error get status

Alibek · Jul 12, 2018

Summary:
Create cluster, add nodes (work only by --use_ssh),
When try look at another node status - request https://node-a:8006/api2/json/nodes/node-b/storage/local/status - fail with error: 596 Connection time out
But if login on each nodes - status returned

Connection is direct, no any proxy, browsers clear in private session and without any extensions.

Code:

proxmox-ve: 5.2-2 (running kernel: 4.15.18-1-pve)
pve-manager: 5.2-5 (running version: 5.2-5/eb24855a)
pve-kernel-4.15: 5.2-4
pve-kernel-4.15.18-1-pve: 4.15.18-15
pve-kernel-4.15.17-3-pve: 4.15.17-14
corosync: 2.4.2-pve5
criu: 2.11.1-1~bpo90
glusterfs-client: 3.8.8-1
ksm-control-daemon: not correctly installed
libjs-extjs: 6.0.1-2
libpve-access-control: 5.0-8
libpve-apiclient-perl: 2.0-5
libpve-common-perl: 5.0-35
libpve-guest-common-perl: 2.0-17
libpve-http-server-perl: 2.0-9
libpve-storage-perl: 5.0-24
libqb0: 1.0.1-1
lvm2: 2.02.168-pve6
lxc-pve: 3.0.0-3
lxcfs: 3.0.0-1
novnc-pve: 1.0.0-1
proxmox-widget-toolkit: 1.0-19
pve-cluster: 5.0-28
pve-container: 2.0-24
pve-docs: 5.2-4
pve-firewall: 3.0-13
pve-firmware: 2.0-5
pve-ha-manager: 2.0-5
pve-i18n: 1.0-6
pve-libspice-server1: 0.12.8-3
pve-qemu-kvm: 2.11.1-5
pve-xtermjs: 1.0-5
qemu-server: 5.0-29
smartmontools: 6.5+svn4324-1
spiceterm: 3.0-5
vncterm: 1.5-3
zfsutils-linux: 0.7.9-pve1~bpo9

Cause:
If use curl have next error:

Code:

$ curl -k -d "username=root@pam&password=......"  [URL]https://node-a:8006/api2/json/access/ticket[/URL]
$ curl -v -k -b "PVE:root@pam:5B477CF0::e......"  [URL]https://node-a:8006/api2/json/nodes/node-b/storage/local/status[/URL]
*   Trying 10.1.12.224...
* TCP_NODELAY set
* Connected to 10.1.12.224 (10.1.12.224) port 8006 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: OU=PVE Cluster Node; O=Proxmox Virtual Environment; CN=node-a
*  start date: Jul 10 16:39:52 2018 GMT
*  expire date: Jul  7 16:39:52 2028 GMT
*  issuer: CN=Proxmox Virtual Environment; OU=eff7585c-aafd-4f32-972b-f5453b85cd1a; O=PVE Cluster Manager CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> GET /api2/json/nodes/node-b/storage/local/status HTTP/1.1
> Host: 10.1.12.224:8006
> User-Agent: curl/7.58.0
> Accept: */*
> Cookie: PVE:root@pam:5B477CF0::e......
>
< HTTP/1.1 401 No ticket
< Cache-Control: max-age=0
< Connection: close
< Date: Thu, 12 Jul 2018 16:09:32 GMT
< Pragma: no-cache
< Server: pve-api-daemon/3.0
< Expires: Thu, 12 Jul 2018 16:09:32 GMT
<
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):

Decision:
The cause of the problem in mtu 9000. After return mtu on bond interfaces to 1500 - trouble is solved.

Conclusion:
I use bonding and vlans over bond for all networks - lan, nodes interconnect, external. And mtu 9000 was set on all bond. But mtu 9000 is need only vlan for interconnect of nodes.

wolfgang · Jul 13, 2018

Hi,

try to restart the http daemon.

Code:

systemctl restart pveproxy.service

Alibek · Jul 13, 2018

I restart pveproxy, restart each service, restart servers, remove all, and reinstall again - no effect

wolfgang · Jul 16, 2018

You did not set the cookie type

curl -v -k -b "PVEAuthCookie=PVE:root@pam:5B4......

see
https://pve.proxmox.com/wiki/Proxmox_VE_API

Alibek · Aug 2, 2018

Yes, my mistake.
With cookie have timeout:

Code:

...
> GET /api2/json/nodes/node-b/storage/local/status HTTP/1.1
> Host: node-a:8006
> User-Agent: curl/7.58.0
> Accept: */*
> Cookie: PVEAuthCookie=PVE:root@pam:5B632A...
>
< HTTP/1.1 596 Connection timed out
...

Then if check status node-a - have reply:

Code:

....
> GET /api2/json/nodes/node-a/storage/local/status HTTP/1.1
> Host: node-a:8006
> User-Agent: curl/7.58.0
> Accept: */*
> Cookie: PVEAuthCookie=PVE:root@pam:5B632A...
>
< HTTP/1.1 200 OK
< Cache-Control: max-age=0
< Connection: Keep-Alive
< Connection: Keep-Alive
< Date: Thu, 02 Aug 2018 16:10:49 GMT
< Pragma: no-cache
< Server: pve-api-daemon/3.0
< Content-Length: 154
< Content-Type: application/json;charset=UTF-8
< Expires: Thu, 02 Aug 2018 16:10:49 GMT
<
* Connection #0 to host node-a left intact
{"data":{"enabled":1,"type":"dir","total":156413468672,"active":1,"avail":148020617216,"shared":0,"content":"vztmpl,images,rootdir,iso","used":376070144}}

And have reply if i check status node-b on node-b with same cookie(ticket):

Code:

> GET /api2/json/nodes/node-b/storage/local/status HTTP/1.1
> Host: node-b:8006
> User-Agent: curl/7.58.0
> Accept: */*
> Cookie: PVEAuthCookie=PVE:root@pam:5B632A...
> 
< HTTP/1.1 200 OK
< Cache-Control: max-age=0
< Connection: Keep-Alive
< Connection: Keep-Alive
< Date: Thu, 02 Aug 2018 16:14:30 GMT
< Pragma: no-cache
< Server: pve-api-daemon/3.0
< Content-Length: 154
< Content-Type: application/json;charset=UTF-8
< Expires: Thu, 02 Aug 2018 16:14:30 GMT
< 
* Connection #0 to host node-b left intact
{"data":{"shared":0,"total":156413468672,"avail":148030951424,"used":365735936,"type":"dir","content":"iso,images,vztmpl,rootdir","active":1,"enabled":1}}

dcsapak · Aug 3, 2018

can your nodes reach each others port 8006 via https ?

Alibek · Aug 8, 2018

dcsapak said:
can your nodes reach each others port 8006 via https ?

Yes:

Code:

root@node-a:~# curl -I -k https://node-b:8006
HTTP/1.1 501 method 'HEAD' not available
Cache-Control: max-age=0
Connection: close
Date: Wed, 08 Aug 2018 11:55:58 GMT
Pragma: no-cache
Server: pve-api-daemon/3.0
Expires: Wed, 08 Aug 2018 11:55:58 GMT

root@node-b:~# curl -I -k https://node-a:8006
HTTP/1.1 501 method 'HEAD' not available
Cache-Control: max-age=0
Connection: close
Date: Wed, 08 Aug 2018 11:54:50 GMT
Pragma: no-cache
Server: pve-api-daemon/3.0
Expires: Wed, 08 Aug 2018 11:54:50 GMT

Alibek · Sep 4, 2018

The cause of the problem in mtu 9000. After return mtu on bond interfaces to 1500 - trouble is solved.
I use bonding and vlans over bond for all networks - lan, nodes interconnect, external. And mtu 9000 was set on all bond. But mtu 9000 is need only vlan for interconnect of nodes.

RocketSam · Feb 13, 2025

Hello Alibek. Can you please explain the mechanism of the problem with API when MTU is set to 9000?
"596 - Broken pipe" error is driving me crazy.

Search

Search

[SOLVED] proxmox cluster error get status

Alibek

Renowned Member

wolfgang

Proxmox Retired Staff

Alibek

Renowned Member

wolfgang

Proxmox Retired Staff

Alibek

Renowned Member

dcsapak

Proxmox Staff Member

Alibek

Renowned Member

Alibek

Renowned Member

RocketSam

New Member

We value your privacy