[SOLVED] Can't access a node via another nodes Web GUI (Error 595)

Nic_

Member
Jan 5, 2022
35
1
13
17
Austria
Hello everyone,
so today I created a cluster on node A and joined node B into that cluster.
Both nodes are on PVE v8.0.3 and worked fine. The only issue is that when I am trying to access anything from node B, while connected via node A's Web GUI I always get a timeout (image), and also the other way around.

1690461370675.png

If you need any logs or config files please tell me because I don't know what to post.
Firewalls are disabled for now, on datacenter-level and on the nodes themselves.

Thanks!
 
what do the nodes hostnames resolve to on each end? anything visible in the logs (journalctl)?
 
The hostnames of each node resolve to the nodes IP address in the /etc/hosts and I also can ping each node via their hostname

I didn't find anything that looks like a pve-cluster error message of any kind in the logs of journalctl


they also show the status of each node and container correctly
 
pve-cluster is not or only barely involved in this, it's pveproxy that does the proxying between nodes..
 
SSH is not needed for proxying API requests, only for cross-node console access and migration..
 
Verify ssh from one node to the other, making sure it works without password prompt.
From nodeA: root@nodeA:~# ssh root@nodeB and then from nodeB: root@nodeB:~# ssh root@nodeA
Ssh connections work without password
pve-cluster is not or only barely involved in this, it's pveproxy that does the proxying between nodes..
The only thing I found from pveproxy is
Code:
pveproxy [1146]: proxy detected vanished client connection
 
can you try stopping the pveproxy service (systemctl stop pveproxy, and starting the pveproxy binary in debug mode (pveproxy start -d and post its output here when you attempt such a failing request?
 
Code:
worker[10030]: PVE::APIServer::AnyEvent +1792: (eval): ACCEPT FH10 CONN1
worker[10030]: PVE::APIServer::AnyEvent +1792: (eval): Setting TLS to autostart
worker[10030]: PVE::APIServer::AnyEvent +1834: client_do_disconnect: close connection AnyEvent::Handle=HASH(0x5594a46494d8)
worker[10030]: PVE::APIServer::AnyEvent +1834: client_do_disconnect: CLOSE FH10 CONN0
worker[10029]: PVE::APIServer::AnyEvent +1834: client_do_disconnect: close connection AnyEvent::Handle=HASH(0x5594a47b00c8)
worker[10029]: PVE::APIServer::AnyEvent +1834: client_do_disconnect: CLOSE FH14 CONN1
proxy detected vanished client connection
worker[10030]: PVE::APIServer::AnyEvent +1792: (eval): ACCEPT FH10 CONN1
worker[10030]: PVE::APIServer::AnyEvent +1792: (eval): Setting TLS to autostart
worker[10030]: PVE::APIServer::AnyEvent +1834: client_do_disconnect: close connection AnyEvent::Handle=HASH(0x5594a4655900)
worker[10030]: PVE::APIServer::AnyEvent +1834: client_do_disconnect: CLOSE FH10 CONN0
thats the output

I also got this now
1690466326390.png
 
I also now got this error while trying to connect to a console on another node
Code:
TASK ERROR: command '/usr/bin/termproxy 5900 --path /nodes/pve-pycmc --perm Sys.Console -- /usr/bin/ssh -e none -t IPV6-ADDRESS-HERE -- /bin/login -f root' failed: exit code 1
 
that error message indicates that you have an NGINX somewhere in front of the PVE API/GUI? I am not sure what's going on, but that definitely means you are using some sort of non-standard customized setup and that is broken..
 
that error message indicates that you have an NGINX somewhere in front of the PVE API/GUI? I am not sure what's going on, but that definitely means you are using some sort of non-standard customized setup and that is broken..
Yes I use nginx proxy manager for my connections but I also tested it without the proxy manager and still get the timeout so its not the proxies fault
 
well, something is messing with your inter-node connections, by default nothing in PVE does, and the only requirement is that the node names are correctly resolving on all nodes.
 
well, something is messing with your inter-node connections, by default nothing in PVE does, and the only requirement is that the node names are correctly resolving on all nodes.
Can you please explain to me what I have to do to make them resolve correctly, maybe I did a mistake.

So what I did was that I added the servers hostnames and IP addresses in each others resolv.conf file, thats it.
 
1690672094724.png

could it be that my certificate on every node only has the hostname of node A and the IPv6 address of node A in it?
I don't know anything about certificates so I can't tell...
 
can you please do the following?

- "pveversion -v" on each node
- "cat /etc/hosts" on each node
- "hostname -f" on each node
- "ss -ltpn" on each node
- "curl -kv 'https://$HOSTNAME_OF_OTHER_NODE:8006'" on each node, for each other node
 
Node A
pveversion -v
Code:
proxmox-ve: 8.0.1 (running kernel: 6.2.16-5-pve)
pve-manager: 8.0.3 (running version: 8.0.3/bbf3993334bfa916)
pve-kernel-6.2: 8.0.4
pve-kernel-5.15: 7.4-4
pve-kernel-6.2.16-5-pve: 6.2.16-6
pve-kernel-5.15.108-1-pve: 5.15.108-1
pve-kernel-5.15.104-1-pve: 5.15.104-2
ceph: 17.2.6-pve1+3
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx3
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.3
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.6
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.4
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.1-1
proxmox-backup-file-restore: 3.0.1-1
proxmox-kernel-helper: 8.0.2
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.2
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-3
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

cat /etc/hosts
Code:
127.0.0.1 localhost
xxx.xx.xx.169 pve-pycmc.pycmc.eu pve-pycmc
xxx.xxx.xxx.146 AX41-1665271
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
xxxx:xxx:261:4fe2::2 AX41-1665271

hostname -f
Code:
AX41-1665271

ss -ltpn
Code:
State                Recv-Q               Send-Q                             Local Address:Port                               Peer Address:Port               Process                                                                                                                                                       
LISTEN               0                    4096                                   127.0.0.1:85                                      0.0.0.0:*                   users:(("pvedaemon worke",pid=1996451,fd=6),("pvedaemon worke",pid=1979737,fd=6),("pvedaemon worke",pid=1962652,fd=6),("pvedaemon",pid=1283,fd=6))           
LISTEN               0                    4096                                     0.0.0.0:111                                     0.0.0.0:*                   users:(("rpcbind",pid=695,fd=4),("systemd",pid=1,fd=36))                                                                                                     
LISTEN               0                    128                                      0.0.0.0:22                                      0.0.0.0:*                   users:(("sshd",pid=917,fd=3))                                                                                                                                
LISTEN               0                    100                                      0.0.0.0:25                                      0.0.0.0:*                   users:(("master",pid=1219,fd=13))                                                                                                                            
LISTEN               0                    4096                                           *:8006                                          *:*                   users:(("pveproxy worker",pid=3789591,fd=6),("pveproxy worker",pid=3789590,fd=6),("pveproxy worker",pid=3789589,fd=6),("pveproxy",pid=1293,fd=6))            
LISTEN               0                    4096                                        [::]:111                                        [::]:*                   users:(("rpcbind",pid=695,fd=6),("systemd",pid=1,fd=38))                                                                                                     
LISTEN               0                    128                                         [::]:22                                         [::]:*                   users:(("sshd",pid=917,fd=4))                                                                                                                                
LISTEN               0                    100                                         [::]:25                                         [::]:*                   users:(("master",pid=1219,fd=14))                                                                                                                            
LISTEN               0                    4096                                           *:3128                                          *:*                   users:(("spiceproxy work",pid=3789588,fd=6),("spiceproxy",pid=1299,fd=6))

curl -kv 'https://$HOSTNAME_OF_OTHER_NODE:8006'
Code:
*   Trying xxx.xx.xx.169:8006...
* Connected to pve-pycmc (xxx.xx.xx.169) port 8006 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: OU=PVE Cluster Node; O=Proxmox Virtual Environment; CN=AX41-1665271
*  start date: Jan 17 10:16:07 2023 GMT
*  expire date: Jan 16 10:16:07 2025 GMT
*  issuer: CN=Proxmox Virtual Environment; OU=8c34a05a-1196-498a-9e10-bba503a64de2; O=PVE Cluster Manager CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET / HTTP/1.1
> Host: pve-pycmc:8006
> User-Agent: curl/7.88.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 200 OK
< Cache-Control: max-age=0
< Connection: Keep-Alive
< Date: Mon, 31 Jul 2023 08:49:20 GMT
< Pragma: no-cache
< Server: pve-api-daemon/3.0
< Content-Length: 2403
< Content-Type: text/html; charset=utf-8
< Expires: Mon, 31 Jul 2023 08:49:20 GMT
< 
<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
    <title>pve-pycmc - Proxmox Virtual Environment</title>
    <link rel="icon" sizes="128x128" href="/pve2/images/logo-128.png" />
    <link rel="apple-touch-icon" sizes="128x128" href="/pve2/images/logo-128.png" />
    <link rel="stylesheet" type="text/css" href="/pve2/ext6/theme-crisp/resources/theme-crisp-all.css?ver=7.0.0" />
    <link rel="stylesheet" type="text/css" href="/pve2/ext6/crisp/resources/charts-all.css?ver=7.0.0" />
    <link rel="stylesheet" type="text/css" href="/pve2/fa/css/font-awesome.css" />
    <link rel="stylesheet" type="text/css" href="/pve2/css/ext6-pve.css?ver=8.0.3" />
    <link rel="stylesheet" type="text/css" href="/pwt/css/ext6-pmx.css?ver=v4.0.6-t1687803897" />
    <link rel="stylesheet" type="text/css" media="(prefers-color-scheme: dark)" href="/pwt/themes/theme-proxmox-dark.css?ver=v4.0.6-t1687803897" />
    
    <script type='text/javascript'>function gettext(buf) { return buf; }</script>
    
    <script type="text/javascript" src="/pve2/ext6/ext-all.js?ver=7.0.0"></script>
    <script type="text/javascript" src="/pve2/ext6/charts.js?ver=7.0.0"></script>
    
    <script type="text/javascript" src="/pve2/js/u2f-api.js"></script>
    <script type="text/javascript" src="/qrcode.min.js"></script>
    <script type="text/javascript">
    Proxmox = {
        Setup: { auth_cookie_name: 'PVEAuthCookie' },
        defaultLang: 'en',
        NodeName: 'pve-pycmc',
        UserName: '',
        CSRFPreventionToken: 'null'
    };
    </script>
    <script type="text/javascript" src="/proxmoxlib.js?ver=v4.0.6-t1687803897"></script>
    <script type="text/javascript" src="/pve2/js/pvemanagerlib.js?ver=8.0.3"></script>
    <script type="text/javascript" src="/pve2/ext6/locale/locale-en.js?ver=7.0.0"></script>

    <script type="text/javascript">
    if (typeof(PVE) === 'undefined') PVE = {};
    Ext.History.fieldid = 'x-history-field';
    Ext.onReady(function() { Ext.create('PVE.StdWorkspace');});
    </script>

  </head>
  <body>
    <!-- Fields required for history management -->
    <form id="history-form" class="x-hidden">
    <input type="hidden" id="x-history-field"/>
    </form>
  </body>
</html>
* Connection #0 to host pve-pycmc left intact
 
Node B
pveversion -v
Code:
proxmox-ve: 8.0.1 (running kernel: 6.2.16-4-pve)
pve-manager: 8.0.3 (running version: 8.0.3/bbf3993334bfa916)
pve-kernel-6.2: 8.0.3
pve-kernel-5.15: 7.4-4
pve-kernel-6.2.16-4-pve: 6.2.16-5
pve-kernel-5.15.108-1-pve: 5.15.108-1
ceph-fuse: 17.2.6-pve1+3
corosync: 3.1.7-pve3
criu: 3.17.1-2
glusterfs-client: 10.3-5
ifupdown: residual config
ifupdown2: 3.2.0-1+pmx3
libjs-extjs: 7.0.0-3
libknet1: 1.25-pve1
libproxmox-acme-perl: 1.4.6
libproxmox-backup-qemu0: 1.4.0
libproxmox-rs-perl: 0.3.0
libpve-access-control: 8.0.3
libpve-apiclient-perl: 3.3.1
libpve-common-perl: 8.0.6
libpve-guest-common-perl: 5.0.3
libpve-http-server-perl: 5.0.4
libpve-rs-perl: 0.8.4
libpve-storage-perl: 8.0.2
libspice-server1: 0.15.1-1
lvm2: 2.03.16-2
lxc-pve: 5.0.2-4
lxcfs: 5.0.3-pve3
novnc-pve: 1.4.0-2
proxmox-backup-client: 3.0.1-1
proxmox-backup-file-restore: 3.0.1-1
proxmox-kernel-helper: 8.0.2
proxmox-mail-forward: 0.2.0
proxmox-mini-journalreader: 1.4.0
proxmox-widget-toolkit: 4.0.6
pve-cluster: 8.0.2
pve-container: 5.0.4
pve-docs: 8.0.4
pve-edk2-firmware: 3.20230228-4
pve-firewall: 5.0.3
pve-firmware: 3.7-1
pve-ha-manager: 4.0.2
pve-i18n: 3.0.5
pve-qemu-kvm: 8.0.2-3
pve-xtermjs: 4.16.0-3
qemu-server: 8.0.6
smartmontools: 7.3-pve1
spiceterm: 3.3.0
swtpm: 0.8.0+pve1
vncterm: 1.8.0
zfsutils-linux: 2.1.12-pve1

cat /etc/hosts
Code:
### Hetzner Online GmbH installimage
127.0.0.1 localhost.localdomain localhost
10.10.10.100 pve-pycmc
xxx.xx.xx.169 pve-pycmc.pycmc.eu pve-pycmc
xxx.xxx.xxx.146 AX41-1665271
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
xxxx:xxx:121:14bd::2 pve-pycmc.pycmc.eu pve-pycmc

hostname -f
Code:
pve-pycmc

ss -ltpn
Code:
State                Recv-Q               Send-Q                             Local Address:Port                               Peer Address:Port               Process                                                                                                                                                       
LISTEN               0                    100                                    127.0.0.1:25                                      0.0.0.0:*                   users:(("master",pid=1096,fd=13))                                                                                                                           
LISTEN               0                    4096                                   127.0.0.1:85                                      0.0.0.0:*                   users:(("pvedaemon worke",pid=3918815,fd=6),("pvedaemon worke",pid=3884909,fd=6),("pvedaemon worke",pid=3805925,fd=6),("pvedaemon",pid=1151,fd=6))           
LISTEN               0                    4096                                     0.0.0.0:111                                     0.0.0.0:*                   users:(("rpcbind",pid=577,fd=4),("systemd",pid=1,fd=117))                                                                                                   
LISTEN               0                    128                                      0.0.0.0:22                                      0.0.0.0:*                   users:(("sshd",pid=780,fd=3))                                                                                                                               
LISTEN               0                    4096                                           *:8006                                          *:*                   users:(("pveproxy worker",pid=3931122,fd=6),("pveproxy worker",pid=3920885,fd=6),("pveproxy worker",pid=3918680,fd=6),("pveproxy",pid=1161,fd=6))           
LISTEN               0                    4096                                        [::]:111                                        [::]:*                   users:(("rpcbind",pid=577,fd=6),("systemd",pid=1,fd=119))                                                                                                   
LISTEN               0                    128                                         [::]:22                                         [::]:*                   users:(("sshd",pid=780,fd=4))                                                                                                                               
LISTEN               0                    4096                                           *:3128                                          *:*                   users:(("spiceproxy work",pid=3885066,fd=6),("spiceproxy",pid=1167,fd=6))                                                                                   
LISTEN               0                    100                                        [::1]:25                                         [::]:*                   users:(("master",pid=1096,fd=14))

curl -kv 'https://$HOSTNAME_OF_OTHER_NODE:8006'
Code:
*   Trying xxx.xxx.xxx.146:8006...
* Connected to AX41-1665271 (xxx.xxx.xxx.146) port 8006 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: OU=PVE Cluster Node; O=Proxmox Virtual Environment; CN=AX41-1665271
*  start date: Jan 17 10:16:07 2023 GMT
*  expire date: Jan 16 10:16:07 2025 GMT
*  issuer: CN=Proxmox Virtual Environment; OU=8c34a05a-1196-498a-9e10-bba503a64de2; O=PVE Cluster Manager CA
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET / HTTP/1.1
> Host: AX41-1665271:8006
> User-Agent: curl/7.88.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/1.1 200 OK
< Cache-Control: max-age=0
< Connection: Keep-Alive
< Date: Mon, 31 Jul 2023 09:05:12 GMT
< Pragma: no-cache
< Server: pve-api-daemon/3.0
< Content-Length: 2409
< Content-Type: text/html; charset=utf-8
< Expires: Mon, 31 Jul 2023 09:05:12 GMT
<
<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
    <title>AX41-1665271 - Proxmox Virtual Environment</title>
    <link rel="icon" sizes="128x128" href="/pve2/images/logo-128.png" />
    <link rel="apple-touch-icon" sizes="128x128" href="/pve2/images/logo-128.png" />
    <link rel="stylesheet" type="text/css" href="/pve2/ext6/theme-crisp/resources/theme-crisp-all.css?ver=7.0.0" />
    <link rel="stylesheet" type="text/css" href="/pve2/ext6/crisp/resources/charts-all.css?ver=7.0.0" />
    <link rel="stylesheet" type="text/css" href="/pve2/fa/css/font-awesome.css" />
    <link rel="stylesheet" type="text/css" href="/pve2/css/ext6-pve.css?ver=8.0.3" />
    <link rel="stylesheet" type="text/css" href="/pwt/css/ext6-pmx.css?ver=v4.0.6-t1687803897" />
    <link rel="stylesheet" type="text/css" media="(prefers-color-scheme: dark)" href="/pwt/themes/theme-proxmox-dark.css?ver=v4.0.6-t1687803897" />
    
    <script type='text/javascript'>function gettext(buf) { return buf; }</script>
    
    <script type="text/javascript" src="/pve2/ext6/ext-all.js?ver=7.0.0"></script>
    <script type="text/javascript" src="/pve2/ext6/charts.js?ver=7.0.0"></script>
    
    <script type="text/javascript" src="/pve2/js/u2f-api.js"></script>
    <script type="text/javascript" src="/qrcode.min.js"></script>
    <script type="text/javascript">
    Proxmox = {
        Setup: { auth_cookie_name: 'PVEAuthCookie' },
        defaultLang: 'en',
        NodeName: 'AX41-1665271',
        UserName: '',
        CSRFPreventionToken: 'null'
    };
    </script>
    <script type="text/javascript" src="/proxmoxlib.js?ver=v4.0.6-t1687803897"></script>
    <script type="text/javascript" src="/pve2/js/pvemanagerlib.js?ver=8.0.3"></script>
    <script type="text/javascript" src="/pve2/ext6/locale/locale-en.js?ver=7.0.0"></script>

    <script type="text/javascript">
    if (typeof(PVE) === 'undefined') PVE = {};
    Ext.History.fieldid = 'x-history-field';
    Ext.onReady(function() { Ext.create('PVE.StdWorkspace');});
    </script>

  </head>
  <body>
    <!-- Fields required for history management -->
    <form id="history-form" class="x-hidden">
    <input type="hidden" id="x-history-field"/>
    </form>
  </body>
</html>
* Connection #0 to host AX41-1665271 left intact
 
Code:
10.10.10.100 pve-pycmc

you have two entries for this hostname, this is likely messing up your proxying..
 
  • Like
Reactions: Nic_

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!