Can't connect to GUI on same network

mecheng

New Member
Jan 26, 2023
8
0
1
Hello,

I have just installed proxmox on debian 11. This is a new headless installation.

Situation: Can't access https://172.16.5.250:8006

Typical culprits:

/etc/network/interfaces

Code:
mecheng@skynet:~$ cat /etc/network/interfaces
# network interface settings; autogenerated
# Please do NOT modify this file directly, unless you know what
# you're doing.
#
# If you want to manage parts of the network configuration manually,
# please utilize the 'source' or 'source-directory' directives to do
# so.
# PVE will preserve these directives, but will NOT read its network
# configuration from sourced files, so do not attempt to move any of
# the PVE managed interfaces into external files!

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet dhcp

/etc/hostname
Code:
mecheng@skynet:~$ cat /etc/hostname
skynet

/etc/hosts
Code:
mecheng@skynet:~$ cat /etc/hosts
172.16.5.250    skynet.skynet   skynet pvelocalhost

# The following lines are desirable for IPv6 capable hosts
# ::1     localhost ip6-localhost ip6-loopback
# ff02::1 ip6-allnodes
# ff02::2 ip6-allrouters

Code:
mecheng@skynet:~$ hostname --ip-address
172.16.5.250

netstat - am I missing a routing table?????
Code:
mecheng@skynet:~$ netstat -an | grep 8006
tcp6       0      0 :::8006                 :::*                    LISTEN
mecheng@skynet:~$ netstat -an | grep 10000
tcp        0      0 0.0.0.0:10000           0.0.0.0:*               LISTEN
tcp6       0      0 :::10000                :::*                    LISTEN
udp        0      0 0.0.0.0:10000           0.0.0.0:*
mecheng@skynet:~$ netstat -an | grep 8000
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN
tcp6       0      0 :::8000                 :::*                    LISTEN
mecheng@skynet:~$ netstat -an | grep 9000
tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN
tcp6       0      0 :::9000                 :::*                    LISTEN
mecheng@skynet:~$

Can ssh into the server and it seems that the webpage is there but only accessible from the machine:
Code:
mecheng@skynet:~$ curl -s -k https://172.16.5.250:8006 | grep title
    <title>skynet - Proxmox Virtual Environment</title>

Output of pveversion -v
Code:
proxmox-ve: 7.3-1 (running kernel: 5.15.83-1-pve)
pve-manager: 7.3-4 (running version: 7.3-4/d69b70d4)
pve-kernel-helper: 7.3-2
pve-kernel-5.15: 7.3-1
pve-kernel-5.15.83-1-pve: 5.15.83-1
ceph-fuse: 14.2.21-1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.3
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.3-1
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-1
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.3.2-1
proxmox-backup-file-restore: 2.3.2-1
proxmox-mini-journalreader: 1.3-1
proxmox-offline-mirror-helper: 0.5.0-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-2
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.6-2
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-2
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.7-pve3


Also have webmin (10000), docker and portainer (8000, 9000, 9443) installed. both webmin and portainer are accessible from the GUI.
 
You've provided a lot of good information, But you did not define what exactly this means:
This is important, as you indicate that SSH and other services are functional and only PVE GUI is not.

Its generally not recommended to install multiple management frameworks on the same machine as Proxmox. Have you tested PVE access on a vanilla installation, before you added the rest of the applications? Although I am familiar with the apps you install, I dont know their installation quirks by heart. Have you checked firewall rules?

You mentioned 3 apps, is there anything else that was installed outside of PVE framework? My recommendation is to reinstall PVE only, ensure you have access and add the apps one by one, testing access after each installation.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Apologies.

I can't access the GUI from any computer on the network. Can access portainer and webmin from other computers and mobile devices.

The web GUI seems to be up and running as indicated by curl command. When I run the curl command on my windows or Mac, nothing.

Do you think that the netstat results indicate anything?
 
Can access portainer and webmin from other computers and mobile devices.
Do you access them by IP? If yes, is it the same IP that you use to SSH and when you SSH do you reach PVE shell?
When I run the curl command on my windows or Mac, nothing.
In the last few years I cant recall a single thread on the forum where OP complained about inability to access PVE GUI and it ended up being a bug in PVE. Every time it was an environmental issue. Sometimes very exoteric.
The netstat output is the same as on any other server I've seen, sans all the extra services you added.

The fact that local curl works indicates that PVE service is running. That means that the issue is likely in IP layer. You have few options:
1) from external host run curl with -vvv -d options, examine output
2) use tcpdump
3) use "nc"
4) disable/remove extra services
5) make a clean PVE installation


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Do you access them by IP? If yes, is it the same IP that you use to SSH and when you SSH do you reach PVE shell?


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

yes, 172.16.5.250:10000 for webmin and 172.16.5.250:9000 for portainer. And can ssh from two different machines into the 172.16.5.250.

curl -vvvv -d -s -k https://172.16.5.250:pORTS | grep title
Code:
MACOS$ curl -vvv -d -s -k https://172.16.5.250:8006 | grep title
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 172.16.5.250:8006...
  0     0    0     0    0     0      0      0 --:--:--  0:00:28 --:--:--     0^C



MACOS$ curl -vvv -d -s -k https://172.16.5.250:10000 | grep title
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 172.16.5.250:10000...
* Connected to 172.16.5.250 (172.16.5.250) port 10000 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):


I think that this is what you are asking for:

Code:
sudo tcpdump -i eno1 -nn -s0 -v port 8006
tcpdump: listening on eno1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
17:26:25.094057 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58523 > 172.16.5.250.8006: Flags [S], cksum 0xaeaf (correct), seq 4201457751, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4134340451 ecr 0,sackOK,eol], length 0
17:26:25.096917 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58524 > 172.16.5.250.8006: Flags [S], cksum 0xb2f3 (correct), seq 397235863, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2958228921 ecr 0,sackOK,eol], length 0
17:26:25.352498 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58526 > 172.16.5.250.8006: Flags [S], cksum 0xea75 (correct), seq 2750160084, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1117053303 ecr 0,sackOK,eol], length 0
17:26:26.095053 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58523 > 172.16.5.250.8006: Flags [S], cksum 0xaac6 (correct), seq 4201457751, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4134341452 ecr 0,sackOK,eol], length 0
17:26:26.096139 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58524 > 172.16.5.250.8006: Flags [S], cksum 0xaf0b (correct), seq 397235863, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2958229921 ecr 0,sackOK,eol], length 0
17:26:26.354038 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58526 > 172.16.5.250.8006: Flags [S], cksum 0xe68c (correct), seq 2750160084, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1117054304 ecr 0,sackOK,eol], length 0
17:26:27.096234 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58523 > 172.16.5.250.8006: Flags [S], cksum 0xa6de (correct), seq 4201457751, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4134342452 ecr 0,sackOK,eol], length 0
17:26:27.096743 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58524 > 172.16.5.250.8006: Flags [S], cksum 0xab22 (correct), seq 397235863, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2958230922 ecr 0,sackOK,eol], length 0
17:26:27.354901 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58526 > 172.16.5.250.8006: Flags [S], cksum 0xe2a3 (correct), seq 2750160084, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1117055305 ecr 0,sackOK,eol], length 0
17:26:28.096226 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58523 > 172.16.5.250.8006: Flags [S], cksum 0xa2f6 (correct), seq 4201457751, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4134343452 ecr 0,sackOK,eol], length 0
17:26:28.097855 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58524 > 172.16.5.250.8006: Flags [S], cksum 0xa739 (correct), seq 397235863, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2958231923 ecr 0,sackOK,eol], length 0
17:26:28.354777 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58526 > 172.16.5.250.8006: Flags [S], cksum 0xdebb (correct), seq 2750160084, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1117056305 ecr 0,sackOK,eol], length 0
17:26:29.096437 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58523 > 172.16.5.250.8006: Flags [S], cksum 0x9f0d (correct), seq 4201457751, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4134344453 ecr 0,sackOK,eol], length 0
17:26:29.098140 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58524 > 172.16.5.250.8006: Flags [S], cksum 0xa351 (correct), seq 397235863, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2958232923 ecr 0,sackOK,eol], length 0
17:26:29.354461 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58526 > 172.16.5.250.8006: Flags [S], cksum 0xdad3 (correct), seq 2750160084, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1117057305 ecr 0,sackOK,eol], length 0
17:26:30.099766 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58524 > 172.16.5.250.8006: Flags [S], cksum 0x9f69 (correct), seq 397235863, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2958233923 ecr 0,sackOK,eol], length 0
17:26:30.099996 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58523 > 172.16.5.250.8006: Flags [S], cksum 0x9b23 (correct), seq 4201457751, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4134345455 ecr 0,sackOK,eol], length 0
17:26:30.355290 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58526 > 172.16.5.250.8006: Flags [S], cksum 0xd6ea (correct), seq 2750160084, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1117058306 ecr 0,sackOK,eol], length 0
17:26:32.100069 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58524 > 172.16.5.250.8006: Flags [S], cksum 0x9798 (correct), seq 397235863, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2958235924 ecr 0,sackOK,eol], length 0
17:26:32.100834 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58523 > 172.16.5.250.8006: Flags [S], cksum 0x9352 (correct), seq 4201457751, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4134347456 ecr 0,sackOK,eol], length 0
17:26:32.357650 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58526 > 172.16.5.250.8006: Flags [S], cksum 0xcf18 (correct), seq 2750160084, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1117060308 ecr 0,sackOK,eol], length 0
17:26:36.100920 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58524 > 172.16.5.250.8006: Flags [S], cksum 0x87f7 (correct), seq 397235863, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 2958239925 ecr 0,sackOK,eol], length 0
17:26:36.101515 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58523 > 172.16.5.250.8006: Flags [S], cksum 0x83b1 (correct), seq 4201457751, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 4134351457 ecr 0,sackOK,eol], length 0
17:26:36.359155 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 64)
    172.16.5.186.58526 > 172.16.5.250.8006: Flags [S], cksum 0xbf76 (correct), seq 2750160084, win 65535, options [mss 1460,nop,wscale 6,nop,nop,TS val 1117064310 ecr 0,sackOK,eol], length 0


nc
Code:
mecheng@skynet:~$ sudo nc -z -v 172.16.5.250 8006
skynet.skynet [172.16.5.250] 8006 (?) open
mecheng@skynet:~$ sudo nc -z -v 172.16.5.250 10000
skynet.skynet [172.16.5.250] 10000 (webmin) open
mecheng@skynet:~$ sudo nc -z -v 172.16.5.250 22
skynet.skynet [172.16.5.250] 22 (ssh) open
mecheng@skynet:~$ sudo nc -z -v 172.16.5.250 9000
skynet.skynet [172.16.5.250] 9000 (?) open
 
Shouldn't this be in the netstat results for the 8006:

Code:
tcp        0      0 0.0.0.0:8006           0.0.0.0:*               LISTEN
 
What side was the tcpdump taken on? You need both sides of tcpdump for proper analyses. Everything else did not reveal anything new.
You still did not answer whether you reviewed firewall rules.
https://unix.stackexchange.com/ques...ons-listening-to-ports-listed-only-in-a-inet6

I recommend, again, trying vanilla installation.


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

That was from the server side.

Here are logs from both sides.

I did not review the firewall rules.
 

Attachments

  • tcpdump172165250_client.log
    8.9 KB · Views: 2
  • tcpdump172165250_server.log
    8.9 KB · Views: 1
Code:
netstat -planA inet
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name   
tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:9443            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:10000           0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      -                   
tcp        0      0 127.0.0.1:85            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8006            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      -                   
tcp        0      0 0.0.0.0:3128            0.0.0.0:*               LISTEN      -                   
tcp        0    324 172.16.5.250:22         172.16.5.186:58516      ESTABLISHED -                   
udp        0      0 0.0.0.0:10000           0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:68              0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:111             0.0.0.0:*                           -
 
Wait
yes, 172.16.5.250:10000 for webmin and 172.16.5.250:9000 for portainer. And can ssh from two different machines into the 172.16.5.250.
Same IP for prox webmin and portainer? So you installed proxmox, then proceeded to install docker / portainer directly onto the host?

Highly suggest you nuke it all and reinstall.
Don't waste time untangling problems.

Proxmox should generally "just work" out of the box.
Fresh install will get you to the webUI

From there make your VMs or LXC containers for docker, portainer, etc.

Let proxmox do proxmox things. keep the host clean.
Install your apps isolated in their own virtual environments.
That's the whole point, right?
 
tcpdump shows a TCP SYN packet sent from client to server every few microseconds using a new source port.
No indication of an ACK reply, or any return traffic. Which is an indication of either broken routing and traffic being sent via wrong interface or firewall dropping the packets/misconfiguration. Neither can possibly be present in a vanilla PVE installation.

My recommendation is to reinstall PVE only, ensure you have access and add the apps one by one, testing access after each installation.
5) make a clean PVE installation
I recommend, again, trying vanilla installation.
Highly suggest you nuke it all and reinstall.

Good luck


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
/etc/hosts
Code:
mecheng@skynet:~$ cat /etc/hosts
172.16.5.250 skynet.skynet skynet pvelocalhost

# The following lines are desirable for IPv6 capable hosts
# ::1 localhost ip6-localhost ip6-loopback
# ff02::1 ip6-allnodes
# ff02::2 ip6-allrouters

I am not sure, if this has anything to do with your specific problem, but the: /etc/hosts for PVE should more look like this (skipped the IPv6-part):
Bash:
127.0.0.1 localhost.localdomain localhost
192.168.1.201 pve01.home pve01
(Of course, adapt it.)

But like others already said: Better start fresh again and leave the PVE-host clean! (Docker, for example, can easily mess up your whole network on the host!)

PS.: It is also recommended to set a static IP directly on the PVE-host; instead of DHCP...
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!