Port 8006 not open, get connection error serve offline? [SOLVED]

[robb01]

I am posting a new thread for my issue as many similar posts have the same issue but for very different situations.
This is a new install from an iso on a NUC7i5 16GB, 1.8TB SSD .
There are no clusters in my setup.
/etc/hosts

127.0.0.1 localhost.localdomain localhost
192.168.1.89 pve.rb.test pve

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

/etc/network/interfaces

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.89/24
        gateway 192.168.1.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

pveversion -v

proxmox-ve: 7.3-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.3-3 (running version: 7.3-3/c3928077)
pve-kernel-helper: 7.3-1
pve-kernel-5.15: 7.2-14
pve-kernel-5.15.74-1-pve: 5.15.74-1
ceph-fuse: 15.2.17-pve1
corosync: 3.1.7-pve1
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve2
libproxmox-acme-perl: 1.4.3
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-5
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.3-1
libpve-guest-common-perl: 4.2-3
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.3-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.3.1-1
proxmox-backup-file-restore: 2.3.1-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.3
pve-cluster: 7.3-1
pve-container: 4.4-2
pve-docs: 7.3-1
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-7
pve-firmware: 3.6-1
pve-ha-manager: 3.5.1
pve-i18n: 2.8-1
pve-qemu-kvm: 7.1.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.3-1
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

The port 8006 shows as closed when the connection to the console fails. An ssh connection attempt also fails with

Unable to negotiate with 192.168.1.89 port 22: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1

Curiously, when I cannot connect, I can run sudo nmap -sS -p 8006 192.168.1.89 on my laptop and the port comes open. An ssh connection then is successful and I can access the web console.
I have run pvecm updatecerts -f after reading
http://server1.sharewiz.net/doku.ph...to_connect_with_server_disconnected_code:1006
I have tried various browsers including Chrome and FF and also on a windows machine rather than my 22.04 laptop. So it seems to not be a browser issue but a connection or network issue.

Is there anything further I can post that will help diagnose this issue?
TIA

 

[bbgeek17]

The port 8006 shows as closed

When you are still unable to connect, from physical console of the PVE run:
curl -ks https://localhost:8006
curl -ks https://192.168.1.89:8006

An ssh connection attempt also fails with

from physical console of PVE run:
ssh root@192.168.1.89

I suspect the first set of commands will return html pages. The second will allow you to login.
If my suspicion is correct, then you have an issue with a router or some other middleware device on your network, or duplicate IP.
Run a direct cable from your laptop/workstation to PVE host - do you still have any issues?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[mr44er]

If my suspicion is correct,

Looks like there is SSL interception between.

no matching key exchange method found. Their offer: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1 should not happen on both machines updated and patched.

And yes, please try a direct cable connection. If SSL interception does not ring a bell, then there is something nasty going on.

 

[robb01]

Thanks for the replies.
"When you are still unable to connect, from physical console of the PVE run:" - I usually am running the device headless but have hooked up a monitor and keyboard to try this. Both curl cases suggested just give html response for the page. Same result when I do this in the node pve shell.

Running the ssh command results in the Bad SSH2 cipher response.
BTW I had to update the .ssh/config file to

Ciphers aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,KexAlgorithms +diffie-hellman-group1-sha1

to be able to ssh connect.

I have just direct-connected to the router bypassing an 8-port tp-link switch TL-SG1008D so I need to give that some time to see if I lose the connection. Looking ok ATM.

 

[robb01]

Hmm. Still getting "Connection error - server offline?" with the direct cable connection even after a systemctl restart networking.
I am open to suggestion about the router but how to diagnose what's going on there?
Looking at the router's web interface I see that the PM server is getting an IPv6 address as well. Could this be what is timing out and closing the connection?

tail -25 /var/log/syslog

Dec 24 13:02:59 pve chronyd[739]: Source 27.124.125.251 online
Dec 24 13:02:59 pve systemd[1]: Reloading Postfix Mail Transport Agent (instance -).
Dec 24 13:02:59 pve postfix/postfix-script[252217]: refreshing the Postfix mail system
Dec 24 13:02:59 pve postfix/master[886]: reload -- version 3.5.17, configuration /etc/postfix
Dec 24 13:02:59 pve systemd[1]: Reloaded Postfix Mail Transport Agent (instance -).
Dec 24 13:02:59 pve systemd[1]: Reloading Postfix Mail Transport Agent.
Dec 24 13:02:59 pve systemd[1]: Reloaded Postfix Mail Transport Agent.
Dec 24 13:02:59 pve systemd[1]: Finished Network initialization.
Dec 24 13:03:02 pve kernel: [52542.134089] vmbr0: port 1(eno1) entered forwarding state
Dec 24 13:03:07 pve chronyd[739]: Selected source 162.159.200.1 (2.debian.pool.ntp.org)
Dec 24 13:03:07 pve systemd[1]: session-290.scope: Succeeded.
Dec 24 13:03:07 pve pvedaemon[972]: <root@pam> end task UPID:pve:0003D811:0050230A:63A65DBD:vncshell::root@pam: OK
Dec 24 13:03:09 pve pvedaemon[971]: <root@pam> successful auth for user 'root@pam'
Dec 24 13:03:16 pve pvedaemon[971]: <root@pam> successful auth for user 'root@pam'
Dec 24 13:03:20 pve pvedaemon[252356]: starting termproxy UPID:pve:0003D9C4:005033B7:63A65DE8:vncshell::root@pam:
Dec 24 13:03:20 pve pvedaemon[970]: <root@pam> starting task UPID:pve:0003D9C4:005033B7:63A65DE8:vncshell::root@pam:
Dec 24 13:03:20 pve pvedaemon[972]: <root@pam> successful auth for user 'root@pam'
Dec 24 13:03:20 pve systemd[1]: Started Session 291 of user root.
Dec 24 13:05:06 pve pveproxy[251862]: problem with client ::ffff:192.168.1.39; No route to host
Dec 24 13:05:06 pve systemd[1]: session-281.scope: Succeeded.
Dec 24 13:05:06 pve pvedaemon[972]: <root@pam> end task UPID:pve:0003C5CB:004EA708:63A659F0:vncshell::root@pam: OK
Dec 24 13:11:10 pve pvedaemon[972]: <root@pam> starting task UPID:pve:0003E2FB:0050EB5F:63A65FBE:vncshell::root@pam:
Dec 24 13:11:10 pve pvedaemon[254715]: starting termproxy UPID:pve:0003E2FB:0050EB5F:63A65FBE:vncshell::root@pam:
Dec 24 13:11:10 pve pvedaemon[970]: <root@pam> successful auth for user 'root@pam'
Dec 24 13:11:10 pve systemd[1]: Started Session 295 of user root.

although there is a line saying "problem with client"

 

[robb01]

I see the following on the router:-

and also

 

[robb01]

Re-assigned the IP address. Looking good.

 

[robb01]

I am marking this as solved. IP address conflict. Thanks for your people guys.