unable to access webgui

nerd

Member
Nov 4, 2021
7
0
6
46
After a power loss I am no longer able to access the web GUI.
VIMs are up and running, SSH access is available, but GUI will not respond.
Server has 2 IP addresses. Eg. 10.1.1.5 with default gateway and eg. 10.10.0.5. I am trying to connect from 10.10.0.x myself so a direct layer 2 connection.
Connecting to 10.10.1.5 address (through FW) also times out.

Bash:
# pveversion -v
proxmox-ve: 7.2-1 (running kernel: 5.15.74-1-pve)
pve-manager: 7.2-11 (running version: 7.2-11/b76d3178)
pve-kernel-5.15: 7.2-14
pve-kernel-helper: 7.2-14
pve-kernel-5.13: 7.1-9
pve-kernel-5.11: 7.0-10
pve-kernel-5.15.74-1-pve: 5.15.74-1
pve-kernel-5.15.35-3-pve: 5.15.35-6
pve-kernel-5.15.35-1-pve: 5.15.35-3
pve-kernel-5.15.30-2-pve: 5.15.30-3
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-1-pve: 5.13.19-3
pve-kernel-5.11.22-7-pve: 5.11.22-12
pve-kernel-5.11.22-5-pve: 5.11.22-10
pve-kernel-5.11.22-4-pve: 5.11.22-9
ceph-fuse: 15.2.14-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-4
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-4
libpve-guest-common-perl: 4.1-4
libpve-http-server-perl: 4.1-5
libpve-storage-perl: 7.2-10
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
proxmox-backup-client: 2.2.7-1
proxmox-backup-file-restore: 2.2.7-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-2
pve-container: 4.2-3
pve-docs: 7.2-2
pve-edk2-firmware: 3.20220526-1
pve-firewall: 4.2-6
pve-firmware: 3.5-6
pve-ha-manager: 3.4.0
pve-i18n: 2.7-2
pve-qemu-kvm: 7.0.0-4
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-5
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.8.0~bpo11+2
vncterm: 1.7-1
zfsutils-linux: 2.1.6-pve1

Browsers throw up "connection has timed out". Tested firefox and chrome. Also tested incognito mode to rule out cache.
Running # service pveproxy restart does not help.
Restarting the server systemctl reboot did not help either
journalctl -f does not give any useful info
It seems some commands hang up the SSH session too. Eg. I simply can not run cat /etc/network/interfaces without locking up the SSH session? If I try using nano to view the same / cat /etc/network/interfaces it says [ Read 81 lines ] but displays less (how many varies) than that and again freezes the SSH session. Doing a more /etc/network/interfaces does give the correct and complete output... WEIRD.
Looking closer at this output I can see nothing wrong with it.

Bash:
nmap -p 8006 10.10.0.5
Starting Nmap 7.80 ( https://nmap.org ) at 2022-12-08 06:08 CET
Nmap scan report for 10.10.0.5
Host is up (0.000035s latency).

PORT     STATE SERVICE
8006/tcp open  wpl-analytics

Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds
Likewise for 10.10.1.5.

Well.. I am no ProxmoxVE specialist, but the above seems pretty darn weird to me.
Hopefully someone here can help me fix this issue?
 
# EDIT #
Seeing as the curl gave SOME output, I got my laptop and tried from there. My apologies, because with that PC I am able to access the web GUI, have zero issues with the different commands locking up the SSH and everything seems to work just fine.
So, the problem seems to be with my main PC instead of the proxmoxVE install. Don't understand the bizarre symptoms I get but unless someone has a good idea, I'll take that offline.

Thank you @bbgeek17 for your effort. I would not have made this 'discovery' without it!
# END EDIT #


FYI: I have since ran # apt install proxmox-ve and # apt dist-upgrade -y which ran without issue.
Now running
# pveversion -v proxmox-ve: 7.3-1 (running kernel: 5.15.74-1-pve)

Confident I do not have any IP conflicts. ARP entries remain the same
All 3 curl command hang up the SSH session 100% of the times I tried. :confused:
When I run those commands from another linux machine I get the following result for both IP addresses.

Bash:
# curl -sk https://10.10.0.5:8006
<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no">
    <title>pve - Proxmox Virtual Environment</title>
    <link rel="icon" sizes="128x128" href="/pve2/images/logo-128.png" />
    <link rel="apple-touch-icon" sizes="128x128" href="/pve2/images/logo-128.png" />
    <link rel="stylesheet" type="text/css" href="/pve2/ext6/theme-crisp/resources/theme-crisp-all.css?ver=7.0.0" />
    <link rel="stylesheet" type="text/css" href="/pve2/ext6/crisp/resources/charts-all.css?ver=7.0.0" />
    <link rel="stylesheet" type="text/css" href="/pve2/fa/css/font-awesome.css" />
    <link rel="stylesheet" type="text/css" href="/pve2/css/ext6-pve.css?ver=7.3-3" />
    <link rel="stylesheet" type="text/css" href="/pwt/css/ext6-pmx.css?ver=3.5.3" />
 
    <script type='text/javascript'>function gettext(buf) { return buf; }</script>
 
    <script type="text/javascript" src="/pve2/ext6/ext-all.js?ver=7.0.0"></script>
    <script type="text/javascript" src="/pve2/ext6/charts.js?ver=7.0.0"></script>
 
    <script type="text/javascript" src="/pve2/js/u2f-api.js"></script>
    <script type="text/javascript" src="/qrcode.min.js"></script>
    <script type="text/javascript">
    Proxmox = {
        Setup: { auth_cookie_name: 'PVEAuthCookie' },
        defaultLang: 'en',
        NodeName: 'pve',
        UserName: '',
        CSRFPreventionToken: 'null'
    };
    </script>
    <script type="text/javascript" src="/proxmoxlib.js?ver=3.5.3"></script>
    <script type="text/javascript" src="/pve2/js/pvemanagerlib.js?ver=7.3-3"></script>
    <script type="text/javascript" src="/pve2/ext6/locale/locale-en.js?ver=7.0.0"></script>

    <script type="text/javascript">
    if (typeof(PVE) === 'undefined') PVE = {};
    Ext.History.fieldid = 'x-history-field';
    Ext.onReady(function() { Ext.create('PVE.StdWorkspace');});
    </script>

  </head>
  <body>
    <!-- Fields required for history management -->
    <form id="history-form" class="x-hidden">
    <input type="hidden" id="x-history-field"/>
    </form>
  </body>
</html>
 
Last edited:
The MAC address does not need to change to highlight IP conflict, the switch (if there is one in place) could be sending traffic to wrong ports.

You have a TCP layer issue, based on the information you presented. No amount of reinstalling userland packages is going to solve it.
The IP conflict could be for your _client_, if another host on the network operates normally.
You said your windows(?) client is unable to access GUI and looses SSH connectivity, but apparently another host on the network is fine? That points to a client issue.

Run continues ping from multiple clients with larger than standard packets (less then current MTU), does everything work as expected except your original client?


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: nerd
ARP entries on the FW (DGW for every VLAN) should show duplicate IP addresses in the ARP table. Still, to make sure I tried with a different IP address on my windows client to rule that out entirely.
And yes, completely agree this is a client issue as I mentioned in the edit.
Still.. can't say I understand what is going wrong on my Win client machine where I have zero issues other than with that single ProxmoxVE server. Even other machines in the same VLAN I can access without problem.
And why would certain command fail hang the SSH session? How does that come in to play?

Mmmmm.. seems my proxmoxVE server has an issue with my jumbo frames. When I disable jumbo frames on my Windows client I am suddenly able to reach it without problems.
 
google: where should jumbo frames be enabled
Code:
Before you enable jumbo frames on your storage system , jumbo frames must be enabled for the switch ports, client interfaces, 
and intermediate routers on the network. If your storage system and the client are on different subnets, the next-hop router must be
 configured for jumbo frames.

https://library.netapp.com/ecmdocs/ECMP1368834/html/GUID-D3AB10A1-D15A-490D-8DCE-34BE73C3DACF.htm

i.e. JF must be enabled _everywhere_


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
  • Like
Reactions: nerd
google: where should jumbo frames be enabled
Code:
Before you enable jumbo frames on your storage system , jumbo frames must be enabled for the switch ports, client interfaces,
and intermediate routers on the network. If your storage system and the client are on different subnets, the next-hop router must be
 configured for jumbo frames.

https://library.netapp.com/ecmdocs/ECMP1368834/html/GUID-D3AB10A1-D15A-490D-8DCE-34BE73C3DACF.htm

i.e. JF must be enabled _everywhere_


Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

Yep, I know. It was enabled everywhere but not with the same values as my client.
Changing the proxmoxve interface mtu's from 9216 to 9014 + reboot seems to have resolved the issue... finally.

Thank you for your assistance and perseverance. Truly appreciated!
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!