Can not login to Proxmox webui or ssh

[insomniac]

Hi,

I'm a Linux newbie so please bear with me...

I'm running a local Proxmox server where all config is pretty much standard out of the box, apart from GPU passthrough IOMMU tweaks and a few days ago I noticed both webui and ssh had stopped working to the Proxmox host and LXC, whereas VMs seemed to be fine.

What I've noticed so far:
- I'm NOT able to SSH from my PC to Proxmox (connection reset by peer)
- I'm NOT able to open the Proxmox webui from my PC
- I'm NOT able to open the webui on 2 VMs
- I'm NOT able to connect to my LXC via ssh but it is up and running (3 games servers in docker and I can connect to those)
- I'm able to ping Proxmox on it's static IP
- I'm able to connect to a guest VM via ssh (and on that guest my docker containers running traefik have access)

 

[Hannes Laimer]

Hey,

if you have direct access to the server could you check that the interface names in /etc/network/interfaces match the ones in the output of ip a. Also, the output of systemctl status pveproxy and systemctl status sshd would be interesting.
The VMs and the host are on vmbr0, right? Does it change anything of you try to SSH into the host from a VM? Does rebooting change anything?

 

[insomniac]

I'm passing through the iGPU so have no console access, but I'll check the outputs of those commands with ssh.

Yes the VMs and host are on vmbr0 and some VMs also on bridge vmbr1 an internal network with mtu=9000 and can communicate normally. However when the webui and ssh stop working I can't access it from the VM.

It takes some time to happen and a reboot fixes it for some time again

auto vmbr0
iface vmbr0 inet static
        address 192.168.100.2/24
        gateway 192.168.100.1
        bridge-ports enp2s0
        bridge-stp off
        bridge-fd 0

2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr0 state UP group default qlen 1000

root@pve:~# systemctl status pveproxy
● pveproxy.service - PVE API Proxy Server
     Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-05-16 12:35:24 AEST; 3h 16min ago
    Process: 1226 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
    Process: 1228 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
   Main PID: 1230 (pveproxy)
      Tasks: 4 (limit: 38131)
     Memory: 179.3M
        CPU: 15.821s
     CGroup: /system.slice/pveproxy.service
             ├─ 1230 pveproxy
             ├─48725 "pveproxy worker"
             ├─49652 "pveproxy worker"
             └─49967 "pveproxy worker"

May 16 15:28:37 pve pveproxy[1230]: starting 1 worker(s)
May 16 15:28:37 pve pveproxy[1230]: worker 48725 started
May 16 15:29:47 pve pveproxy[1232]: worker exit
May 16 15:29:47 pve pveproxy[1230]: worker 1232 finished
May 16 15:29:47 pve pveproxy[1230]: starting 1 worker(s)
May 16 15:29:47 pve pveproxy[1230]: worker 49652 started
May 16 15:31:42 pve pveproxy[1231]: worker exit
May 16 15:31:42 pve pveproxy[1230]: worker 1231 finished
May 16 15:31:42 pve pveproxy[1230]: starting 1 worker(s)
May 16 15:31:42 pve pveproxy[1230]: worker 49967 starte

root@pve:~# systemctl status sshd
● ssh.service - OpenBSD Secure Shell server
     Loaded: loaded (/lib/systemd/system/ssh.service; enabled; preset: enabled)
     Active: active (running) since Thu 2024-05-16 12:35:19 AEST; 3h 16min ago
       Docs: man:sshd(8)
             man:sshd_config(5)
    Process: 964 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
   Main PID: 1010 (sshd)
      Tasks: 1 (limit: 38131)
     Memory: 5.1M
        CPU: 151ms
     CGroup: /system.slice/ssh.service
             └─1010 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"

May 16 12:36:03 pve sshd[1519]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
May 16 12:36:03 pve sshd[1519]: pam_env(sshd:session): deprecated reading of user environment enabled
May 16 15:17:42 pve sshd[31573]: Accepted password for root from 192.168.100.254 port 52858 ssh2
May 16 15:17:42 pve sshd[31573]: pam_unix(sshd:session): session opened for user root(uid=0) by (uid=0)
May 16 15:17:42 pve sshd[31573]: pam_env(sshd:session): deprecated reading of user environment enabled

 

[insomniac]

Seems the Proxmox host is doing something weird... the webui can't connect and my ssh session is still connected however no commands work....





It does respond to ping from another VM though:



This VM has the nvme passed through and doesn't rely on the Proxmox hosts storage so still functions.

The other VMs and LXC can not be reached via ssh.

 

[insomniac]

PVE is installed on an external M.2 SSD connected via USB-C, something is happening with the USB connection and it is dropping off, hence no communication.