Suddenly no GUI access

StanM

Member
Mar 5, 2021
22
0
6
77
I have three Proxmox servers in a cluster. One has been turned off for a few months. Suddenly, this morning, I don't have browser access to the two which are active. I've restarted both to no effect. One of the servers hosts a virtual Windows machine, which I can access. The other hosts an LXC hosting my Unifi Contoller. I can't reach the Unifi Controller.

I started the third, long dormant, Proxmox server and have Web access to that. The other two show up under Datacenter, but without the green circled checkmark. When I go to the Shell of each of the other two, I receive the message "Connection failed (Error 500: closing file '/var/tmp/pve-reserved-ports.tmp.850' failed - No space left on device". I'm not sure whether that's referring to the third server or the other two.

I do have CLI access. Does anyone have a suggestion of what to do?
 
I have three Proxmox servers in a cluster. One has been turned off for a few months. Suddenly, this morning, I don't have browser access to the two which are active. I've restarted both to no effect. One of the servers hosts a virtual Windows machine, which I can access. The other hosts an LXC hosting my Unifi Contoller. I can't reach the Unifi Controller.

I started the third, long dormant, Proxmox server and have Web access to that. The other two show up under Datacenter, but without the green circled checkmark. When I go to the Shell of each of the other two, I receive the message "Connection failed (Error 500: closing file '/var/tmp/pve-reserved-ports.tmp.850' failed - No space left on device". I'm not sure whether that's referring to the third server or the other two.

I do have CLI access. Does anyone have a suggestion of what to do?
Give us output from command df -h from servers
 
Here is the result from one of the servers. I don't have a way to snip or copy and paste.

Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 9.2M 3.2G 1% /run
/dev/mapper/pve-root 94G 4.4G 85G 5% /
tmpfs 16G 45M 16G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/nvme0n1p2 511M 328K 511M 1% /boot/efi
Common 779G 128K 779G 1% /Common
Common/subvol-112-disk-0 32G 618M 32G 2% /Common/subvol-112-disk-0
Common/subvol-111-disk-0 8G 1.4G 6.7G 18% /Common/subvol-111-disk-0
/dev/fuse 30M 32K 30M 1% /etc/pve
192.168.10.4:/VMs 397G 310G 87G 79% /mnt/pve/ISO_NAS
tpmfs 3.2G 0 3.2G 0% /run/user/0
 
Here is the result from one of the servers. I don't have a way to snip or copy and paste.

Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 9.2M 3.2G 1% /run
/dev/mapper/pve-root 94G 4.4G 85G 5% /
tmpfs 16G 45M 16G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/nvme0n1p2 511M 328K 511M 1% /boot/efi
Common 779G 128K 779G 1% /Common
Common/subvol-112-disk-0 32G 618M 32G 2% /Common/subvol-112-disk-0
Common/subvol-111-disk-0 8G 1.4G 6.7G 18% /Common/subvol-111-disk-0
/dev/fuse 30M 32K 30M 1% /etc/pve
192.168.10.4:/VMs 397G 310G 87G 79% /mnt/pve/ISO_NAS
tpmfs 3.2G 0 3.2G 0% /run/user/0

Go tu console of server, then run commend df-h . Idea is to find some mount point that is used 100 %.
When find that mount point, run du -h --max-depth 1
For example
If output say

Code:
/dev/mapper/pve-root   94G            85G            85G          100%              /

run

Code:
du -h --max-depth 1 /

You will see size of folder and files .....
Go to location and delete unused files ...
 
Thanks, but I'm not sure how to search for mound points with the problem. When I change directories and run du -h, I always get the same response as I found above, with nothing showing 100%.
 
The message I see on the server that I can still get to on the GUI says: "Error 500: closing file '/var/tmp/pve-reserved-ports.tmp.851' failed - -No space left on device". So on one of the servers that I can't get to on the GUI, in the CLI I went to the folder "/var/tmp" and listed all the files (ls -a). There is a file named "pve-reserved-ports", but nothing with ".tmp.851" appended to the end. The "pve-reserved-ports" file consists of one line that reads "5900 1614381237". I hope this additional information helps.
 
The message I see on the server that I can still get to on the GUI says: "Error 500: closing file '/var/tmp/pve-reserved-ports.tmp.851' failed - -No space left on device". So on one of the servers that I can't get to on the GUI, in the CLI I went to the folder "/var/tmp" and listed all the files (ls -a). There is a file named "pve-reserved-ports", but nothing with ".tmp.851" appended to the end. The "pve-reserved-ports" file consists of one line that reads "5900 1614381237". I hope this additional information helps.
What say output of command
Code:
lsof / | grep deleted
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!