Suddenly no GUI access

[StanM]

I have three Proxmox servers in a cluster. One has been turned off for a few months. Suddenly, this morning, I don't have browser access to the two which are active. I've restarted both to no effect. One of the servers hosts a virtual Windows machine, which I can access. The other hosts an LXC hosting my Unifi Contoller. I can't reach the Unifi Controller.

I started the third, long dormant, Proxmox server and have Web access to that. The other two show up under Datacenter, but without the green circled checkmark. When I go to the Shell of each of the other two, I receive the message "Connection failed (Error 500: closing file '/var/tmp/pve-reserved-ports.tmp.850' failed - No space left on device". I'm not sure whether that's referring to the third server or the other two.

I do have CLI access. Does anyone have a suggestion of what to do?

 

[2707974]

I have three Proxmox servers in a cluster. One has been turned off for a few months. Suddenly, this morning, I don't have browser access to the two which are active. I've restarted both to no effect. One of the servers hosts a virtual Windows machine, which I can access. The other hosts an LXC hosting my Unifi Contoller. I can't reach the Unifi Controller.

I started the third, long dormant, Proxmox server and have Web access to that. The other two show up under Datacenter, but without the green circled checkmark. When I go to the Shell of each of the other two, I receive the message "Connection failed (Error 500: closing file '/var/tmp/pve-reserved-ports.tmp.850' failed - No space left on device". I'm not sure whether that's referring to the third server or the other two.

I do have CLI access. Does anyone have a suggestion of what to do?

Give us output from command df -h from servers

 

[StanM]

Here is the result from one of the servers. I don't have a way to snip or copy and paste.

Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 9.2M 3.2G 1% /run
/dev/mapper/pve-root 94G 4.4G 85G 5% /
tmpfs 16G 45M 16G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/nvme0n1p2 511M 328K 511M 1% /boot/efi
Common 779G 128K 779G 1% /Common
Common/subvol-112-disk-0 32G 618M 32G 2% /Common/subvol-112-disk-0
Common/subvol-111-disk-0 8G 1.4G 6.7G 18% /Common/subvol-111-disk-0
/dev/fuse 30M 32K 30M 1% /etc/pve
192.168.10.4:/VMs 397G 310G 87G 79% /mnt/pve/ISO_NAS
tpmfs 3.2G 0 3.2G 0% /run/user/0

 

[2707974]

Here is the result from one of the servers. I don't have a way to snip or copy and paste.

Filesystem Size Used Avail Use% Mounted on
udev 16G 0 16G 0% /dev
tmpfs 3.2G 9.2M 3.2G 1% /run
/dev/mapper/pve-root 94G 4.4G 85G 5% /
tmpfs 16G 45M 16G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 16G 0 16G 0% /sys/fs/cgroup
/dev/nvme0n1p2 511M 328K 511M 1% /boot/efi
Common 779G 128K 779G 1% /Common
Common/subvol-112-disk-0 32G 618M 32G 2% /Common/subvol-112-disk-0
Common/subvol-111-disk-0 8G 1.4G 6.7G 18% /Common/subvol-111-disk-0
/dev/fuse 30M 32K 30M 1% /etc/pve
192.168.10.4:/VMs 397G 310G 87G 79% /mnt/pve/ISO_NAS
tpmfs 3.2G 0 3.2G 0% /run/user/0

Go tu console of server, then run commend df-h . Idea is to find some mount point that is used 100 %.
When find that mount point, run du -h --max-depth 1
For example
If output say

/dev/mapper/pve-root   94G            85G            85G          100%              /

run

du -h --max-depth 1 /

You will see size of folder and files .....
Go to location and delete unused files ...

 

[StanM]

Thanks, but I'm not sure how to search for mound points with the problem. When I change directories and run du -h, I always get the same response as I found above, with nothing showing 100%.

 

[StanM]

The message I see on the server that I can still get to on the GUI says: "Error 500: closing file '/var/tmp/pve-reserved-ports.tmp.851' failed - -No space left on device". So on one of the servers that I can't get to on the GUI, in the CLI I went to the folder "/var/tmp" and listed all the files (ls -a). There is a file named "pve-reserved-ports", but nothing with ".tmp.851" appended to the end. The "pve-reserved-ports" file consists of one line that reads "5900 1614381237". I hope this additional information helps.

 

[2707974]

The message I see on the server that I can still get to on the GUI says: "Error 500: closing file '/var/tmp/pve-reserved-ports.tmp.851' failed - -No space left on device". So on one of the servers that I can't get to on the GUI, in the CLI I went to the folder "/var/tmp" and listed all the files (ls -a). There is a file named "pve-reserved-ports", but nothing with ".tmp.851" appended to the end. The "pve-reserved-ports" file consists of one line that reads "5900 1614381237". I hope this additional information helps.

What say output of command

lsof / | grep deleted