No space left on device and can't login to web ui

[Gabeben]

My proxmox server is full, which as a result it has automatically locked the web ui disabling access to it.

My plan is to use an external hard drive to move some of my virtual machines off of the server freeing up some space.

I also need to figure aside from the virtual machines what is eating up all my space. I believe it is logs but I don't know how to get rid of them.

 

[bbgeek17]

Can you access the PVE via CLI, ie SSH or physical console?

If you can, run "df -h", followed by "du -h -d1 -x /".
That should give you an idea where space is being used. For example, if you see /var as an abuser, you can repeat the command as following:
du -h -d1 -x /var

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Gabeben]

look like /dev/mapper/pve-root is full

what can I do about this?

 

[bbgeek17]

Have you run the commands? Do you understand the output? Can you provide the output? Preferably as text, but screenshot will do in worst case.

/dev/mapper/pve-root is what / (root) points to, thats your root filesystem. You can clean things up to free up space. But first you need to find out what is taking the space.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Gabeben]

df -h output:
Filesystem            Size  Used Avail Use% Mounted on
udev                  6.8G     0  6.8G   0% /dev
tmpfs                 1.4G   41M  1.4G   3% /run
/dev/mapper/pve-root   94G   94G     0 100% /
tmpfs                 6.8G   28M  6.8G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
/dev/sdb2            1022M  344K 1022M   1% /boot/efi
/dev/sdc1             4.6T  247M  4.6T   1% /media/pi
/dev/sda1             1.9T  1.3T  599G  68% /mnt/pve/samsungevo
/dev/fuse             128M   88K  128M   1% /etc/pve
tmpfs                 1.4G     0  1.4G   0% /run/user/0

 output of du -h -d1 -x /dev/mapper/pve-root:
0       /dev/mapper/pve-root

 

[Gabeben]

look like the symbolic links to vm disks

total 0
crw------- 1 root root 10, 236 Mar 13 21:37 control
lrwxrwxrwx 1 root root       7 Mar 13 21:37 pve-data -> ../dm-5
lrwxrwxrwx 1 root root       7 Mar 13 21:37 pve-data_tdata -> ../dm-3
lrwxrwxrwx 1 root root       7 Mar 13 21:37 pve-data_tmeta -> ../dm-2
lrwxrwxrwx 1 root root       7 Mar 13 21:37 pve-data-tpool -> ../dm-4
lrwxrwxrwx 1 root root       7 Mar 13 21:37 pve-root -> ../dm-1
lrwxrwxrwx 1 root root       7 Mar 13 21:37 pve-swap -> ../dm-0
lrwxrwxrwx 1 root root       7 Mar 13 21:37 pve-vm--100--disk--0 -> ../dm-6
lrwxrwxrwx 1 root root       7 Mar 13 21:37 pve-vm--100--disk--1 -> ../dm-7
lrwxrwxrwx 1 root root       7 Mar 13 21:37 pve-vm--101--disk--0 -> ../dm-8
lrwxrwxrwx 1 root root       7 Mar 13 21:37 pve-vm--101--disk--1 -> ../dm-9
lrwxrwxrwx 1 root root       8 Mar 13 21:37 pve-vm--101--disk--2 -> ../dm-10
lrwxrwxrwx 1 root root       8 Mar 13 21:37 pve-vm--102--disk--0 -> ../dm-18
lrwxrwxrwx 1 root root       8 Mar 13 21:37 pve-vm--102--disk--1 -> ../dm-17
lrwxrwxrwx 1 root root       8 Mar 13 21:37 pve-vm--102--disk--2 -> ../dm-13
lrwxrwxrwx 1 root root       8 Mar 13 21:37 pve-vm--102--disk--3 -> ../dm-14
lrwxrwxrwx 1 root root       8 Mar 13 21:37 pve-vm--102--disk--4 -> ../dm-16
lrwxrwxrwx 1 root root       8 Mar 13 21:37 pve-vm--102--state--a81923730 -> ../dm-15
lrwxrwxrwx 1 root root       8 Mar 13 21:37 pve-vm--104--disk--0 -> ../dm-11
lrwxrwxrwx 1 root root       8 Mar 13 21:37 pve-vm--104--disk--1 -> ../dm-12

 

[bbgeek17]

Please run the commands exactly as shown in comment #2
du -h -d1 -x /

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Gabeben]

I apologize, did not see that command

4.0K    /media
314M    /boot
36K     /tmp
90G     /var
171M    /root
4.0K    /srv
16K     /lost+found
5.5M    /etc
4.0K    /home
4.2G    /usr
4.0K    /opt
28K     /mnt
94G     /

 

[bbgeek17]

90G /var

there you go, most of your space is used in /var

That should give you an idea where space is being used. For example, if you see /var as an abuser, you can repeat the command as following:
du -h -d1 -x /var

continue adding to path to drill down to where most space has gone. Most likely /var/lib/vz, occupied by VMs, ISO or other files. Be prepared to figure out what you can delete to free up some space.

You can immediately get rid of some system logs by doing:
journalctl --flush --rotate --vacuum-time=1s
journalctl --user --flush --rotate --vacuum-time=1s

You can also do:
rm -rf /var/log/*.gz

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox