Things quickly went catastrophic - No space left on device 99.99% Full

T

Tyson_Sean

New Member
Mar 26, 2024
4
0
1
After having setup a PVE with a couple of test VM's I started seeing the local LVM fill up fast. So I moved the ISO's to another disk, or deleted them all together . Pruned some backups tried to free up space . But it quickly went catastrophic and I had a RunAway issue . other than that, the unit ran fine without any issues for a month.

The VM's themselves are still running fine . I have no Web GUI login ability . and the Proxmox Backup Server 3.1-4 still seems to be working, I'm able to log in via We GUI.

So last night before bed I set the backups to run. this morning was unable to go into GUI / Web Interface . [ ERROR: Login failed. Please try again ] No mater which Realm I chose.So I SSH into it , run service pvedaemon --full-restart & service pvestatd --full-restart & pveproxy --full-restart & Reboot. Still not able to log into GUI.

I can't even install ncdu
root@pve:/# apt install ncdu ... Error! E: Could not create temporary file for /var/cache/apt/pkgcache.bin - mkstemp (28: No space left on device)

1st . I do not know how to safely delete files from CLI via SSH & which files I "can" safely delete.
2nd. This is my first PVE setup . This is all new to me. I have googled some things, and found how to find the large files sys info , etc. But as far as how to remove what, and what is safe to remove, I'm lost .

any detailed explanation on safely removing large files with CLI in SSH would be greatly appreciated . I may need a bit of hand holding guidance per se .

Visuals: https://imgur.com/a/RHpy4ow
Other Possibly Useful Info:



journalctl
Mar 12 00:33:25 pve kernel: Linux version 6.5.11-8-pve (build@proxmox) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (G>
Mar 12 00:33:25 pve kernel: Command line: BOOT_IMAGE=/boot/vmlinuz-6.5.11-8-pve root=/dev/mapper/pve-root ro qui>



root@pve:/dev/mapper# cd /dev/mapper/
root@pve:/dev/mapper# ls
control pve-data_tdata pve-data-tpool pve-swap Synology-Synology Synology-Synology_tmeta Synology-vm--100--disk--0 Synology-vm--101--disk--0
pve-data pve-data_tmeta pve-root pve-vm--104--disk--0 Synology-Synology_tdata Synology-Synology-tpool Synology-vm--100--disk--1


root@pve:/# apt install ncdu
Reading package lists... Error!
E: Could not create temporary file for /var/cache/apt/pkgcache.bin - mkstemp (28: No space left on device)
E: The package lists or status file could not be parsed or opened.



root@pve:/# systemctl status pveproxy.service
● pveproxy.service - PVE API Proxy Server
Loaded: loaded (/lib/systemd/system/pveproxy.service; enabled; preset: enabled)
Active: active (running) since Tue 2024-03-26 08:30:32 EDT; 2min 49s ago
Process: 1747 ExecStartPre=/usr/bin/pvecm updatecerts --silent (code=exited, status=0/SUCCESS)
Process: 1749 ExecStart=/usr/bin/pveproxy start (code=exited, status=0/SUCCESS)
Main PID: 1752 (pveproxy)
Tasks: 4 (limit: 28552)
Memory: 158.8M
CPU: 3.986s
CGroup: /system.slice/pveproxy.service
├─1752 pveproxy
├─2122 "pveproxy worker"
├─2152 "pveproxy worker"
└─2186 "pveproxy worker"

Mar 26 08:32:25 pve pveproxy[1752]: starting 1 worker(s)
Mar 26 08:32:25 pve pveproxy[1752]: worker 2152 started
Mar 26 08:32:34 pve pveproxy[2123]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl>
Mar 26 08:32:34 pve pveproxy[2123]: error writing access log
Mar 26 08:32:38 pve pveproxy[2123]: worker exit
Mar 26 08:32:38 pve pveproxy[1752]: worker 2123 finished
Mar 26 08:32:38 pve pveproxy[1752]: starting 1 worker(s)
Mar 26 08:32:38 pve pveproxy[1752]: worker 2186 started
Mar 26 08:32:41 pve pveproxy[2122]: Warning: unable to close filehandle GEN5 properly: No space left on device at /usr/share/perl>
Mar 26 08:32:41 pve pveproxy[2122]: error writing access log
...skipping...



root@pve:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 12G 0 12G 0% /dev
tmpfs 2.4G 1.4M 2.4G 1% /run
/dev/mapper/pve-root 12G 12G 272K 100% /
tmpfs 12G 40M 12G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
efivarfs 320K 66K 250K 21% /sys/firmware/efi/efivars
/dev/nvme1n1p2 1022M 12M 1011M 2% /boot/efi
/dev/sdd1 457G 288M 434G 1% /mnt/pve/WD_USB_500GB
/dev/sdb1 298G 94G 205G 32% /mnt/pve/WD_320GB
VM_Storage16GB 16G 128K 16G 1% /VM_Storage16GB
SynologyDisk2 159G 128K 159G 1% /SynologyDisk2
SynologyDisk1 159G 128K 159G 1% /SynologyDisk1
/dev/fuse 128M 24K 128M 1% /etc/pve
tmpfs 2.4G 0 2.4G 0% /run/user/0
 
12GB root partition is very easy to fill up, it should be enough, but with little experience its easy to make mistakes.

You dont need to install anything to track down your space.
Unmount each of your external disks: umount /[path]
Run "du -h -d1 -x /" - examine the output. If you see anything under "/[path]" you previously unmounted delete it all "rm -rf /[path]/*"
Vacuum your log to get a short relieve with a bit of space:
journalctl --flush --rotate --vacuum-time=1s and journalctl --user --flush --rotate --vacuum-time=1s

Follow the crumbs in the "du" output, if "/var" is showing a lot of occupied space, re-run du: du -h -d1 -x /var

good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Hey, thanks for the quick reply . this sounds like a great way to clear some room!
Question 1.

bbgeek17 said:
Unmount each of your external disks: umount /[path]
Click to expand...
  • Do I do the from SSH Root?
  • /
  • /mnt
  • /dev
  • ?
While I understand English very well and the words you have typed, I'm very much a GUI guy.
Little to no experience in SSH & Terminal . I'm a sign maker and graphic designer . everything I do is point/click/push the mouse around the keyboard.

Thanks for your time . Much appreciated . Please be patient with my navigation .
 

Attachments

  • Screenshot 2024-03-26 at 8.10.56 PM.png
    Screenshot 2024-03-26 at 8.10.56 PM.png
    268.4 KB · Views: 9
  • Screenshot 2024-03-26 at 8.33.58 PM.png
    Screenshot 2024-03-26 at 8.33.58 PM.png
    243.5 KB · Views: 9
Last edited:
Tyson_Sean said:
While I understand English very well and the words you have typed, I'm very much a GUI guy.
Click to expand...
Unfortunately you glanced over some pertinent information and veered a bit off track.
bbgeek17 said:
Unmount each of your external disks
Click to expand...
The critical word was "external". In your case those are:
Tyson_Sean said:
/dev/sdd1 457G 288M 434G 1% /mnt/pve/WD_USB_500GB
/dev/sdb1 298G 94G 205G 32% /mnt/pve/WD_320GB
VM_Storage16GB 16G 128K 16G 1% /VM_Storage16GB
SynologyDisk2 159G 128K 159G 1% /SynologyDisk2
SynologyDisk1 159G 128K 159G 1% /SynologyDisk1
Click to expand...
You did not need to unmount root devices or special devices. Luckily the system did not let you.

Its much easier to work with text than screenshots. Next time use the CODE button of the edit box and paste the text output there.
The initial df output shows that you have 2.3G in /usr - thats your OS, nothing you can do about it.
The remaining space is occupied by /var - 9.6GB.
Combined, that pretty much the entirety of your 12GB root partition.

As you correctly drilled down the space is taken by data in /var/tmp and is probably a temp ISO file your tried to upload. Examine the location with "ls -al /var/tmp" and remove the stale data.

Good luck

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox
 
Thank You Kind Human .
I managed to get somewhere . I think .

There were 3 ISO attempted uploads that look as if they failed. (Did not = Sum of original KB)
ISO storage was moved off of local soon after setup .
Assumed Proxmox would delete the breadcrumbs of failed uploads. (Probably a cron for that)
In the web UI local(pve)/ISO_Images was empty . So was backup . So was CT Templates .
I really didn't know where the disk space was going.

VM's are back up and running . local (pve) is now 25%




code_language.shell: 
........
root@pve:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   12G     0   12G   0% /dev
tmpfs                 2.4G   73M  2.3G   4% /run
/dev/mapper/pve-root   12G   12G   20K 100% /
tmpfs                  12G   43M   12G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
efivarfs              320K   66K  250K  21% /sys/firmware/efi/efivars
/dev/nvme1n1p2       1022M   12M 1011M   2% /boot/efi
/dev/sdb1             298G  105G  194G  36% /mnt/pve/WD_320GB
SynologyDisk2         159G  128K  159G   1% /SynologyDisk2
SynologyDisk1         159G  128K  159G   1% /SynologyDisk1
/dev/fuse             128M   24K  128M   1% /etc/pve
tmpfs                 2.4G     0  2.4G   0% /run/user/0
VM_Storage16GB         16G  128K   16G   1% /VM_Storage16GB




root@pve:/# umount /dev/sdb1
umount: /mnt/pve/WD_320GB: target is busy.
root@pve:/# umount /VM_Storage16GB
root@pve:/# umount VM_Storage16GB
root@pve:/# umount /SynologyDisk2
root@pve:/# umount /SynologyDisk1



 
root@pve:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   12G     0   12G   0% /dev
tmpfs                 2.4G   73M  2.3G   4% /run
/dev/mapper/pve-root   12G   12G   20K 100% /
tmpfs                  12G   43M   12G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
efivarfs              320K   66K  250K  21% /sys/firmware/efi/efivars
/dev/nvme1n1p2       1022M   12M 1011M   2% /boot/efi
/dev/sdb1             298G  105G  194G  36% /mnt/pve/WD_320GB
/dev/fuse             128M   24K  128M   1% /etc/pve
tmpfs                 2.4G     0  2.4G   0% /run/user/0
VM_Storage16GB         16G  128K   16G   1% /VM_Storage16GB
SynologyDisk1         159G  128K  159G   1% /SynologyDisk1
SynologyDisk2         159G  128K  159G   1% /SynologyDisk2




root@pve:/# umount /dev/sdb1
umount: /mnt/pve/WD_320GB: target is busy.
root@pve:/# sudo umount /dev/sdb1
-bash: sudo: command not found
root@pve:/# umount /dev/sdb1
umount: /mnt/pve/WD_320GB: target is busy.



__________________________________________________________________________________________#  SHUTDOWN PBS VM VIA WEB GUI   .  SO I COULD :
umount: /mnt/pve/WD_320GB
root@pve:/# umount /dev/sdb1


root@pve:/# du -h -d1 -x /
94M    /boot
9.6G    /var
0    /mnt
3.9M    /etc
0    /home
0    /media
0    /opt
52K    /root
0    /srv
128K    /tmp
2.3G    /usr
12G    /




root@pve:/# ls -al /var/tmp
total 9341104
drwxrwxrwt  4 root     root           4096 Mar 26 22:04 .
drwxr-xr-x 11 root     root            139 Feb  5 11:38 ..
-rw-r--r--  1 root     root             16 Mar 25 23:58 pve-reserved-ports
-rw-------  1 www-data www-data     999424 Mar 24 23:43 pveupload-2929f878978a9b80cdc2943464792e7c
-rw-------  1 www-data www-data     163840 Mar 24 23:43 pveupload-3fdd392080b87fcc39b3a3d6ff2405ef
-rw-------  1 www-data www-data 5399396352 Mar 24 23:43 pveupload-ba9672893ce6e238f4d2e726e7884863
-rw-------  1 www-data www-data 4164714496 Mar 17 22:48 pveupload-d3496ce23492c94798059ea2163299ca
drwx------  3 root     root             17 Mar 26 22:04 systemd-private-038ca2fe044040b484bb2b341857af68-chrony.service-JnIFsB
drwx------  3 root     root             17 Mar 26 22:04 systemd-private-038ca2fe044040b484bb2b341857af68-systemd-logind.service-AHHtAO



root@pve:/# rm -f /var/tmp/pveupload-d3496ce23492c94798059ea2163299ca
root@pve:/# rm -f /var/tmp/pveupload-ba9672893ce6e238f4d2e726e7884863
root@pve:/# rm -f /var/tmp/pveupload-3fdd392080b87fcc39b3a3d6ff2405ef



 
root@pve:/# ls -al /var/tmp
total 980
drwxrwxrwt  4 root     root        246 Mar 27 12:06 .
drwxr-xr-x 11 root     root        139 Feb  5 11:38 ..
-rw-r--r--  1 root     root         16 Mar 25 23:58 pve-reserved-ports
-rw-------  1 www-data www-data 999424 Mar 24 23:43 pveupload-2929f878978a9b80cdc2943464792e7c
drwx------  3 root     root         17 Mar 26 22:04 systemd-private-038ca2fe044040b484bb2b341857af68-chrony.service-JnIFsB
drwx------  3 root     root         17 Mar 26 22:04 systemd-private-038ca2fe044040b484bb2b341857af68-systemd-logind.service-AHHtAO





root@pve:/# ls -al /var/tmp
total 4
drwxrwxrwt  4 root root 196 Mar 27 12:06 .
drwxr-xr-x 11 root root 139 Feb  5 11:38 ..
-rw-r--r--  1 root root  16 Mar 25 23:58 pve-reserved-ports
drwx------  3 root root  17 Mar 26 22:04 systemd-private-038ca2fe044040b484bb2b341857af68-chrony.service-JnIFsB
drwx------  3 root root  17 Mar 26 22:04 systemd-private-038ca2fe044040b484bb2b341857af68-systemd-logind.service-AHHtAO





root@pve:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   12G     0   12G   0% /dev
tmpfs                 2.4G   73M  2.3G   4% /run
/dev/mapper/pve-root   12G  3.1G  9.0G  26% /
tmpfs                  12G   40M   12G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
efivarfs              320K   66K  250K  21% /sys/firmware/efi/efivars
/dev/nvme1n1p2       1022M   12M 1011M   2% /boot/efi
/dev/fuse             128M   24K  128M   1% /etc/pve
tmpfs                 2.4G     0  2.4G   0% /run/user/0
VM_Storage16GB         16G  128K   16G   1% /VM_Storage16GB
SynologyDisk1         159G  128K  159G   1% /SynologyDisk1
SynologyDisk2         159G  128K  159G   1% /SynologyDisk2







root@pve:/# journalctl --user --flush --rotate --vacuum-time=1s
No journal files were found.
Deleted archived journal /run/log/journal/2b14c9decf9343978b87f0fa8ed59910/system@1a6e5d1fdc1a436094c13d511f2485dd-0000000000014fe8-000614a6ad84b8ac.journal (876.0K).
Vacuuming done, freed 876.0K of archived journals from /run/log/journal/2b14c9decf9343978b87f0fa8ed59910.
Vacuuming done, freed 0B of archived journals from /var/log/journal.
Vacuuming done, freed 0B of archived journals from /run/log/journal.
Vacuuming done, freed 0B of archived journals from /var/log/journal/2b14c9decf9343978b87f0fa8ed59910.





root@pve:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   12G     0   12G   0% /dev
tmpfs                 2.4G  9.3M  2.4G   1% /run
/dev/mapper/pve-root   12G  3.0G  9.0G  25% /
tmpfs                  12G   40M   12G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
efivarfs              320K   66K  250K  21% /sys/firmware/efi/efivars
/dev/nvme1n1p2       1022M   12M 1011M   2% /boot/efi
/dev/fuse             128M   24K  128M   1% /etc/pve
tmpfs                 2.4G     0  2.4G   0% /run/user/0
VM_Storage16GB         16G  128K   16G   1% /VM_Storage16GB
SynologyDisk1         159G  128K  159G   1% /SynologyDisk1
SynologyDisk2         159G  128K  159G   1% /SynologyDisk2
root@pve:/#










root@pve:/#
root@pve:/# lsblk
NAME                              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda                                 8:0    0   2.7T  0 disk
├─sda1                              8:1    0   2.7T  0 part
└─sda9                              8:9    0     8M  0 part
sdb                                 8:16   0 298.1G  0 disk
└─sdb1                              8:17   0 298.1G  0 part
sdc                                 8:32   0   2.7T  0 disk
├─sdc1                              8:33   0   2.7T  0 part
└─sdc9                              8:41   0     8M  0 part
sdd                                 8:48   1     0B  0 disk
zd0                               230:0    0   2.4T  0 disk
├─zd0p1                           230:1    0     8G  0 part
├─zd0p2                           230:2    0     2G  0 part
└─zd0p5                           230:5    0   2.4T  0 part
zd16                              230:16   0   2.4T  0 disk
├─zd16p1                          230:17   0     8G  0 part
├─zd16p2                          230:18   0     2G  0 part
└─zd16p5                          230:21   0   2.4T  0 part
zd32                              230:32   0    16G  0 disk
zd48                              230:48   0     5G  0 disk
nvme0n1                           259:0    0 476.9G  0 disk
├─nvme0n1p1                       259:1    0   260M  0 part
├─nvme0n1p2                       259:2    0    16M  0 part
├─nvme0n1p3                       259:3    0 475.7G  0 part
└─nvme0n1p4                       259:4    0   995M  0 part
nvme1n1                           259:5    0 238.5G  0 disk
├─nvme1n1p1                       259:6    0  1007K  0 part
├─nvme1n1p2                       259:7    0     1G  0 part /boot/efi
├─nvme1n1p3                       259:8    0 128.7G  0 part
│ ├─pve-swap                      252:0    0    12G  0 lvm  [SWAP]
│ ├─pve-root                      252:1    0    12G  0 lvm  /
│ ├─pve-data_tmeta                252:2    0     1G  0 lvm
│ │ └─pve-data-tpool              252:6    0    10G  0 lvm
│ │   ├─pve-data                  252:10   0    10G  1 lvm
│ │   └─pve-vm--104--disk--0      252:11   0    16G  0 lvm
│ └─pve-data_tdata                252:4    0    10G  0 lvm
│   └─pve-data-tpool              252:6    0    10G  0 lvm
│     ├─pve-data                  252:10   0    10G  1 lvm
│     └─pve-vm--104--disk--0      252:11   0    16G  0 lvm
├─nvme1n1p4                       259:9    0  38.5G  0 part
│ ├─Synology-Synology_tmeta       252:3    0     1G  0 lvm
│ │ └─Synology-Synology-tpool     252:7    0  36.3G  0 lvm
│ │   ├─Synology-Synology         252:8    0  36.3G  1 lvm
│ │   ├─Synology-vm--100--disk--0 252:9    0     1G  0 lvm
│ │   ├─Synology-vm--100--disk--1 252:12   0    40G  0 lvm
│ │   └─Synology-vm--101--disk--0 252:13   0    10G  0 lvm
│ └─Synology-Synology_tdata       252:5    0  36.3G  0 lvm
│   └─Synology-Synology-tpool     252:7    0  36.3G  0 lvm
│     ├─Synology-Synology         252:8    0  36.3G  1 lvm
│     ├─Synology-vm--100--disk--0 252:9    0     1G  0 lvm
│     ├─Synology-vm--100--disk--1 252:12   0    40G  0 lvm
│     └─Synology-vm--101--disk--0 252:13   0    10G  0 lvm
├─nvme1n1p5                       259:10   0  39.1G  0 part
├─nvme1n1p6                       259:11   0  15.6G  0 part
└─nvme1n1p7                       259:12   0  15.6G  0 part
root@pve:/#
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back