[SOLVED] pve local becomes full in a short period

muyue · Jun 9, 2022

Hi all,
I have PVE installed on a 500GB ssd with swap set to be 0 in installer. However, after a week, local in the node becomes full. What gives?
This local has 100GB but the usage reaches nearly 100%. I have never mounted any network storage to PVE nor made backup of VMs on PVE. However, I have a 8TB HDD and another 500GB SSD passthoughed to different VMs using methods provided in https://pve.proxmox.com/wiki/Passthrough_Physical_Disk_to_Virtual_Machine_(VM)
I am using PVE 7.2-4.
After trying to search for answers, I tried the following commands:

Code:

root@pve:~# du -sx /*
0       /bin
172328  /boot
0       /dev
5172    /etc
114116  /home
0       /lib
0       /lib32
0       /lib64
0       /libx32
16      /lost+found
4       /media
4       /mnt
4       /opt
du: cannot access '/proc/41564': No such file or directory
du: cannot access '/proc/41565/task/41565/fd/4': No such file or directory
du: cannot access '/proc/41565/task/41565/fdinfo/4': No such file or directory
du: cannot access '/proc/41565/fd/3': No such file or directory
du: cannot access '/proc/41565/fdinfo/3': No such file or directory
0       /proc
60      /root
1636    /run
0       /sbin
4       /srv
0       /sys
40      /tmp
3377188 /usr
16286732        /var

Code:

root@pve:~# find / -type f -size +1000M
/var/lib/vz/template/iso/manjaro-kde-21.2.6-minimal-220416-linux515.iso
/var/lib/vz/template/iso/ThinPC_110415_EVAL_x86fre.iso
/var/lib/vz/template/iso/Win8.1_English_x64.iso
/var/lib/vz/template/iso/19044.1288.211006-0501.21h2_release_svc_refresh_CLIENT_LTSC_EVAL_x64FRE_en-us.iso
/proc/kcore
find: ‘/proc/42485/task/42485/fdinfo/5’: No such file or directory
find: ‘/proc/42485/fdinfo/6’: No such file or directory

Code:

root@pve:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                  7.7G     0  7.7G   0% /dev
tmpfs                 1.6G  1.6M  1.6G   1% /run
/dev/mapper/pve-root   94G   88G  1.9G  98% /
tmpfs                 7.8G   46M  7.7G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
/dev/nvme1n1p2        511M  328K  511M   1% /boot/efi
/dev/fuse             128M   20K  128M   1% /etc/pve
tmpfs                 1.6G     0  1.6G   0% /run/user/0

Code:

root@pve:~# lvs -a
  LV                       VG  Attr       LSize    Pool Origin                   Data%  Meta%  Move Log Cpy%Sync Convert
  data                     pve twi-aotz-- <346.20g                               11.92  0.88
  [data_tdata]             pve Twi-ao---- <346.20g
  [data_tmeta]             pve ewi-ao----   <3.54g
  [lvol0_pmspare]          pve ewi-------   <3.54g
  root                     pve -wi-ao----   96.00g
  snap_vm-104-disk-0_clean pve Vri---tz-k   32.00g data
  vm-100-disk-0            pve Vwi-aotz--   32.00g data                          71.23
  vm-100-disk-1            pve Vwi-aotz--    4.00m data                          14.06
  vm-100-disk-2            pve Vwi-a-tz--    1.00g data                          0.00
  vm-101-disk-0            pve Vwi-a-tz--    4.00m data                          14.06
  vm-102-disk-0            pve Vwi-a-tz--   60.00g data                          11.66
  vm-103-disk-0            pve Vwi-a-tz--    4.00g data                          3.89
  vm-104-disk-0            pve Vwi-a-tz--   32.00g data snap_vm-104-disk-0_clean 32.67

after running

Code:

ncdu -x /

I got the following result

and for ncdu /

I still can't figure out what takes up so much space.
Any help would be appreciate.

oguz · Jun 9, 2022

hi,

muyue said:
I have PVE installed on a 500GB ssd with swap set to be 0 in installer. However, after a week, local in the node becomes full. What gives?

as you noticed, you only have 100G for the root partition itself.

what does lsblk -f look like?

your /var/ folder is taking around 15.5G, that probably includes the ISO files inside /var/lib/vz/ directory. i would take a look here to see if there's something that can be moved/deleted.

i would also check the /var/log directory for any big files (auth.log tends to get big if your servers SSH port is exposed, because of many login tries) and delete/move them if unnecessary.

running fstrim -av could also help getting some space back.

hope this helps

bbgeek17 · Jun 9, 2022

I recommend you to read through this https://unix.stackexchange.com/ques...rge-files-that-are-open-but-have-been-deleted
it could be contributing to your issue

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

muyue · Jun 9, 2022

Hi oguz, thanks for your reply
I know that local has only 100GB, but one thing I don't understand is that the sum of numbers shown in ncdu is far less than the usage reported in PVE web GUI

oguz said:
what does lsblk -f look like?

Code:

root@pve:~# lsblk -f
NAME                         FSTYPE      FSVER    LABEL UUID                                   FSAVAIL FSUSE% MOUNTPOINT
sda
└─sda1                       btrfs                      8fb74b3c-0655-439f-b3b3-bf27f5842dea
nvme0n1
├─nvme0n1p1                  vfat        FAT32          3A5A-04C7
├─nvme0n1p2
└─nvme0n1p3                  ntfs                       108C9E3E8C9E1DF2
nvme1n1
├─nvme1n1p1
├─nvme1n1p2                  vfat        FAT32          5D14-3E76                               510.7M     0% /boot/efi
└─nvme1n1p3                  LVM2_member LVM2 001       Wfc370-FgnY-KbNQ-MHxq-eVE9-4M5B-ImNYV4
  ├─pve-root                 ext4        1.0            6accb1aa-9f88-4c6d-acaa-0dd1295e9ba2      1.9G    93% /
  ├─pve-data_tmeta
  │ └─pve-data-tpool
  │   ├─pve-data
  │   ├─pve-vm--100--disk--0
  │   ├─pve-vm--100--disk--1
  │   ├─pve-vm--101--disk--0
  │   ├─pve-vm--102--disk--0
  │   ├─pve-vm--100--disk--2
  │   ├─pve-vm--104--disk--0
  │   └─pve-vm--103--disk--0 ext4        1.0            a5fa6cb3-f02b-4f13-9252-e76ffaa78307
  └─pve-data_tdata
    └─pve-data-tpool
      ├─pve-data
      ├─pve-vm--100--disk--0
      ├─pve-vm--100--disk--1
      ├─pve-vm--101--disk--0
      ├─pve-vm--102--disk--0
      ├─pve-vm--100--disk--2
      ├─pve-vm--104--disk--0
      └─pve-vm--103--disk--0 ext4        1.0            a5fa6cb3-f02b-4f13-9252-e76ffaa78307

oguz said:
your /var/ folder is taking around 15.5G, that probably includes the ISO files inside /var/lib/vz/ directory. i would take a look here to see if there's something that can be moved/deleted.

As I uploaded the ISOs using web GUI, so I guess the ISOs are placed in the intended place.

oguz said:
i would also check the /var/log directory for any big files (auth.log tends to get big if your servers SSH port is exposed, because of many login tries) and delete/move them if unnecessary.

I did find a folder call journal which took around 800MB of space.

oguz said:
running fstrim -av could also help getting some space back.

It did help a little bit, I get around 2GB of free space now.

bbgeek17 said:
I recommend you to read through this https://unix.stackexchange.com/ques...rge-files-that-are-open-but-have-been-deleted
it could be contributing to your issue

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

It worked! I reclaimed around 80GB by truncating the logs. Thanks for your suggestion!

Search

Search

[SOLVED] pve local becomes full in a short period

muyue

Member

oguz

Proxmox Retired Staff

bbgeek17

Distinguished Member

muyue

Member

We value your privacy