Proxmox local disk full

fredamn76 · Jul 10, 2022

Hi,
My local proxmox disk is full and I am trying to fix the issue but does not know how to fix it. I've looked in forums but cannot find a solution. Hope someone can help me. I've been able to ssh into my machine but webui does not work.

I ran df -h command (results below) and it shows the /dev/mapper/pve-root is full.

Code:

root@pve:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   20G     0   20G   0% /dev
tmpfs                 4.0G   18M  3.9G   1% /run
/dev/mapper/pve-root   55G   55G     0 100% /
tmpfs                  20G     0   20G   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
/dev/sdc1             5.5T  5.0T  468G  92% /mnt/data_1
tmpfs                 4.0G     0  4.0G   0% /run/user/0
root@pve:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   20G     0   20G   0% /dev
tmpfs                 4.0G   26M  3.9G   1% /run
/dev/mapper/pve-root   55G   55G     0 100% /
tmpfs                  20G     0   20G   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
/dev/sdc1             5.5T  5.0T  468G  92% /mnt/data_1
tmpfs                 4.0G     0  4.0G   0% /run/user/0

Code:

root@pve:~# fdisk -l
Disk /dev/sda: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: WD1004FBYZ-23YC
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 69047576-1C91-5F4E-AF61-9E3C64C3B0FF

Device          Start        End    Sectors   Size Type
/dev/sda1        2048 1953507327 1953505280 931.5G Solaris /usr & Apple ZFS
/dev/sda9  1953507328 1953523711      16384     8M Solaris reserved 1


Disk /dev/sdb: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: WD1004FBYZ-23YC
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 3BB5DE07-376C-CE4E-8044-C862F83704D2

Device          Start        End    Sectors   Size Type
/dev/sdb1        2048 1953507327 1953505280 931.5G Solaris /usr & Apple ZFS
/dev/sdb9  1953507328 1953523711      16384     8M Solaris reserved 1


Disk /dev/sdc: 5.46 TiB, 6001175126016 bytes, 11721045168 sectors
Disk model: WDC WD60EFRX-68L
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 5BF5B862-5A04-40AE-A8DB-17B76075F195

Device     Start         End     Sectors  Size Type
/dev/sdc1     64 11721045134 11721045071  5.5T Linux filesystem


Disk /dev/sdd: 223.57 GiB, 240057409536 bytes, 468862128 sectors
Disk model: WDC WDS240G2G0A-
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: E18EE190-6CC3-447B-A3B1-795F8DF0C51B

Device       Start       End   Sectors   Size Type
/dev/sdd1       34      2047      2014  1007K BIOS boot
/dev/sdd2     2048   1050623   1048576   512M EFI System
/dev/sdd3  1050624 468862094 467811471 223.1G Linux LVM


Disk /dev/mapper/pve-swap: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/pve-root: 55.75 GiB, 59861106688 bytes, 116916224 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/pve-vm--108--disk--0: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-vm--105--disk--1: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-vm--101--disk--0: 67 GiB, 71940702208 bytes, 140509184 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: 5FB70816-BFCE-4A98-A2F8-BFA38BF1A38E

Device                                 Start       End   Sectors Size Type
/dev/mapper/pve-vm--101--disk--0-part1  2048      4095      2048   1M BIOS boot
/dev/mapper/pve-vm--101--disk--0-part2  4096 140509150 140505055  67G Linux filesystem


Disk /dev/mapper/pve-vm--101--disk--1: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes

Code:

root@pve:~# pvs
  PV         VG  Fmt  Attr PSize    PFree
  /dev/sdd3  pve lvm2 a--  <223.07g <16.00g


  -------------------------

  root@pve:~# vgs
  VG  #PV #LV #SN Attr   VSize    VFree
  pve   1   9   0 wz--n- <223.07g <16.00g

Code:

  root@pve:~# lvs
  LV                                  VG  Attr       LSize   Pool Origin        Data%  Meta%  Move Log Cpy%Sync Convert
  data                                pve twi-aotz-- 140.45g                    30.58  2.30
  root                                pve -wi-ao----  55.75g
  snap_vm-108-disk-0_beforebetaupgrad pve Vri---tz-k   8.00g data vm-108-disk-0
  snap_vm-108-disk-0_beforeupdate     pve Vri---tz-k   8.00g data vm-108-disk-0
  swap                                pve -wi-ao----   8.00g
  vm-101-disk-0                       pve Vwi-a-tz--  67.00g data               45.07
  vm-101-disk-1                       pve Vwi-a-tz--  15.00g data               1.49
  vm-105-disk-1                       pve Vwi-a-tz--   4.00m data               1.56
  vm-108-disk-0                       pve Vwi-a-tz--   8.00g data               86.25

Code:

root@pve:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=20454900k,nr_inodes=5113725,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=4097868k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15902)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/sdc1 on /mnt/data_1 type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=4097864k,nr_inodes=1024466,mode=700,inode64)

fredamn76 · Jul 10, 2022

I looked at my log files and i hade two big kern.log files. Then i deleted them and the system started just fine.

Dunuin · Jul 10, 2022

Browse through your folders and look if you find files that can be deleted. Good start would be /var/log (all files ending with ".gz", ".1", ".2", ".3" and so on) as well as /var/tmp (look for failed ISO uploads) and mountpoints where data was written to the root filesystem because the mount failed.
apt autoremove also might free up some space or a fstrim -a.
And you can use goole to look for stuff like "find biggest files folders linux" to find out whats using all that space.

fredamn76 · Jul 10, 2022

Dunuin said:
Browse through your folders and look if you find files that can be deleted. Good start would be /var/log (all files ending with ".gz", ".1", ".2", ".3" and so on) as well as /var/tmp (look for failed ISO uploads) and mountpoints where data was written to the root filesystem because the mount failed.
apt autoremove also might free up some space or a fstrim -a.
And you can use goole to look for stuff like "find biggest files folders linux" to find out whats using all that space.

Thank you. I manage to free up som more space now.

fredamn76 · Jul 10, 2022

Is it possible to limit the size of log files?

Dunuin · Jul 10, 2022

You can edit several syslog and systemd parameters. I for example do this for my Debian VM and LXCs and I guess it should work for PVE too as PVE is bsed on Debian.

Limit logging

Limit systemd logs to 20MB:
nano /etc/systemd/journald.conf
Search for:
#SystemMaxUse=
Replace with:
SystemMaxUse=20M
Restart journald:
systemctl restart systemd-journald
enable compression for rotated logs:
nano /etc/logrotate.conf
Search for:
# uncomment this if you want your log files compressed #compress
Replace with:
# uncomment this if you want your log files compressed
compress
do a reboot:
reboot

And there are alot of other options in those files you could edit. Like how many rotated logfiles to keep, when to rotate and so on.

Search

Search

Proxmox local disk full

fredamn76

New Member

fredamn76

New Member

Dunuin

Distinguished Member

fredamn76

New Member

fredamn76

New Member

Dunuin

Distinguished Member

Limit logging

We value your privacy

Proxmox local disk full

fredamn76

New Member

fredamn76

New Member

Dunuin

Distinguished Member

fredamn76

New Member

fredamn76

New Member

Dunuin

Distinguished Member

Limit logging​

We value your privacy

Limit logging