Proxmox local disk full

F

fredamn76

New Member
Aug 6, 2020
9
0
1
48
Hi,
My local proxmox disk is full and I am trying to fix the issue but does not know how to fix it. I've looked in forums but cannot find a solution. Hope someone can help me. I've been able to ssh into my machine but webui does not work.

I ran df -h command (results below) and it shows the /dev/mapper/pve-root is full.

Code: 
root@pve:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   20G     0   20G   0% /dev
tmpfs                 4.0G   18M  3.9G   1% /run
/dev/mapper/pve-root   55G   55G     0 100% /
tmpfs                  20G     0   20G   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
/dev/sdc1             5.5T  5.0T  468G  92% /mnt/data_1
tmpfs                 4.0G     0  4.0G   0% /run/user/0
root@pve:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   20G     0   20G   0% /dev
tmpfs                 4.0G   26M  3.9G   1% /run
/dev/mapper/pve-root   55G   55G     0 100% /
tmpfs                  20G     0   20G   0% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
/dev/sdc1             5.5T  5.0T  468G  92% /mnt/data_1
tmpfs                 4.0G     0  4.0G   0% /run/user/0


Code: 
root@pve:~# fdisk -l
Disk /dev/sda: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: WD1004FBYZ-23YC
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 69047576-1C91-5F4E-AF61-9E3C64C3B0FF

Device          Start        End    Sectors   Size Type
/dev/sda1        2048 1953507327 1953505280 931.5G Solaris /usr & Apple ZFS
/dev/sda9  1953507328 1953523711      16384     8M Solaris reserved 1


Disk /dev/sdb: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: WD1004FBYZ-23YC
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 3BB5DE07-376C-CE4E-8044-C862F83704D2

Device          Start        End    Sectors   Size Type
/dev/sdb1        2048 1953507327 1953505280 931.5G Solaris /usr & Apple ZFS
/dev/sdb9  1953507328 1953523711      16384     8M Solaris reserved 1


Disk /dev/sdc: 5.46 TiB, 6001175126016 bytes, 11721045168 sectors
Disk model: WDC WD60EFRX-68L
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 5BF5B862-5A04-40AE-A8DB-17B76075F195

Device     Start         End     Sectors  Size Type
/dev/sdc1     64 11721045134 11721045071  5.5T Linux filesystem


Disk /dev/sdd: 223.57 GiB, 240057409536 bytes, 468862128 sectors
Disk model: WDC WDS240G2G0A-
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: E18EE190-6CC3-447B-A3B1-795F8DF0C51B

Device       Start       End   Sectors   Size Type
/dev/sdd1       34      2047      2014  1007K BIOS boot
/dev/sdd2     2048   1050623   1048576   512M EFI System
/dev/sdd3  1050624 468862094 467811471 223.1G Linux LVM


Disk /dev/mapper/pve-swap: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/pve-root: 55.75 GiB, 59861106688 bytes, 116916224 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mapper/pve-vm--108--disk--0: 8 GiB, 8589934592 bytes, 16777216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-vm--105--disk--1: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes


Disk /dev/mapper/pve-vm--101--disk--0: 67 GiB, 71940702208 bytes, 140509184 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 65536 bytes / 65536 bytes
Disklabel type: gpt
Disk identifier: 5FB70816-BFCE-4A98-A2F8-BFA38BF1A38E

Device                                 Start       End   Sectors Size Type
/dev/mapper/pve-vm--101--disk--0-part1  2048      4095      2048   1M BIOS boot
/dev/mapper/pve-vm--101--disk--0-part2  4096 140509150 140505055  67G Linux filesystem


Disk /dev/mapper/pve-vm--101--disk--1: 15 GiB, 16106127360 bytes, 31457280 sectors
Units: sectors of 1 * 512 = 512 bytes


Code: 
root@pve:~# pvs
  PV         VG  Fmt  Attr PSize    PFree
  /dev/sdd3  pve lvm2 a--  <223.07g <16.00g


  -------------------------

  root@pve:~# vgs
  VG  #PV #LV #SN Attr   VSize    VFree
  pve   1   9   0 wz--n- <223.07g <16.00g

Code: 
  root@pve:~# lvs
  LV                                  VG  Attr       LSize   Pool Origin        Data%  Meta%  Move Log Cpy%Sync Convert
  data                                pve twi-aotz-- 140.45g                    30.58  2.30
  root                                pve -wi-ao----  55.75g
  snap_vm-108-disk-0_beforebetaupgrad pve Vri---tz-k   8.00g data vm-108-disk-0
  snap_vm-108-disk-0_beforeupdate     pve Vri---tz-k   8.00g data vm-108-disk-0
  swap                                pve -wi-ao----   8.00g
  vm-101-disk-0                       pve Vwi-a-tz--  67.00g data               45.07
  vm-101-disk-1                       pve Vwi-a-tz--  15.00g data               1.49
  vm-105-disk-1                       pve Vwi-a-tz--   4.00m data               1.56
  vm-108-disk-0                       pve Vwi-a-tz--   8.00g data               86.25

Code: 
root@pve:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=20454900k,nr_inodes=5113725,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=4097868k,mode=755,inode64)
/dev/mapper/pve-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15902)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
/dev/sdc1 on /mnt/data_1 type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota)
lxcfs on /var/lib/lxcfs type fuse.lxcfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=4097864k,nr_inodes=1024466,mode=700,inode64)
 
Last edited:
I looked at my log files and i hade two big kern.log files. Then i deleted them and the system started just fine.
 
Browse through your folders and look if you find files that can be deleted. Good start would be /var/log (all files ending with ".gz", ".1", ".2", ".3" and so on) as well as /var/tmp (look for failed ISO uploads) and mountpoints where data was written to the root filesystem because the mount failed.
apt autoremove also might free up some space or a fstrim -a.
And you can use goole to look for stuff like "find biggest files folders linux" to find out whats using all that space.
 
Last edited:
  • Like
Reactions: fredamn76
Dunuin said:
Browse through your folders and look if you find files that can be deleted. Good start would be /var/log (all files ending with ".gz", ".1", ".2", ".3" and so on) as well as /var/tmp (look for failed ISO uploads) and mountpoints where data was written to the root filesystem because the mount failed.
apt autoremove also might free up some space or a fstrim -a.
And you can use goole to look for stuff like "find biggest files folders linux" to find out whats using all that space.
Click to expand...
Thank you. I manage to free up som more space now.
 
You can edit several syslog and systemd parameters. I for example do this for my Debian VM and LXCs and I guess it should work for PVE too as PVE is bsed on Debian.

Limit logging​

  • Limit systemd logs to 20MB:
    nano /etc/systemd/journald.conf
    Search for:
    #SystemMaxUse=
    Replace with:
    SystemMaxUse=20M
    Restart journald:
    systemctl restart systemd-journald
  • enable compression for rotated logs:
    nano /etc/logrotate.conf
    Search for:
    # uncomment this if you want your log files compressed #compress
    Replace with:
    # uncomment this if you want your log files compressed
    compress
  • do a reboot:
    reboot

And there are alot of other options in those files you could edit. Like how many rotated logfiles to keep, when to rotate and so on.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom