/dev/mapper/pve-root is full

[Rune2504]

Hello,

So I have used proxmox for 1 week now and I really enjoy it. But my disk for Proxmox is apparently full, and as I checked up on this, I can see, that /dev/mapper/pve-root is taking up all my space.

I have no experience in servers, i'm very new to this, so all help I can get is appreciated. See attached picture of the pve-root using 94Gb of space.

Thank you for your help.

 

[_gabriel]

delete some iso

 

[Rune2504]

What do you mean? I have one ISO-image on my server-drive, and this is only 1.47 Gb.

 

[_gabriel]

check backups too.
then delete it temporarly then from Shell

apt install ncdu
ncdu /

to scan folder tree to find which files eat space.

 

[Rune2504]

Thank you for the help already!

I'm not allowed to install ncdu. When I try, I get the "No such file or directory" failure. I have tried to google this problem without luck.

 

[Dunuin]

You probably can't install it because your root filesystem is read-only because it is 100% full.
Browse your folders with cd .., cd YourSubfolder and ls -la and try to find what consumes all that space. Good point to start would be "/var/log", "/mnt", "/var/tmp" and "/var/lib/vz".

 

[_gabriel]

yes delete iso temporarly

 

[bbgeek17]

du -h -x -d1 /

assuming /var is the problem, move on to it: du -h -x -d1 /var

keep going until you find what to delete.

A quick way to get a few kb: journalctl --vacuum-time=1h

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Rune2504]

Thank you so much guys. I can now see that it's my "log" that takes up 90Gb. How do I clear the log files?

 

[Dunuin]

How do I clear the log files?

You can destroy files with the rm /path/to/file command. Files ending with ".1", ".2", ".3", ".gz" should be safe to delete.
But just deleting your logs won't help. you should read your logs and find out what is spamming the log files and solve that problem. Otherwise your logs will continue to grow and you run out of space soon again.

 

[SoonerDave]

BTW, don't know if this even ties to your situation, but I *just* had this same problem recur and revealed it be a super-dummy on me situation

I had dumped the devault thin storage pool for my home setup, did the *lvresize* of the root - but not the *fs* resize. It was just kinda this little ticking bomb waiting to go off becuase my base setup for my system was probably using around 90-95GB of a 100GB default partition, then the logs slowly started filling it up while everything ran just fine...until it didn't. And none of the solutions i found fixed my problem, *UNTIL* I realized I hadn't done the fs resize when I built it a couple of months ago. Total dummy on me.

 

[366pu]

In my case, I had no web ui and when i ran

pveversion

from command line I got

Write error - write (28: No space left on device)

.

Turns out my var/log files kept growing and took all the space because I didnt setup any log rotation. I deleted the log files and add this config to

/etc/systemd/journald.conf

.

[Journal]
SystemMaxUse=1G
RuntimeMaxUse=1G
MaxRetentionSec=3month

 

[wuudn]

Hey guys, i have the same problem, 73gb of 100gb are full on local, but i dont know why.




But there is nothing big in here.

 

[bbgeek17]

Unmount backup : umount /mnt/backup
Check /mnt/backup location or re-run the du command

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[wuudn]

Worked! Backups was somehow double stored, on Backup disc and local, but not listed in GUI.
Thank you.

 

[bbgeek17]

Backups was somehow double stored

At one point you run your computer without disk being mounted. Which means /mnt/backup was a regular folder, i.e. not a mountpoint.
You proceeded to store data there. Then, you mounted the disk and "hidden" the data under the mount.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[wuudn]

Okay i see, good to know.
Thank you

 

[_gabriel]

you can prevent using mount point, while unmounted, with :
chattr +i /mnt/backup

 

[Dunuin]

And you should use something like this to tell PVE that "/mnt/backup" is a mountpoint so it won't try to write backups to the local storage in case that folder isn't mounted: pvesm set <IdOfYourDirectoryStorage> --is_mountpoint /mnt/backup