pve-root full

[rlm42]

Hello guys.

I set up my Proxmox to backup files to my Google Drive, which is still working. I went away for a few months, and the local drive is now full, with my Transmission container not starting. I have a HDD connected to my router and then mounted to my Proxmox (sda1).

Running df -h shows this:

root@pve:~# df -h
Filesystem                     Size  Used Avail Use% Mounted on
udev                           7.7G     0  7.7G   0% /dev
tmpfs                          1.6G  2.0M  1.6G   1% /run
/dev/mapper/pve-root            94G   92G     0 100% /
tmpfs                          7.7G   31M  7.7G   1% /dev/shm
tmpfs                          5.0M     0  5.0M   0% /run/lock
efivarfs                       192K  100K   88K  54% /sys/firmware/efi/efivars
/dev/nvme0n1p2                1022M   12M 1011M   2% /boot/efi
/dev/sdc                        57G   60K   54G   1% /backups
//192.168.1.1/sda1             5.5T  5.3T  238G  96% /media/share
/dev/fuse                      128M   32K  128M   1% /etc/pve
gdrive:                        100G   63G   38G  63% /mnt/gdrive
192.168.1.35:/mnt/pool/backup   53G   26G   28G  48% /mnt/pve/truenas-backup
tmpfs                          1.6G     0  1.6G   0% /run/user/0

Running du -hd1 2>/dev/null | sort -h shows this:

0       ./proc
0       ./sys
4.0K    ./home
4.0K    ./opt
4.0K    ./srv
16K     ./lost+found
36K     ./tmp
56K     ./backups
2.0M    ./run
15M     ./etc
18M     ./root
31M     ./dev
106M    ./boot
2.8G    ./usr
12G     ./var
56G     ./mnt
5.3T    .
5.3T    ./media

I'm a little confused. Been a good 8 months since I set up the server. Inside /mnt I have this. Are these being saved on the local drive as well as the Google drive?

4.0K    ./vzsnap0
26G     ./pve
31G     ./gdrive
56G

I can't remember how I set it all up it's been a long time. Everything is still working, all of my containers, except for one (transmission), which fails to start, and I cannot update or download anything in the shell as the drive is full.

If you can help, it would be much appreciated. Thanks!

 

[bbgeek17]

Hi @rlm42 , welcome to the forum.

What, likely, happened is that your external disk was not mounted when the data was dumped into the mountpoint, filling your root disk instead.
This has happened to more than a few people. Here are several results for google search term "proxmox root full ":

Post in thread '/dev/mapper/pve-root is full'

Feb 2, 2023

du -h -x -d1 /

assuming /var is the problem, move on to it: du -h -x -d1 /var

keep going until you find what to delete.

A quick way to get a few kb: journalctl --vacuum-time=1h

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

• bbgeek17

• 

Post in thread '/dev/mapper/pve-root is full'

Oct 4, 2024

Unmount backup : umount /mnt/backup
Check /mnt/backup location or re-run the du command

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

• bbgeek17

• 

Post in thread 'pve-root full'

Mar 6, 2024

Stop your VMs:
qm list
qm stop [id]

Stop PVE services:
killall -9 corosync
systemctl stop pve-cluster
systemctl stop pvedaemon
systemctl stop pvestatd
systemctl stop pveproxy

Unmount your external drives:
umount /Backup01
(repeat for others)

Examine unmounted locations:
ls -al /Backup01

Confirm the locations are not mounted, there should be no output for:
mount|grep Backup01

Remove the stranded data:
rm -rf /Backup01/*

Reboot

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

• bbgeek17

• 

[SOLVED] Post in thread 'Almost full "pve-root"'

Dec 15, 2023

please see this thread reply:
https://forum.proxmox.com/threads/local-fs-full-ghost-files-edition.137882/#post-614594

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

• bbgeek17

• 

[SOLVED] Post in thread 'Local storage /dev/mapper/pve-root full'

Mar 17, 2023

For others who will come across this later:

I suspect you marked it solved because you found that /mnt at 83G is the culprit due to data being located in root volume, rather than a disk which you thought was mounted?

Its always best to run "du" with -x option, that way it wont cross filesystem boundaries and will give you an accurate information about the specific parent mountpoint.

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

• bbgeek17

[SOLVED] Post in thread 'pve-root full'

May 16, 2024

What happens when you execute:
umount /mnt/HD1
umount /mnt/HD2
umount /mnt/HD3
df -h -d1 /mnt

?

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

• bbgeek17

Post in thread 'No space left on device and can't login to web ui'

Mar 14, 2024

90G /var

there you go, most of your space is used in /var

That should give you an idea where space is being used. For example, if you see /var as an abuser, you can repeat the command as following:
du -h -d1 -x /var

continue adding to path to drill down to where most space has gone. Most likely /var/lib/vz, occupied by VMs, ISO or other files. Be prepared to figure out what you can delete to free up some space.

You can immediately get rid of some system logs by doing:
journalctl --flush --rotate --vacuum-time=1s
journalctl --user --flush --rotate --vacuum-time=1s...

• bbgeek17

• 

[SOLVED] Post in thread 'rpool/root/pve-1 taking up entire disk'

Jun 27, 2024

Please review https://forum.proxmox.com/threads/no-space-left-on-device.120901/#post-525313

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

• bbgeek17

• 

Cheers

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[rlm42]

Hi bbgeek. Thanks for your reply!

Ok fixed that issue. I unmounted all the drives and checked them, it was my 'gdrive'. Using

mount|grep /mnt/gdrive

and

du -h -d1 /mnt/gdrive

I saw the files were saved locally. I deleted and it cleared the space.

But my Transmission container still fails to start with the following log:

run_buffer: 322 Script exited with status 1
lxc_init: 844 Failed to run lxc.hook.pre-start for container "103"
__lxc_start: 2027 Failed to initialize container "103"
TASK ERROR: startup for container '103' failed

Any ideas how to fix this?

 

[UK SPEED]

Hello
I remember a topic about how to keep the /etc/pve/qemu-server/*.conf files synchronise with Google Drive to be able to keep this important file safe, can someone share this file or the link or the idea of that, please?