pve-root full

[Travion]

Hi there,

I am fairly new to proxmox and have run into an interesting situation.
I have a 128GB SSD which I use for my proxmox OS installation. All the other data is on another SSD and also HDD's.
At the moment the pve-root shows that it is at 110.2G

I have done some research and found these commands to troubleshoot:
ncdu -x / --exclude /Backup01 --exclude /Backup02 --exclude /Backup03 --exclude /Bootdisk --exclude /CT-Backups/ --exclude /HDD01 --exclude /HDD02 --exclude HDD03 --exclude /HDD04 in order to exclude the other attached drives.
Output:
2.7 GiB [########################] /usr
546.3 MiB [#### ] /var
93.7 MiB [ ] /boot
5.2 MiB [ ] /etc
280.0 KiB [ ] /root
40.0 KiB [ ] /tmp
24.0 KiB [ ] /home
16.0 KiB [ ] /mnt
16.0 KiB [ ] /lost+found
4.0 KiB [ ] /srv
4.0 KiB [ ] /opt
4.0 KiB [ ] /media


This is the output of lsblk:
nvme0n1 259:3 0 119.2G 0 disk

├─nvme0n1p1 259:4 0 1007K 0 part

├─nvme0n1p2 259:5 0 1G 0 part /boot/efi

└─nvme0n1p3 259:6 0 118.2G 0 part

├─pve-swap 252:0 0 8G 0 lvm [SWAP]

└─pve-root 252:1 0 110.2G 0 lvm /


I have also used this command to get an overview of the mounts, but don't really understand what I would be looking for:
systemctl list-units "*.mount"


In another forum somebody wrote that " Chances are that you wrote directly to one of the folders under /mnt when one of your external disks was NOT mounted.
Unmount all your disks, one at a time or all at the same time, then re-run your ndcu, or "du -h -x -d1 /mnt". I'd bet a donut that you will find your missing files.
https://forum.proxmox.com/threads/local-fs-full-ghost-files-edition.137882/

Maybe this is exactly the same problem I am having. My question is how would I unmount all drives (Can I even do that without messing up my set up) to see if this is the case ? I am terrified that I will mess up my install.
Any advice would be great ! Thanks a lot.

 

[bbgeek17]

Stop your VMs:
qm list
qm stop [id]

Stop PVE services:
killall -9 corosync
systemctl stop pve-cluster
systemctl stop pvedaemon
systemctl stop pvestatd
systemctl stop pveproxy

Unmount your external drives:
umount /Backup01
(repeat for others)

Examine unmounted locations:
ls -al /Backup01

Confirm the locations are not mounted, there should be no output for:
mount|grep Backup01

Remove the stranded data:
rm -rf /Backup01/*

Reboot

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Travion]

Thank you so much for your help ! It worked. I am back down to my install files on the root disk.
Just to understand what happens:
I have a script which runs nightly over cron, backing up files. So when a mounted drives unmounts the script still copies the files , but it chooses the
root disk? Would this be correct?
I am wondering if some command could be added to the script to check if the drive is mounted. If yes, do the backup, if no , then don't.

Thanks again for helping out. I really appreciate it.

 

[bbgeek17]

I am wondering if some command could be added to the script to check if the drive is mounted. If yes, do the backup, if no , then don't.

A google search for "bash check if directory is mounted" leads to this nice overview:
https://www.baeldung.com/linux/bash-is-directory-mounted

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox

 

[Kingneutron]

Maybe this is exactly the same problem I am having. My question is how would I unmount all drives (Can I even do that without messing up my set up) to see if this is the case ? I am terrified that I will mess up my install.
Any advice would be great ! Thanks a lot.

Now that it's working again, MAKE A BACKUP of your system so you can restore if something goes sideways...

 

[Travion]

Are you able to point me in the right direction on how to backup the whole system?
So far I have been backing up my containers and VMs

 

[Kingneutron]

I have homegrown scripts for bare-metal backup, so might not be the best resource to ask.
https://github.com/kneutron/ansitest/tree/master/proxmox

Clonezilla is frequently mentioned for Linux backup; not sure how well it works with LVM. You could also look into Veeam.

Worst-case scenario you could put the host system into ' telinit 1 ' after shutting down all VMs and containers and dd the whole hard drive to a compressed file on NAS or another drive. READ THE MAN PAGE for telinit before trying this, it puts the server into single-user mode and you will probably have to reboot after the backup. (This also means you need to be at the PVE console, as ssh won't work unless you restart the sshd daemon.)


Zeroing out the free space on root / mounted filesystems beforehand will save space on the destination file:

cd somedir; time dd if=/dev/zero of=tmpZEROfile bs=1M status=progress; date; /bin/rm -f tmpZEROfile

# Repeat for mounted filesystems off the root directory / on the same disk, not necessary for external drives


To backup your root/boot disk (may not work for ZFS root unless you do it from a live dvd / recovery environment with root pool exported)

apt install -y zstd lzop

# only run this once

# substitute sda with your actual drive

time dd if=/dev/sda bs=1M status=progress |zstd -T0 -o /mnt/some-external-drive-or-nas/destinationdir/bkp-$(date +%Y%m%d)-proxmox-root-lvm.dd.zstd; date

--Restoring is similar - NOTE don't use the star "*" in filename below if more than one backup, substitute date of backup instead:

# again, substitute sda with your actual boot drive designation, BE AWARE this will overwrite the entire disk!

time zstd -dcf  /mnt/some-external-drive-or-nas/destinationdir/bkp-*-proxmox-root-lvm.dd.zstd |dd of=/dev/sda bs=1M status=progress; date

NOTE these commands are from memory/man pages and I have not tested them personally.

I am also NOT RESPONSIBLE for data loss.

If you have any reservations about these commands, install PVE in a VM and TEST THEM IN A VM FIRST before you do it on your primary install!

 

[Travion]

Wow, thank you for all your help. I will look into all the proposed options. This is amazing.

 

[Travion]

I ran into the same issue with the GUI not been available. I could ssh into the machine and reboot the machine. I did this a few times but it didn't help.
So I went through all the steps from bbgeek17 again and when checking all the mounts with mount|grep Backup01 no data was displayed. So I guess all good so far.
When rebooting the machine the GUI worked again. One thing I noticed was that in the GUI under pve1 / disks / LVM it shows this as in the screenshot. Does this mean that all the space is just allocated to the OS or is the disk completely full?
When running the ncdu again it shows :

3.6 GiB [###############################] /usr
1.5 GiB [############# ] /var
172.3 MiB [# ] /boot
5.3 MiB [ ] /etc
284.0 KiB [ ] /root
40.0 KiB [ ] /tmp
24.0 KiB [ ] /home
16.0 KiB [ ] /lost+found
12.0 KiB [ ] /mnt
4.0 KiB [ ] /srv
4.0 KiB [ ] /opt
4.0 KiB [ ] /media
0.0 B [ ] lib64
0.0 B [ ] sbin
0.0 B [ ] lib
0.0 B [ ] bin
0.0 B [ ] /sys
0.0 B [ ] /run
0.0 B [ ] /proc
0.0 B [ ] /dev
0.0 B [ ] HDD04
0.0 B [ ] HDD03
0.0 B [ ] HDD02
0.0 B [ ] HDD01
0.0 B [ ] /Data-Drive

Thanks a lot for any advice and feedback.