[SOLVED] Local storage all of a sudden nearly full.

[lpj608]

Hi All,

I'm fairly new to Proxmox, so hopefully I will be able to get my issue across clearly.

Background: I have one GKMtek G3 setup running PVE. This is all installed on the internal NVMe 256GB SSD. I have a few containers set up and running smoothly, and I’ve mounted some shared folders from my NAS to allow downloaded files from a torrent client to save directly onto the NAS. This has all been working perfectly.

Yesterday, I installed an extra 250GB M.2 SATA SSD into the G3, as it has a spare slot. The idea is to store all of the downloaded torrent files on this SSD, rather than the NAS, as having it downloading and seeding from the NAS was preventing it from ever hibernating.

I was able to get the SSD mounted into the Node and passed through to the container fine. This all seemed to be working well.

I then proceeded to copy the seeding files from the NAS to the new SSD to redirect the torrent client to find them there and stop it from constantly accessing the NAS. This also seemed to work perfectly, but this is where the issue started.

After checking the Proxmox GUI, I could see the new storage and the increase in usage as I had transferred the files, but the total space was not 256GB; it was only 72GB. Thinking this was strange, I decided to reboot the node to see if it changed. Surprisingly, it did—the capacity increased to the correct 256GB. However, at the same time, all of the files I had just copied over from the NAS were removed.

I then realized that the Local storage seemed to have increased in size at the exact same time as when I was transferring these files but has never gone back down, and I cannot find where they could be stored.

Pics of the two storage graphs are here:

SSD Storage -

https://photos.app.goo.gl/GkQUHMSoXwWSfcF19

Local Storage -

https://photos.app.goo.gl/7Agkuy8vgWztEFsc7

I have run the df command to try to find the missing files and get this result -

root@pve:~# df -h
Filesystem                         Size  Used Avail Use% Mounted on
udev                               3.8G     0  3.8G   0% /dev
tmpfs                              769M  1.7M  768M   1% /run
/dev/mapper/pve-root                68G   52G   14G  80% /
tmpfs                              3.8G   46M  3.8G   2% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
efivarfs                           192K   98K   90K  52% /sys/firmware/efi/efivars
/dev/nvme0n1p2                    1022M   12M 1011M   2% /boot/efi
/dev/sda                           234G  9.0G  213G   5% /mnt/ssd_storage
192.168.20.111:/volume1/Media      3.6T  473G  3.2T  13% /mnt/lxc_shares/media
192.168.20.111:/volume1/Books      3.6T  473G  3.2T  13% /mnt/lxc_shares/books
192.168.20.111:/volume1/Downloads  3.6T  473G  3.2T  13% /mnt/lxc_shares/downloads
/dev/fuse                          128M   24K  128M   1% /etc/pve
192.168.20.111:/volume1/Proxmox    3.6T  473G  3.2T  13% /mnt/pve/synology-nas
tmpfs                              769M     0  769M   0% /run/user/0

It seems like they may be contained within the /dev/mapper/pve-root directory based on the size, but I have had no luck being able to locate them to remove them..

Any ideas to try to clear this storage back to normal are appreciated.

Cheers
Luke

 

[sw-omit]

try using this command:
du -sh /*
This should generate a summary of all the folders in the root and how large they are (/dev/mapper/pve-root is "just" the entire disk, as you can see by the mount-point behind it)
If you find a folder that is large, for example /tmp, run this to get a list of all sub-folders below THAT:
du -sh /tmp/*
Once you find where the space-usage is going, report back here.

I'm suspecting that you might not have the storage of the drive properly linked to the container and/or that it was still busy transferring (and things were still in cache) when you rebooted, causing the files to indeed be stuck on your main drive instead of the new drive.

I'm not too familiar with containers and their storages, only using full VM's, so you'll have to wait for some other input for help with that part, but we might at least be able to find where the disk-usage is going.

 

[lpj608]

Thanks for the reply.

I have run that and this is the output -

root@pve:~# du -sh /*
0       /bin
188M    /boot
46M     /dev
5.7M    /etc
4.0K    /home
0       /lib
0       /lib64
16K     /lost+found
4.0K    /media
499G    /mnt
4.0K    /opt
du: cannot access '/proc/34475/task/34475/fd/4': No such file or directory
du: cannot access '/proc/34475/task/34475/fdinfo/4': No such file or directory
du: cannot access '/proc/34475/fd/3': No such file or directory
du: cannot access '/proc/34475/fdinfo/3': No such file or directory
0       /proc
100K    /root
1.7M    /run
0       /sbin
4.0K    /srv
0       /sys
36K     /tmp
3.0G    /usr
4.5G    /var

/mnt is my NAS and the 499G corrosponds with that.

From what I can see nothing else seems like it could be using too much>?

 

[sw-omit]

There is a (small-ish) 17 GB difference between what it shows here in /mnt and what the "df" showed previously in anything mounted there, so we could do a quick check there just in case with du -sh /mnt/*, but that won't be all of it of course
That said though, maybe also try doing this, to see if there is anything directly in root:

ls -l -a -h /

 

[lpj608]

It NAS is currently restoring from a backup as I also put a new HDD in it yesterday, and it is slowly filing, that would account for the incrase of 17gb.

root@pve:~# ls -l -a -h /
total 68K
drwxr-xr-x  18 root root 4.0K Jul 20 13:06 .
drwxr-xr-x  18 root root 4.0K Jul 20 13:06 ..
lrwxrwxrwx   1 root root    7 Apr 24 21:26 bin -> usr/bin
drwxr-xr-x   5 root root 4.0K Jul 13 22:26 boot
drwxr-xr-x  19 root root 4.8K Jul 21 21:15 dev
drwxr-xr-x  97 root root 4.0K Jul 21 21:15 etc
drwxr-xr-x   2 root root 4.0K Jan 29 07:20 home
lrwxrwxrwx   1 root root    7 Apr 24 21:26 lib -> usr/lib
lrwxrwxrwx   1 root root    9 Apr 24 21:26 lib64 -> usr/lib64
drwx------   2 root root  16K Jun 29 11:08 lost+found
drwxr-xr-x   2 root root 4.0K Apr 24 21:26 media
drwxr-xr-x   5 root root 4.0K Jul 20 13:10 mnt
drwxr-xr-x   2 root root 4.0K Apr 24 21:26 opt
dr-xr-xr-x 473 root root    0 Jul 21 21:15 proc
drwx------   4 root root 4.0K Jul 21 22:25 root
drwxr-xr-x  29 root root 1.4K Jul 21 22:55 run
lrwxrwxrwx   1 root root    8 Apr 24 21:26 sbin -> usr/sbin
drwxr-xr-x   2 root root 4.0K Apr 24 21:26 srv
dr-xr-xr-x  13 root root    0 Jul 21 21:15 sys
drwxrwxrwt   8 root root 4.0K Jul 21 21:15 tmp
drwxr-xr-x  12 root root 4.0K Apr 24 21:26 usr
drwxr-xr-x  11 root root 4.0K Apr 24 21:26 var

To me it looks like there are no large files directly in root?

 

[sw-omit]

Doesn't seem like it, no, very strange indeed.

Looking through the numbers and screenshots some more though, what I think at least happened, is that when you changed the ssd_storage to this new SSD, it didn't update all the things / restarted all the processes yet, which caused it to write to the old location (or a non-existing mount) on the local disk somewhere.
I'm not sure if the files will be in "normal" format on the disk, but if you have the name of an unique file that you copied over yesterday, you could try to see if you can find it with
find / -name UniqueFileName.extension

 

[alexskysilk]

Its possible (maybe probable) that you have stuff hiding in /mnt under the mount.

the way to deal with that is EITHER to dismount the filesystem sitting there and recheck, OR you can bind mount /dev/mapper/pve-root in a temporary location (eg, /temproot)

 

[lpj608]

Thanks for all the help so far.

I tried the 'find /' command on a file I know was copied, but it could not find anything.

I tried bind mounting as alex suggested above and was able to find all of the files stored in there on the local drive.

I have now moved them to the correct place on the ssd_storage, and the space has gone back to as it should be.

Thanks for the help!