Proxmox says my storage is full but it's not

[MrGeezer]

Hi,

My PVE server has a single nvme drive of 1 Tb. I can't remember exactly how I set it up when I installed but it has two storage locations - one LVM-thin and one directory. The directory is the only one which has ISO images allowed and according to the GUI its path is /var/lib/vz

The GUI shows the lvm as having 365 out of 850Gb used, which is about what I expect. However it is showing the directory as having 92 out of 100 Gb in use. Not only is this way more than I would expect (I have 3 ISO images stored there - each less than 4GB0 but it won't let me upload a 5Gb ISO - which should fit even if it really were using 92Gb.

I've looked in /var/lib/vz - the only thing I could find apart from the iso's was in /var/lib/vz/cache there were a couple of 2-ish Gb files that I deleted but that's not made any difference.

Can anyone please tell me where this supposed 90-odd Gb of data is that's using my storage up?

Thanks

 

[Dunuin]

Not only is this way more than I would expect (I have 3 ISO images stored there - each less than 4GB0 but it won't let me upload a 5Gb ISO - which should fit even if it really were using 92Gb.

An ISO has to be stored twice on the "local" storage. Once for the temorary file upload and then a second time for the final file.

Can anyone please tell me where this supposed 90-odd Gb of data is that's using my storage up?

Could be all kind of stuff:
1.) VMs/LXCs stored on "local"
2.) failed ISO uploads in "/var/tmp"
3.) big log files in "/var/log"
4.) backups in "/var/lib/vz/dump"
5.) in case you manually mounted a internal/external disk, pen drive or SMB/NFS share as a directroy storage and forgot the set the "is_mountpoint" option and the mount failed, stuff you thought would end up on that other storage/share might end up on your "local" storage
...

Try to look for the biggest files/folders: https://www.cyberciti.biz/faq/how-d...ilesdirectories-on-a-linuxunixbsd-filesystem/

 

[MrGeezer]

An ISO has to be stored twice on the "local" storage. Once for the temorary file upload and then a second time for the final file.

Makes sense - thanks. Will bear in mind.

There's no VM's/LXC's on the 'local' directory they are all stored on the local-lvm. So I guess must be log files or something.

Can you explain WHY the 'local' is 100Gb only? Would I have set that when I created it? I can't understand why, if the 'directory' is /var/lib/vz, the used space would be affected by anything other than the contents of /var/lib/vz. Why would files in /var/log be filling it? Or var/tmp?

I guess I could change the lvm-local to allow ISO uploads but is there any way of increasing the size of 'local'? It's all on the one HDD anyway!

Thanks

* EDIT TO ADD *

I'm not very experienced with linux but when I run du -h /var it lists a whole load of sub folders with their sizes and at the end the line


7.2G /var

This means the whole /var folder is only 7Gb, right? Do I need to look ABOVE the /var folder for a likely culprit?

 

[MrGeezer]

Further to the above - after a little poking around I have found a file in /root called server.raw that is 64 Gb in size, and another called haos_ova-8.3.qcow2

One of my VM's is called 'server' and another is called 'haos' - are these the files for these virtual machines? I have other virtual machines as well, so if that's where the VM's are being stored then I would have expected to see many other files here.

Are these files safe to delete? If they are, in fact, the files for my VM's then will I be unable to delete them if the VMs are running? I don't want to lose my VM's! Is there any way to check where proxmox is storing the files for my VM's so I can make sure these are not them?

 

[Dunuin]

Can you explain WHY the 'local' is 100Gb only? Would I have set that when I created it? I can't understand why, if the 'directory' is /var/lib/vz, the used space would be affected by anything other than the contents of /var/lib/vz. Why would files in /var/log be filling it? Or var/tmp?

I guess I could change the lvm-local to allow ISO uploads but is there any way of increasing the size of 'local'?

PVE installs 3 partitions. 1MB and 512MB partitions for booting and third partition with the remaining space for LVM. That LVM then consists of a VG and that VG got 3 LVs. One LV is "root" and this is your root filesystem. There all files/folders will be stored. The directory storage "local" is part of that "root" LV. This root LV is usully around 1/4 of your VGs size.
Then there is a LV called "swap" which is used for swapping.
And there is a special LV that is a LVM-Thin pool. This usually uses around 3/4 of the VG. This is your "local-lvm" storage where you can only store virtual disks as it is a block storage and got no file system.

So your "local" is probably 100GB because your "root" LV is just 100 GB.

And "local" shows you the size of the entire root filesystem, not the size of the "/var/lib/vz" folder on that filesystem.

I'm not very experienced with linux but when I run du -h /var it lists a whole load of sub folders with their sizes and at the end the line


7.2G /var

This means the whole /var folder is only 7Gb, right? Do I need to look ABOVE the /var folder for a likely culprit?

Jup, then the entire var folder is 7.2. You will have to run it on multiple folder to see whats going on. Start with the root folder "/" and then go down to the folders until you find the big files/folders.

Further to the above - after a little poking around I have found a file in /root called server.raw that is 64 Gb in size, and another called haos_ova-8.3.qcow2

"raw" and "qcow2" are virtual disks. So yes, you put them there.

One of my VM's is called 'server' and another is called 'haos' - are these the files for these virtual machines? I have other virtual machines as well, so if that's where the VM's are being stored then I would have expected to see many other files here.

Those virtual disks were not created by PVE. You probably downloaded them from somewhere and then manually attached them to an existing VM. Or you used it script that created the VM for you and that script put them there.

Are these files safe to delete? If they are, in fact, the files for my VM's then will I be unable to delete them if the VMs are running? I don't want to lose my VM's! Is there any way to check where proxmox is storing the files for my VM's so I can make sure these are not them?

They are probably used by your VMs. You should have a look at your VMs config files stored in "/etc/pve/qemu-server".