Where did my disk space go?

[Alvarny]

Hello guys!
I've ran into a storage issue I'm not able to fix by myself.

My problem is that the storage 'storagepool' shows 91,87% usage although far less is actually in use.


Even the content tab shows less storage in total than what appears to be "used" somehow:




I recently deleted a 1TiB disk image that may still be somewhere in the nether and take up the space but I am finding no trace of it. I would like to create another 1TiB disk image but I cannot, since the disk "is too full".

/etc/pve/storage.cfg shows this:


When looking into the mountpoint /storagepool, this is what I find:


So only the 100GiB of the container for the 103 LXC. I'm not sure why the vm-107-disk-0 disk image is not listed although supossedly it is stored there. I checked /var/lib/vz/images too but to no avail. There seems to be something fundamental that I'm missing here.


For good measure some more screenshots of du and df commands:




I know there is space available but I don't know why proxmox does not list it as available. I tried to include all the relevant information but if there's something missing to the puzzle I'll include it ASAP.
Would love for you guys to point me into another direction since I don't know where else to look. Thanks!

 

[Dunuin]

So only the 100GiB of the container for the 103 LXC. I'm not sure why the vm-107-disk-0 disk image is not listed although supossedly it is stored there. I checked /var/lib/vz/images too but to no avail. There seems to be something fundamental that I'm missing here.

You should read more about how ZFS works. VMs are using zvols which are block devices so they are not part of any filesystem so you can't find them with ls and df/du also can't see them. LXCs use datasets which are filesystems and you can see them because they are mounted at your root filesystem.

If you want to see what is using how much space use the zfs commands like zfs list



If you think ZFS uses too much space you should check 3 things:
1.) check that you don't use raidz1, raidz2 or raidz3 with a too low volblocksize. Using raidz1/2/3 with the default 8k volblocksize will always waste alot of space when using zvols (or using VMs) because of padding overhead. Use the forums search function for more information, I explained that dozens of times.
2.) check that you don't got old snapshots. If you got a snapshot that is 1 month old that means that nothing can be deleted/freed up that was deleted or overwritten for one month. So basically, the older the snapshot is, the more space it will waste. So a old snapshot can easily use up a multiple of the size of the actual virtual disk that you snapshotted. Don't keep snapshots for more than several days or weeks. If you want long term backups setup a Proxmox Backup Server for that as this might save some space and in contrast to snapshots the backup won't be lost if the pool fails (snapshots/raid never replaces a backup).
3.) make sure that discard is working or ZFS won't be able to free up deleted data. So all VMs need to use a protocol like "SCSI" that can handle TRIM commands, every virtual disk needs the "discard" checkbox checked and you need to setup every guest os to actually use discard/trim.

 

[Alvarny]

Mate thank you!!
File systems are definitely one of my weaker areas of expertise, I'm gonna read up on that.

 

[Alvarny]

I narrowed it down to the vm-107-disk-0.

'zfs list' and 'zfs list -t snapshot' show this:


zfs list -o ... shows this:

So there are no snapshots taking up any space. Discard is set on the 107 disk.

Checking in the VM, 1.8T of the allocated 3.9T are in use:


I don't understand the 4.8T being 'used' and referenced by storagepool/vm-107-disk-0, especially when the disk should discard unused blocks and no snapshots/checkpoints are used. I do understand that the volblocksize of 8K is suboptimal but that should not make up the 3T.

To me it seems like snapshots are still somewhere in the nether that are not displayed for some reason. The 107-disk used to use twice the storage (3.6T) and there used to be another subvol-disk using 1T. Together with the still existing subvol-disk-103 (0.2T) that is exactly the 4.8T that are still shown.
But for some reason neither the reduction of the 107-disk, nor the deletion of the 1T subvol-disk, is taken into account by ZFS.

If anybody can make sense of this, I'll appreciate it lots. I'm on the verge of just setting up everything from scratch.

 

[tunhube]

"1.) check that you don't use raidz1, raidz2 or raidz3 with a too low volblocksize. Using raidz1/2/3 with the default 8k volblocksize will always waste alot of space when using zvols (or using VMs) because of padding overhead. Use the forums search function for more information, I explained that dozens of times."

I just created a 1TB VMdisk on a completely new RAIDZ-1 Datastore and it's taking up 1.6TB of space. I tried to find volblocksize setting on the GUI, but I couldn't. Is it me overlooking something, or you basically can't use RAIDZ-1 on proxmox if you don't want to waste 60% of space or tweak it yourself using the shell?

 

[Dunuin]

Volblocksize can only be set at creation of zvols. Every time you create a new virtual disk (so zvol) PVE will create it with a volblocksize that is defined in webUI at "Datacenter -> Storage -> YourZFSStorage -> Edit -> Block size". So to change it you will have to edit the "Block size" there and then destroy and recreate all virtual disks (restoring a backup or doing a migration will also recreate a zvol).