Proxmox showing incorrect disk usage

[powersupport]

We have a single-node Proxmox setup with one VM, which is a Proxmox Backup Server.

The node has a total storage capacity of 20 TB, of which 14 TB is allocated to the VM. Currently, the VM has used 6 TB of storage, leaving about 8 TB available within the VM. However, Proxmox VE is showing a total usage of 18.93 TB out of 20 TB. the disk configured with btrfs

I’m not sure where the remaining storage has gone. Does anyone have any insights?

 

[dcsapak]

can you show where you get these number (e.g. the commands + output or screenshots) ?

also do you maybe have snapshots on the vm ? (that + data written will consume more disk space than the disks ofcourse)

did you maybe do backups on the same disks?

 

[Fantu]

Can be useful post also the output of

btrfs fi usage

.
You have to be careful and manage well when using snapshots because it is easy to underestimate the space issue (unless you use temporary snapshots or ones kept for a very short time).

 

[Impact]

You might also be lacking Discard.

 

[powersupport]

HI,
We have thoroughly checked and confirmed that no backup snapshots are stored in the storage. There is only one VM, which is using 6 TB, but the remaining usage is still unexplained.


btrfs fi usage /
Overall:
Device size: 18.19TiB
Device allocated: 18.18TiB
Device unallocated: 5.00GiB
Device missing: 0.00B
Device slack: 7.00KiB
Used: 17.24TiB
Free (estimated): 966.43GiB (min: 966.43GiB)
Free (statfs, df): 966.42GiB
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 512.00MiB (used: 0.00B)
Multiple profiles: no

Data,RAID0: Size:18.16TiB, Used:17.22TiB (94.83%)
/dev/sda3 9.08TiB
/dev/sdb3 9.08TiB

Metadata,RAID0: Size:26.00GiB, Used:20.90GiB (80.38%)
/dev/sda3 13.00GiB
/dev/sdb3 13.00GiB

System,RAID0: Size:16.00MiB, Used:1.27MiB (7.91%)
/dev/sda3 8.00MiB
/dev/sdb3 8.00MiB

Unallocated:
/dev/sda3 2.50GiB
/dev/sdb3 2.50GiB

 

[Fantu]

The non -allocated space is dangerously low (only 5GB), the difference with the free one can be resolved with a balance, it would be better first to delete something in order to free space, the screenshots you posted do not make it clear about what it can be.
After for the balance I suggest do before on chunk with allocation <=10%:
btrfs balance start -v -dusage=10 /
and after up to 50%:
btrfs balance start -v -dusage=50 /
can require big time if done on big size on hdd like I suppose are they.
In any case, the maximum free space is actually not much and there is certainly something that does not consider itself well (in most cases they are snapshots)
A strange thing I saw from partial vm config is presence of virtio scsi single controller but disk setted as sata, why?

Edit:
from what you wrote:

We have thoroughly checked and confirmed that no backup snapshots are stored in the storage. There is only one VM

With only that vm with 13TB raw (from the little is visible from these screenshot) do not come back and there is certainly anything else, sure that there are no snapshot BTRFS? There are no other big files in the root or storage (are in the same btrfs volume)?
In the storage on all the other parts outside the VM disk are really empty? You should also check any things done outside the Proxmox console and not visible in it
If there is no btrfs snapshot this should list nothing:

btrfs subvolume list -s /

and not other things, this should only have the size of the vm disk in /var/lib/pve/local-btrfs/images

du -h -d 3 /var/lib/pve/local-btrfs/

 

[powersupport]

HI,

THere is nothing to delete, confirmed everything.

 

[Fantu]

FWIK if disk is raw preallocated should be 13tb from start and not grow, if not preallocated it grows over time and to make it free up unused space you need to use discard/trim.
But it seems strange to me that it grows beyond the limit on raw format, the only ones I've seen grow beyond the maximum size are the qcow2 with internal snapshots.
Try to check with

qemu-img info <path of vm disk0>

In any case, I think that disk configuration in vm disk I think should be improved.
In the meantime, I hope you at least did the balance to avoid errors for full space (to deallocate free space) on btrfs volume.


EDIT:
I did a fast test creating a test vm inside latest proxmox I'm preparing for production on btrfs storage that I will use to store iso and templates to see if there are bugs in proxmox, but I didn't find, here are some details:

vm disk size before start, by default I saw that create it not preallocated, so is zero and taken also the usage of btrfs volume

qemu-img info /mnt/nvme-data/images/100/vm-100-disk-0/disk.raw
image: /mnt/nvme-data/images/100/vm-100-disk-0/disk.raw
file format: raw
virtual size: 50 GiB (53687091200 bytes)
disk size: 0 B
Child node '/file':
    filename: /mnt/nvme-data/images/100/vm-100-disk-0/disk.raw
    protocol type: file
    file length: 50 GiB (53687091200 bytes)
    disk size: 0 B

btrfs fi usage /mnt/nvme-data/
Overall:
    Device size:                 300.00GiB
    Device allocated:              5.02GiB
    Device unallocated:          294.98GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                          1.22GiB
    Free (estimated):            296.77GiB      (min: 149.28GiB)
    Free (statfs, df):           296.77GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:                5.50MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:3.01GiB, Used:1.21GiB (40.36%)
   /dev/mapper/nvme-data           3.01GiB

Metadata,DUP: Size:1.00GiB, Used:1.62MiB (0.16%)
   /dev/mapper/nvme-data           2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
   /dev/mapper/nvme-data          16.00MiB

Unallocated:
   /dev/mapper/nvme-data         294.98GiB

inside the vm formatted the disk as ext4 and created a file of 10gb

qemu-img info /mnt/nvme-data/images/100/vm-100-disk-0/disk.raw
image: /mnt/nvme-data/images/100/vm-100-disk-0/disk.raw
file format: raw
virtual size: 50 GiB (53687091200 bytes)
disk size: 10.5 GiB
Child node '/file':
    filename: /mnt/nvme-data/images/100/vm-100-disk-0/disk.raw
    protocol type: file
    file length: 50 GiB (53687091200 bytes)
    disk size: 10.5 GiB

btrfs fi usage /mnt/nvme-data/
Overall:
    Device size:                 300.00GiB
    Device allocated:             16.02GiB
    Device unallocated:          283.98GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                         11.96GiB
    Free (estimated):            286.04GiB      (min: 144.06GiB)
    Free (statfs, df):           286.04GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               11.53MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:14.01GiB, Used:11.94GiB (85.23%)
   /dev/mapper/nvme-data          14.01GiB

Metadata,DUP: Size:1.00GiB, Used:11.73MiB (1.15%)
   /dev/mapper/nvme-data           2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
   /dev/mapper/nvme-data          16.00MiB

Unallocated:
   /dev/mapper/nvme-data         283.98GiB

after I deleted the file of 10gb in the vm and did fstrim, vm disk image decrease and also allocation on btrfs on proxmox

qemu-img info /mnt/nvme-data/images/100/vm-100-disk-0/disk.raw
image: /mnt/nvme-data/images/100/vm-100-disk-0/disk.raw
file format: raw
virtual size: 50 GiB (53687091200 bytes)
disk size: 1.04 GiB
Child node '/file':
    filename: /mnt/nvme-data/images/100/vm-100-disk-0/disk.raw
    protocol type: file
    file length: 50 GiB (53687091200 bytes)
    disk size: 1.04 GiB

btrfs fi usage /mnt/nvme-data/
Overall:
    Device size:                 300.00GiB
    Device allocated:              6.02GiB
    Device unallocated:          293.98GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                          2.25GiB
    Free (estimated):            295.73GiB      (min: 148.74GiB)
    Free (statfs, df):           295.73GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               11.53MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:4.01GiB, Used:2.25GiB (56.18%)
   /dev/mapper/nvme-data           4.01GiB

Metadata,DUP: Size:1.00GiB, Used:1.77MiB (0.17%)
   /dev/mapper/nvme-data           2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
   /dev/mapper/nvme-data          16.00MiB

Unallocated:
   /dev/mapper/nvme-data         293.98GiB

so I have not spotted bugs or unexpected case

my test vm configuration is:

less /etc/pve/qemu-server/100.conf
agent: 1
balloon: 0
boot: order=scsi0;ide2;net0
cores: 4
cpu: host
ide2: nvme-data:iso/gparted-live-1.6.0-10-amd64.iso,media=cdrom,size=550M
memory: 8192
meta: creation-qemu=9.0.2,ctime=1732190472
name: test
net0: virtio=BC:24:11:85:BA:5B,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: nvme-data:100/vm-100-disk-0.raw,backup=0,discard=on,iothread=1,size=50G
scsihw: virtio-scsi-single
smbios1: uuid=64806fed-8b2c-4fc6-828b-235e60508ed2
sockets: 1
vmgenid: 7978b3c9-3ddf-4266-93a9-6767c0ce0d14

If you enable discard in vm disk and do a fstrim inside the vm I think should unallocate the free space of the vm disk, I suggest also to set disk as virtio scsi instead sata unless you use it for some special problem, I usually use ide or sata temporarily just for p2v vm windows


-----------------
EDIT2:
I did some another fast tests trying to exceed the disk limit, but I only exceeded it with snapshots, taking 2 snapshots and emptying and filling in the middle the vm disk.
What I saw is not a bug but normal (the important is need to do a good management), as I have seen there would be space management problems with the use of snapshots even with lvmthin and I suppose also with zfs.
These are not system problems, but management deficiencies that sysadmin or users do.

Here used space > total vm disk space because of snapshots and full disk rewrite in the middle

btrfs sub list /mnt/nvme-data/
ID 256 gen 153 top level 5 path images/100/vm-100-disk-0
ID 258 gen 100 top level 5 path images/100/vm-100-disk-0@snaptest1
ID 259 gen 126 top level 5 path images/100/vm-100-disk-0@snaptest2

btrfs fi usage /mnt/nvme-data/
Overall:
    Device size:                 300.00GiB
    Device allocated:             85.02GiB
    Device unallocated:          214.98GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                         80.54GiB
    Free (estimated):            217.60GiB      (min: 110.11GiB)
    Free (statfs, df):           217.60GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               81.06MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:83.01GiB, Used:80.38GiB (96.84%)
   /dev/mapper/nvme-data          83.01GiB

Metadata,DUP: Size:1.00GiB, Used:81.94MiB (8.00%)
   /dev/mapper/nvme-data           2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
   /dev/mapper/nvme-data          16.00MiB

Unallocated:
   /dev/mapper/nvme-data         214.98GiB

And after delete of snapshot, remove the test files from vm disk, doing fstrim and waiting garbage collector to fully remove the snapshots returned normally and it turns out it has also freed up correctly on btrfs

btrfs fi usage /mnt/nvme-data/
Overall:
    Device size:                 300.00GiB
    Device allocated:              6.02GiB
    Device unallocated:          293.98GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                          2.26GiB
    Free (estimated):            295.73GiB      (min: 148.74GiB)
    Free (statfs, df):           295.72GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:               81.06MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:4.01GiB, Used:2.26GiB (56.34%)
   /dev/mapper/nvme-data           4.01GiB

Metadata,DUP: Size:1.00GiB, Used:1.83MiB (0.18%)
   /dev/mapper/nvme-data           2.00GiB

System,DUP: Size:8.00MiB, Used:16.00KiB (0.20%)
   /dev/mapper/nvme-data          16.00MiB

Unallocated:
   /dev/mapper/nvme-data         293.98GiB

I didn't find a bug but is normal, as I saw you would have space management problems with the use of snapshots even with lvmthin (and I suppose also with zfs) without a good management.
There are other cases when using not preallocated disk that can require more management.
These normally are not system problems but management mistake that sysadmin or users do.


However, I am curious to know in the case reported if once the VM configuration has been fixed and the necessary operations have been carried out, host disk usage returns to normal or if there is a real bug in the version used.
In fact on proxmox I have already found some regressions after updates, partly it is normal because on the systems where it happened to me I do not have a subscription (and therefore it is a bit like a "beta tester"), then being partially "rolling release" on different main components such as kernel and qemu I have seen more regressions compared to the use of debian stable to which I was used.
Regarding btrfs itself in proxmox its implementation still needs to be improved in some things, for example if someone wanted to take snapshots outside of vm disks would need to configure the subvolumes regarding the root to reduce possible space and performance problems.

 

[Locon]

I have the same problem as the OP, neither a fstrim nor a balance has solved it.

My setup is a raid-0 with BTRFS.

 

[Fantu]

My analysis and tests of the previous message were done based on the little data provided on the previous person's case, and I reported a possible problem based on that data.
Your case may be different but without any data it is difficult to help, any people that want to reply can only make vague assumptions.
You have default BTRFS for both root and vm disks like the previous case?
If yes, can you start to post at least out of these command?

btrfs fi usage /

btrfs sub list /

du -h -d 3 /var/lib/pve/local-btrfs/

 

[Locon]

My analysis and tests of the previous message were done based on the little data provided on the previous person's case, and I reported a possible problem based on that data.
Your case may be different but without any data it is difficult to help, any people that want to reply can only make vague assumptions.
You have default BTRFS for both root and vm disks like the previous case?
If yes, can you start to post at least out of these command?

btrfs fi usage /

btrfs sub list /

du -h -d 3 /var/lib/pve/local-btrfs/

I am using btrfs for both root and datastore virtual machines in a raid 0 of two 2TB disks each.
Attached is the output of the commands, as you can see it tells me that I am using 2.8TB when I am really only using 2TB.

root@proxmox:/var/lib/pve# btrfs fi usage /
Overall:
    Device size:           3.64TiB
    Device allocated:           3.12TiB
    Device unallocated:         526.00GiB
    Device missing:             0.00B
    Device slack:           7.00KiB
    Used:               2.81TiB
    Free (estimated):         848.14GiB    (min: 848.14GiB)
    Free (statfs, df):         848.14GiB
    Data ratio:                  1.00
    Metadata ratio:              1.00
    Global reserve:         512.00MiB    (used: 0.00B)
    Multiple profiles:                no

Data,RAID0: Size:3.12TiB, Used:2.80TiB (89.91%)
   /dev/nvme0n1p3       1.56TiB
   /dev/sda3       1.56TiB

Metadata,RAID0: Size:5.00GiB, Used:4.12GiB (82.47%)
   /dev/nvme0n1p3       2.50GiB
   /dev/sda3       2.50GiB

System,RAID0: Size:32.00MiB, Used:272.00KiB (0.83%)
   /dev/nvme0n1p3      16.00MiB
   /dev/sda3      16.00MiB

Unallocated:
   /dev/nvme0n1p3     263.00GiB
   /dev/sda3     263.00GiB
 
 
root@proxmox:/var/lib/pve# btrfs sub list /
ID 256 gen 1715989 top level 5 path var/lib/pve/local-btrfs
ID 259 gen 1716034 top level 256 path var/lib/pve/local-btrfs/images/101/vm-101-disk-0
ID 262 gen 1716026 top level 256 path var/lib/pve/local-btrfs/images/100/vm-100-disk-0
ID 275 gen 1713541 top level 256 path var/lib/pve/local-btrfs/images/102/vm-102-disk-0
ID 276 gen 1716026 top level 256 path var/lib/pve/local-btrfs/images/102/vm-102-disk-1
ID 277 gen 1713541 top level 256 path var/lib/pve/local-btrfs/images/102/vm-102-disk-2


root@proxmox:/var/lib/pve# du -h -d 3 /var/lib/pve/local-btrfs/
0    /var/lib/pve/local-btrfs/dump
6.1G    /var/lib/pve/local-btrfs/template/iso
122M    /var/lib/pve/local-btrfs/template/cache
6.2G    /var/lib/pve/local-btrfs/template
0    /var/lib/pve/local-btrfs/private
2.0T    /var/lib/pve/local-btrfs/images/101/vm-101-disk-0
2.0T    /var/lib/pve/local-btrfs/images/101
2.1G    /var/lib/pve/local-btrfs/images/100/vm-100-disk-0
2.1G    /var/lib/pve/local-btrfs/images/100
528K    /var/lib/pve/local-btrfs/images/102/vm-102-disk-0
33G    /var/lib/pve/local-btrfs/images/102/vm-102-disk-1
12K    /var/lib/pve/local-btrfs/images/102/vm-102-disk-2
33G    /var/lib/pve/local-btrfs/images/102
2.0T    /var/lib/pve/local-btrfs/images
2.0T    /var/lib/pve/local-btrfs/

 

[Fantu]

Thanks, seems you are not using snapshot, for that data posted for now is strange also your case, there are over 700gb used more, calculating how than if you had full VM disks and no additional data in the rest of the root, but if vm disks are not pre-allocated (as by default) there are many more.

Discard is enabled on all the 3 vm and working?
In case there is no continuously active discard/trim on the vm filesystems you need to do fstrim (for example on linux), if you delete something significant inside the vm with trim working you should see the respective unallocated space on the host's btrfs volume increase similarly to the tests I posted, can you try to check them?
After testing the vm where it does not work, try to post the configuration and say which operating system is installed.
If instead it works on all 3 and the space is deallocated correctly in btrfs (after the deletions in the VMs), then it will be more complicated to find the problem.

 

[Fantu]

Another possibility of "anomalous" space usage would be the fragmentation given by the cow filesystem (except for the parts where the cow is deactivated) due to the fact that it always writes new extents and if the old ones are still in use by snapshots or parts they cannot be deleted.
Without using snapshots it should not be so high, and I didn't know how to identify it in a simple and fast way, but yesterday from a search I found btdu, and it seems very useful to identify that case too in a simple and fast way too, showed as "unreachable".
https://github.com/CyberShadow/btdu
There isn't a package for Debian now, the fast and easy things is download static binary of the latest version.
I tried it on some servers where I have btrfs volumes from long time, major have very low "unreachable" (also few mb), I found a high "unreachable" size on raid 1 of 3tb where storage iso, vm template, some installers, other files and major usage are the backup of vm using acronis, in particular the backup archives setted as "always incremental" (that work like a cow fs) have big size of "unreachable". Cow inside a cow is not optimal and without defrag (even if I can use it as hdd) they got me about 800gb of unreachable in over 4 years.


You can try using btdu and see how much space it gives you as "unreachable"? I suppose it's likely the problem, and it should be all or most of it related to the vm disks.

@Locon: can you tell me if trim/discard in the vm did you activate it and it's been working only recently (for example seeing this topic) or was it activated and working since you created the vm?
How long have you been creating those vm's?

 

[Locon]

I have solved it as follows.

First downloading the btdu software to see space used by unreachable parts of extents.

Once the culprit files were located I defragmented them with “btrfs filesystem defragment /path/file”.

Once I finished the defragmentation I ran a balance of the whole filesystem with:

btrfs balance start /

It has to be in this order because if not, there is no effect and you can't free the space.

First defragment and then balance.

I have the impression that COW file systems are not very good for this, I don't know if ZFS has the same fragmentation problems.

 

[Fantu]

Good to know you solved.
Some data would be useful to better understand the situation and see what can be improved in the btrfs support in proxmox (currently very limited) and in the management (by sysadmin and users).
Near all "unreachable" was in vm disks, right? can you tell roughly the quantities it was showing? how long ago were the vm created?
discard/trim in the vm was enabled and working recently or since vm creation?

About solve is right defragment specific file and after do a balance.
Anyway, regarding the complete balance it is not recommended as it would move most (or near all) of the data taking longer (especially if hdd) and shortening the life due to the many writes (if ssd). It is always better to limit it, in most cases do it only on the data and reallocate only the occupied chunks <=50% >I think is enough.

There is also another possibility to reduce the times, mainly instead defragment (even by a lot in some cases), if you have enough free space, the files in question are not in use and there are no snapshots (or you can delete them, usually to be done even before the first method) or reflink copies, I did also today on a folder with backup files of the 3tb raid1 mentioned in my previous message: copy without reflink, delete original and rename the copy
EDIT: proxmox do subvolume on folder of vm disks so I think is to avoid in these cases or doing only on single files (of vm disks)
changed procedure from DIR to file
cp --reflink=never <PATH FILE> <PATH FILE tmp>
rm <PATH FILE>
mv <PATH FILE tmp> <PATH FILE>

About avoid/limit this problem libvirt for example disable cow on btrfs storage for vm by default, anyway this will disable many useful btrfs features for these files.
I think it is good to understand how frequent and impactful the problem is to be able to evaluate whether it would be a good idea to disable cow by default in /var/lib/pve/local-btrfs/images (and images of other btrfs storage) and the possible consequences must also be carefully evaluated.

About zfs is also a cow fs, but it works differently for different parts, I don't remember enough from when I did some research and tested zfs years ago, I seem to remember a write amplification that is much higher than btrfs, so I suppose it could do certain operations by default that could mitigate these problems, but it's better if some zfs expert answers this.

 

[powersupport]

I have solved it as follows.

First downloading the btdu software to see space used by unreachable parts of extents.

Once the culprit files were located I defragmented them with “btrfs filesystem defragment /path/file”.

Once I finished the defragmentation I ran a balance of the whole filesystem with:

btrfs balance start /

It has to be in this order because if not, there is no effect and you can't free the space.

First defragment and then balance.

I have the impression that COW file systems are not very good for this, I don't know if ZFS has the same fragmentation problems.

Could you please share the steps, how you have installed btdu, I tried it and seems not working

 

[Fantu]

There is nothing to install. There is a static binary to download and make executable.
wget https://github.com/CyberShadow/btdu/releases/download/v0.5.1/btdu-static-x86_64
chmod +x btdu-static-x86_64
and after execute it as root (or with sudo) on the top level (to make possibile scan all), now that proxmox don't have implemented good subvolume structure is default, if you instead manually added a good subvolume structure you need to mount it on other point and use it
./btdu-static-x86_64 /

EDIT:
from a fast search I found an alternative for find more fragmented files: https://helmundwalter.de/en/blog/btrfs-finding-and-fixing-highly-fragmented-files
I changed to show the files with over 5000 extent instead only 500 that seem to me too low in case of big file, probably with very big files should be also 50000, !warning: can take long time, to restrict search instead all change path of find to specific folder

find / -xdev -type f| xargs filefrag 2>/dev/null | sed 's/^\(.*\): \([0-9]\+\) extent.*/\2 \1/' | awk -F ' ' '$1 > 5000' | sort -n -r

commands of the linked howto can be useful for a scheduled script that automatically defrag files if needed in periods of non-use of the server, obviously the values should be modified to avoid excessive and/or unhelpful use, or it could be useful to see how fragmented the files are to decide where it could be useful to disable cow on them.
It must be taken into account that the number of extents can be higher for many factors, for example file size, limited and fragmented free space, presence of snapshots or reflink copies, whether pre-allocated or not (if not pre-allocated it can also impact the fact that the space is not deallocated if trim/discard is not used)