Very strange file behavior with a qcow2 file

[Red Squirrel]

So a little background, I have a Proxmox VE 6.4-13 server at OVH. I chose default partitioning when I set it up, only to realize the hard way that they only allocated 20GB to the / partition.

I made a /localdata/vms/ folder to set as my VM store and quickly ran out of space while building my VM which caused some corruption and other issues.

I ended up finding out via "df" command when checking disk space that they allocated everything to some odd ball folder in /var

So I moved the VM there but the disk space was not being freed on /. A reboot ended up fixing that. Needless to say after all of that I ended up with lot of corruption such as no longer being able to start/stop VMs etc and the whole server is basically broke.

Now the VM file. It's just a qcow2 file, about 17GB. But here is the weird part, if I try to copy that file anywhere, it just keeps copying past 17GB. Basically it will keep copying until it fills up the destination. I am trying to get the file over to another server so I can format the host and then pull the file back and restore the VM. But no luck.

In "dir" and stat it also shows the file is much bigger but in DU it shows the proper size. I've never seen anything like this before, this is the weirdest thing.

root@server04:100# du -h vm-100-disk-0.qcow2
17G vm-100-disk-0.qcow2


root@server04:100# stat vm-100-disk-0.qcow2
File: vm-100-disk-0.qcow2
Size: 3221717254144 Blocks: 33723240 IO Block: 4096 regular file
Device: fd00h/64768d Inode: 179961860 Links: 1
Access: (0640/-rw-r-----) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2022-01-25 13:07:43.003980187 +0000
Modify: 2022-01-24 02:40:04.772365413 +0000
Change: 2022-01-24 02:40:04.772365413 +0000
Birth: -

root@server04:100# dir -h
total 17G <--- here it's normal
0 -rw-r--r-- 1 root root 0 Jan 24 21:45 test
17G -rw-r----- 1 root root 3.0T Jan 24 02:40 vm-100-disk-0.qcow2 <--- but here it's not?

At this point I'm probably going to cut my losses and format the box without saving the VM but I still want to understand what is going on, and I also want to ensure that I never run into this again. Is this something specific to Proxmox and how it handles qcow2 files? What would be the beset way to avoid this sort of weirdness when I setup the server again? I want to ensure that my files are normal and that I can easily back them up or copy them if needed.

 

[fiona]

Hi,

So a little background, I have a Proxmox VE 6.4-13 server at OVH. I chose default partitioning when I set it up, only to realize the hard way that they only allocated 20GB to the / partition.

I made a /localdata/vms/ folder to set as my VM store and quickly ran out of space while building my VM which caused some corruption and other issues.

You also need to tell Proxmox VE about this storage (e.g. Datacenter > Storage > Add > Directory).

I ended up finding out via "df" command when checking disk space that they allocated everything to some odd ball folder in /var

Was it /var/lib/vz? That is the folder for the default local storage.

So I moved the VM there but the disk space was not being freed on /. A reboot ended up fixing that. Needless to say after all of that I ended up with lot of corruption such as no longer being able to start/stop VMs etc and the whole server is basically broke.

Now the VM file. It's just a qcow2 file, about 17GB. But here is the weird part, if I try to copy that file anywhere, it just keeps copying past 17GB. Basically it will keep copying until it fills up the destination. I am trying to get the file over to another server so I can format the host and then pull the file back and restore the VM. But no luck.

The qcow2 file is sparse, which means that it doesn't actually use as much space on the disk compared to its "true" size (which is ~3TiB) in your case. How do you try to copy the file? Maybe the destination doesn't support sparse files?

In "dir" and stat it also shows the file is much bigger but in DU it shows the proper size. I've never seen anything like this before, this is the weirdest thing.





At this point I'm probably going to cut my losses and format the box without saving the VM but I still want to understand what is going on, and I also want to ensure that I never run into this again. Is this something specific to Proxmox and how it handles qcow2 files? What would be the beset way to avoid this sort of weirdness when I setup the server again? I want to ensure that my files are normal and that I can easily back them up or copy them if needed.

 

[Red Squirrel]

I recently learned about "sparse" files in this troubleshooting, but still baffled at the concept. I know about thin provisioning but never seen it done this way, I figured the file would just grow in size and that the hypervisor itself handled that, but in this case it's almost like it's happening at the file system level or something, is that right? How is a file smaller but yet not small at same time?

I did add the custom storage path when I originally tried to use another path, but when I found out they only allocated 20GB to it I got burned by that so I moved the file back to the default location. And yeah it was /var/lib/vz if I recall. I don't like having data in oddball places so it's why I tried to move it originally.

I was trying to use rsync and scp to move the file as I was trying to get it on another server to back it up. It kept trying to produce a file that is bigger than what it actually is.

I pretty much gave up and I will just format the server and start over but I do worry I run into this in the future, whether it's with PVE or anything else. This would ruin my backup jobs if even 1 file like this happens to show up on one of my servers. Will I prevent that if I use raw format?

Is it some kind of kernel flag or file system flag that causes files like this to happen or is it specific to PVE?

 

[fiona]

I recently learned about "sparse" files in this troubleshooting, but still baffled at the concept. I know about thin provisioning but never seen it done this way, I figured the file would just grow in size and that the hypervisor itself handled that, but in this case it's almost like it's happening at the file system level or something, is that right? How is a file smaller but yet not small at same time?

Yes, a filesystem might or might not support it. Short introduction.

I did add the custom storage path when I originally tried to use another path, but when I found out they only allocated 20GB to it I got burned by that so I moved the file back to the default location. And yeah it was /var/lib/vz if I recall. I don't like having data in oddball places so it's why I tried to move it originally.

I was trying to use rsync and scp to move the file as I was trying to get it on another server to back it up. It kept trying to produce a file that is bigger than what it actually is.

I pretty much gave up and I will just format the server and start over but I do worry I run into this in the future, whether it's with PVE or anything else. This would ruin my backup jobs if even 1 file like this happens to show up on one of my servers. Will I prevent that if I use raw format?

By default, Proxmox VE should also create a local-lvm storage alongside the local storage, which is intended for VM/container images. The easiest way to avoid problems is to not make virtual disks larger than what you have available.

Is it some kind of kernel flag or file system flag that causes files like this to happen or is it specific to PVE?

Not sure if it can be deactivated at that level. It is supported by default on most modern file systems AFAIK. In recent versions of Proxmox VE, there is a preallocation setting (advanced storage settings). When set to full it should avoid sparse VM images.

 

[Red Squirrel]

Yeah recently read on sparse files, never heard of them before this incident. Not quite sure I understand it yet, or how it happens. Ex: how did the file end up that way?

Is there a way to make it like vmware? The vmdks are regular files. The hypervisor will just grow it as needed and handle the thin provisioning. If you try to copy a thin provisioned vmdk that is a certain size, it will only copy that much data. The hypervisor itself handles thin provisioning and does not do anything oddball with the file system.

Though I suppose another option for me is to just do thick provisioning and split OS from data. I can do OS drive like 30GB then that's a reasonable file to back up vs 3TB+ or whatever the sparse file was trying to do when I was copying it. It just kept going until it filled up the destination.

 

[fiona]

Yeah recently read on sparse files, never heard of them before this incident. Not quite sure I understand it yet, or how it happens. Ex: how did the file end up that way?

Because by default Proxmox VE uses qemu-img with preallocation=metadata to create images, which only allocates the metadata up front.

Is there a way to make it like vmware? The vmdks are regular files. The hypervisor will just grow it as needed and handle the thin provisioning. If you try to copy a thin provisioned vmdk that is a certain size, it will only copy that much data. The hypervisor itself handles thin provisioning and does not do anything oddball with the file system.

AFAIK Proxmox VE does support VMDK images, but I'm not sure if you can get the type of sparse image you mention. Regarding the copying of sparse files, It seems like scp doesn't support it, but you should be fine if you use rsync --sparse.

Though I suppose another option for me is to just do thick provisioning and split OS from data. I can do OS drive like 30GB then that's a reasonable file to back up vs 3TB+ or whatever the sparse file was trying to do when I was copying it. It just kept going until it filled up the destination.

 

[Red Squirrel]

Is there a way to change that default so it just preallocates or at least does not create sparse files? That would probably be my best bet. I just don't like the weirdness of sparse files and how they are basically impossible to copy anywhere using normal means.

 

[fiona]

Is there a way to change that default so it just preallocates or at least does not create sparse files? That would probably be my best bet. I just don't like the weirdness of sparse files and how they are basically impossible to copy anywhere using normal means.

Yes

In recent versions of Proxmox VE, there is a preallocation setting (advanced storage settings). When set to full it should avoid sparse VM images.

 

[Red Squirrel]

Where would I go for that? I can't seem to find it. I don't really see anything called settings anywhere. I'm on 6.4-13 so I wonder if maybe that's not recent enough? (it's what OVH installs by default, there is no way to install anything not in the list)

 

[Red Squirrel]

I tried to use raw files. If I create them manually I seem to be ok, but minute I try to do anything within PVM such as add disk space it starts treating it as a sparse file again. This is so annoying. I hate the idea of having unpredictable disk space reporting and hindering the ability at doing basic file management because of files acting strange.

Is there some kind of kernel flag to disable this functionality altogether? I want files to actually be the size that they actually are... if that even makes sense. Ex: when I run df, du etc I want the proper sizes to show up.

 

[fiona]

Where would I go for that? I can't seem to find it. I don't really see anything called settings anywhere. I'm on 6.4-13 so I wonder if maybe that's not recent enough? (it's what OVH installs by default, there is no way to install anything not in the list)

Yes, you need pve-manager 7.0-14 or later.

 

[Red Squirrel]

Oh ok guess I'm out of luck then. Well hopefully they add that version as an option to install later down the road.

I guess I will just need to find a different way to backup for now. Technically I only need an initial backup of the OS once everything is configured so I will just do some downtime so I can take an acronis image.

 

[Whitterquick]

Not sure if it can be deactivated at that level. It is supported by default on most modern file systems AFAIK. In recent versions of Proxmox VE, there is a preallocation setting (advanced storage settings). When set to full it should avoid sparse VM images.

That Preallocation option could do with some documentation. I could not find it in the online Wiki or the Documentation when you click ‘Help’ from the Storage dilog.

What is the difference between ’Full’ and ‘Full (posix_fallocate)’ ??

 

[fiona]

That Preallocation option could do with some documentation. I could not find it in the online Wiki or the Documentation when you click ‘Help’ from the Storage dilog.

What is the difference between ’Full’ and ‘Full (posix_fallocate)’ ??

From the fallocate man page (technically not completely the same as posix_fallocate, but this should still apply):

For filesystems which support the fallocate system call, preallocation is done quickly by allocating blocks and marking them as uninitialized, requiring no IO to the data blocks. This is much faster than creating a file by filling it with zeroes.

 

[Whitterquick]

From the fallocate man page (technically not completely the same as posix_fallocate, but this should still apply):

Thanks, so this posix_fallocate is better to use than ‘Full’ I guess…

I don’t suppose it’s possible to mix and match (different Preallocation for different VMs)?

If I have it on default and I switch to Full will all VMs suddenly take up all their required space?

 

[fiona]

Thanks, so this posix_fallocate is better to use than ‘Full’ I guess…

I don’t suppose it’s possible to mix and match (different Preallocation for different VMs)?

No, it's a per-storage setting.

If I have it on default and I switch to Full will all VMs suddenly take up all their required space?

It will only affect newly allocated images (which does include moved or fully cloned images).