VM Disks grown bigger than allocated size?

[rendili]

I need a bit of guidance on where I have gone wrong with storage.

I have had Proxmox up and running for about 6 months and all been ok until this week when one of my VMs suddenly started fail, and looking in to it, it looks to be a disk space issue. Getting the following errors soon after re-starting the failed VM:
proxmox kernel: device-mapper: thin: 253:4: reached low water mark for data device: sending event. proxmox kernel: device-mapper: thin: 253:4: switching pool to out-of-data-space (queue IO) mode

I really don't understand how the storage works with Proxmox, I thought I had configured my storage for around 50GB for the system and the rest for VM storage, and within that split the remaining storage between the only two VMs that i need to host, filling up pretty much all the storage, But now the size of the disks looks to have grown to larger than I allocated?

I have only one 480GB SSD in a NUC with the following setup:

Under Storage I have
local at 52GB:

and local-lvm 392GB:

Adding those two up should still leave a bit of space not allocated (around 35GB)?

For the two VMs I have allocated:
vm100 32GB


vm101 350GB

So again, adding those together there should be a little free space out of the 392GB.

But here is where I am confused, looking at VM Disks view, it looks like the 32GB disk has grown larger to 34GB and the 350GB disk has grown larger to 375GB:


So I am assuming this is where the issue is, the containing volume has no free space left for them to grow, but then they should not grow larger than what has been allocated?

Within each VM, vm100 is only 26% used and vm101 is 68% used, so they are not all that close to filling up, so I dont understand what is going on really.


Any advise would be greatly appreciated.

 

[Neobin]

I guess, the "problem"/confusion here is GB vs GiB.

Your local-lvm shows a total available of 392,11 GB, which is 365,18 GiB.
You assigned both of your VMs a total of 382 GiB, which is 410,17 GB.

	Total assigned	Total available
GB	410,17	392,11
GiB	382	365,18

So basically, you assigned your VMs more space, than it is available.

For reference:
https://bugzilla.proxmox.com/show_bug.cgi?id=4607

 

[Dunuin]

And using snapshots or trimming/discarding not frequently enough will add some overhead too.

 

[rendili]

ah, thank you, that all makes sense, I didn't notice that different screens were showing in different units, it certainly would help to have them all consistently the same (thanks for the link).

So the question now is, is it possible to shrink disks?
I have looked in the GUI and can see only an option for increasing a disk size but not reducing.
Are you able to point me in the right direction for a simple / straight forward guide to shrinking a VM disk (if its possible)?
(both guests are Linux)


With trimming / discarding, is it enough that I select the Discard option on both VM disks (both are unchecked), or is there a manual process I need to run (on the host?) every now and again?


Thanks for your help, much appreciated

 

[Dunuin]

With trimming / discarding, is it enough that I select the Discard option on both VM disks (both are unchecked), or is there a manual process I need to run (on the host?) every now and again?

All virtual disks need the discard checkbox checked. And in addition to that you need to set up each guest OS to trim regularly (for linux for example a daily fstrim -a via cron job) or mount the guests filesystems with the "discard" option (for ext4/xfs).

So the question now is, is it possible to shrink disks?
I have looked in the GUI and can see only an option for increasing a disk size but not reducing.
Are you able to point me in the right direction for a simple / straight forward guide to shrinking a VM disk (if its possible)?
(both guests are Linux)

Shinking a disk can always cause data loss. First you would need to shrink the filesystems inside the guests, then shink the partitions inside the guest and only then you can try to shink the LVs or zvols on the host. Don't forget to create backups first.

 

[rendili]

Thanks Dunuin, I checked the Discard checkbox on both disks and forced an fstrim on the larger disk (which is standard ubuntu, the other vm I am not sure about, it's some custom linux) and now have some free space and a bit of breathing room:

At least on the VM with the larger disk it looks like there is a weekly schedule, so hopefully now that the Discard option is checked it should keep it trimmed weekly.

I will see if I can find a good article for shrinking filesystems and partitions for ubuntu, but for the other VM i think it would be easier to backup, destroy, install, restore.

I had a search but cant find any articles on shrinking the Proxmox VM disks, do you have a link to an article?

 

[Dunuin]

Google for the "lvresize" or "lvreduce" command. Here is for example one result: http://www.microhowto.info/howto/reduce_the_size_of_an_lvm_logical_volume.html

 

[rendili]

Thanks Dunuin, much appreciated