LVM-Thin volume keeps getting filled up to 100%

[jaceqp]

Hi there.
I have a ~500GB LVM_Thin for vm disks. Both vdisks have total 450GB capacity (150+300). Yet after a week or so lvm-thin fills up to 100% crashing vm using it as storage location.

I'm using scheduled vm backups on external nas device (NFS) - could backup possibly consume rest of free space on that lvm-thin?
I'm about to expand server with another Raid1 volume with 2 larger disks but still wonder what affects the real usage?

 

[dylanw]

If backups are going to another device, this shouldn't fill up disk space in the long term. Is the storage purely for VMs? Do you have it over provisioned by any chance?
Do you have trim enabled [1] on the VM drives?
If it's something that is happening continuously, I would also look for the VM that seems to rapidly fill it's storage.

[1] https://pve.proxmox.com/pve-docs/pve-admin-guide.html#qm_hard_disk_discard

 

[jaceqp]

Forgot to mention that this lvm-thin also contained 2 oooold snapshots from initial install process of vm. Snaps took additional ca 20GB yet both snapshots+vdisks left over 30GB free space.
The 'only' solution to recover some of free space was to delete one of snapshots. Yet after a week or so free space was 'eaten' again. Had to delete another (last) snapshot in order to recover VM back online. Luckily I managed to add another storage (2x1.2TB Raid1), created another lvm-thin and move one of vdisks to a new location. Although there were no scheduled snapshots etc. I'm thinking if having ANY previous snapshots might be the issue. Yet adding all numbers (overall sizes) did not exceed lvm capacity...

BTW: are there any downsides of using full capacity for lvm-thin? If not what about overall vdisks capacity?

 

[dylanw]

I'm thinking if having ANY previous snapshots might be the issue. Yet adding all numbers (overall sizes) did not exceed lvm capacity...

Following a snapshot, only differences are stored in the current state, so previous snapshots should only really be an issue if the data is changing rapidly and significantly. In your case where you had two old snapshots, I don't know what would suddenly cause the operating system to deviate so far from them. Are you still seeing rapid usage in capacity? If so, I would really try to track down what is happening in the VMs that is using so much storage.

BTW: are there any downsides of using full capacity for lvm-thin? If not what about overall vdisks capacity?

Not really, if you've carefully planned your storage requirements. lvm-thin just offers the convenience of creating a virtual drive that could potentially reach a certain capacity, without having to reserve that capacity there and then. However, there is an emphasis on "carefully" here, as things like snapshots can sometimes lead to unpredictable usage

 

[jaceqp]

Following a snapshot, only differences are stored in the current state, so previous snapshots should only really be an issue if the data is changing rapidly and significantly. In your case where you had two old snapshots, I don't know what would suddenly cause the operating system to deviate so far from them. Are you still seeing rapid usage in capacity? If so, I would really try to track down what is happening in the VMs that is using so much storage.

These snapshots were made during or short after initial VM setup after install. So yes, those might vary quite significantly compared to 'current' VM state.

Fortunately I managed to install additional (also larger) RAID-1 volume and use it all for vm vd's. I've partitioned, it then created vg, LVM etc. Moved one of vm's VDs - problem solved... Or that's what I hoped.
After a week or so VM suddenly stuck at boot loop (no bootable device). New LVM crashed (possibly caused by metadata space filled) ditching VM's OS vd...
Awaiting further suggestions on recovering lvm-thin - see my other topic:
https://forum.proxmox.com/threads/my-lvm-thin-gone-vdisks-missing-empty-lvscan-inactive.86280/