local-lvm steadily filling up

[danielo515]

Hello.
I have a proxmox installation with a few VMs and just two containers. The thing is that, for some reason that I can not understand, the local-lvm disk is being filled at an steady pace.
It got 100% full, so I moved out some VM disks and removed some templates, which gave me back to 85%. After 2 days it is again at 87% and continues filling. What can be causing this? Here is a graph of the last month:



While this was on the last 24h:


As you can see, it is not growing very fast, but it keeps growing, and I didn't added any new VM or anything.
What can be the root cause?

 

[dylanw]

Hi,

What do you have running in the VMs and Containers? Could it be that this is contributing to the growing amounts of storage used? Local-lvm is by default LVM-thin storage type, which only allocates blocks of storage when they are written [1].

https://pve.proxmox.com/pve-docs/pve-admin-guide.html#storage_lvmthin

 

[danielo515]

Right now I only have one lxc running, which uses ubuntu for a pihole + etcd (one node) services
and a virtual machine with home-assistant.

This is the reported content of the local-lvm, which doesn't seem to grow at all:

I think they are all "pre-allocated"

 

[dcsapak]

you do make snapshots of your vms (visible with the 'state-YYY' disks)

if you make snapshots, every block the vm writes over the existing data, has to be allocated newly

 

[danielo515]

Yes, I have two snapshots. What does that mean? That every block piece of modified data will be reallocated? Does it happen only the first time or for every modification? Imagine that I have a picture that it's on the snapshot, the first time I replace the picture it will be "reallocated", I can understand that. Will it happen if I overwrite it again? or does it only happen once?

 

[dylanw]

The reallocation will happen only once for each block that changes.
After the snapshot has been created, it does not store new data, but rather it makes a copy of old data before it gets changed. This is known as copy-on-write and is done so that the snapshot takes up minimal storage space, while still being able to revert the system back to a previous point in time.

 

[danielo515]

Ok, that makes sense.
I just made some basic maths and I saw that the sum of images + snapshots that I have is about 80GB while the size of the disk is around 70GB, so if any of the virtual disks gets fully rewritten it will fill up the entire lvm.

 

[danielo515]

Here is an screenshot of the contents of local-lvm, 7 days have passed since the latest one, and content sizes hasn't changed, however it was 86% and now is fillup to 91%

 

[spirit]

Here is an screenshot of the contents of local-lvm, 7 days have passed since the latest one, and content sizes hasn't changed, however it was 86% and now is fillup to 91%

View attachment 21552

currently, the size in the gui, is the max size of a disk, not the current allocated size.

 

[iT4RZ4N]

I am having the same issue.

Hello.
I have a proxmox installation with a few VMs and just two containers. The thing is that, for some reason that I can not understand, the local-lvm disk is being filled at an steady pace.
It got 100% full, so I moved out some VM disks and removed some templates, which gave me back to 85%. After 2 days it is again at 87% and continues filling. What can be causing this? Here is a graph of the last month:

View attachment 21347

While this was on the last 24h:
View attachment 21348

As you can see, it is not growing very fast, but it keeps growing, and I didn't added any new VM or anything.
What can be the root cause?

I am having the exact same issue. Did you end up figuring it out? It seems like an issue with Home Assistant?

 

[danielo515]

I am having the same issue.

I am having the exact same issue. Did you end up figuring it out? It seems like an issue with Home Assistant?

Well, it is hard to say.
Since then I did some things that may have impacted it:
- Moved all my backups to an external location, so I don't have any snapsot on the local storage anymore
- Added two more nodes to the cluster, not sure if now it is load balancing some of the data (I don't think so)

 

[gama1]

Did you ever figure out what the issue was? I have the exact same problem. 2 vms, one ubuntu server, and one nextcloud server, I'm not running backups or snapshots, not creating any files on the vm hosts, yet the local-lvm is steadily growing. I'm almost at 100% which i am trying to avoid as it will bring my system to a stand still...

 

[Dunuin]

If its not snapshots that are preventing LVM-Thin from freeing up stuff its probably missing TRIM/discard. Make sure you setup discard/TRIM in each guest OS, that you enable "discard" for the virtual disk, that you use a protocol like SCSI that supports TRIM (IDE and virtio block won`t) and that your physical disk controller support TRIM (no all raid cards do). Only if LVM receives TRIM commands from the guest OS it will try to delete stuff.