local-lvm steadily filling up

danielo515

Member
Nov 20, 2020
32
1
13
38
Hello.
I have a proxmox installation with a few VMs and just two containers. The thing is that, for some reason that I can not understand, the local-lvm disk is being filled at an steady pace.
It got 100% full, so I moved out some VM disks and removed some templates, which gave me back to 85%. After 2 days it is again at 87% and continues filling. What can be causing this? Here is a graph of the last month:

1605852046075.png

While this was on the last 24h:
1605852074766.png

As you can see, it is not growing very fast, but it keeps growing, and I didn't added any new VM or anything.
What can be the root cause?
 
Right now I only have one lxc running, which uses ubuntu for a pihole + etcd (one node) services
and a virtual machine with home-assistant.

This is the reported content of the local-lvm, which doesn't seem to grow at all:
1605876350052.png
I think they are all "pre-allocated"
 
you do make snapshots of your vms (visible with the 'state-YYY' disks)

if you make snapshots, every block the vm writes over the existing data, has to be allocated newly
 
Yes, I have two snapshots. What does that mean? That every block piece of modified data will be reallocated? Does it happen only the first time or for every modification? Imagine that I have a picture that it's on the snapshot, the first time I replace the picture it will be "reallocated", I can understand that. Will it happen if I overwrite it again? or does it only happen once?
 
The reallocation will happen only once for each block that changes.
After the snapshot has been created, it does not store new data, but rather it makes a copy of old data before it gets changed. This is known as copy-on-write and is done so that the snapshot takes up minimal storage space, while still being able to revert the system back to a previous point in time.
 
Ok, that makes sense.
I just made some basic maths and I saw that the sum of images + snapshots that I have is about 80GB while the size of the disk is around 70GB, so if any of the virtual disks gets fully rewritten it will fill up the entire lvm.
 
Here is an screenshot of the contents of local-lvm, 7 days have passed since the latest one, and content sizes hasn't changed, however it was 86% and now is fillup to 91%

1606482529233.png
 
Here is an screenshot of the contents of local-lvm, 7 days have passed since the latest one, and content sizes hasn't changed, however it was 86% and now is fillup to 91%

View attachment 21552
currently, the size in the gui, is the max size of a disk, not the current allocated size.
 
I am having the same issue.
Hello.
I have a proxmox installation with a few VMs and just two containers. The thing is that, for some reason that I can not understand, the local-lvm disk is being filled at an steady pace.
It got 100% full, so I moved out some VM disks and removed some templates, which gave me back to 85%. After 2 days it is again at 87% and continues filling. What can be causing this? Here is a graph of the last month:

View attachment 21347

While this was on the last 24h:
View attachment 21348

As you can see, it is not growing very fast, but it keeps growing, and I didn't added any new VM or anything.
What can be the root cause?
I am having the exact same issue. Did you end up figuring it out? It seems like an issue with Home Assistant?
 
I am having the same issue.

I am having the exact same issue. Did you end up figuring it out? It seems like an issue with Home Assistant?
Well, it is hard to say.
Since then I did some things that may have impacted it:
- Moved all my backups to an external location, so I don't have any snapsot on the local storage anymore
- Added two more nodes to the cluster, not sure if now it is load balancing some of the data (I don't think so)
 
Did you ever figure out what the issue was? I have the exact same problem. 2 vms, one ubuntu server, and one nextcloud server, I'm not running backups or snapshots, not creating any files on the vm hosts, yet the local-lvm is steadily growing. I'm almost at 100% which i am trying to avoid as it will bring my system to a stand still...
 
If its not snapshots that are preventing LVM-Thin from freeing up stuff its probably missing TRIM/discard. Make sure you setup discard/TRIM in each guest OS, that you enable "discard" for the virtual disk, that you use a protocol like SCSI that supports TRIM (IDE and virtio block won`t) and that your physical disk controller support TRIM (no all raid cards do). Only if LVM receives TRIM commands from the guest OS it will try to delete stuff.
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!