LVM-Thin Storage filling up why?

[pietroaretino]

I have two RAID-10 spans configured on my server (4 x 1TB HDDs per RAID 10)

One stores the local and local-lvm (2TB storage)

The second span I turned into its own LVM-thin storage, i called it data2 (2TB storage)



On data2 I've created only one container. I wanted my data2 lvm-thin to only be used as a dedicated storage space for this one container.

However my lvm-thin storage (data2) is filling up quickly and i dont understand why?

When I look inside my container it says its only using 454GB out of 2TB leaving 1.4TB available for use. Ok that sounds good...

Filesystem                          Size  Used Avail Use% Mounted on
/dev/mapper/data2-vm--109--disk--0  2.0T  454G  1.4T  25% /
none                                492K  4.0K  488K   1% /dev
udev                                 48G     0   48G   0% /dev/tty
tmpfs                                48G   16K   48G   1% /dev/shm
tmpfs                               9.5G  224K  9.5G   1% /run
tmpfs                               5.0M     0  5.0M   0% /run/lock
tmpfs                                48G     0   48G   0% /sys/fs/cgroup

OK, but when I look at my LVM-Thin storage its almost completely full!?



This is the only container and data living on this LVM-thin (data2)



When I look at the container's resources I get even more confused...it says it has 1990GB dedicated in size to this LVM-thin, but in the CT Volumes menu its 2.14TB?



Here's an lvs command output

LV                          VG    Attr       LSize   Pool  Origin                      Data%  Meta%  Move Log Cpy%Sync Convert
  data2                       data2 twi-aotz--  <1.79t                                   86.83  4.82                          
  vm-109-disk-0               data2 Vwi-aotz--   1.94t data2                             79.84                                
  data                        pve   twi-aotz--  <1.67t                                   8.08   0.58                          
  root                        pve   -wi-ao----  96.00g                                                                        
  snap_vm-100-disk-0_Prima    pve   Vri---tz-k  40.00g data  vm-100-disk-0                                                    
  snap_vm-101-disk-1_Working  pve   Vri---tz-k  32.00g data  vm-101-disk-1                                                    
  snap_vm-103-disk-0_PassBolt pve   Vri---tz-k  12.00g data                                                                    
  snap_vm-107-disk-1_Working  pve   Vri---tz-k  12.00g data  vm-107-disk-1                                                    
  snap_vm-109-disk-1_Working  pve   Vri---tz-k   1.23t data  vm-109-disk-1                                                    
  snap_vm-110-disk-0_Certbot  pve   Vri---tz-k 100.00g data  vm-110-disk-0                                                    
  swap                        pve   -wi-ao----   8.00g                                                                        
  vm-100-disk-0               pve   Vwi-aotz--  40.00g data                              5.85                                  
  vm-100-state-Prima          pve   Vwi-a-tz-- <32.49g data                              5.65                                  
  vm-101-disk-1               pve   Vwi-aotz--  32.00g data                              30.93                                
  vm-102-disk-0               pve   Vwi-aotz--  12.00g data                              28.66                                
  vm-103-disk-0               pve   Vwi-aotz--  12.00g data  snap_vm-103-disk-0_PassBolt 28.06                                
  vm-104-disk-0               pve   Vwi-aotz--  12.00g data                              20.52                                
  vm-105-disk-0               pve   Vwi-aotz--  12.00g data                              45.09                                
  vm-107-disk-1               pve   Vwi-aotz--  12.00g data                              26.01                                
  vm-109-disk-1               pve   Vwi-a-tz--   1.23t data                              4.97                                  
  vm-110-disk-0               pve   Vwi-a-tz-- 100.00g data                              11.16                                
  vm-111-disk-2               pve   Vwi-aotz--  50.00g data                              29.33                                
  vm-111-state-First          pve   Vwi-a-tz--  <8.49g data                              46.13                                
  vm-112-disk-1               pve   Vwi-a-tz--  25.00g data                              8.23                                  
  vm-113-disk-0               pve   Vwi-aotz--  10.00g data                              13.29

I don't understand what could be filling up this LVM-thin so much?

Was LVM-thin a poor choice as a storage pool for a single large container?

Thanks guys.

 

[Dunuin]

One thing that will make LVM-Thin/ZFS storages grow and grow is using snapshots. As long as there is a snapshot no deleted data will be really deleted because that freeing up space is hold back by the snapshot.
Another thing is that thin-provisioned storages like LVM-Thin/ZFS need discard/TRIM support to be able to free up deleted data. So you need to use a protocol that supports discard (like virtio SCSI and not virtio block or IDE), make sure that the "discard" checkbox is set for each virtual disk and you need to setup every guest to actually use discard.

You should also check if your raid controller supports TRIM. Often they don't do so TRIM commands never reach the actual disks.

 

[pietroaretino]

One thing that will make LVM-Thin/ZFS storages grow and grow is using snapshots. As long as there is a snapshot no deleted data will be really deleted because that freeing up space is hold back by the snapshot.
Another thing is that thin-provisioned storages like LVM-Thin/ZFS need discard/TRIM support to be able to free up deleted data. So you need to use a protocol that supports discard (like virtio SCSI and not virtio block or IDE), make sure that the "discard" checkbox is set for each virtual disk and you need to setup every guest to actually use discard.

You should also check if your raid controller supports TRIM. Often they don't do so TRIM commands never reach the actual disks.

I appreciate this information a lot, thank you.

Where can I find the "discard" checkbox you're speaking of? Looking at the various options for the lvm-thin datastore and the container itself, I was unable to find any of this configuration.

Is there anyway I can save this lvm-thin storage? Or should I just backup and export the lxc-container to a separate storage, wipe the lvm-thin and rebuild it with the appropriate settings?

When I began a backup it gave me this information,

INFO: create storage snapshot 'vzdump'

  WARNING: You have not turned on protection against thin pools running out of space.

 WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.

  Logical volume "snap_vm-109-disk-0_vzdump" created.

 WARNING: Sum of all thin volume sizes (<3.89 TiB) exceeds the size of thin pool data2/data2 and the size of whole volume group (<1.82 TiB).

INFO: creating vzdump archive '/mnt/pve/AdriaNAS2/dump/vzdump-lxc-109-2022_02_05-11_19_41.tar.gz'

which I'm assuming has to do with the fact the lvm-thin is not getting rid of any blocks. deleted the one and only snapshot it had.

 

[Dunuin]

You only got that discard checkbox with VMs as these use block storages and not filesystems like LXCs.
You can force a trim of a LXC with this commmand: pct fstrim <VMIDofYourLXC>

 

[pietroaretino]

You only got that discard checkbox with VMs as these use block storages and not filesystems like LXCs.
You can force a trim of a LXC with this commmand: pct fstrim <VMIDofYourLXC>

Thanks again man. I am currently doing a backup of the container to a NAS. Once the backup has completed i will give that command a shot and return with the results.
Once again I appreciate you taking the time to respond and your assistance.
Cheers!

 

[pietroaretino]

What would be a preferable storage type for a single lxc-container with a fixed size?

Seems like lvm-thin would just be an infinite headache to manage this.

Should I format the storage pool and make it into a "Directory"?

 

[pietroaretino]

So I was able to successfully backup to my NAS and run the

pct fstrim

command.



This cleaned up my LVM-Thin. However I am still very confused as to why this LVM continues to grow and grow and grow when I am not adding any data to my only LXC container that lives in this LVM-Thin...if you see the graph above, you can see it began increasing in size towards the end.

After trimming it, it dropped from ~1.8TB to 751GB.

I began resizing the LXC container. Its root-disk had been set to 1990GB.
The actual size of the LXC was only 644GB, so I am shrinking it down to 700GB using e2fsck following this guide: https://yomis.blog/proxmox-resizing-a-disk/

I needed to shrink it down so I could move it to my other LVM-thin (local-lvm sda), because I want to delete and reconfigure my data2 LVM-thin (sdb)

As I began shrinking it I noticed that the size of the LVM-Thin increased by over 100GB?!

It went from 751GB to 873GB as you can see in the graph there, towards the end it began increasing, and its still going?!

This data2 (lvm-thin sdb) has no snapshots, it only has the one LXC container (109), and that LXC container only has a single back up to my external NAS via an NFS share.

What is still causing this data to infinitely increase when nothing is actively adding data to it?

If I move the LXC container to my local-lvm, is there a way I can wipe sdb and merge that RAID span into the sda lvm-thin?

I am extremely confused with the inconsistencies with storage as far as this lvm-thin and LXC container go.

Here it tells me this LXC container is 2.14TB?

While over here under the LXC container itself, it tells me its 1990GB

But when I boot up the container and go inside of it, a df -h command gives me the following output.

/dev/mapper/data2-vm--109--disk--0  2.0T  644G  1.2T  35% /
none                                492K  4.0K  488K   1% /dev
udev                                 48G     0   48G   0% /dev/tty
tmpfs                                48G   16K   48G   1% /dev/shm
tmpfs                               9.5G  128K  9.5G   1% /run
tmpfs                               5.0M     0  5.0M   0% /run/lock
tmpfs                                48G     0   48G   0% /sys/fs/cgroup

Any help in explaining this would be much appreciated. I just want to get this container into a stable storage pool that doesn't infinitely increase on its own. Ideally I'd just want to get reformat sdb into something more stable to host this container, but I have no idea if i should configure it into an lvm or a directory? Or if theres a way to merge those disks with the sda lvm-thin pool and then keep all my containers and vm on the same volume group? I'm fairly ignorant on logical-volume-management, pardon my ignorance.

Thanks again to anyone out there! Hopefully this might help someone else down the road. I will continue to update with my findings.

 

[Dunuin]

Its normal that thin-provisioned disks grow with every write (edit or even every edit in case of copy-on-write). Thats why you eighter mount you filesystem with discard so a trim is done after every single write or you setup a cron to hourly/daily trim your filesystem. You really should considering creating a hourly/daily cron running pct fstrim 109 so it gets trimmed each hour or day.
So something like adding this to your "/etc/crontab" for a hourly trim:

0  *   * * *   root      /sbin/pct fstrim 109 > /dev/null

 

[pietroaretino]

Its normal that thin-provisioned disks grow with every write (edit or even every edit in case of copy-on-write). Thats why you eighter mount you filesystem with discard so a trim is done after every single write or you setup a cron to hourly/daily trim your filesystem. You really should considering creating a hourly/daily cron running pct fstrim 109 so it gets trimmed each hour or day.
So something like adding this to your "/etc/crontab" for a hourly trim:

0  *   * * *   root      /sbin/pct fstrim 109 > /dev/null

Got it, so let me get this straight...
dev/sda/ has mounted partition dev/sda3, that holds the local-lvm, with "discard", while the lvm-thin data2 (dev/sdb) pool i created, wasn't mounted with "discard" which is why the pool is consistently filling up? Did I get that right?

So is there a way I can just wipe and re-initialize the data2 LVM-thin and then just merge it into the local-lvm pool, so I just have one large volume group? I don't know if that makes sense.

Thank you sir.

 

[pietroaretino]

In case anyone is reading this and having the same problems and want to do the same thing this is what I ended up doing.

I moved my LXC container to the separate RAID span. I am using a Dell Poweredge with Open Manage (OMSA) installed.

i was able to login to my open manage server panel at port 1311 and re-initialize the 2nd RAID10 span. This wiped the disks. Make sure youre not wiping the disks with your proxmox OS.

I SSHd in and created a new partition for sdb using fdisk.

I then turned it into an LVM by following this guide:
https://www.hostfav.com/blog/index.php/2017/02/01/add-a-new-physical-hard-drive-to-proxmox-ve-4x-5x/

I then moved my LXC container back to the new LVM storage, and now its living there happily without the LVM-Thin issue I was having before (data filling up entire drive).

I've lost the ability to do snapshots, but that wasn't very important for me. I may try to rebuild a stable LVM-Thin in the future to regain snapshot functionality, but until that becomes a priority a stable storage system is better for me.

I need to figure out if you can create a new lvm-thin storage and have it mounted with the "discard" or "fstrim" option Dunuin mentioned above. It seemed that when I was running fstrim manually it stopped my container, which was not a viable option for me, as this is a 24/7 production container that needs to be up.

If anyone can point me to an article or wiki discussing the best way to configure new lvm-thin storage provisioning with automatic "discard" and or "fstrim" for containers I'd much appreciate it. It is obviously possible as the pre-created "local-lvm" that Proxmox builds for you on first install, is an LVM-thin partition, and I have never experienced that partition becoming over-filled with data by itself, which means that there must be an fstrim or discard happening automatically on it, if i've understood anything about lvm-thin provisioning.

I'd also like to ask anyone out there, if they ever successfully merged two RAID10 hardware spans, into a single logical volume group?

I was thinking about making the second RAID10 span into a single physical volume, and then adding that physical volume to the pre-made proxmox "data" lvm-thin group, but I didn't know if this would cause some sort of problem seeing as its a hardware RAID handling two separate spans on the same server. Which is why I just made the second one into its own logical volume group.