LVM at 100% full on 1 node

[squeezerie]

I'm relatively new to the Proxmox tool and not that well versed with Linux environment particularly with how LVM storage work.
I built a 3 node cluster and 1 host got 100% full on the LVM (local-lvm), this node now has a constant Connection error - Timeout.

Name                      Type     Status           Total            Used       Available        %
local                      dir     active        38414944        27648008         8890324   71.97%
local-lvm                  lvm     active        76615680        76615680               0  100.00%
lvm-thin               lvmthin     active        26918912            2691        26916220    0.01%
truenas-gp01               nfs     active      5487517056      1349715840      4137801216   24.60%
truenas-pve-gp01           nfs     active      4344080000       206278912      4137801088    4.75%
truenas-pve-images         nfs     active      4146728832         8927616      4137801216    0.22%

 

[Dunuin]

You should set up proper monitoring. When working with thin-provisioned storages like LVM-Thin it's your duty to monitor it so this will never happen in the first place as you should delete stuff or expand the storage before this happens. Filling it completely up means downtime and more or less data-loss.

First I would see if there are any snapshots you could delete and if discard/TRIM is properly set up within each VMs guestOS, if virtio SCSI is used with "discard" checkbox enabled for the virtual disks and if you run "pct fstrim YourVMID" regularily.

If thats all done and you still need space, backup and destroy VMs/LXCs (or move them to another storage) until you are below 80% capacity again.