VM io-error, local-lvm out of memory

[CAppa2]

Hi everyone, recently my VMs freeze and proxmox marks io-error status. I think it is due to the local-lvm volume being full

Here are some dumps:

root@pve:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   12G     0   12G   0% /dev
tmpfs                 2.4G  9.1M  2.4G   1% /run
/dev/mapper/pve-root   29G  5.2G   23G  19% /
tmpfs                  12G   43M   12G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                  12G     0   12G   0% /sys/fs/cgroup
/dev/nvme0n1p2        511M  312K  511M   1% /boot/efi
/dev/fuse              30M   20K   30M   1% /etc/pve
tmpfs                 2.4G     0  2.4G   0% /run/user/0

root@pve:~# vgs
  VG  #PV #LV #SN Attr   VSize    VFree
  pve   1   7   0 wz--n- <118.74g 14.75g

root@pve:~# pvs
  PV             VG  Fmt  Attr PSize    PFree
  /dev/nvme0n1p3 pve lvm2 a--  <118.74g 14.75g

root@pve:~# lvs
  LV            VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data          pve twi-aotzD- <67.49g             100.00 4.62                           
  root          pve -wi-ao----  29.50g                                                   
  swap          pve -wi-ao----   5.00g                                                   
  vm-100-disk-0 pve Vwi-aotz--  32.00g data        81.81                                 
  vm-100-disk-1 pve Vwi-aotz--  20.00g data        48.46                                 
  vm-102-disk-0 pve Vwi-aotz--  32.00g data        98.81                                 
  vm-102-disk-2 pve Vwi-aotz--   4.00m data        3.12    

root@pve:~# pvesm status
Name              Type     Status           Total            Used       Available        %
Backup_VMs         dir   disabled               0               0               0      N/A
local              dir     active        30316484         5443500        23309952   17.96%
local-lvm      lvmthin     active        70766592        70766592               0  100.00%

As I understand it the logical volume data (which should be the volume where the virtual disks are saved) is full. From the dumps I also see that there is still ~ 15GB in the volume group and physical volume.

Unfortunately I have no way to immediately have more space for virtual disks, until then I would like to make do with these 15GB still free until I buy a new SSD.

I know that I can extend the size of the data volume via LVM commands lvresize -L [+|-][Size] <vgname>/<lvname> or something like that. The problem is that I don't quite understand how, since it is an LVM-thin volume and so there is also the volume of metadata.
I don't want to screw everything up due to incorrect commands, so:

1. Is it safe to expand the volume this way?
   
2. Otherwise what commands should i give to expand the volume safely?
3. If I don't get it wrong, eventually I should also resize the filesystem with something like this resize2fs /<absolute_path_volume>?
4. Is there an easier and safer way using pvesm? I couldn't figure it out from here: PVE-admin-guide-storage

All VMs run under proxmox 6.3-2, Thank you very much in advance for your time.

 

[CAppa2]

No one knows how to do it?

 

[Dunuin]

The PVE installer always leaves some space unused so LVM can use it for snapshots and so on. So that unused space is there on purpose.
If I were you, I would first try to free up some stuff. You could delete old logs (easily can be some GB), remove unused packages (apt autoremove), run a fstrim -a inside your VMs and check that discard is enabled for your virtual disk and that a protocol like SCSI is used that supports discard, look if you find some files you don't need anymore, ...

 

[CAppa2]

Thank you for your answer. In order to do this I should have access to the VMs or to one of them, which I don't have because they don't start, they go into io-error immediately and crash.

 

[SawKyrom]

@CAppa2 I feel your pain. Proxmox evidently doesn't have a safe guard to prevent LVM-thin volume from exceeding storage capacity and once this occurs, VM's are rendered inoperable with I/O errors. From experience, I would definitely NOT try lvresize command. In my case, doing so resulted in an EXT4-fs error that I've still haven't recovered.

Instead, try deleting old or unused VM's or even a snapshot. Once you free up space and can access VM, you can do some housekeeping and regenerate the snapshot lost. I would recommend not setting up backups/snapshots on the LVM-thin volume. See link below for instructions on removing snapshots. Good luck!

https://forum.proxmox.com/threads/removing-snapshots-cli.103102/#post-443953