VM io-error, local-lvm out of memory

CAppa2

New Member
Feb 27, 2021
8
0
1
Hi everyone, recently my VMs freeze and proxmox marks io-error status. I think it is due to the local-lvm volume being full

Here are some dumps:
Code:
root@pve:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
udev                   12G     0   12G   0% /dev
tmpfs                 2.4G  9.1M  2.4G   1% /run
/dev/mapper/pve-root   29G  5.2G   23G  19% /
tmpfs                  12G   43M   12G   1% /dev/shm
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                  12G     0   12G   0% /sys/fs/cgroup
/dev/nvme0n1p2        511M  312K  511M   1% /boot/efi
/dev/fuse              30M   20K   30M   1% /etc/pve
tmpfs                 2.4G     0  2.4G   0% /run/user/0

root@pve:~# vgs
  VG  #PV #LV #SN Attr   VSize    VFree
  pve   1   7   0 wz--n- <118.74g 14.75g

root@pve:~# pvs
  PV             VG  Fmt  Attr PSize    PFree
  /dev/nvme0n1p3 pve lvm2 a--  <118.74g 14.75g

root@pve:~# lvs
  LV            VG  Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  data          pve twi-aotzD- <67.49g             100.00 4.62                           
  root          pve -wi-ao----  29.50g                                                   
  swap          pve -wi-ao----   5.00g                                                   
  vm-100-disk-0 pve Vwi-aotz--  32.00g data        81.81                                 
  vm-100-disk-1 pve Vwi-aotz--  20.00g data        48.46                                 
  vm-102-disk-0 pve Vwi-aotz--  32.00g data        98.81                                 
  vm-102-disk-2 pve Vwi-aotz--   4.00m data        3.12    

root@pve:~# pvesm status
Name              Type     Status           Total            Used       Available        %
Backup_VMs         dir   disabled               0               0               0      N/A
local              dir     active        30316484         5443500        23309952   17.96%
local-lvm      lvmthin     active        70766592        70766592               0  100.00%

As I understand it the logical volume data (which should be the volume where the virtual disks are saved) is full. From the dumps I also see that there is still ~ 15GB in the volume group and physical volume.

Unfortunately I have no way to immediately have more space for virtual disks, until then I would like to make do with these 15GB still free until I buy a new SSD.

I know that I can extend the size of the data volume via LVM commands lvresize -L [+|-][Size] <vgname>/<lvname> or something like that. The problem is that I don't quite understand how, since it is an LVM-thin volume and so there is also the volume of metadata.
I don't want to screw everything up due to incorrect commands, so:
  1. Is it safe to expand the volume this way?
  2. Otherwise what commands should i give to expand the volume safely?
  3. If I don't get it wrong, eventually I should also resize the filesystem with something like this resize2fs /<absolute_path_volume>?
  4. Is there an easier and safer way using pvesm? I couldn't figure it out from here: PVE-admin-guide-storage
All VMs run under proxmox 6.3-2, Thank you very much in advance for your time.
 

Dunuin

Famous Member
Jun 30, 2020
9,015
2,315
156
Germany
The PVE installer always leaves some space unused so LVM can use it for snapshots and so on. So that unused space is there on purpose.
If I were you, I would first try to free up some stuff. You could delete old logs (easily can be some GB), remove unused packages (apt autoremove), run a fstrim -a inside your VMs and check that discard is enabled for your virtual disk and that a protocol like SCSI is used that supports discard, look if you find some files you don't need anymore, ...
 
Last edited:

CAppa2

New Member
Feb 27, 2021
8
0
1
Thank you for your answer. In order to do this I should have access to the VMs or to one of them, which I don't have because they don't start, they go into io-error immediately and crash.
 

SawKyrom

New Member
Jun 3, 2021
20
1
3
49
@CAppa2 I feel your pain. Proxmox evidently doesn't have a safe guard to prevent LVM-thin volume from exceeding storage capacity and once this occurs, VM's are rendered inoperable with I/O errors. From experience, I would definitely NOT try lvresize command. In my case, doing so resulted in an EXT4-fs error that I've still haven't recovered.

Instead, try deleting old or unused VM's or even a snapshot. Once you free up space and can access VM, you can do some housekeeping and regenerate the snapshot lost. I would recommend not setting up backups/snapshots on the LVM-thin volume. See link below for instructions on removing snapshots. Good luck!

https://forum.proxmox.com/threads/removing-snapshots-cli.103102/#post-443953
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!