[SOLVED] io-error on all VMs- HDD Full?

[passuff]

Hello everybody
I'm a real noob with Proxmox and I've no clue about different storages. Maybe someone can support me with finding the issue about io-error/ storage is full.

Here is a first overview:

root@pve:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 12G 0 12G 0% /dev
tmpfs 2.4G 9.7M 2.4G 1% /run
/dev/mapper/pve-root 27G 12G 15G 44% /
tmpfs 12G 43M 12G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 12G 0 12G 0% /sys/fs/cgroup
/dev/sda2 511M 324K 511M 1% /boot/efi
/dev/fuse 30M 20K 30M 1% /etc/pve
//192.168.1.44/Proxmox 2.6T 1.4T 1.2T 54% /mnt/pve/MegaServer
tmpfs 2.4G 0 2.4G 0% /run/user/0

 

[oguz]

hi,

try df -hi to see inodes usage. maybe you ran out of inodes. if you see 100% inode usage for a storage, this usually means you have a lot of small files somewhere on the disk. deleting those will free it

 

[passuff]

root@pve:~# df -hi
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 3.0M 554 3.0M 1% /dev
tmpfs 3.0M 1.1K 3.0M 1% /run
/dev/mapper/pve-root 1.8M 121K 1.7M 7% /
tmpfs 3.0M 99 3.0M 1% /dev/shm
tmpfs 3.0M 16 3.0M 1% /run/lock
tmpfs 3.0M 18 3.0M 1% /sys/fs/cgroup
/dev/sda2 0 0 0 - /boot/efi
/dev/fuse 9.8K 40 9.8K 1% /etc/pve
tmpfs 3.0M 10 3.0M 1% /run/user/0


Seems that is not the case...

 

[oguz]

okay. where are you getting the io errors then? can you give some more context? how often does it happen?

 

[passuff]

currently all VMs are halted that have local-LVM as storage source. Maybe this is an hint.

 

[oguz]

lvs -a
vgs -a
lsblk

should give an overview

 

[passuff]

root@pve:/etc/apt/sources.list.d# lvs -a
  LV              VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vm-101-disk-0   Volume2 -wi-a-----  <2.62t
  data            pve     twi-aotzD- <59.50g             100.00 4.47
  [data_tdata]    pve     Twi-ao---- <59.50g
  [data_tmeta]    pve     ewi-ao----   1.00g
  [lvol0_pmspare] pve     ewi-------   1.00g
  root            pve     -wi-ao----  27.50g
  swap            pve     -wi-ao----   8.00g
  vm-100-disk-0   pve     Vwi-a-tz--   8.00g data        57.66
  vm-102-disk-0   pve     Vwi-aotz--  32.00g data        25.71
  vm-103-disk-0   pve     Vwi-aotz--  32.00g data        30.35
  vm-107-disk-0   pve     Vwi-a-tz--  32.00g data        100.00
  vm-110-disk-0   pve     Vwi-aotz--  16.25g data        30.42

root@pve:/etc/apt/sources.list.d# vgs -a
 VG      #PV #LV #SN Attr   VSize    VFree
  Volume2   1   1   0 wz--n-   <2.62t <1.25g
  pve       1   8   0 wz--n- <110.75g 13.75g

root@pve:/etc/apt/sources.list.d# lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 111.3G  0 disk
├─sda1                         8:1    0  1007K  0 part
├─sda2                         8:2    0   512M  0 part /boot/efi
└─sda3                         8:3    0 110.8G  0 part
  ├─pve-swap                 253:1    0     8G  0 lvm  [SWAP]
  ├─pve-root                 253:2    0  27.5G  0 lvm  /
  ├─pve-data_tmeta           253:3    0     1G  0 lvm
  │ └─pve-data-tpool         253:5    0  59.5G  0 lvm
  │   ├─pve-data             253:6    0  59.5G  0 lvm
  │   ├─pve-vm--100--disk--0 253:7    0     8G  0 lvm
  │   ├─pve-vm--102--disk--0 253:8    0    32G  0 lvm
  │   ├─pve-vm--103--disk--0 253:9    0    32G  0 lvm
  │   ├─pve-vm--107--disk--0 253:10   0    32G  0 lvm
  │   └─pve-vm--110--disk--0 253:11   0  16.3G  0 lvm
  └─pve-data_tdata           253:4    0  59.5G  0 lvm
    └─pve-data-tpool         253:5    0  59.5G  0 lvm
      ├─pve-data             253:6    0  59.5G  0 lvm
      ├─pve-vm--100--disk--0 253:7    0     8G  0 lvm
      ├─pve-vm--102--disk--0 253:8    0    32G  0 lvm
      ├─pve-vm--103--disk--0 253:9    0    32G  0 lvm
      ├─pve-vm--107--disk--0 253:10   0    32G  0 lvm
      └─pve-vm--110--disk--0 253:11   0  16.3G  0 lvm
sdb                            8:16   0   2.6T  0 disk
└─Volume2-vm--101--disk--0   253:0    0   2.6T  0 lvm

 

[oguz]

root@pve:/etc/apt/sources.list.d# lvs -a
  LV              VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vm-101-disk-0   Volume2 -wi-a-----  <2.62t
  data            pve     twi-aotzD- <59.50g             100.00 4.47

your pve/data lv is full. check what VM is using that storage and maybe remove some stuff from there (backups, snapshots, unneeded vm etc.). you can also try to extend the size of the lv if there's nothing you can remove.

 

[passuff]

That exactly was the problem. Thx a lot. I removed one not needed VM and everything is running again. But still I wonder why it was not like that earlier, as the configuration was untouched for months. I did not create any disk.
How can I extend a lv? Does it work in proxmox with gui or do I have to use cmd?

 

[oguz]

But still I wonder why it was not like that earlier, as the configuration was untouched for months. I did not create any disk.

disks of VMs can get filled up over time with logs etc.

How can I extend a lv? Does it work in proxmox with gui or do I have to use cmd?

you will need to do it over the command line. check here[0] for instructions.

[0]: https://www.linuxtechi.com/extend-lvm-partitions/

 

[passuff]

thx alot for your great support!

 

[oguz]

you're welcome!

please mark the thread as [SOLVED] by editing the thread prefix, that way others who run into the same problem know what to expect!

 

[Rasel11]

root@pve:/etc/apt/sources.list.d# lvs -a
  LV              VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vm-101-disk-0   Volume2 -wi-a-----  <2.62t
  data            pve     twi-aotzD- <59.50g             100.00 4.47

your pve/data lv is full. check what VM is using that storage and maybe remove some stuff from there (backups, snapshots, unneeded vm etc.). you can also try to extend the size of the lv if there's nothing you can remove.

Really apriciatig your kind support. I just want to know to can i check which VM is using that storage and how can i remove some stuff from that.if you can give me a link or solution that will be very helpfull.

 

[oguz]

Really apriciatig your kind support. I just want to know to can i check which VM is using that storage and how can i remove some stuff from that.if you can give me a link or solution that will be very helpfull.

if your VM are stored on LVM storage then lvs -a should give you an overview

 

[Rasel11]

if your VM are stored on LVM storage then lvs -a should give you an overview

Thanks i have found and solve the issue. Can you please tell me how i can solve the problem without deleting the VM or stuffs? how i could extend my lv

 

[semutmikro]

if your VM are stored on LVM storage then lvs -a should give you an overview

can i get into disk and delete some file ?