[SOLVED] io-error on all VMs- HDD Full?

passuff

Member
Aug 15, 2019
19
1
8
39
Hello everybody
I'm a real noob with Proxmox and I've no clue about different storages. Maybe someone can support me with finding the issue about io-error/ storage is full.

Here is a first overview:

root@pve:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 12G 0 12G 0% /dev
tmpfs 2.4G 9.7M 2.4G 1% /run
/dev/mapper/pve-root 27G 12G 15G 44% /
tmpfs 12G 43M 12G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 12G 0 12G 0% /sys/fs/cgroup
/dev/sda2 511M 324K 511M 1% /boot/efi
/dev/fuse 30M 20K 30M 1% /etc/pve
//192.168.1.44/Proxmox 2.6T 1.4T 1.2T 54% /mnt/pve/MegaServer
tmpfs 2.4G 0 2.4G 0% /run/user/0
 

oguz

Proxmox Retired Staff
Retired Staff
Nov 19, 2018
5,207
692
118
hi,

try df -hi to see inodes usage. maybe you ran out of inodes. if you see 100% inode usage for a storage, this usually means you have a lot of small files somewhere on the disk. deleting those will free it
 

passuff

Member
Aug 15, 2019
19
1
8
39
root@pve:~# df -hi
Filesystem Inodes IUsed IFree IUse% Mounted on
udev 3.0M 554 3.0M 1% /dev
tmpfs 3.0M 1.1K 3.0M 1% /run
/dev/mapper/pve-root 1.8M 121K 1.7M 7% /
tmpfs 3.0M 99 3.0M 1% /dev/shm
tmpfs 3.0M 16 3.0M 1% /run/lock
tmpfs 3.0M 18 3.0M 1% /sys/fs/cgroup
/dev/sda2 0 0 0 - /boot/efi
/dev/fuse 9.8K 40 9.8K 1% /etc/pve
tmpfs 3.0M 10 3.0M 1% /run/user/0


Seems that is not the case...
 

oguz

Proxmox Retired Staff
Retired Staff
Nov 19, 2018
5,207
692
118
okay. where are you getting the io errors then? can you give some more context? how often does it happen?
 

passuff

Member
Aug 15, 2019
19
1
8
39
currently all VMs are halted that have local-LVM as storage source. Maybe this is an hint.
 

oguz

Proxmox Retired Staff
Retired Staff
Nov 19, 2018
5,207
692
118
Code:
lvs -a
vgs -a
lsblk

should give an overview
 

passuff

Member
Aug 15, 2019
19
1
8
39
Code:
root@pve:/etc/apt/sources.list.d# lvs -a
  LV              VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vm-101-disk-0   Volume2 -wi-a-----  <2.62t
  data            pve     twi-aotzD- <59.50g             100.00 4.47
  [data_tdata]    pve     Twi-ao---- <59.50g
  [data_tmeta]    pve     ewi-ao----   1.00g
  [lvol0_pmspare] pve     ewi-------   1.00g
  root            pve     -wi-ao----  27.50g
  swap            pve     -wi-ao----   8.00g
  vm-100-disk-0   pve     Vwi-a-tz--   8.00g data        57.66
  vm-102-disk-0   pve     Vwi-aotz--  32.00g data        25.71
  vm-103-disk-0   pve     Vwi-aotz--  32.00g data        30.35
  vm-107-disk-0   pve     Vwi-a-tz--  32.00g data        100.00
  vm-110-disk-0   pve     Vwi-aotz--  16.25g data        30.42
Code:
root@pve:/etc/apt/sources.list.d# vgs -a
 VG      #PV #LV #SN Attr   VSize    VFree
  Volume2   1   1   0 wz--n-   <2.62t <1.25g
  pve       1   8   0 wz--n- <110.75g 13.75g
Code:
root@pve:/etc/apt/sources.list.d# lsblk
NAME                         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                            8:0    0 111.3G  0 disk
├─sda1                         8:1    0  1007K  0 part
├─sda2                         8:2    0   512M  0 part /boot/efi
└─sda3                         8:3    0 110.8G  0 part
  ├─pve-swap                 253:1    0     8G  0 lvm  [SWAP]
  ├─pve-root                 253:2    0  27.5G  0 lvm  /
  ├─pve-data_tmeta           253:3    0     1G  0 lvm
  │ └─pve-data-tpool         253:5    0  59.5G  0 lvm
  │   ├─pve-data             253:6    0  59.5G  0 lvm
  │   ├─pve-vm--100--disk--0 253:7    0     8G  0 lvm
  │   ├─pve-vm--102--disk--0 253:8    0    32G  0 lvm
  │   ├─pve-vm--103--disk--0 253:9    0    32G  0 lvm
  │   ├─pve-vm--107--disk--0 253:10   0    32G  0 lvm
  │   └─pve-vm--110--disk--0 253:11   0  16.3G  0 lvm
  └─pve-data_tdata           253:4    0  59.5G  0 lvm
    └─pve-data-tpool         253:5    0  59.5G  0 lvm
      ├─pve-data             253:6    0  59.5G  0 lvm
      ├─pve-vm--100--disk--0 253:7    0     8G  0 lvm
      ├─pve-vm--102--disk--0 253:8    0    32G  0 lvm
      ├─pve-vm--103--disk--0 253:9    0    32G  0 lvm
      ├─pve-vm--107--disk--0 253:10   0    32G  0 lvm
      └─pve-vm--110--disk--0 253:11   0  16.3G  0 lvm
sdb                            8:16   0   2.6T  0 disk
└─Volume2-vm--101--disk--0   253:0    0   2.6T  0 lvm
 

oguz

Proxmox Retired Staff
Retired Staff
Nov 19, 2018
5,207
692
118
Code:
root@pve:/etc/apt/sources.list.d# lvs -a
  LV              VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vm-101-disk-0   Volume2 -wi-a-----  <2.62t
  data            pve     twi-aotzD- <59.50g             100.00 4.47

your pve/data lv is full. check what VM is using that storage and maybe remove some stuff from there (backups, snapshots, unneeded vm etc.). you can also try to extend the size of the lv if there's nothing you can remove.
 

passuff

Member
Aug 15, 2019
19
1
8
39
That exactly was the problem. Thx a lot. I removed one not needed VM and everything is running again. But still I wonder why it was not like that earlier, as the configuration was untouched for months. I did not create any disk.
How can I extend a lv? Does it work in proxmox with gui or do I have to use cmd?
 

oguz

Proxmox Retired Staff
Retired Staff
Nov 19, 2018
5,207
692
118

oguz

Proxmox Retired Staff
Retired Staff
Nov 19, 2018
5,207
692
118
you're welcome!

please mark the thread as [SOLVED] by editing the thread prefix, that way others who run into the same problem know what to expect!
 

Rasel11

New Member
May 25, 2022
2
0
1
Code:
root@pve:/etc/apt/sources.list.d# lvs -a
  LV              VG      Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  vm-101-disk-0   Volume2 -wi-a-----  <2.62t
  data            pve     twi-aotzD- <59.50g             100.00 4.47

your pve/data lv is full. check what VM is using that storage and maybe remove some stuff from there (backups, snapshots, unneeded vm etc.). you can also try to extend the size of the lv if there's nothing you can remove.
Really apriciatig your kind support. I just want to know to can i check which VM is using that storage and how can i remove some stuff from that.if you can give me a link or solution that will be very helpfull.
 

oguz

Proxmox Retired Staff
Retired Staff
Nov 19, 2018
5,207
692
118
Really apriciatig your kind support. I just want to know to can i check which VM is using that storage and how can i remove some stuff from that.if you can give me a link or solution that will be very helpfull.
if your VM are stored on LVM storage then lvs -a should give you an overview
 

Rasel11

New Member
May 25, 2022
2
0
1
if your VM are stored on LVM storage then lvs -a should give you an overview
Thanks i have found and solve the issue. Can you please tell me how i can solve the problem without deleting the VM or stuffs? how i could extend my lv
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!