proxmox-vm "status error: io-error", local storage is full.

[breakandfix]

Hi there,

My proxmox VM is not accessible and it has io-error status with yellow attention. I think I by mistake used local storage instead of the disk I attached to my proxmox node.
and now local storage is completely full. What to do, please help ASAP.

how to extend the local storage with another storage type or how to fix this issue?
Thank you.

df -H

Filesystem           Size  Used Avail Use% Mounted on
udev                 136G     0  136G   0% /dev
tmpfs                 28G   17M   27G   1% /run
/dev/md3              21G  5.6G   15G  28% /
tmpfs                136G   52M  136G   1% /dev/shm
tmpfs                5.3M     0  5.3M   0% /run/lock
/dev/md2             1.1G   95M  871M  10% /boot
/dev/sdb1            536M  173k  536M   1% /boot/efi
/dev/mapper/vg-data  448G  448G     0 100% /var/lib/vz
/dev/fuse            135M   29k  135M   1% /etc/pve
storage              7.6T  132k  7.6T   1% /storage
tmpfs                 28G     0   28G   0% /run/user/0

/var/lib/vz - I only see the image of VM which is fully used.


lsblk

NAME          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda             8:0    1 447.1G  0 disk
├─sda1          8:1    1   511M  0 part
├─sda2          8:2    1     1G  0 part
│ └─md2         9:2    0  1022M  0 raid1 /boot
├─sda3          8:3    1    20G  0 part
│ └─md3         9:3    0    20G  0 raid1 /
├─sda4          8:4    1     1G  0 part  [SWAP]
├─sda5          8:5    1 424.6G  0 part
│ └─md5         9:5    0 424.5G  0 raid1
│   └─vg-data 253:0    0 424.5G  0 lvm   /var/lib/vz
└─sda6          8:6    1     2M  0 part
sdb             8:16   1 447.1G  0 disk
├─sdb1          8:17   1   511M  0 part  /boot/efi
├─sdb2          8:18   1     1G  0 part
│ └─md2         9:2    0  1022M  0 raid1 /boot
├─sdb3          8:19   1    20G  0 part
│ └─md3         9:3    0    20G  0 raid1 /
├─sdb4          8:20   1     1G  0 part  [SWAP]
└─sdb5          8:21   1 424.6G  0 part
  └─md5         9:5    0 424.5G  0 raid1
    └─vg-data 253:0    0 424.5G  0 lvm   /var/lib/vz
nvme0n1       259:0    0   3.5T  0 disk
├─nvme0n1p1   259:6    0   3.5T  0 part
└─nvme0n1p9   259:7    0     8M  0 part
nvme3n1       259:1    0   3.5T  0 disk
├─nvme3n1p1   259:10   0   3.5T  0 part
└─nvme3n1p9   259:11   0     8M  0 part
nvme2n1       259:2    0   3.5T  0 disk
├─nvme2n1p1   259:8    0   3.5T  0 part
└─nvme2n1p9   259:9    0     8M  0 part
nvme1n1       259:3    0   3.5T  0 disk
├─nvme1n1p1   259:4    0   3.5T  0 part
└─nvme1n1p9   259:5    0     8M  0 part

please let me know if you require more details.

 

[bbgeek17]

Hi @breakandfix , welcome to the forum.

You are in a standard Linux management territory. You have indeed filled up your LVM slice dedicated to /var/lib/vz.
There are multiple avenues you can pursue:
a) take one of your NVMe disks, place a filesystem on it, mount it to a temp location, move data manually from /var/lib/vz to /mnt/temp, permanently mount new disk to /var/lib/vz
b) expand your LVM VG with a disk

You have a few complications to consider:
1) your root disk is an MD device
2) root disks do not appear to be NVMe
Given this, I would not recommend option (b) as it will create mix of disk types and disk formats.

You could also do partial (a) by moving all data to new disk, then adding this disk as storage pool in PVE. You will need to update your VM configuration.

Good luck

---

Blockbridge : Ultra low latency all-NVME shared storage for Proxmox - https://www.blockbridge.com/proxmox