[SOLVED] ZFS IO disk error

Toni Blanch

Member
May 8, 2019
27
0
6
49
Hello everyone, I need your help, I have a 4 disc raid,
Captura de Pantalla 2019-10-07 a les 15.41.53.png

With a 15TB of storage.

Captura de Pantalla 2019-10-07 a les 15.52.22.png

although the machine disk is 15000 GB, it gives me that error message.

More info.
Captura de Pantalla 2019-10-07 a les 15.41.37.png
Captura de Pantalla 2019-10-07 a les 16.00.47.png



the available space is kb and I think that's the problem, can you help me?

can i reduce the disk size?
 

Attachments

  • Captura de Pantalla 2019-10-07 a les 15.41.30.png
    Captura de Pantalla 2019-10-07 a les 15.41.30.png
    17 KB · Views: 12
Do you have some snapshots for that VM?

What's the output of zfs list -t snapshot -r almacen?

You can copy and paste the output into CODE tags.
 
Do you have some snapshots for that VM?

What's the output of zfs list -t snapshot -r almacen?

You can copy and paste the output into CODE tags.

root@FTPSERVER:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
almacen 15,3T 599K 140K /almacen
almacen/vm-142-disk-0 15,3T 599K 15,3T -
rpool 18,4G 443G 96K /rpool
rpool/ROOT 13,9G 443G 96K /rpool/ROOT
rpool/ROOT/pve-1 13,9G 443G 13,9G /
rpool/data 96K 443G 96K /rpool/data
rpool/swap 4,25G 446G 974M -
root@FTPSERVER:~# zfs list -t snapshot -r almacen
no datasets available
root@FTPSERVER:~#
 
Can you post the contents of your /etc/pve/storage.cfg file?
How full does the VM report it's disk?
 
Can you post the contents of your /etc/pve/storage.cfg file?
How full does the VM report it's disk?

Hello, this is the disk datos.ftp(F:)


Captura de pantalla 2019-10-07 a las 18.28.33.png


The storage.cfg:

zfspool: almacen
pool almacen
blocksize 32
content images
nodes FTPSERVER
sparse 0
 
Okay, so the Storage does not have "Thin" enabled and you created a disk the size of the volume in it.
This means that the full size of the VMs disk is reserved on the zpool and therefore it already appears full.

The rule of thumb for zpools is to not fill them up to more than around 80% to avoid performance reduction.

If you want to get rid of the warning the safest option would be to backup the data of that disk and recreate it with a smaller size.
 
Thank you very much, for your answer, but the vm does not start, with the error message, and I can not get the data, is there any way to reduce the size of the raw file? Can you help me with this?
 
but the vm does not start, with the error message,
Is there something missing in the first post? I don't see an error message?

You can remove the reservation for that VMs disk. This will make ZFS show more available space and probably help with whatever error you get when you want to start that VM.
Code:
 zfs set refreservation=none almacen/vm-142-disk-0

Just don't leave it like that because with this the disk can still grow to 15TB if you fill it.
 
Is there something missing in the first post? I don't see an error message?

You can remove the reservation for that VMs disk. This will make ZFS show more available space and probably help with whatever error you get when you want to start that VM.
Code:
 zfs set refreservation=none almacen/vm-142-disk-0

Just don't leave it like that because with this the disk can still grow to 15TB if you fill it.

It does not start, I think the only option is to reduce the raw file, is it possible?

Captura de Pantalla 2019-10-09 a les 15.10.00.png
 
At the bottom you have the task logs (maybe collapsed). You should see the start log of that VM if you double click on it. What does it say?
 
Ah okay. I took me a bit to fully realize what situation you are in.

Let me summarize to make sure we are on the same page.
The VM will not boot because at some point it will stop with an IO error?

Unfortunately you cannot reduze the size of the disk on the zpool from outside. The problem with almost no available space with ZFS is, that it is a copy on write file system. Each new change is written to a new block. Old blocks are only overwritten if they are not in use anymore. That is why the rule of thumb is to not fill it up more than 80%.

Is that VM disk the only location where you have those files?
If no then destroy it and recreate it with a smaller size or use something other than ZFS underneath it.

If yes then you could try
  • to boot the VM from a rescue ISO and shrink the partition in that disk
  • enable Discard and SSD Emulation for that VM disk and make sure Trim is enabled in Windows (might release some space on ZFS if the disk is not filled completely with files)
  • boot the VM from a rescue ISO and copy the data somewhere else in order to then destroy and recreate the disk smaller.
 
Ah okay. I took me a bit to fully realize what situation you are in.

Let me summarize to make sure we are on the same page.
The VM will not boot because at some point it will stop with an IO error?

Unfortunately you cannot reduze the size of the disk on the zpool from outside. The problem with almost no available space with ZFS is, that it is a copy on write file system. Each new change is written to a new block. Old blocks are only overwritten if they are not in use anymore. That is why the rule of thumb is to not fill it up more than 80%.

Is that VM disk the only location where you have those files?
If no then destroy it and recreate it with a smaller size or use something other than ZFS underneath it.

If yes then you could try
  • to boot the VM from a rescue ISO and shrink the partition in that disk
  • enable Discard and SSD Emulation for that VM disk and make sure Trim is enabled in Windows (might release some space on ZFS if the disk is not filled completely with files)
  • boot the VM from a rescue ISO and copy the data somewhere else in order to then destroy and recreate the disk smaller.
Hello!
I am already recovering the files inside the raw !, thank you very much, finally with a live cd with the appropriate drivers I have been able to access the info and I am already transferring the information.
When I finish, I will remove the raw disk and create it at 80% of the pool capacity.
Your help has been great!
 
You're welcome. Please mark the thread as solved :)
 
  • Like
Reactions: Toni Blanch

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!