[SOLVED] ZFS IO disk error

[Toni Blanch]

Hello everyone, I need your help, I have a 4 disc raid,


With a 15TB of storage.



although the machine disk is 15000 GB, it gives me that error message.

More info.





the available space is kb and I think that's the problem, can you help me?

can i reduce the disk size?

 

[aaron]

Do you have some snapshots for that VM?

What's the output of zfs list -t snapshot -r almacen?

You can copy and paste the output into CODE tags.

 

[Toni Blanch]

Do you have some snapshots for that VM?

What's the output of zfs list -t snapshot -r almacen?

You can copy and paste the output into CODE tags.

root@FTPSERVER:~# zfs list
NAME USED AVAIL REFER MOUNTPOINT
almacen 15,3T 599K 140K /almacen
almacen/vm-142-disk-0 15,3T 599K 15,3T -
rpool 18,4G 443G 96K /rpool
rpool/ROOT 13,9G 443G 96K /rpool/ROOT
rpool/ROOT/pve-1 13,9G 443G 13,9G /
rpool/data 96K 443G 96K /rpool/data
rpool/swap 4,25G 446G 974M -
root@FTPSERVER:~# zfs list -t snapshot -r almacen
no datasets available
root@FTPSERVER:~#

 

[aaron]

Can you post the contents of your /etc/pve/storage.cfg file?
How full does the VM report it's disk?

 

[Toni Blanch]

Can you post the contents of your /etc/pve/storage.cfg file?
How full does the VM report it's disk?

Hello, this is the disk datos.ftp(F





The storage.cfg:

zfspool: almacen
pool almacen
blocksize 32
content images
nodes FTPSERVER
sparse 0

 

[aaron]

Okay, so the Storage does not have "Thin" enabled and you created a disk the size of the volume in it.
This means that the full size of the VMs disk is reserved on the zpool and therefore it already appears full.

The rule of thumb for zpools is to not fill them up to more than around 80% to avoid performance reduction.

If you want to get rid of the warning the safest option would be to backup the data of that disk and recreate it with a smaller size.

 

[Toni Blanch]

Thank you very much, for your answer, but the vm does not start, with the error message, and I can not get the data, is there any way to reduce the size of the raw file? Can you help me with this?

 

[aaron]

but the vm does not start, with the error message,

Is there something missing in the first post? I don't see an error message?

You can remove the reservation for that VMs disk. This will make ZFS show more available space and probably help with whatever error you get when you want to start that VM.

 zfs set refreservation=none almacen/vm-142-disk-0

Just don't leave it like that because with this the disk can still grow to 15TB if you fill it.

 

[Toni Blanch]

Is there something missing in the first post? I don't see an error message?

You can remove the reservation for that VMs disk. This will make ZFS show more available space and probably help with whatever error you get when you want to start that VM.

 zfs set refreservation=none almacen/vm-142-disk-0

Just don't leave it like that because with this the disk can still grow to 15TB if you fill it.

It does not start, I think the only option is to reduce the raw file, is it possible?

 

[aaron]

At the bottom you have the task logs (maybe collapsed). You should see the start log of that VM if you double click on it. What does it say?

 

[Toni Blanch]

At the bottom you have the task logs (maybe collapsed). You should see the start log of that VM if you double click on it. What does it say?

All I see is the io-error seen in the machine. I show you the screenshot.

 

[aaron]

Ah okay. I took me a bit to fully realize what situation you are in.

Let me summarize to make sure we are on the same page.
The VM will not boot because at some point it will stop with an IO error?

Unfortunately you cannot reduze the size of the disk on the zpool from outside. The problem with almost no available space with ZFS is, that it is a copy on write file system. Each new change is written to a new block. Old blocks are only overwritten if they are not in use anymore. That is why the rule of thumb is to not fill it up more than 80%.

Is that VM disk the only location where you have those files?
If no then destroy it and recreate it with a smaller size or use something other than ZFS underneath it.

If yes then you could try

• to boot the VM from a rescue ISO and shrink the partition in that disk
• enable Discard and SSD Emulation for that VM disk and make sure Trim is enabled in Windows (might release some space on ZFS if the disk is not filled completely with files)
• boot the VM from a rescue ISO and copy the data somewhere else in order to then destroy and recreate the disk smaller.

 

[Toni Blanch]

Ah okay. I took me a bit to fully realize what situation you are in.

Let me summarize to make sure we are on the same page.
The VM will not boot because at some point it will stop with an IO error?

Unfortunately you cannot reduze the size of the disk on the zpool from outside. The problem with almost no available space with ZFS is, that it is a copy on write file system. Each new change is written to a new block. Old blocks are only overwritten if they are not in use anymore. That is why the rule of thumb is to not fill it up more than 80%.

Is that VM disk the only location where you have those files?
If no then destroy it and recreate it with a smaller size or use something other than ZFS underneath it.

If yes then you could try

• to boot the VM from a rescue ISO and shrink the partition in that disk
• enable Discard and SSD Emulation for that VM disk and make sure Trim is enabled in Windows (might release some space on ZFS if the disk is not filled completely with files)
• boot the VM from a rescue ISO and copy the data somewhere else in order to then destroy and recreate the disk smaller.

Hello!
I am already recovering the files inside the raw !, thank you very much, finally with a live cd with the appropriate drivers I have been able to access the info and I am already transferring the information.
When I finish, I will remove the raw disk and create it at 80% of the pool capacity.
Your help has been great!

 

[aaron]

You're welcome. Please mark the thread as solved