VM I/O error when mounting a RAIDZ partition

[Edu]

Hello everybody,

First of all, I should say I'm a total beginner regarding these matters, so please bear with me for a moment. Thank you.

My company accidentally fill one RAIDZ partition (sdb1) up to 100% of its capacity yesterday causing the VM in which such partition was originally mounted to go unresponsive.
We stopped and then started the VM succesfully, but the part is now unmounted as it is absent in df (see below).


When checking with lsblk I can see the disk and its partition without a mount point indeed (see figure below for details).


The problem is that when I try to mount the partition from the console using the typical mount /dev/sdb1 <directory>, the terminal hangs indefinitely, and when checking it out in the proxmox VE dashboard the VM appears with a warning icon indicating io-error, although apparently the VM is still running as indicated in the Status flag (I don't know why it says that, because it effectively goes unresponsive). The figure below shows the specific error.


I assume the problem comes from the lack of any free space in the partition, so, do you have any ideas about how can I proceed without losing the data? I have searched and searched for solutions and had no luck yet... The fact I'm only a begginer does not help at all, to be honest.

Thank you very much in advance for your time!

 

[Edu]

Bumping... I'd be very grateful if someone could take a shot at this, thanks.

 

[aaron]

A ZFS raidz is never just one partition on a disk.

You will have to use the ZFS tools. What output do you get if you run

zpool status

and

zpool import

?

 

[Edu]

Thank you very much aaron, I appreciate your time a lot.

Running zpool status outputs "no pools available".

Running zpool import outputs "no pools available to import"

Cheers.

 

[aaron]

Okay, so for me to understand it correctly. The disk inside the VM ran full? The screenshots are from within the VM?

Did you run the zpool commands on the Proxmox VE host or within the VM?

 

[Edu]

Thanks again aaron. I'll try to explain in a bit more detail.

One of the VM's drives ran full because someone uploaded way too many files via ftp. That was what originated the problem as far as we know.

The disk in question is a ZFS SSD12TB-RAIDZ with 10.82 TiB allocated (and 357.5 free GiB). Here is a screenshot taken from the Disks tab of the node:


Now, the disk is installed inside the VM, showing 8.65 TiB of size (as far as I know this reduction from the supposedly 10.82 TiB is due to the RAIDZ setup and is OK), can be inspected in the Summary tab and of course it shows a 100% storage usage (see image below):


In the other hand, in the Content tab, it shows the following:



Now, regarding your questions:

The disk inside the VM ran full?

Yes, I believe there must be about 200 MB of free space or something like that, but to all effects it seems as it's fully occupied.

The screenshots are from within the VM?

The screenshots are taken from the terminal of the VM, you are right.

Did you run the zpool commands on the Proxmox VE host or within the VM?

I ran the commands from the VM terminal. I will try using the Proxmox VE host if you suggest it.

Hoping to hear from you again! Thanks in advance.

 

[aaron]

Okay. Well, to get a more complete picture, please run the zpool status and zfs list commands on the Proxmox VE host.
As well as zpool get all <pool name>.
In the output of the zfs list command, you will see the vm-5001-disk-0 dataset. Please run zfs get all <pool>/vm-5001-disk-0. The path might be a bit different.
Please also share the storage configuration: cat /etc/pve/storage.cfg

With that, we should get a good picture of the current state. The big problem is, that once ZFS is full, IO will be problematic as it needs a bit of free space since it is copy on write.

 

[Dunuin]

For the future you might want to use some kind of quota. ZFS supports this and with that you can for example limit that your ZFS pool never can be filled up more than 90%. That way you can't run into a situation like now where ZFS stops working because it completely ran out of space. And beside that, a ZFS pool gets slow when it gets full. Because of that it is recommended to not fill your pool more than 80%. So another point why it makes sense to use quotas.

 

[Edu]

Thanks @Dunuin for the suggestion, it seems something we should have thought about earlier... Very useful indeed.

@aaron, I ran the commands from the Proxmox VE Shell with the following outputs:

• zpool status: the first result is the problematic disk; it seems no errors are found...
  
  
  
• zfs list: here we can see 0B available in the disk
  
  
• Additionally, the zpool get all SSD12TB-RAIDZ command shows (I divide the screenshot in two images)
  
  
  
  
  
• zfs get all SSD12TB-RAIDZ/vm-5001-disk-0:
  
  
  
  
  
• Finally, the storage configuration as displayed by the /etc/pve/storage.cfg:
  

Thanks again for you invaluable help! Cheers!