VM Crashing with io-error

[ErkDog]

OK so precursors. ProxMox seems to AGGRESSIVELY Reserve excess space when you create ZFS volumes for some reason.

Therefore in order to get the usable space out of the arrays, I had to tweak the reserved space values etc.

However, this VM ran FINE for months. Then all of a sudden it gets stuck "running (io-error) and of course is non-responsive.

I tracked it down to a specific ZFS Pool that I saw some other issues on, but every time I scrub it there's no errors or problems.

I can't find the applicable log files for where this type of thing gets logged, but for now I 'detached' the volume in the hardware panel so I could use the virtual-machine period.

Here are the stats on the ZPool and Volume in question.

These are sitting on a raid controller with each of the 4 2TB SSDs in single JBOD Pass-through then put in a ZFS+1 Array with the full disk minus a few megabytes provisioned.

Please advise:

SIXTBSATA type filesystem -
SIXTBSATA creation Wed Mar 11 13:52 2020 -
SIXTBSATA used 5.25T -
SIXTBSATA available 1.51M -
SIXTBSATA referenced 140K -
SIXTBSATA compressratio 1.00x -
SIXTBSATA mounted yes -
SIXTBSATA quota none default
SIXTBSATA reservation none default
SIXTBSATA recordsize 128K default
SIXTBSATA mountpoint /SIXTBSATA default
SIXTBSATA sharenfs off default
SIXTBSATA checksum on default
SIXTBSATA compression lz4 local
SIXTBSATA atime on default
SIXTBSATA devices on default
SIXTBSATA exec on default
SIXTBSATA setuid on default
SIXTBSATA readonly off default
SIXTBSATA zoned off default
SIXTBSATA snapdir hidden default
SIXTBSATA aclinherit restricted default
SIXTBSATA createtxg 1 -
SIXTBSATA canmount on default
SIXTBSATA xattr on default
SIXTBSATA copies 1 default
SIXTBSATA version 5 -
SIXTBSATA utf8only off -
SIXTBSATA normalization none -
SIXTBSATA casesensitivity sensitive -
SIXTBSATA vscan off default
SIXTBSATA nbmand off default
SIXTBSATA sharesmb off default
SIXTBSATA refquota none default
SIXTBSATA refreservation none default
SIXTBSATA guid 11203910512181312855 -
SIXTBSATA primarycache all default
SIXTBSATA secondarycache all default
SIXTBSATA usedbysnapshots 0B -
SIXTBSATA usedbydataset 140K -
SIXTBSATA usedbychildren 5.25T -
SIXTBSATA usedbyrefreservation 0B -
SIXTBSATA logbias latency default
SIXTBSATA objsetid 54 -
SIXTBSATA dedup off default
SIXTBSATA mlslabel none default
SIXTBSATA sync standard default
SIXTBSATA dnodesize legacy default
SIXTBSATA refcompressratio 1.00x -
SIXTBSATA written 140K -
SIXTBSATA logicalused 3.61T -
SIXTBSATA logicalreferenced 42K -
SIXTBSATA volmode default default
SIXTBSATA filesystem_limit none default
SIXTBSATA snapshot_limit none default
SIXTBSATA filesystem_count none default
SIXTBSATA snapshot_count none default
SIXTBSATA snapdev hidden default
SIXTBSATA acltype off default
SIXTBSATA context none default
SIXTBSATA fscontext none default
SIXTBSATA defcontext none default
SIXTBSATA rootcontext none default
SIXTBSATA relatime off default
SIXTBSATA redundant_metadata all default
SIXTBSATA overlay off default
SIXTBSATA encryption off default
SIXTBSATA keylocation none default
SIXTBSATA keyformat none default
SIXTBSATA pbkdf2iters 0 default
SIXTBSATA special_small_blocks 0 default
SIXTBSATA/vm-100-disk-0 type volume -
SIXTBSATA/vm-100-disk-0 creation Wed Mar 11 14:06 2020 -
SIXTBSATA/vm-100-disk-0 used 5.25T -
SIXTBSATA/vm-100-disk-0 available 1.51M -
SIXTBSATA/vm-100-disk-0 referenced 5.25T -
SIXTBSATA/vm-100-disk-0 compressratio 1.00x -
SIXTBSATA/vm-100-disk-0 reservation none default
SIXTBSATA/vm-100-disk-0 volsize 5.18T local
SIXTBSATA/vm-100-disk-0 volblocksize 8K default
SIXTBSATA/vm-100-disk-0 checksum on default
SIXTBSATA/vm-100-disk-0 compression lz4 inherited from SIXTBSATA
SIXTBSATA/vm-100-disk-0 readonly off default
SIXTBSATA/vm-100-disk-0 createtxg 194 -
SIXTBSATA/vm-100-disk-0 copies 1 default
SIXTBSATA/vm-100-disk-0 refreservation 5.18T local
SIXTBSATA/vm-100-disk-0 guid 16831928596383082767 -
SIXTBSATA/vm-100-disk-0 primarycache all default
SIXTBSATA/vm-100-disk-0 secondarycache all default
SIXTBSATA/vm-100-disk-0 usedbysnapshots 0B -
SIXTBSATA/vm-100-disk-0 usedbydataset 5.25T -
SIXTBSATA/vm-100-disk-0 usedbychildren 0B -
SIXTBSATA/vm-100-disk-0 usedbyrefreservation 0B -
SIXTBSATA/vm-100-disk-0 logbias latency default
SIXTBSATA/vm-100-disk-0 objsetid 651 -
SIXTBSATA/vm-100-disk-0 dedup off default
SIXTBSATA/vm-100-disk-0 mlslabel none default
SIXTBSATA/vm-100-disk-0 sync standard default
SIXTBSATA/vm-100-disk-0 refcompressratio 1.00x -
SIXTBSATA/vm-100-disk-0 written 5.25T -
SIXTBSATA/vm-100-disk-0 logicalused 3.61T -
SIXTBSATA/vm-100-disk-0 logicalreferenced 3.61T -
SIXTBSATA/vm-100-disk-0 volmode default default
SIXTBSATA/vm-100-disk-0 snapshot_limit none default
SIXTBSATA/vm-100-disk-0 snapshot_count none default
SIXTBSATA/vm-100-disk-0 snapdev hidden default
SIXTBSATA/vm-100-disk-0 context none default
SIXTBSATA/vm-100-disk-0 fscontext none default
SIXTBSATA/vm-100-disk-0 defcontext none default
SIXTBSATA/vm-100-disk-0 rootcontext none default
SIXTBSATA/vm-100-disk-0 redundant_metadata all default
SIXTBSATA/vm-100-disk-0 encryption off default
SIXTBSATA/vm-100-disk-0 keylocation none default
SIXTBSATA/vm-100-disk-0 keyformat none default
SIXTBSATA/vm-100-disk-0 pbkdf2iters 0 default

 

[mira]

Looks like your pool is full:

SIXTBSATA used 5.25T -
SIXTBSATA available 1.51M -

 

[ErkDog]

Well right, the pool is full because it's allocated to a storage volume, which is of a fixed size. It's worked fine for 3 months and now all of a sudden is a problem.

Also Available 000000 would indicate full to me, that says there is 1.51Megs available.

Perhaps something is somehow over provisioned.

However, none of this information helps me solve the problem at hand:

SIXTBSATA/vm-100-disk-0 used 5.25T -
SIXTBSATA/vm-100-disk-0 available 1.51M -
SIXTBSATA/vm-100-disk-0 referenced 5.25T -
SIXTBSATA/vm-100-disk-0 compressratio 1.00x -
SIXTBSATA/vm-100-disk-0 reservation none default
SIXTBSATA/vm-100-disk-0 volsize 5.18T local

That's the actual vm-100-disk volume, so how am I supposed to remove data from it /slash/ make it smaller if the VM won't boot.

Can anyone assist me with how I would mount this volume to a folder directly inside of ProxMox shell itself so I can delete some data?

Also any assistance with what values I need to change to prevent it from happening again on the zpool would be greatly appreciated as well.

Thanks,

 

[ErkDog]

Ugh:

root@pmox:~# zfs set mountpoint=/mnt/tempmount SIXTBSATA/vm-100-disk-0
cannot set property for 'SIXTBSATA/vm-100-disk-0': 'mountpoint' does not apply to datasets of this type

 

[mira]

You could boot a live CD and run fstrim if the disk has discard enabled.

 

[ErkDog]

Mira, I appreciate your responses, but they are very brief and lack any actual explanation of how to do things.1

A -disk- is not the problem, the problem is the sub volume vm-100-disk-0 is apparently full, but how do I mount it so I can make it unfull?

fstrim doesn't help me because I can't access anything because I can't mount it.

Maybe I'm missing something, it's hard to tell though, because of your brief responses.

 

[mira]

1. Activate discard on the disk
2. Add a Live CD (whichever you prefer) to your cdrom drive
3. Boot the VM into the live CD
4. Run fstrim on all disks

Without discard enabled on the disk, files that are deleted in the VM will not be deleted on the underlying disk as it has no knowledge of it. But with discard enabled when you delete a file inside the guest OS, it will be passed through to the underlying storage and will be deleted there as well. This could help you free some space on your disk as long as you don't use the whole 5.25T inside the guest.

 

[ErkDog]

Thanks for the more detailed information mira.

Trying this now.