Buffer I/O error ... lost async page write

[reto]

FYI; here some recent experience with PVE running ZFS

Solution: ZFS Pool was full

Symptoms:

• All VMs froze, reboot didn't help, shutdown stuck
• kernel log: Buffer I/O error on dev zd224, logical block 0, lost async page write
  • zd224 pointed to cloud init device, error got only printed for them
  • no other logs!
• yellow warning sign "io-error" on all running vms
• zpool status was healthy and happy
• zpool get capacity zfsraid1 reported 93% usage

Fix:

• removed some old vms and snapshots
• restarted all vms

Conclusion:

• Never let your disk run out (makes sense)
• Error message quiet strange, IMHO it would be nice if the admin could be more assisted in this topic with ZFS