How to recover from 100% disk use

pool: Backup_Storage
state: DEGRADED
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
config:

NAME STATE READ WRITE CKSUM
Backup_Storage DEGRADED 0 0 0
sdb DEGRADED 0 0 0 too many errors

errors: No known data errors
 
!! EVERYTHING AT YOUR RISK!!
I would do the following:

- identify the affected disk with zpool status -x (see also link in your posted message)
- Replace the cable on this plate.
- zpool clear Backup_Storage <diskwitherror>. After that the status should be changed from DEGRADED to ONLINE. Check this with zpool status.
- Observe the system for a while. If the error no longer occurs, then the matter is over.
- If the error still occurs again, replace the disk (see link).
- Now you can run zfs set reservation=2g Backup_Storage.
 
  • Like
Reactions: ioB_Newcomer
I made a script to help other out
If you end up filling up your space run this with privileged user like root

Steps:
1. Set the Paths "SRC_DIR", "DEST_DIR" and "BACKUP_FILE"
2. Run the script and select "MOVE", you need to free up at least 100mb
3. Manually run a Garbage Collect Job now in PBS, it should work (ignore the warnings, we will restore them in step 4.)
4. Run this script and select "RESTORE", to restore your moved file as they were.
5. Run a full Verify in PBS

Thank you!

Just had this issue, been trying to solve it for a couple of days and using this script cracked it for me!