PANIC: zfs: attempting to increase fill beyond max; probable double add in segment

[PeterZ]

Hi all

My system is not booting anymore and I get this message:
PANIC: zfs: attempting to increase fill beyond max; probable double add in segment

Does that imply that a disc is full (which could be possible due to automatic backups)
If so, how can get access to the system to delete some of these backups?

Thanks a lot!

 

[Dunuin]

Maybe you can't delete anything at all. ZFS uses copy-on-write, so you need free space, to add new records, to be able to delete something. A ZFS pool should NEVER EVER be filled to 100%. Atleast 20% should be kept always free for ZFS to work optimally. Thats why you should set a pool wide quota so that it is impossible to fill it up completely, setup monitoring and add new disks when the pool gets close to 80% usage.

What I would first try is to delete snapshots and to run fstrim -a to free up some space. If you can't boot PVE anymore you could try to boot a Live Linux ISO that got ZFS support, then try to mount that pool in rescue mode.
If nothing works and you can't delete anything you would need to buy the same number of disks but bigger, clone all data on block level from the smaller to the bigger disks, so ZFS can try to expand the pool.

 

[PeterZ]

Hi Dunuin, thank you for your help!

Is that message that I quoted 100% because of a full disk?
I remembered later that I had a script to offload these backups and after that they are deleted from the server.

I was able to boot the server in command-line: is there something I can do from there?

 

[Dunuin]

You should run zpool list. The "CAP" column is what percentage your pool is filled. zfs list -o space is also useful to see how much space is used by snapshots and refreservation.

 

[PeterZ]

After installing another system with some backups that I had to be able to restore functionality, I'm looking at this issue again.

I'm booting with ubuntu live and see all disks.
The system has 2 disks:
- SSD (rpool) for OS & VM's
- HDD for data

I was able to import the HDD but the SSD takes forever: could this mean that the SSD has a physical problem and is it obvious that this is reason why my server isn't booting?