PANIC: zfs: attempting to increase fill beyond max; probable double add in segment

PeterZ

Member
Jul 18, 2019
31
0
11
52
Hi all

My system is not booting anymore and I get this message:
PANIC: zfs: attempting to increase fill beyond max; probable double add in segment

Does that imply that a disc is full (which could be possible due to automatic backups)
If so, how can get access to the system to delete some of these backups?

Thanks a lot!
 

Dunuin

Famous Member
Jun 30, 2020
8,235
2,071
151
Germany
Maybe you can't delete anything at all. ZFS uses copy-on-write, so you need free space, to add new records, to be able to delete something. A ZFS pool should NEVER EVER be filled to 100%. Atleast 20% should be kept always free for ZFS to work optimally. Thats why you should set a pool wide quota so that it is impossible to fill it up completely, setup monitoring and add new disks when the pool gets close to 80% usage.

What I would first try is to delete snapshots and to run fstrim -a to free up some space. If you can't boot PVE anymore you could try to boot a Live Linux ISO that got ZFS support, then try to mount that pool in rescue mode.
If nothing works and you can't delete anything you would need to buy the same number of disks but bigger, clone all data on block level from the smaller to the bigger disks, so ZFS can try to expand the pool.
 
Last edited:
  • Like
Reactions: PeterZ

PeterZ

Member
Jul 18, 2019
31
0
11
52
Hi Dunuin, thank you for your help!

Is that message that I quoted 100% because of a full disk?
I remembered later that I had a script to offload these backups and after that they are deleted from the server.

I was able to boot the server in command-line: is there something I can do from there?
 

Dunuin

Famous Member
Jun 30, 2020
8,235
2,071
151
Germany
You should run zpool list. The "CAP" column is what percentage your pool is filled. zfs list -o space is also useful to see how much space is used by snapshots and refreservation.
 
Last edited:
  • Like
Reactions: PeterZ

PeterZ

Member
Jul 18, 2019
31
0
11
52
After installing another system with some backups that I had to be able to restore functionality, I'm looking at this issue again.

I'm booting with ubuntu live and see all disks.
The system has 2 disks:
- SSD (rpool) for OS & VM's
- HDD for data

I was able to import the HDD but the SSD takes forever: could this mean that the SSD has a physical problem and is it obvious that this is reason why my server isn't booting?
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!