can't verify chunk, load failed - store 'backup', unable to load chunk '' - Too many open files (os error 24)

[layer7.net]

Hi,

during backup via PVE to an external PBS server we see:

2024-12-30T08:35:02+01:00: Automatically verifying newly added snapshot
2024-12-30T08:35:02+01:00: verify backup:vm/221/2024-12-30T06:48:24Z
2024-12-30T08:35:02+01:00:   check qemu-server.conf.blob
2024-12-30T08:35:02+01:00:   check drive-scsi0.img.fidx
2024-12-31T01:44:20+01:00: can't verify chunk, load failed - store 'backup', unable to load chunk '6590794ba87b710b83f6cafefae9045a0bd2403afe7580fa22173ae51417cb36' - Too many open files (os error 24)
2024-12-31T01:44:22+01:00: corrupted chunk renamed to "/backup-zfs/backup/.chunks/6590/6590794ba87b710b83f6cafefae9045a0bd2403afe7580fa22173ae51417cb36.0.bad"
2024-12-31T01:44:22+01:00: can't verify chunk, load failed - store 'backup', unable to load chunk '5bbdc5053c3e46c26842a7cff02e079d2f8d350443c0426f99cc0dfc78692f20' - Too many open files (os error 24)

Where is this too many open files error coming from?

Thank you! And have a nice turn over to 2025!

 

[waltar]

Check "ulimit -a" and create /etc/security/limits.d/pbs.conf
with line
* - nofile 65536
and reboot pbs, check with "ulimit -a".

 

[layer7.net]

Hello @waltar

Thank you very much for your time and response!

I actually checked it already and the open files are limited to the default 1024.

But the system is working with a quiet higher open files count without issues:

lsof | wc -l
10992

As far as i know zfs has its own open files limitation and that the 1024 limit will not count for files opened on the zfs pool.

Otherwise things would have had to explode a lot earlier i think. I assume(d) that the limitation might come from the proxmox software itself.
Thats why i opened this thread. But maybe i am wrong here.

I will follow your suggestion it wont hurt anyway.

Thank you!

 

[waltar]

Otherwise things would have had to explode a lot earlier i think. I assume(d) that the limitation might come from the proxmox software itself.

Some things with zfs will explode a lot later but hopefully never happen ...
I don't assume pbs would have a file limit reached now as it happily would "support" much more files for bigger pve clusters as you are running yet.
But no ideas for you on my side yet, maybe another has further tip to your pbs backup run problem "... - Too many open files (os error 24)" ...

 

[tcabernoch]

You aren't using an EMC storage back end, are you? Some older EMC gear isn't compatible with .chunks.

What version of PBS? I read in the release notes that early 3.x didn't really deal with bad chunks, and the new release actually does something about them now. It still didn't sound reassuring, but the latest release of PBS has the latest bad .chunk management, and it sounds like you might need that.