Too many open files (os error 24)

[villain666]

Hello

On second PBS (replicate from main PBS) i got next error (two times for last day)

Jul 17 13:33:27 pbsnfs proxmox-backup-proxy[6750]: GET /api2/json/status/tasks: 400 Bad Request: [client [::ffff:192.168.21.96]:54748] EMFILE: Too many open files
Jul 17 13:33:28 pbsnfs proxmox-backup-proxy[6750]: server error: error accepting connection: Too many open files (os error 24)
Jul 17 13:33:28 pbsnfs proxmox-backup-proxy[6750]: SET SHUTDOWN MODE
Jul 17 13:33:28 pbsnfs proxmox-backup-proxy[6750]: daemon shutting down...
Jul 17 13:33:28 pbsnfs proxmox-backup-proxy[6750]: server shutting down, waiting for active workers to complete
Jul 17 13:33:28 pbsnfs proxmox-backup-proxy[6750]: done - exit server
Jul 17 13:33:28 pbsnfs systemd[1]: proxmox-backup-proxy.service: Succeeded.

and web interface is gone

root@pbsnfs:~# ps axwf|grep proxy
7983 pts/0 S+ 0:00 \_ grep proxy
root@pbsnfs:~# ss -tulnp|grep 800
root@pbsnfs:~#

for start again need restarting service

root@pbsnfs:~# systemctl restart proxmox-backup-proxy
root@pbsnfs:~# ss -tulnp|grep 800
tcp LISTEN 0 1024 *:8007 *:* users("proxmox-backup-",pid=7993,fd=13),("proxmox-backup-",pid=7993,fd=12))

 

[DerDanilo]

Maybe the amount of chunks is to much for default values. Configuring this could be handled by the debian package right?

 

[villain666]

again shutdown and errors from task logging

2020-07-17T21:06:00+05:00: Starting datastore sync job 'repl0'
2020-07-17T21:06:00+05:00: task triggered by schedule 'Mon..Sun *:06'
2020-07-17T21:06:00+05:00: Sync datastore 'nfs' from 'pbsmain/backup'
2020-07-17T21:06:00+05:00: sync group vm/102 failed - last_successful_backup: unexpected error - EMFILE: Too many open files
2020-07-17T21:06:00+05:00: sync group vm/11003 failed - last_successful_backup: unexpected error - EMFILE: Too many open files
2020-07-17T21:06:00+05:00: sync group vm/11004 failed - last_successful_backup: unexpected error - EMFILE: Too many open files
2020-07-17T21:06:00+05:00: sync group vm/11005 failed - last_successful_backup: unexpected error - EMFILE: Too many open files
2020-07-17T21:06:00+05:00: sync group vm/11007 failed - last_successful_backup: unexpected error - EMFILE: Too many open files
2020-07-17T21:06:00+05:00: sync group vm/11009 failed - last_successful_backup: unexpected error - EMFILE: Too many open files
2020-07-17T21:06:00+05:00: sync group vm/11010 failed - last_successful_backup: unexpected error - EMFILE: Too many open files
2020-07-17T21:06:00+05:00: sync group vm/11011 failed - last_successful_backup: unexpected error - EMFILE: Too many open files
2020-07-17T21:06:00+05:00: sync group vm/11012 failed - last_successful_backup: unexpected error - EMFILE: Too many open files
2020-07-17T21:06:00+05:00: sync group vm/11013 failed - last_successful_backup: unexpected error - EMFILE: Too many open files
2020-07-17T21:06:00+05:00: sync group vm/11253 failed - last_successful_backup: unexpected error - EMFILE: Too many open files
2020-07-17T21:06:00+05:00: error during cleanup: EMFILE: Too many open files
2020-07-17T21:06:00+05:00: TASK ERROR: sync failed with some errors.

add LimitNOFILE=10240:524288 in /etc/systemd/system/multi-user.target.wants/proxmox-backup-proxy.service

and go to test...

 

[villain666]

Again fail with Too many open files
after two days

I think there leak file descriptor ....

 

[fabian]

could you open a bug over at https://bugzilla.proxmox.com ? please include details like installed versions, config files and in this case especially, number of chunks in the chunk stores