[SOLVED] Backup fails with "exceeded allowed number of file entries"

mbosma

Member
Dec 3, 2018
95
10
13
28
Hi,

I've been testing pbs for a while now and I'm backing up production vm's and lxc containers to really test pbs in our environment.

One container has been giving errors regarding a file entries limit. This seems to be limited to pbs.

I just upgraded to pbs 9.0 (client and server) but had this issue on 8.x as well.

Here's the log file of the pbs backup:
Code:
INFO: starting new backup job: vzdump 172 --mode snapshot --remove 0 --node pve4 --storage backup2
INFO: Starting Backup of VM 172 (lxc)
INFO: Backup started at 2020-10-07 12:51:38
INFO: status = running
INFO: CT Name: web14
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/home') in backup
INFO: excluding volume mount point mp1 ('/backup') from backup (disabled)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-172-disk-0_vzdump" created.
  WARNING: Sum of all thin volume sizes (3.72 TiB) exceeds the size of thin pool pve/data and the size of whole volume group (3.49 TiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-172-disk-1_vzdump" created.
  WARNING: Sum of all thin volume sizes (<3.92 TiB) exceeds the size of thin pool pve/data and the size of whole volume group (3.49 TiB).
INFO: resume vm
INFO: guest is online again after 1 seconds
INFO: creating Proxmox Backup Server archive 'ct/172/2020-10-07T10:51:38Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp40510/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --include-dev /mnt/vzsnap0/./home --skip-lost-and-found --backup-type ct --backup-id 172 --backup-time 1602067898 --repository hodc1@pbs@10.250.2.32:hodc1
INFO: Starting backup: ct/172/2020-10-07T10:51:38Z
INFO: Client name: pve4
INFO: Starting backup protocol: Wed Oct  7 12:51:39 2020
INFO: Upload config file '/var/tmp/vzdumptmp40510/etc/vzdump/pct.conf' to 'hodc1@pbs@10.250.2.32:8007:hodc1' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'hodc1@pbs@10.250.2.32:8007:hodc1' as root.pxar.didx
INFO: Error: error at "home/*******/public_html/*******.nl/var/session": exceeded allowed number of file entries (> 1048576)
INFO: remove vzdump snapshot
  Logical volume "snap_vm-172-disk-0_vzdump" successfully removed
  Logical volume "snap_vm-172-disk-1_vzdump" successfully removed
ERROR: Backup of VM 172 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp40510/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --include-dev /mnt/vzsnap0/./home --skip-lost-and-found --backup-type ct --backup-id 172 --backup-time 1602067898 --repository hodc1@pbs@10.250.2.32:hodc1' failed: exit code 255
INFO: Failed at 2020-10-07 13:06:54
INFO: Backup job finished with errors
TASK ERROR: job errors

And the one from a successful backup to nfs:
Code:
INFO: starting new backup job: vzdump 172 --storage nas03 --node pve4 --compress lzo --remove 0 --mode snapshot
INFO: Starting Backup of VM 172 (lxc)
INFO: Backup started at 2020-10-07 12:06:41
INFO: status = running
INFO: CT Name: web14
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/home') in backup
INFO: excluding volume mount point mp1 ('/backup') from backup (disabled)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-172-disk-0_vzdump" created.
  WARNING: Sum of all thin volume sizes (3.72 TiB) exceeds the size of thin pool pve/data and the size of whole volume group (3.49 TiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-172-disk-1_vzdump" created.
  WARNING: Sum of all thin volume sizes (<3.92 TiB) exceeds the size of thin pool pve/data and the size of whole volume group (3.49 TiB).
INFO: resume vm
INFO: guest is online again after 2 seconds
INFO: creating vzdump archive '/mnt/pve/nas03/dump/vzdump-lxc-172-2020_10_07-12_06_41.tar.lzo'
INFO: Total bytes written: 169297162240 (158GiB, 64MiB/s)
INFO: archive file size: 118.86GB
INFO: remove vzdump snapshot
  Logical volume "snap_vm-172-disk-0_vzdump" successfully removed
  Logical volume "snap_vm-172-disk-1_vzdump" successfully removed
INFO: Finished Backup of VM 172 (00:42:44)
INFO: Backup finished at 2020-10-07 12:49:25
INFO: Backup job finished successfully
TASK OK

If there's any way I can help to resolve this issue please let me know.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,619
1,432
164
I'd evaluate whether you really need that directory or whether you can exclude it from your backup. it might make sense to make this limit configurable on the PVE side (the client has a parameter to set it, but PVE does not expose it).
 

mbosma

Member
Dec 3, 2018
95
10
13
28
The folder appeared to have over 2 million files, since these were old session files they could be deleted.
Is there any way to exclude a single folder or increase the file limit in case something like this happens again?
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,619
1,432
164
  • Like
Reactions: mbosma

Joe How

Member
Nov 18, 2019
6
3
8
34
Hi folks,

for me to exclude the folder is not a solution as I do a backup of my /var/mail/vhost/mail.domain/

Maybe some other approach?

THX.
 
Jan 20, 2016
14
0
21
Athens, Greece
Now in 2022 and since apparently "--entries-max" can be passed to the proxmox-client-backup utility according to the docs, is there perhaps some general config file for that utility?
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,619
1,432
164
no, this is still not exposed in vzdump and only settable via the PBS client CLI parameters.
 

fabriciopc7

New Member
May 27, 2022
6
0
1
I need to make the backup using the PROXMOX BACKUP SERVER tool, so I would need to know if there is any file that could be changing this value.
 

fabian

Proxmox Staff Member
Staff member
Jan 7, 2016
7,619
1,432
164
it's a client-side parameter, so you need to set it when calling "proxmox-backup-client"
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!