[SOLVED] Backup fails with "exceeded allowed number of file entries"

mbosma · Oct 7, 2020

Hi,

I've been testing pbs for a while now and I'm backing up production vm's and lxc containers to really test pbs in our environment.

One container has been giving errors regarding a file entries limit. This seems to be limited to pbs.

I just upgraded to pbs 9.0 (client and server) but had this issue on 8.x as well.

Here's the log file of the pbs backup:

Code:

INFO: starting new backup job: vzdump 172 --mode snapshot --remove 0 --node pve4 --storage backup2
INFO: Starting Backup of VM 172 (lxc)
INFO: Backup started at 2020-10-07 12:51:38
INFO: status = running
INFO: CT Name: web14
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/home') in backup
INFO: excluding volume mount point mp1 ('/backup') from backup (disabled)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-172-disk-0_vzdump" created.
  WARNING: Sum of all thin volume sizes (3.72 TiB) exceeds the size of thin pool pve/data and the size of whole volume group (3.49 TiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-172-disk-1_vzdump" created.
  WARNING: Sum of all thin volume sizes (<3.92 TiB) exceeds the size of thin pool pve/data and the size of whole volume group (3.49 TiB).
INFO: resume vm
INFO: guest is online again after 1 seconds
INFO: creating Proxmox Backup Server archive 'ct/172/2020-10-07T10:51:38Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp40510/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --include-dev /mnt/vzsnap0/./home --skip-lost-and-found --backup-type ct --backup-id 172 --backup-time 1602067898 --repository hodc1@pbs@10.250.2.32:hodc1
INFO: Starting backup: ct/172/2020-10-07T10:51:38Z
INFO: Client name: pve4
INFO: Starting backup protocol: Wed Oct  7 12:51:39 2020
INFO: Upload config file '/var/tmp/vzdumptmp40510/etc/vzdump/pct.conf' to 'hodc1@pbs@10.250.2.32:8007:hodc1' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'hodc1@pbs@10.250.2.32:8007:hodc1' as root.pxar.didx
INFO: Error: error at "home/*******/public_html/*******.nl/var/session": exceeded allowed number of file entries (> 1048576)
INFO: remove vzdump snapshot
  Logical volume "snap_vm-172-disk-0_vzdump" successfully removed
  Logical volume "snap_vm-172-disk-1_vzdump" successfully removed
ERROR: Backup of VM 172 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp40510/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --include-dev /mnt/vzsnap0/./home --skip-lost-and-found --backup-type ct --backup-id 172 --backup-time 1602067898 --repository hodc1@pbs@10.250.2.32:hodc1' failed: exit code 255
INFO: Failed at 2020-10-07 13:06:54
INFO: Backup job finished with errors
TASK ERROR: job errors

And the one from a successful backup to nfs:

Code:

INFO: starting new backup job: vzdump 172 --storage nas03 --node pve4 --compress lzo --remove 0 --mode snapshot
INFO: Starting Backup of VM 172 (lxc)
INFO: Backup started at 2020-10-07 12:06:41
INFO: status = running
INFO: CT Name: web14
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/home') in backup
INFO: excluding volume mount point mp1 ('/backup') from backup (disabled)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-172-disk-0_vzdump" created.
  WARNING: Sum of all thin volume sizes (3.72 TiB) exceeds the size of thin pool pve/data and the size of whole volume group (3.49 TiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-172-disk-1_vzdump" created.
  WARNING: Sum of all thin volume sizes (<3.92 TiB) exceeds the size of thin pool pve/data and the size of whole volume group (3.49 TiB).
INFO: resume vm
INFO: guest is online again after 2 seconds
INFO: creating vzdump archive '/mnt/pve/nas03/dump/vzdump-lxc-172-2020_10_07-12_06_41.tar.lzo'
INFO: Total bytes written: 169297162240 (158GiB, 64MiB/s)
INFO: archive file size: 118.86GB
INFO: remove vzdump snapshot
  Logical volume "snap_vm-172-disk-0_vzdump" successfully removed
  Logical volume "snap_vm-172-disk-1_vzdump" successfully removed
INFO: Finished Backup of VM 172 (00:42:44)
INFO: Backup finished at 2020-10-07 12:49:25
INFO: Backup job finished successfully
TASK OK

If there's any way I can help to resolve this issue please let me know.

fabian · Oct 7, 2020

I'd evaluate whether you really need that directory or whether you can exclude it from your backup. it might make sense to make this limit configurable on the PVE side (the client has a parameter to set it, but PVE does not expose it).

mbosma · Oct 8, 2020

The folder appeared to have over 2 million files, since these were old session files they could be deleted.
Is there any way to exclude a single folder or increase the file limit in case something like this happens again?

fabian · Oct 8, 2020

yeah, you can exclude with https://pbs.proxmox.com/docs/administration-guide.html#excluding-files-folders-from-a-backup (inside the container) or https://pve.proxmox.com/pve-docs/pve-admin-guide.html#_file_exclusions (on the PVE/vzdump side). the setting to increase this limit is not (yet) exposed in vzdump, feel free to file an enhancement request over at https://bugzilla.proxmox.com.

mbosma · Oct 8, 2020

Thank you, I did dive into the manpages but didn't find the .pxarexclude solution.
This issue is fixed for me as I can simply exclude this folder.

I created a enhancement request in case someone ever runs into this issue with a more valuable folder.
https://bugzilla.proxmox.com/show_bug.cgi?id=3069

Corwin · Oct 15, 2020

I also missed it while browsing through the documentation. Please note the correct URL https://pbs.proxmox.com/docs/backup-client.html#excluding-files-folders-from-a-backup

Joe How · Nov 18, 2020

Hi folks,

for me to exclude the folder is not a solution as I do a backup of my /var/mail/vhost/mail.domain/

Maybe some other approach?

THX.

fabian · Nov 18, 2020

Joe How said:
Hi folks,

for me to exclude the folder is not a solution as I do a backup of my /var/mail/vhost/mail.domain/

Maybe some other approach?

THX.

https://bugzilla.proxmox.com/show_bug.cgi?id=3069

fotisevangelou · Mar 28, 2022

Now in 2022 and since apparently "--entries-max" can be passed to the proxmox-client-backup utility according to the docs, is there perhaps some general config file for that utility?

fabian · Mar 28, 2022

no, this is still not exposed in vzdump and only settable via the PBS client CLI parameters.

fabriciopc7 · May 27, 2022

Good afternoon what would these CLI parameters be

Dunuin · May 27, 2022

fabriciopc7 said:
Good afternoon what would these CLI parameters be

See: https://pbs.proxmox.com/docs/command-syntax.html#proxmox-backup-client

--entries-max <integer> (default=1048576)
Max number of entries to hold in memory.

fabriciopc7 · May 27, 2022

I need to make the backup using the PROXMOX BACKUP SERVER tool, so I would need to know if there is any file that could be changing this value.

fabian · May 30, 2022

it's a client-side parameter, so you need to set it when calling "proxmox-backup-client"

fotisevangelou · Dec 17, 2022

Circling back at this...

What's the point of configuring backup jobs using the Web UIs (PBS & Proxmox) if you can't pass on any additional parameters to the underlying CLI commands?

And if I manually add backup jobs in a shell script, it really defeats the purpose of using a UI in the first place.

Unless this is a commercial/subscription feature I'm not aware of.

P.S. As time passes (since PBS was released) and more people use it and backups get larger, this is bound to become a real issue for your support team.

fabian · Dec 19, 2022

it shouldn't be too hard to make this somehow configurable for PVE/vzdump as well, just updated the bugzilla entry accordingly..

Search

Search

[SOLVED] Backup fails with "exceeded allowed number of file entries"

mbosma

Well-Known Member

fabian

Proxmox Staff Member

mbosma

Well-Known Member

fabian

Proxmox Staff Member

mbosma

Well-Known Member

Corwin

Renowned Member

Joe How

Member

fabian

Proxmox Staff Member

fotisevangelou

Renowned Member

fabian

Proxmox Staff Member

fabriciopc7

New Member

Dunuin

Distinguished Member

fabriciopc7

New Member

fabian

Proxmox Staff Member

fotisevangelou

Renowned Member

fabian

Proxmox Staff Member