[SOLVED] Backup fails with "exceeded allowed number of file entries"

M

mbosma

Renowned Member
Dec 3, 2018
124
25
68
30
Hi,

I've been testing pbs for a while now and I'm backing up production vm's and lxc containers to really test pbs in our environment.

One container has been giving errors regarding a file entries limit. This seems to be limited to pbs.

I just upgraded to pbs 9.0 (client and server) but had this issue on 8.x as well.

Here's the log file of the pbs backup:
Code: 
INFO: starting new backup job: vzdump 172 --mode snapshot --remove 0 --node pve4 --storage backup2
INFO: Starting Backup of VM 172 (lxc)
INFO: Backup started at 2020-10-07 12:51:38
INFO: status = running
INFO: CT Name: web14
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/home') in backup
INFO: excluding volume mount point mp1 ('/backup') from backup (disabled)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-172-disk-0_vzdump" created.
  WARNING: Sum of all thin volume sizes (3.72 TiB) exceeds the size of thin pool pve/data and the size of whole volume group (3.49 TiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-172-disk-1_vzdump" created.
  WARNING: Sum of all thin volume sizes (<3.92 TiB) exceeds the size of thin pool pve/data and the size of whole volume group (3.49 TiB).
INFO: resume vm
INFO: guest is online again after 1 seconds
INFO: creating Proxmox Backup Server archive 'ct/172/2020-10-07T10:51:38Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp40510/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --include-dev /mnt/vzsnap0/./home --skip-lost-and-found --backup-type ct --backup-id 172 --backup-time 1602067898 --repository hodc1@pbs@10.250.2.32:hodc1
INFO: Starting backup: ct/172/2020-10-07T10:51:38Z
INFO: Client name: pve4
INFO: Starting backup protocol: Wed Oct  7 12:51:39 2020
INFO: Upload config file '/var/tmp/vzdumptmp40510/etc/vzdump/pct.conf' to 'hodc1@pbs@10.250.2.32:8007:hodc1' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'hodc1@pbs@10.250.2.32:8007:hodc1' as root.pxar.didx
INFO: Error: error at "home/*******/public_html/*******.nl/var/session": exceeded allowed number of file entries (> 1048576)
INFO: remove vzdump snapshot
  Logical volume "snap_vm-172-disk-0_vzdump" successfully removed
  Logical volume "snap_vm-172-disk-1_vzdump" successfully removed
ERROR: Backup of VM 172 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp40510/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --include-dev /mnt/vzsnap0/./home --skip-lost-and-found --backup-type ct --backup-id 172 --backup-time 1602067898 --repository hodc1@pbs@10.250.2.32:hodc1' failed: exit code 255
INFO: Failed at 2020-10-07 13:06:54
INFO: Backup job finished with errors
TASK ERROR: job errors

And the one from a successful backup to nfs:
Code: 
INFO: starting new backup job: vzdump 172 --storage nas03 --node pve4 --compress lzo --remove 0 --mode snapshot
INFO: Starting Backup of VM 172 (lxc)
INFO: Backup started at 2020-10-07 12:06:41
INFO: status = running
INFO: CT Name: web14
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/home') in backup
INFO: excluding volume mount point mp1 ('/backup') from backup (disabled)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: suspend vm to make snapshot
INFO: create storage snapshot 'vzdump'
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-172-disk-0_vzdump" created.
  WARNING: Sum of all thin volume sizes (3.72 TiB) exceeds the size of thin pool pve/data and the size of whole volume group (3.49 TiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "snap_vm-172-disk-1_vzdump" created.
  WARNING: Sum of all thin volume sizes (<3.92 TiB) exceeds the size of thin pool pve/data and the size of whole volume group (3.49 TiB).
INFO: resume vm
INFO: guest is online again after 2 seconds
INFO: creating vzdump archive '/mnt/pve/nas03/dump/vzdump-lxc-172-2020_10_07-12_06_41.tar.lzo'
INFO: Total bytes written: 169297162240 (158GiB, 64MiB/s)
INFO: archive file size: 118.86GB
INFO: remove vzdump snapshot
  Logical volume "snap_vm-172-disk-0_vzdump" successfully removed
  Logical volume "snap_vm-172-disk-1_vzdump" successfully removed
INFO: Finished Backup of VM 172 (00:42:44)
INFO: Backup finished at 2020-10-07 12:49:25
INFO: Backup job finished successfully
TASK OK

If there's any way I can help to resolve this issue please let me know.
 
I'd evaluate whether you really need that directory or whether you can exclude it from your backup. it might make sense to make this limit configurable on the PVE side (the client has a parameter to set it, but PVE does not expose it).
 
The folder appeared to have over 2 million files, since these were old session files they could be deleted.
Is there any way to exclude a single folder or increase the file limit in case something like this happens again?
 
  • Like
Reactions: mbosma
Hi folks,

for me to exclude the folder is not a solution as I do a backup of my /var/mail/vhost/mail.domain/

Maybe some other approach?

THX.
 
Now in 2022 and since apparently "--entries-max" can be passed to the proxmox-client-backup utility according to the docs, is there perhaps some general config file for that utility?
 
no, this is still not exposed in vzdump and only settable via the PBS client CLI parameters.
 
I need to make the backup using the PROXMOX BACKUP SERVER tool, so I would need to know if there is any file that could be changing this value.
 
it's a client-side parameter, so you need to set it when calling "proxmox-backup-client"
 
Circling back at this...

What's the point of configuring backup jobs using the Web UIs (PBS & Proxmox) if you can't pass on any additional parameters to the underlying CLI commands?

And if I manually add backup jobs in a shell script, it really defeats the purpose of using a UI in the first place.

Unless this is a commercial/subscription feature I'm not aware of.

P.S. As time passes (since PBS was released) and more people use it and backups get larger, this is bound to become a real issue for your support team.
 
Last edited:
it shouldn't be too hard to make this somehow configurable for PVE/vzdump as well, just updated the bugzilla entry accordingly..
 
So what is the state of this? It still seems that in vzdump.conf there is no variable to increase file limit above 1048576 ?
on bugzilla there is a patch from 2023, but it is not implemented, right?

https://bugzilla.proxmox.com/show_bug.cgi?id=3069

i run blockchain archival nodes, which sometimes store a lot of files..

INFO: starting new backup job: vzdump 123 --mode snapshot --storage kurb-store --notification-mode auto --remove 0 --notes-template '{{guestname}}' --node Erbse
INFO: Starting Backup of VM 123 (lxc)
INFO: Backup started at 2024-08-21 16:37:05
INFO: status = running
INFO: CT Name: avalanche3
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: bandwidth limit: 250000 KB/s
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/123/2024-08-21T14:37:05Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp2950886_123/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 123 --backup-time 1724251025 --repository cluster@pbs@192.168.128.20:cluster
INFO: Starting backup: ct/123/2024-08-21T14:37:05Z
INFO: Client name: Erbse
INFO: Starting backup protocol: Wed Aug 21 16:37:05 2024
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp2950886_123/etc/vzdump/pct.conf' to 'cluster@pbs@192.168.128.20:8007:cluster' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'cluster@pbs@192.168.128.20:8007:cluster' as root.pxar.didx
INFO: catalog upload error - channel closed
INFO: Error: error at "root/db/mainnet/v1.4.5": exceeded allowed number of file entries (> 1048576)
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 123 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp2950886_123/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 123 --backup-time 1724251025 --repository cluster@pbs@192.168.128.20:cluster' failed: exit code 255
INFO: Failed at 2024-08-21 16:37:36
INFO: Backup job finished with errors
INFO: skipping disabled matcher 'default-matcher'
INFO: notified via target `netcup`
TASK ERROR: job errors

Eidt: here is my solution for a manual backup without vzdump:

Code: 
mkdir -p /var/tmp/backup_lxc_123/etc/vzdump
cp /etc/pve/lxc/123.conf /var/tmp/backup_lxc_123/etc/vzdump/pct.conf
/usr/bin/proxmox-backup-client backup pct.conf:/var/tmp/backup_lxc_123/etc/vzdump/pct.conf root.pxar:/zfs/subvol-123-disk-0 --backup-type ct --backup-id 123 --backup-time $(date +%s) --repository cluster@pbs@192.168.128.20:cluster --entries-max 9999999
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back