[SOLVED] Unable to start garbage collect

[ronaldzaz]

I have a daily backup with a lot of vm's and a weekly garbage collect.
Since 2 weeks the garbage collect fails each time.
Manually starting it with

proxmox-backup-manager garbage-collection start pbs-daily

Results in:

starting garbage collection on store pbs-daily
TASK ERROR: unable to get exclusive lock - EAGAIN: Try again

I've looked in every log file I could find and looked with journalctl but found nothing more than

pbs proxmox-backup-proxy[539]: Detected stopped task 'UPIDbs:0000021B:0000018A:000000F9:600039CA:garbage_collectionbs\x2ddaily:root@pam:'

Anyone any idea what I should check next? Something to trigger more verbose logging?
I've already restarted everything a couple of times so that can't be it.
And now I had to disable my backups because the folder has exceeded 11 TB and I'm running out of space now.

 

[fiona]

Hi,
are there other tasks running on the datastore at the same time (e.g. a backup)? Could you share the output of proxmox-backup-manager versions --verbose? What does lslocks show?

 

[ronaldzaz]

Hi,
are there other tasks running on the datastore at the same time (e.g. a backup)? Could you share the output of proxmox-backup-manager versions --verbose? What does lslocks show?

No other tasks are running.
Version:

proxmox-backup             1.0-4        running kernel: 5.4.78-2-pve
proxmox-backup-server      1.0.6-1      running version: 1.0.6     
pve-kernel-5.4             6.3-3                                   
pve-kernel-helper          6.3-3                                   
pve-kernel-5.4.78-2-pve    5.4.78-2                                
pve-kernel-5.4.65-1-pve    5.4.65-1                                
ifupdown2                  3.0.0-1+pve3                            
libjs-extjs                6.0.1-10                                
proxmox-backup-docs        1.0.6-1                                 
proxmox-backup-client      1.0.6-1                                 
proxmox-mini-journalreader 1.1-1                                   
proxmox-widget-toolkit     2.4-3                                   
pve-xtermjs                4.7.0-3                                 
smartmontools              7.1-pve2                                
zfsutils-linux             0.8.5-pve1

Locks:

COMMAND           PID  TYPE SIZE MODE  M START END PATH
rpcbind           446 FLOCK      WRITE 0     0   0 /run/rpcbind.lock
iscsid            520 POSIX   4B WRITE 0     0   0 /run/iscsid.pid
zed               455 POSIX   4B WRITE 0     0   0 /run/zed.pid
zed               455 POSIX      WRITE 0     0   0 /run/zed.state
cron              685 FLOCK   4B WRITE 0     0   0 /run/crond.pid
proxmox-backup-   539 POSIX      READ  0     0   0 /mnt/nas/pbs-daily/.nfs000000000020299f00000002 (deleted)
master            714 FLOCK  33B WRITE 0     0   0 /var/spool/postfix/pid/master.pid
master            714 FLOCK  33B WRITE 0     0   0 /var/lib/postfix/master.lock

 

[fiona]

Could you also share the output of findmnt /mnt/nas (assuming that /mnt/nas is the mount point) and tell us more about the share?

It's probably this bug you run into, although in your case the error is EAGAIN instead of EACCES.

EDIT: Could you also try using the nolock mount option and see if it works then?

 

[ronaldzaz]

Could you also share the output of findmnt /mnt/nas (assuming that /mnt/nas is the mount point) and tell us more about the share?

It's probably this bug you run into, although in your case the error is EAGAIN instead of EACCES.

EDIT: Could you also try using the nolock mount option and see if it works then?

The output was

/mnt/nas 10.100.7.4:/volume1/pbs nfs    rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.100.7.4,mountvers=3,mountport=892,mountproto=udp,local_lock=none,addr=10.100.7.4

But now I've tried the nolock mount option and it looks like that did it! Thanks. It's collecting garbage now.