Garbage Collecor - TASK ERROR: unexpected error on datastore traversal: Not a directory (os error 20)

[l.ansaloni]

Hi there,
for some days the Garbage Collector operation has failed with the following error:

TASK ERROR: unexpected error on datastore traversal: Not a directory (os error 20)

Backups work fine.

How can I fix the garbage collector error?

Package version:

proxmox-backup: 2.1-1 (running kernel: 5.13.19-6-pv
proxmox-backup-server: 2.1.8-1 (running version: 2.1.
pve-kernel-helper: 7.2
pve-kernel-5.13: 7.1-9
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.13.19-1-pve: 5.13.19-3
ifupdown2: 3.1.0-1+pmx3
libjs-extjs: 7.0.0-1
proxmox-backup-docs: 2.1.8-1
proxmox-backup-client: 2.1.8-1
proxmox-mini-journalreader: 1.2-1
proxmox-widget-toolkit: 3.4-10
pve-xtermjs: 4.16.0-1
smartmontools: 7.2-pve3
zfsutils-linux: 2.1.4-pve1

 

[dcsapak]

what fs is the datastore exactly?
can you post the complete task log of the gc ?

 

[l.ansaloni]

The datastore is a shared folder from a QNAP NAS mounted as cifs:

//NAS-IP/backup-cluster /backup-cluster cifs auto,username=backup-cluster,password=XXXXXXX,vers=1.0,uid=34,noforceuid,gid=34,noforcegid 0 0

In the Task Viewer Output I only see these log lines:

2022-05-13T00:00:00+02:00: starting garbage collection on store backup-cluster
2022-05-13T00:00:00+02:00: task triggered by schedule 'daily'
2022-05-13T00:00:00+02:00: Start GC phase1 (mark used chunks)
2022-05-13T00:17:22+02:00: TASK ERROR: unexpected error on datastore traversal: Input/output error (os error 5)

 

[dcsapak]

2022-05-13T00:17:22+02:00: TASK ERROR: unexpected error on datastore traversal: Input/output error (os error 5)

that is a different error from the first one. input/output errors are most often a broken storage or (in case of network storage) some network issue. are you sure the connection to the cifs is stable and working properly?

 

[l.ansaloni]

that is a different error from the first one. input/output errors are most often a broken storage or (in case of network storage) some network issue. are you sure the connection to the cifs is stable and working properly?

On the same storage I have scheduled the backup of 16VM which works regularly, this for example is the log of a backup:

2022-05-15T21:22:40+02:00: starting new backup on datastore 'backup-cluster': "vm/505/2022-05-15T19:22:39Z"
2022-05-15T21:22:40+02:00: download 'index.json.blob' from previous backup.
2022-05-15T21:22:40+02:00: register chunks in 'drive-virtio0.img.fidx' from previous backup.
2022-05-15T21:22:40+02:00: download 'drive-virtio0.img.fidx' from previous backup.
2022-05-15T21:22:40+02:00: created new fixed index 1 ("vm/505/2022-05-15T19:22:39Z/drive-virtio0.img.fidx")
2022-05-15T21:22:40+02:00: register chunks in 'drive-virtio2.img.fidx' from previous backup.
2022-05-15T21:22:40+02:00: download 'drive-virtio2.img.fidx' from previous backup.
2022-05-15T21:22:40+02:00: created new fixed index 2 ("vm/505/2022-05-15T19:22:39Z/drive-virtio2.img.fidx")
2022-05-15T21:22:40+02:00: add blob "/backup-cluster/vm/505/2022-05-15T19:22:39Z/qemu-server.conf.blob" (413 bytes, comp: 413)
2022-05-15T21:23:13+02:00: Upload statistics for 'drive-virtio2.img.fidx'
2022-05-15T21:23:13+02:00: UUID: 638e947ea4b242feb74d59b0f87db0d1
2022-05-15T21:23:13+02:00: Checksum: e05a17b5aeb571d1ba8a542f66d4a238d13b36f2e396248723cc63b6c9840500
2022-05-15T21:23:13+02:00: Size: 1392508928
2022-05-15T21:23:13+02:00: Chunk count: 332
2022-05-15T21:23:13+02:00: Upload size: 1384120320 (99%)
2022-05-15T21:23:13+02:00: Duplicates: 2+1 (0%)
2022-05-15T21:23:13+02:00: Compression: 20%
2022-05-15T21:23:13+02:00: successfully closed fixed index 2
2022-05-15T21:23:18+02:00: Upload statistics for 'drive-virtio0.img.fidx'
2022-05-15T21:23:18+02:00: UUID: 5f0b929e558247faaaccd03b5efc61ca
2022-05-15T21:23:18+02:00: Checksum: a29d0d8f4faeac0581d5208fe4a0fb48abbadd560c5537c780cea7f267986d1e
2022-05-15T21:23:18+02:00: Size: 675282944
2022-05-15T21:23:18+02:00: Chunk count: 161
2022-05-15T21:23:18+02:00: Upload size: 595591168 (88%)
2022-05-15T21:23:18+02:00: Duplicates: 19+6 (15%)
2022-05-15T21:23:18+02:00: Compression: 18%
2022-05-15T21:23:18+02:00: successfully closed fixed index 1
2022-05-15T21:23:18+02:00: add blob "/backup-cluster/vm/505/2022-05-15T19:22:39Z/index.json.blob" (373 bytes, comp: 373)
2022-05-15T21:23:18+02:00: successfully finished backup
2022-05-15T21:23:18+02:00: backup finished successfully
2022-05-15T21:23:18+02:00: TASK OK

I think network issue can be excluded, moreover the server and NAS are on the same switch which also provides other network services without problems.

The error log is not always the same, these two errors alternate:

2022-05-16T00:00:00+02:00: starting garbage collection on store backup-cluster
2022-05-16T00:00:00+02:00: task triggered by schedule 'daily'
2022-05-16T00:00:00+02:00: Start GC phase1 (mark used chunks)
2022-05-16T00:07:01+02:00: TASK ERROR: unexpected error on datastore traversal: Not a directory (os error 20)

2022-05-15T00:00:02+02:00: starting garbage collection on store backup-cluster
2022-05-15T00:00:02+02:00: task triggered by schedule 'daily'
2022-05-15T00:00:02+02:00: Start GC phase1 (mark used chunks)
2022-05-15T00:07:26+02:00: TASK ERROR: unexpected error on datastore traversal: Input/output error (os error 5)

 

[l.ansaloni]

I solved the problem.
In the same folder as the backup there was another subfolder that had nothing to do with the backup, removed that subfolder the garbege collector started regularly.