Probleme mit großem Sync Job

[ednt]

Wir habe einen PBS1 auf den die Backups gemacht werden.
Ein PBS2 an einem anderen Standort soll dort alles mittels 2er Sync-Jobs holen.

Das Problem ist anscheinend, dass eine VM, unser file server, 2.7TB hat und es schon 6 (inkrementelle) Backups gibt.
Der Sync job brach schon des öfteren ab:

2021-03-31T19:55:11+02:00: sync snapshot "vm/100/2021-03-26T22:00:02Z" done
2021-03-31T19:55:11+02:00: percentage done: 50.91% (11 of 22 groups, 1 of 5 group snapshots)
2021-03-31T19:55:11+02:00: sync group vm/100 failed - write failed: No space left on device (os error 28)
2021-03-31T19:55:11+02:00: sync group vm/108 failed - write failed: No space left on device (os error 28)
2021-03-31T19:55:12+02:00: sync group vm/1113 failed - write failed: No space left on device (os error 28)
2021-03-31T19:55:12+02:00: sync group vm/114 failed - write failed: No space left on device (os error 28)
2021-03-31T19:55:12+02:00: sync group vm/123 failed - write failed: No space left on device (os error 28)
2021-03-31T19:55:12+02:00: sync group vm/128 failed - write failed: No space left on device (os error 28)
2021-03-31T19:55:13+02:00: sync group vm/143 failed - write failed: No space left on device (os error 28)
2021-03-31T19:55:13+02:00: sync group vm/148 failed - write failed: No space left on device (os error 28)
2021-03-31T19:55:13+02:00: sync group vm/162 failed - write failed: No space left on device (os error 28)
2021-03-31T19:55:13+02:00: sync group vm/170 failed - write failed: No space left on device (os error 28)
2021-03-31T19:55:14+02:00: sync group vm/175 failed - write failed: No space left on device (os error 28)
2021-03-31T19:55:14+02:00: TASK ERROR: sync failed with some errors.

Aber:


df auf PBS2
data-1 15T 1.6T 13T 12% /mnt/datastore/data-1

df auf PBS1
data-1 57T 2.3T 55T 5% /mnt/datastore/data-1


Wenn ich nun den Job manuell starten will bekomme ich auf dem PBS2
TASK ERROR: Failed to retrieve backup groups from remote - EMFILE: Too many open files

Der PBS1 war per Web nicht mehr ereichbar, nur noch über ssh.

Darum habe ich ihn nun rebootet.

Irgendwer eine Idee?

 

[ednt]

Neuer Versuch nach reboots von beiden PBS:

Die große VM will nicht:

2021-04-01T15:56:03+02:00: sync snapshot "vm/100/2021-03-26T22:00:02Z"
2021-04-01T15:56:03+02:00: sync archive qemu-server.conf.blob
2021-04-01T15:56:03+02:00: sync archive drive-scsi4.img.fidx
2021-04-01T15:56:22+02:00: downloaded 0 bytes (0.00 MiB/s)
2021-04-01T15:56:22+02:00: sync archive drive-scsi3.img.fidx
2021-04-01T16:07:23+02:00: downloaded 0 bytes (0.00 MiB/s)
2021-04-01T16:07:23+02:00: sync archive drive-scsi1.img.fidx
2021-04-01T16:07:23+02:00: percentage done: 50.76% (11 of 22 groups, 1 of 6 group snapshots)
2021-04-01T16:07:23+02:00: sync group vm/100 failed - broken pipe

Alle anderen gingen.