Impossible Sync Job

Aug 14, 2024
8
3
3
Hi community !

I'm facing my first problem in PBS today.

I've got a 3 nodes cluster with PVE.

I use PBS to backup them in a datastore called "pbs_hdd_pool".

Then I wanted to use 2 hard drives, in rotation to do "external backups".

So I used 2x16To Seagate Iron Wolf disks.

Initalized through PBS.

Mounted auto with UUID and /etc/fstab.

Then, I use x2 "PULL" sync jobs.

Everything was going well, and suddenly, it stopped working.

I didn't changed anything.

Already rebooted.

And at the reboot, I receive this email : 'SMART error (ErrorCount) detected on host: pbs'

1760953689873.png

Code:
2025-10-17T22:00:00+02:00: Starting datastore sync job '-:pbs_hdd_pool:backup_impaire::s-67c51382-621c'
2025-10-17T22:00:00+02:00: task triggered by schedule 'fri 22:00'
2025-10-17T22:00:00+02:00: sync datastore 'backup_impaire' from 'pbs_hdd_pool'
2025-10-17T22:00:00+02:00: ----
2025-10-17T22:00:00+02:00: Syncing datastore 'pbs_hdd_pool', root namespace into datastore 'backup_impaire', root namespace
2025-10-17T22:00:00+02:00: found 54 groups to sync (out of 54 total)
2025-10-17T22:00:00+02:00: skipped: 28 snapshot(s) (2025-05-24T22:06:13Z .. 2025-10-15T22:08:09Z) - older than the newest snapshot present on sync target
2025-10-17T22:00:00+02:00: re-sync snapshot ct/1000/2025-10-16T22:07:29Z
2025-10-17T22:00:00+02:00: no data changes
2025-10-17T22:00:00+02:00: percentage done: 1.85% (1/54 groups)
2025-10-17T22:00:01+02:00: skipped: 28 snapshot(s) (2025-05-24T22:00:01Z .. 2025-10-15T22:00:02Z) - older than the newest snapshot present on sync target
2025-10-17T22:00:01+02:00: re-sync snapshot vm/100/2025-10-16T22:00:06Z
2025-10-17T22:00:01+02:00: no data changes
2025-10-17T22:00:01+02:00: percentage done: 3.70% (2/54 groups)
2025-10-17T22:00:01+02:00: skipped: 28 snapshot(s) (2025-05-24T22:00:02Z .. 2025-10-15T22:00:03Z) - older than the newest snapshot present on sync target
2025-10-17T22:00:01+02:00: re-sync snapshot vm/101/2025-10-16T22:00:04Z
2025-10-17T22:00:01+02:00: no data changes
2025-10-17T22:00:01+02:00: percentage done: 5.56% (3/54 groups)
2025-10-17T22:00:01+02:00: skipped: 27 snapshot(s) (2025-05-24T22:00:08Z .. 2025-10-15T22:00:39Z) - older than the newest snapshot present on sync target
2025-10-17T22:00:01+02:00: re-sync snapshot vm/102/2025-10-16T22:00:28Z
2025-10-17T22:00:01+02:00: no data changes
2025-10-17T22:00:01+02:00: percentage done: 7.41% (4/54 groups)
2025-10-17T22:00:01+02:00: skipped: 28 snapshot(s) (2025-05-24T22:01:31Z .. 2025-10-15T22:01:01Z) - older than the newest snapshot present on sync target
2025-10-17T22:00:01+02:00: re-sync snapshot vm/103/2025-10-16T22:00:47Z
2025-10-17T22:00:01+02:00: no data changes
2025-10-17T22:00:01+02:00: percentage done: 9.26% (5/54 groups)
2025-10-17T22:00:02+02:00: skipped: 28 snapshot(s) (2025-05-24T22:01:46Z .. 2025-10-15T22:01:22Z) - older than the newest snapshot present on sync target
2025-10-17T22:00:02+02:00: re-sync snapshot vm/107/2025-10-16T22:01:13Z
2025-10-17T22:00:02+02:00: no data changes
2025-10-17T22:00:02+02:00: percentage done: 11.11% (6/54 groups)
2025-10-17T22:00:02+02:00: skipped: 26 snapshot(s) (2025-05-24T22:01:54Z .. 2025-10-15T22:01:36Z) - older than the newest snapshot present on sync target
2025-10-17T22:00:02+02:00: re-sync snapshot vm/108/2025-10-16T22:01:12Z
2025-10-17T22:00:02+02:00: no data changes
2025-10-17T22:00:02+02:00: percentage done: 12.96% (7/54 groups)
2025-10-17T22:00:02+02:00: skipped: 28 snapshot(s) (2025-05-24T22:00:53Z .. 2025-10-15T22:04:47Z) - older than the newest snapshot present on sync target
2025-10-17T22:00:02+02:00: re-sync snapshot vm/109/2025-10-16T22:01:49Z
2025-10-17T22:00:02+02:00: no data changes
2025-10-17T22:00:02+02:00: percentage done: 14.81% (8/54 groups)
2025-10-17T22:00:02+02:00: skipped: 11 snapshot(s) (2025-05-24T22:02:06Z .. 2025-09-28T22:01:28Z) - older than the newest snapshot present on sync target
2025-10-17T22:00:02+02:00: re-sync snapshot vm/110/2025-09-29T22:01:28Z due to corruption
2025-10-17T22:00:02+02:00: sync archive qemu-server.conf.blob
2025-10-17T22:00:03+02:00: sync archive drive-scsi0.img.fidx
2025-10-17T22:00:03+02:00: downloaded 0 B (0 B/s)
2025-10-17T22:00:03+02:00: percentage done: 14.92% (8/54 groups, 1/18 snapshots in group #9)
2025-10-17T22:00:03+02:00: re-sync snapshot vm/110/2025-09-30T22:01:20Z due to corruption
2025-10-17T22:00:03+02:00: sync archive qemu-server.conf.blob
2025-10-17T22:00:03+02:00: sync archive drive-scsi0.img.fidx
2025-10-17T22:00:03+02:00: percentage done: 15.02% (8/54 groups, 2/18 snapshots in group #9)
2025-10-17T22:00:03+02:00: sync group vm/110 failed - Input/output error (os error 5)
2025-10-17T22:00:03+02:00: sync group vm/111 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/112 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/113 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/114 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/115 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/116 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/117 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/118 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/119 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/120 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/121 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/122 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/123 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/124 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/125 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/126 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/127 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/128 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/129 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/130 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/131 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/132 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/133 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/134 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/135 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/136 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/137 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/138 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/139 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/140 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/141 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/142 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/143 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/144 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/145 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/146 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/147 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/148 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/149 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/150 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/151 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/152 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/153 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/154 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: sync group vm/161 failed - group lock failed: File exists (os error 17)
2025-10-17T22:00:03+02:00: create_locked_backup_group failed
2025-10-17T22:00:03+02:00: Finished syncing root namespace, current progress: 53 groups, 0 snapshots
2025-10-17T22:00:03+02:00: queued notification (id=a72893cc-cf81-4fae-98ea-cf132a58f1f6)
2025-10-17T22:00:03+02:00: TASK ERROR: sync failed with some errors.


Here is my config (not licenced yet + fully up to date) :

Code:
proxmox-backup: 3.2.0 (running kernel: 6.8.12-4-pve)
proxmox-backup-server: 3.3.0-1 (running version: 3.3.0)
proxmox-kernel-helper: 8.1.0
proxmox-kernel-6.8: 6.8.12-4
proxmox-kernel-6.8.12-4-pve-signed: 6.8.12-4
ifupdown2: 3.2.0-1+pmx11
libjs-extjs: 7.0.0-5
proxmox-backup-docs: 3.3.0-1
proxmox-backup-client: 3.3.0-1
proxmox-mail-forward: 0.3.1
proxmox-mini-journalreader: 1.4.0
proxmox-offline-mirror-helper: 0.6.7
proxmox-widget-toolkit: 4.3.3
pve-xtermjs: 5.3.0-3
smartmontools: 7.3-pve1
zfsutils-linux: 2.2.6-pve1

1760948951207.png

Can you please guys help me to debug this situation ? I don't know were to start :)

Have a nice day,

EXODUS
 
Last edited: