Fehlermeldung beim Backup

[GMBauer]

Hallo beinand,

ich habe bei einem Backup einen Abbruch mit folgender Fehlermeldung auf dem PVE:

INFO: sata0: dirty-bitmap status: OK (30.6 GiB of 500.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 30.6 GiB dirty of 500.0 GiB total
INFO: 1% (592.0 MiB of 30.6 GiB) in 3s, read: 197.3 MiB/s, write: 197.3 MiB/s
INFO: 3% (1.1 GiB of 30.6 GiB) in 6s, read: 186.7 MiB/s, write: 186.7 MiB/s
INFO: 5% (1.8 GiB of 30.6 GiB) in 9s, read: 233.3 MiB/s, write: 233.3 MiB/s
INFO: 7% (2.4 GiB of 30.6 GiB) in 12s, read: 198.7 MiB/s, write: 198.7 MiB/s
INFO: 9% (3.0 GiB of 30.6 GiB) in 15s, read: 197.3 MiB/s, write: 197.3 MiB/s
INFO: 11% (3.5 GiB of 30.6 GiB) in 18s, read: 177.3 MiB/s, write: 177.3 MiB/s
INFO: 13% (4.1 GiB of 30.6 GiB) in 21s, read: 206.7 MiB/s, write: 206.7 MiB/s
INFO: 15% (4.6 GiB of 30.6 GiB) in 24s, read: 186.7 MiB/s, write: 186.7 MiB/s
INFO: 16% (5.2 GiB of 30.6 GiB) in 27s, read: 176.0 MiB/s, write: 173.3 MiB/s
INFO: 18% (5.7 GiB of 30.6 GiB) in 30s, read: 200.0 MiB/s, write: 200.0 MiB/s
INFO: 20% (6.3 GiB of 30.6 GiB) in 33s, read: 196.0 MiB/s, write: 194.7 MiB/s
INFO: 22% (6.9 GiB of 30.6 GiB) in 36s, read: 201.3 MiB/s, write: 201.3 MiB/s
INFO: 24% (7.5 GiB of 30.6 GiB) in 39s, read: 192.0 MiB/s, write: 192.0 MiB/s
INFO: 26% (8.0 GiB of 30.6 GiB) in 42s, read: 192.0 MiB/s, write: 192.0 MiB/s
INFO: 27% (8.6 GiB of 30.6 GiB) in 45s, read: 184.0 MiB/s, write: 184.0 MiB/s
INFO: 29% (9.2 GiB of 30.6 GiB) in 48s, read: 208.0 MiB/s, write: 208.0 MiB/s
INFO: 31% (9.7 GiB of 30.6 GiB) in 51s, read: 174.7 MiB/s, write: 174.7 MiB/s
INFO: 33% (10.1 GiB of 30.6 GiB) in 54s, read: 148.0 MiB/s, write: 148.0 MiB/s
ERROR: backup write data failed: command error: error:0A0003FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../ssl/record/rec_layer_s3.c:1605:SSL alert number 20
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 101 failed - backup write data failed: command error: error:0A0003FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac:../ssl/record/rec_layer_s3.c:1605:SSL alert number 20
INFO: Failed at 2025-06-04 22:30:57

Weil ich zwei PBS hintereinander schalten wollte, ist der hier verwendete PBS nur ein kleiner Dell Optiplex mit einer Samsung 863a drin mit 99% "Restqualität".

Was meint Ihr? Defekte SSD?

Hier das Log vom PBS von den automatischen Backups von 23.30 und 2.30:

Jun 04 22:30:03 optipbs proxmox-backup-proxy[616]: starting new backup on datastore 'backups' from ::ffff:192.168.230.23: "ns/Fujitsu/vm/101/2025-06-04T20:30:02Z"
Jun 04 22:30:56 optipbs proxmox-backup-proxy[616]: TASK ERROR: connection error: error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:622:: error:0A000119:SSL routines:ssl3_get_record:decryption failed or bad record mac:../ssl/record/ssl3_record.c:622:
Jun 04 22:30:57 optipbs proxmox-backup-proxy[616]: starting new backup on datastore 'backups' from ::ffff:192.168.230.23: "ns/Fujitsu/vm/110/2025-06-04T20:30:57Z"
Jun 04 22:31:07 optipbs proxmox-backup-proxy[616]: Upload backup log to datastore 'backups', namespace 'Fujitsu' vm/110/2025-06-04T20:30:57Z/client.log.blob
Jun 04 22:35:37 optipbs smartd[472]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 51 to 50
Jun 04 22:35:37 optipbs smartd[472]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 51 to 50
Jun 04 22:35:39 optipbs proxmox-backup-proxy[616]: rrd journal successfully committed (25 files in 0.011 seconds)

Jun 04 23:05:37 optipbs smartd[472]: Device: /dev/sda [SAT], SMART Usage Attribute: 190 Airflow_Temperature_Cel changed from 50 to 51
Jun 04 23:05:37 optipbs smartd[472]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 50 to 51

Jun 05 02:30:03 optipbs proxmox-backup-proxy[616]: starting new backup on datastore 'backups' from ::ffff:192.168.230.23: "ns/Fujitsu/vm/101/2025-06-05T00:30:03Z"
Jun 05 02:30:22 optipbs proxmox-backup-proxy[616]: TASK ERROR: connection error: bytes remaining on stream: bytes remaining on stream
Jun 05 02:30:23 optipbs proxmox-backup-proxy[616]: starting new backup on datastore 'backups' from ::ffff:192.168.230.23: "ns/Fujitsu/vm/110/2025-06-05T00:30:23Z"
Jun 05 02:30:26 optipbs proxmox-backup-proxy[616]: Upload backup log to datastore 'backups', namespace 'Fujitsu' vm/110/2025-06-05T00:30:23Z/client.log.blob

Dankeschön für eure Hilfe!

 

[GMBauer]

Von Hand gestartet läuft alles durch...

INFO: starting new backup job: vzdump 101 --notes-template '{{guestname}}' --notification-mode auto --mode snapshot --remove 0 --node pvefuji --storage OptiPBS
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2025-06-05 11:39:34
INFO: status = running
INFO: VM Name: xxxxDocBoxAppServ
INFO: include disk 'sata0' 'lvm4tb:vm-101-disk-0' 500G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: skip unused drive 'lvm4tb:vm-101-disk-1' (not included into backup)
INFO: creating Proxmox Backup Server archive 'vm/101/2025-06-05T09:39:34Z'
INFO: started backup task 'fa3b8207-fd28-4a7c-a085-46075a37c331'
INFO: resuming VM again
INFO: sata0: dirty-bitmap status: OK (35.8 GiB of 500.0 GiB dirty)
INFO: using fast incremental mode (dirty-bitmap), 35.8 GiB dirty of 500.0 GiB total
INFO: 1% (592.0 MiB of 35.8 GiB) in 3s, read: 197.3 MiB/s, write: 197.3 MiB/s
INFO: 2% (948.0 MiB of 35.8 GiB) in 6s, read: 118.7 MiB/s, write: 118.7 MiB/s
INFO: 4% (1.6 GiB of 35.8 GiB) in 9s, read: 214.7 MiB/s, write: 214.7 MiB/s
INFO: 6% (2.2 GiB of 35.8 GiB) in 12s, read: 205.3 MiB/s, write: 205.3 MiB/s
INFO: 7% (2.7 GiB of 35.8 GiB) in 15s, read: 200.0 MiB/s, write: 200.0 MiB/s
INFO: 9% (3.3 GiB of 35.8 GiB) in 18s, read: 186.7 MiB/s, write: 186.7 MiB/s
INFO: 10% (3.8 GiB of 35.8 GiB) in 21s, read: 184.0 MiB/s, write: 184.0 MiB/s
INFO: 12% (4.4 GiB of 35.8 GiB) in 24s, read: 197.3 MiB/s, write: 197.3 MiB/s
INFO: 13% (5.0 GiB of 35.8 GiB) in 27s, read: 192.0 MiB/s, write: 192.0 MiB/s
INFO: 15% (5.5 GiB of 35.8 GiB) in 30s, read: 166.7 MiB/s, write: 165.3 MiB/s
INFO: 16% (6.0 GiB of 35.8 GiB) in 33s, read: 182.7 MiB/s, write: 182.7 MiB/s
INFO: 18% (6.6 GiB of 35.8 GiB) in 36s, read: 197.3 MiB/s, write: 197.3 MiB/s
INFO: 19% (7.1 GiB of 35.8 GiB) in 39s, read: 189.3 MiB/s, write: 189.3 MiB/s
INFO: 21% (7.7 GiB of 35.8 GiB) in 42s, read: 194.7 MiB/s, write: 194.7 MiB/s
INFO: 23% (8.3 GiB of 35.8 GiB) in 45s, read: 196.0 MiB/s, write: 196.0 MiB/s
INFO: 24% (8.8 GiB of 35.8 GiB) in 48s, read: 190.7 MiB/s, write: 190.7 MiB/s
INFO: 26% (9.3 GiB of 35.8 GiB) in 51s, read: 173.3 MiB/s, write: 173.3 MiB/s
INFO: 27% (9.9 GiB of 35.8 GiB) in 54s, read: 204.0 MiB/s, write: 204.0 MiB/s
INFO: 29% (10.5 GiB of 35.8 GiB) in 57s, read: 181.3 MiB/s, write: 181.3 MiB/s
INFO: 30% (11.0 GiB of 35.8 GiB) in 1m, read: 173.3 MiB/s, write: 173.3 MiB/s
INFO: 32% (11.5 GiB of 35.8 GiB) in 1m 3s, read: 185.3 MiB/s, write: 184.0 MiB/s
INFO: 33% (12.0 GiB of 35.8 GiB) in 1m 6s, read: 177.3 MiB/s, write: 177.3 MiB/s
INFO: 34% (12.5 GiB of 35.8 GiB) in 1m 9s, read: 154.7 MiB/s, write: 154.7 MiB/s
INFO: 36% (12.9 GiB of 35.8 GiB) in 1m 12s, read: 148.0 MiB/s, write: 148.0 MiB/s
INFO: 37% (13.4 GiB of 35.8 GiB) in 1m 15s, read: 150.7 MiB/s, write: 150.7 MiB/s
INFO: 38% (13.8 GiB of 35.8 GiB) in 1m 18s, read: 152.0 MiB/s, write: 152.0 MiB/s
INFO: 39% (14.3 GiB of 35.8 GiB) in 1m 21s, read: 152.0 MiB/s, write: 152.0 MiB/s
INFO: 41% (14.8 GiB of 35.8 GiB) in 1m 24s, read: 198.7 MiB/s, write: 198.7 MiB/s
INFO: 42% (15.3 GiB of 35.8 GiB) in 1m 27s, read: 172.0 MiB/s, write: 172.0 MiB/s
INFO: 44% (15.8 GiB of 35.8 GiB) in 1m 30s, read: 153.3 MiB/s, write: 153.3 MiB/s
INFO: 45% (16.3 GiB of 35.8 GiB) in 1m 33s, read: 160.0 MiB/s, write: 160.0 MiB/s
INFO: 46% (16.7 GiB of 35.8 GiB) in 1m 36s, read: 165.3 MiB/s, write: 165.3 MiB/s
INFO: 48% (17.2 GiB of 35.8 GiB) in 1m 39s, read: 162.7 MiB/s, write: 162.7 MiB/s
INFO: 49% (17.7 GiB of 35.8 GiB) in 1m 42s, read: 166.7 MiB/s, write: 166.7 MiB/s
INFO: 50% (18.2 GiB of 35.8 GiB) in 1m 45s, read: 158.7 MiB/s, write: 158.7 MiB/s
INFO: 51% (18.6 GiB of 35.8 GiB) in 1m 48s, read: 146.7 MiB/s, write: 146.7 MiB/s
INFO: 53% (19.1 GiB of 35.8 GiB) in 1m 51s, read: 157.3 MiB/s, write: 157.3 MiB/s
INFO: 54% (19.5 GiB of 35.8 GiB) in 1m 54s, read: 157.3 MiB/s, write: 157.3 MiB/s
INFO: 55% (20.0 GiB of 35.8 GiB) in 1m 57s, read: 158.7 MiB/s, write: 158.7 MiB/s
INFO: 57% (20.5 GiB of 35.8 GiB) in 2m, read: 160.0 MiB/s, write: 160.0 MiB/s
INFO: 58% (20.9 GiB of 35.8 GiB) in 2m 3s, read: 162.7 MiB/s, write: 162.7 MiB/s
INFO: 59% (21.4 GiB of 35.8 GiB) in 2m 6s, read: 161.3 MiB/s, write: 161.3 MiB/s
INFO: 61% (21.9 GiB of 35.8 GiB) in 2m 9s, read: 154.7 MiB/s, write: 154.7 MiB/s
INFO: 62% (22.3 GiB of 35.8 GiB) in 2m 12s, read: 153.3 MiB/s, write: 153.3 MiB/s
INFO: 63% (22.8 GiB of 35.8 GiB) in 2m 15s, read: 154.7 MiB/s, write: 154.7 MiB/s
INFO: 65% (23.3 GiB of 35.8 GiB) in 2m 18s, read: 190.7 MiB/s, write: 190.7 MiB/s
INFO: 66% (23.8 GiB of 35.8 GiB) in 2m 21s, read: 158.7 MiB/s, write: 158.7 MiB/s
INFO: 67% (24.3 GiB of 35.8 GiB) in 2m 24s, read: 164.0 MiB/s, write: 164.0 MiB/s
INFO: 69% (24.7 GiB of 35.8 GiB) in 2m 27s, read: 157.3 MiB/s, write: 157.3 MiB/s
INFO: 70% (25.2 GiB of 35.8 GiB) in 2m 30s, read: 160.0 MiB/s, write: 160.0 MiB/s
INFO: 71% (25.7 GiB of 35.8 GiB) in 2m 33s, read: 173.3 MiB/s, write: 173.3 MiB/s
INFO: 73% (26.2 GiB of 35.8 GiB) in 2m 36s, read: 160.0 MiB/s, write: 160.0 MiB/s
INFO: 74% (26.6 GiB of 35.8 GiB) in 2m 39s, read: 162.7 MiB/s, write: 162.7 MiB/s
INFO: 75% (27.2 GiB of 35.8 GiB) in 2m 42s, read: 173.3 MiB/s, write: 173.3 MiB/s
INFO: 77% (27.6 GiB of 35.8 GiB) in 2m 45s, read: 162.7 MiB/s, write: 162.7 MiB/s
INFO: 78% (28.1 GiB of 35.8 GiB) in 2m 48s, read: 164.0 MiB/s, write: 164.0 MiB/s
INFO: 79% (28.6 GiB of 35.8 GiB) in 2m 51s, read: 158.7 MiB/s, write: 158.7 MiB/s
INFO: 81% (29.1 GiB of 35.8 GiB) in 2m 54s, read: 180.0 MiB/s, write: 180.0 MiB/s
INFO: 82% (29.6 GiB of 35.8 GiB) in 2m 57s, read: 174.7 MiB/s, write: 174.7 MiB/s
INFO: 84% (30.1 GiB of 35.8 GiB) in 3m, read: 172.0 MiB/s, write: 172.0 MiB/s
INFO: 85% (30.6 GiB of 35.8 GiB) in 3m 3s, read: 173.3 MiB/s, write: 173.3 MiB/s
INFO: 86% (31.2 GiB of 35.8 GiB) in 3m 6s, read: 180.0 MiB/s, write: 180.0 MiB/s
INFO: 88% (31.7 GiB of 35.8 GiB) in 3m 9s, read: 190.7 MiB/s, write: 186.7 MiB/s
INFO: 90% (32.3 GiB of 35.8 GiB) in 3m 12s, read: 204.0 MiB/s, write: 204.0 MiB/s
INFO: 91% (32.9 GiB of 35.8 GiB) in 3m 15s, read: 190.7 MiB/s, write: 190.7 MiB/s
INFO: 93% (33.4 GiB of 35.8 GiB) in 3m 18s, read: 180.0 MiB/s, write: 180.0 MiB/s
INFO: 94% (33.9 GiB of 35.8 GiB) in 3m 21s, read: 176.0 MiB/s, write: 176.0 MiB/s
INFO: 96% (34.4 GiB of 35.8 GiB) in 3m 24s, read: 165.3 MiB/s, write: 165.3 MiB/s
INFO: 97% (35.0 GiB of 35.8 GiB) in 3m 27s, read: 198.7 MiB/s, write: 198.7 MiB/s
INFO: 99% (35.6 GiB of 35.8 GiB) in 3m 30s, read: 198.7 MiB/s, write: 198.7 MiB/s
INFO: 100% (35.8 GiB of 35.8 GiB) in 3m 33s, read: 89.3 MiB/s, write: 89.3 MiB/s
INFO: Waiting for server to finish backup validation...
INFO: backup was done incrementally, reused 464.20 GiB (92%)
INFO: transferred 35.82 GiB in 218 seconds (168.3 MiB/s)
INFO: adding notes to backup
INFO: Finished Backup of VM 101 (00:03:38)
INFO: Backup finished at 2025-06-05 11:43:12
INFO: Backup job finished successfully
INFO: skipping disabled target 'mail-to-root'
TASK OK

 

[Chris]

Hallo, welche NIC ist im PBS verbaut und in Verwendung? TCP segmentation offloading vom Linux Kernel an manche NICs hat schon des öfteren zu solchen Netzwerk-Problemen geführt. Eventuell mal versuchen dies zu deaktivieren, siehe dazu auch https://forum.proxmox.com/threads/pbs-sync-failed-each-time.113921/#post-573939

Edit: Die notwendige Konfiguration um das offloading zu deaktivieren ist hier beschrieben https://forum.proxmox.com/threads/e1000e-reset-adapter-unexpectedly.87769/post-384609

 

[GMBauer]

Aaaarghl, ich fall vom Glauben ab... Das ist eine Auswirkung eines e1000e-Bugs? Ich habe noch - um den zu umgehen - alles mit einem Realtek-Adapter am USB getestet...

 

[GMBauer]

Das war es noch nicht...

Mit einem externen USB-Realtek-Netzwerkdings dran (Log vom PVE)

INFO: starting new backup job: vzdump 110 --notification-mode auto --mode snapshot --node pvefuji --remove 0 --storage OptiPBS --notes-template '{{guestname}}'
INFO: Starting Backup of VM 110 (qemu)
INFO: Backup started at 2025-06-05 15:26:55
INFO: status = running
INFO: VM Name: xxxxxHost02
INFO: include disk 'scsi0' 'lvm4tb:vm-110-disk-0' 30G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/110/2025-06-05T13:26:55Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '940bcb49-ba69-4776-ab1a-70f8e9b72a11'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO: 2% (876.0 MiB of 30.0 GiB) in 3s, read: 292.0 MiB/s, write: 282.7 MiB/s
INFO: 6% (1.9 GiB of 30.0 GiB) in 6s, read: 361.3 MiB/s, write: 182.7 MiB/s
INFO: 8% (2.6 GiB of 30.0 GiB) in 9s, read: 218.7 MiB/s, write: 205.3 MiB/s
INFO: 10% (3.1 GiB of 30.0 GiB) in 12s, read: 201.3 MiB/s, write: 201.3 MiB/s
INFO: 11% (3.6 GiB of 30.0 GiB) in 15s, read: 141.3 MiB/s, write: 141.3 MiB/s
INFO: 13% (3.9 GiB of 30.0 GiB) in 18s, read: 125.3 MiB/s, write: 125.3 MiB/s
INFO: 13% (4.0 GiB of 30.0 GiB) in 19s, read: 64.0 MiB/s, write: 64.0 MiB/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest.
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 110 failed - backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest.
INFO: Failed at 2025-06-05 15:27:14
INFO: Backup job finished with errors
INFO: skipping disabled target 'mail-to-root'
TASK ERROR: job errors

Log vom PBS:

Jun 05 15:26:55 optipbs proxmox-backup-proxy[711]: starting new backup on datastore 'backups' from ::ffff:192.168.230.23: "ns/fujitsu/vm/110/2025-06-05T13:26:55Z"
Jun 05 15:26:55 optipbs proxmox-backup-proxy[711]: created new fixed index 1 ("ns/fujitsu/vm/110/2025-06-05T13:26:55Z/drive-scsi0.img.fidx")
Jun 05 15:26:55 optipbs proxmox-backup-proxy[711]: add blob "/mnt/datastore/backups/ns/fujitsu/vm/110/2025-06-05T13:26:55Z/qemu-server.conf.blob" (307 bytes, comp: 307)
Jun 05 15:27:14 optipbs proxmox-backup-proxy[711]: backup failed: connection error: connection reset
Jun 05 15:27:14 optipbs proxmox-backup-proxy[711]: removing failed backup

Wenn das passiert, geht die Festplatten-LED in ein Dauerblinken mit etwa 2 Hz, auch wenn ich den Vorgang schon lange abgebrochen habe.


Zum Ausschließen von Fehlern habe ich auf dem PVE einen virtuellen PBS installiert. Auf den sichert der PVE perfekt. (Ist aber natürlich keine akzeptable Lösung.)

Hat jemand einen Tipp für mich? Ein Live-Linux oder meinetwegen auch ein ISO, mit irgendeinem Tool zum Dauerprüfen der SSDs? Würde die nicht gerne ausbauen, sondern den kleinen Rechner so wie er ist einem (SSD-)Belastungstest unterziehen.

 

[GMBauer]

Wenn ich beispielsweise ISO-Dateien in einen PVE uploade, dann kommen die ja auch zuerst in irgendein Verzeichnis und werden dann erst dahin geschrieben wo sie final hin sollen, habe ich hier im Forum gelernt. (Sorry für die einfache Ausdrucksweise....) Passiert das gleiche denn auch beim PBS? Macht es Sinn, die root-SSD ebenfalls zu prüfen?

 

[Chris]

Das war es noch nicht...
...

ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest.
INFO: aborting backup job

Der Fehler ist nun jedoch ein anderer, der hochgeladene Daten-chunk entspricht nicht dem erwarteten Inhalt. Dies kann verschiedene Ursachen haben, als ersten Schritt würde ich jedoch einen erweiterten memory check empfehlen. Eventuell auch die RAM sticks ausbauen, Kontakte säubern und wieder einbauen.

Wenn ich beispielsweise ISO-Dateien in einen PVE uploade, dann kommen die ja auch zuerst in irgendein Verzeichnis und werden dann erst dahin geschrieben wo sie final hin sollen, habe ich hier im Forum gelernt. (Sorry für die einfache Ausdrucksweise....) Passiert das gleiche denn auch beim PBS? Macht es Sinn, die root-SSD ebenfalls zu prüfen?

Nein, Chunks und Metadaten files werden direkt in den datastore geschrieben, nicht in ein temporäres Verzeichnis.

 

[GMBauer]

Können alle Fehler an einem defekten Core in einem i5 8400 (o.ä.) im Server liegen? Serverhardware gewechselt, weil mir das zu heikel war, und jetzt backupt er auf den gleichen PBS ohne Probleme.

Ich werde den herausgenommenen Rechner mal zuhause in einen Dauerlauf schicken. Einen normalen Memtest über 24 Stunden hat er vorher problemlos absolviert...