Backups schlagen häufig wegen falschem Digest fehl

P

Poehlmann

New Member
Sep 4, 2023
10
0
1
Hallo,

ich habe bei unserem PBS das Problem, dass die Backups häufig, aber nicht immer mit folgendem Fehler fehlschlagen.
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest.
Click to expand...

Ich weiß nicht genau, wie ich so etwas debuggen kann. Falls ihr hier mehr Hinweise zu habt, wäre euch sehr verbunden.

Das System ist eine Neuinstallation des PBS, das System ist unten aufgeführt.

CPU: 8 x Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz (1 Socket)
Kernel Version: Linux 6.8.4-3-pve (2024-05-02T11:55Z)
RAM: 64GB ECC
RAID-Controller: LSI MegaRAID SAS-3 3108 (JBOD Mode)
Drives: 2x 16TB HDD, 4x 1TB SSD

NFO: starting kvm to execute backup task
INFO: started backup task '5c8a165f-fd52-4caf-8334-945719b022db'
INFO: resuming VM again after 6 seconds
INFO: scsi0: dirty-bitmap status: created new
INFO: 0% (540.0 MiB of 200.0 GiB) in 3s, read: 180.0 MiB/s, write: 134.7 MiB/s
INFO: 1% (2.1 GiB of 200.0 GiB) in 11s, read: 196.5 MiB/s, write: 105.0 MiB/s
INFO: 2% (4.3 GiB of 200.0 GiB) in 24s, read: 172.6 MiB/s, write: 158.2 MiB/s
INFO: 3% (6.2 GiB of 200.0 GiB) in 33s, read: 222.2 MiB/s, write: 203.6 MiB/s
INFO: 4% (9.7 GiB of 200.0 GiB) in 36s, read: 1.2 GiB/s, write: 196.0 MiB/s
INFO: 5% (10.2 GiB of 200.0 GiB) in 39s, read: 178.7 MiB/s, write: 164.0 MiB/s
INFO: 6% (12.2 GiB of 200.0 GiB) in 49s, read: 203.6 MiB/s, write: 158.4 MiB/s
INFO: 7% (14.1 GiB of 200.0 GiB) in 1m, read: 178.5 MiB/s, write: 149.8 MiB/s
INFO: 8% (16.0 GiB of 200.0 GiB) in 1m 14s, read: 139.1 MiB/s, write: 104.9 MiB/s
INFO: 9% (18.1 GiB of 200.0 GiB) in 1m 28s, read: 150.3 MiB/s, write: 88.6 MiB/s
INFO: 10% (20.0 GiB of 200.0 GiB) in 1m 41s, read: 155.1 MiB/s, write: 108.9 MiB/s
INFO: 11% (22.1 GiB of 200.0 GiB) in 1m 53s, read: 175.3 MiB/s, write: 117.3 MiB/s
INFO: 12% (24.1 GiB of 200.0 GiB) in 2m 9s, read: 128.0 MiB/s, write: 89.8 MiB/s
INFO: 13% (26.1 GiB of 200.0 GiB) in 2m 27s, read: 113.3 MiB/s, write: 67.6 MiB/s
INFO: 14% (28.0 GiB of 200.0 GiB) in 2m 38s, read: 182.9 MiB/s, write: 133.5 MiB/s
INFO: 15% (30.1 GiB of 200.0 GiB) in 2m 54s, read: 131.0 MiB/s, write: 127.0 MiB/s
INFO: 16% (32.1 GiB of 200.0 GiB) in 3m 11s, read: 122.6 MiB/s, write: 115.5 MiB/s
INFO: 17% (34.1 GiB of 200.0 GiB) in 3m 35s, read: 85.2 MiB/s, write: 80.8 MiB/s
INFO: 18% (36.1 GiB of 200.0 GiB) in 3m 56s, read: 94.7 MiB/s, write: 67.0 MiB/s
INFO: 19% (38.0 GiB of 200.0 GiB) in 4m 9s, read: 152.3 MiB/s, write: 93.5 MiB/s
INFO: 20% (40.1 GiB of 200.0 GiB) in 4m 24s, read: 140.5 MiB/s, write: 77.3 MiB/s
INFO: 21% (42.0 GiB of 200.0 GiB) in 4m 36s, read: 169.0 MiB/s, write: 160.3 MiB/s
INFO: 22% (44.1 GiB of 200.0 GiB) in 4m 55s, read: 112.0 MiB/s, write: 92.0 MiB/s
INFO: 23% (46.1 GiB of 200.0 GiB) in 5m 11s, read: 124.5 MiB/s, write: 92.5 MiB/s
INFO: 24% (48.0 GiB of 200.0 GiB) in 5m 22s, read: 183.6 MiB/s, write: 133.8 MiB/s
INFO: 25% (50.1 GiB of 200.0 GiB) in 5m 43s, read: 102.9 MiB/s, write: 85.5 MiB/s
INFO: 26% (52.1 GiB of 200.0 GiB) in 6m 5s, read: 89.5 MiB/s, write: 54.0 MiB/s
INFO: 27% (54.2 GiB of 200.0 GiB) in 6m 20s, read: 144.5 MiB/s, write: 103.7 MiB/s
INFO: 28% (56.2 GiB of 200.0 GiB) in 6m 36s, read: 131.8 MiB/s, write: 78.5 MiB/s
INFO: 29% (58.1 GiB of 200.0 GiB) in 6m 50s, read: 136.9 MiB/s, write: 89.4 MiB/s
INFO: 30% (60.1 GiB of 200.0 GiB) in 7m 8s, read: 113.1 MiB/s, write: 76.2 MiB/s
INFO: 31% (62.0 GiB of 200.0 GiB) in 7m 26s, read: 109.1 MiB/s, write: 70.9 MiB/s
INFO: 32% (64.2 GiB of 200.0 GiB) in 7m 44s, read: 123.6 MiB/s, write: 90.7 MiB/s
INFO: 33% (66.1 GiB of 200.0 GiB) in 7m 57s, read: 149.5 MiB/s, write: 104.6 MiB/s
INFO: 34% (68.1 GiB of 200.0 GiB) in 8m 21s, read: 85.0 MiB/s, write: 58.8 MiB/s
INFO: 35% (70.1 GiB of 200.0 GiB) in 8m 40s, read: 110.1 MiB/s, write: 103.6 MiB/s
INFO: 36% (72.0 GiB of 200.0 GiB) in 8m 56s, read: 120.0 MiB/s, write: 114.0 MiB/s
INFO: 37% (74.1 GiB of 200.0 GiB) in 9m 11s, read: 144.0 MiB/s, write: 100.0 MiB/s
INFO: 38% (76.1 GiB of 200.0 GiB) in 9m 32s, read: 96.6 MiB/s, write: 89.0 MiB/s
INFO: 39% (78.1 GiB of 200.0 GiB) in 9m 52s, read: 104.0 MiB/s, write: 85.4 MiB/s
INFO: 40% (80.1 GiB of 200.0 GiB) in 10m 11s, read: 104.4 MiB/s, write: 96.4 MiB/s
INFO: 41% (82.1 GiB of 200.0 GiB) in 10m 27s, read: 132.0 MiB/s, write: 126.0 MiB/s
INFO: 41% (82.3 GiB of 200.0 GiB) in 10m 30s, read: 61.3 MiB/s, write: 61.3 MiB/s
ERROR: backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest.
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 200 failed - backup write data failed: command error: write_data upload error: pipelined request failed: detected chunk with wrong digest.
INFO: Failed at 2024-05-29 08:39:27
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors
 
Last edited:
Hi,
bitte nachschauen ob im systemd journal des PBS hosts Fehler im Zeitraum des fehlerhaften Backups gelogged werden. journalctl -r -b listet das journal seit Boot in umgekehrter chronologischer Reihenfolge. Eventuell auch einen Memory Test sowohl auf Proxmox VE Seite als auch auf PBS Seite durchführen. Kaputter Speicher kann auch dazu führen, dass ein digest falsch berechnet wird.
 
Danke für die schnelle Antwort.
Folgende Fehler stehen im journal. Memorytest habe ich vor ein paar Tagen durchgeführt und da scheint es keine Probleme zu geben. Allerdings habe ich den RAM nur im Live-Betrieb getestet, was ja immer nur so semi viel hilft. Ich kann den Virtualisierungsserver aber auch nicht einfach herunterfahren, deshalb kann ich da erst in ein paar Tagen genauere Auskunft geben.

May 29 08:39:26 pbs proxmox-backup-proxy[850]: TASK ERROR: backup ended but finished flag is not set.
May 29 08:39:26 pbs proxmox-backup-proxy[850]: removing backup snapshot "/mnt/datastore/backups-01/vm/200/2024-05-29T06:28:51Z"
May 29 08:39:26 pbs proxmox-backup-proxy[850]: removing unfinished backup
May 29 08:39:26 pbs proxmox-backup-proxy[850]: backup ended and finish failed: backup ended but finished flag is not set.
May 29 08:39:26 pbs proxmox-backup-proxy[850]: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: stream error received: stream no longer needed
May 29 08:39:26 pbs proxmox-backup-proxy[850]: POST /fixed_chunk: 400 Bad Request: error reading a body from connection: stream error received: stream no longer needed
May 29 08:39:26 pbs proxmox-backup-proxy[850]: POST /fixed_chunk: 400 Bad Request: detected chunk with wrong digest.
May 29 08:39:22 pbs proxmox-backup-proxy[850]: error during snapshot file listing: 'unable to load blob '"/mnt/datastore/backups-01/vm/200/2024-05-29T06:28:51Z/index.json.blob"' - No such file or directory (os error 2)'
May 29 08:39:21 pbs proxmox-backup-proxy[850]: error during snapshot file listing: 'unable to load blob '"/mnt/datastore/backups-01/vm/200/2024-05-29T06:28:51Z/index.json.blob"' - No such file or directory (os error 2)'
May 29 08:34:29 pbs proxmox-backup-proxy[850]: error during snapshot file listing: 'unable to load blob '"/mnt/datastore/backups-01/vm/200/2024-05-29T06:28:51Z/index.json.blob"' - No such file or directory (os error 2)'
May 29 08:30:25 pbs systemd[1]: Finished man-db.service - Daily man-db regeneration.
May 29 08:30:25 pbs systemd[1]: man-db.service: Deactivated successfully.
May 29 08:30:25 pbs systemd[1]: Starting man-db.service - Daily man-db regeneration...
May 29 08:29:42 pbs proxmox-backup-proxy[850]: error during snapshot file listing: 'unable to load blob '"/mnt/datastore/backups-01/vm/200/2024-05-29T06:28:51Z/index.json.blob"' - No such file or directory (os error 2)'
May 29 08:28:57 pbs proxmox-backup-proxy[850]: add blob "/mnt/datastore/backups-01/vm/200/2024-05-29T06:28:51Z/qemu-server.conf.blob" (382 bytes, comp: 382)
May 29 08:28:57 pbs proxmox-backup-proxy[850]: created new fixed index 1 ("vm/200/2024-05-29T06:28:51Z/drive-scsi0.img.fidx")
May 29 08:28:57 pbs proxmox-backup-proxy[850]: GET /previous: 400 Bad Request: no valid previous backup
 
Oh, ich habe gerade nochmal den memtester auf dem PBS gestartet und bekomme jetzt tatsächlich eine ganze Reihe an Fehlern. Das wird dann hoffentlich das Problem sein. Komisch, dass es sich jetzt anders verhält, als noch vor einer Woche, aber das ist wohl das Problem mit dem memtester.
FAILURE: possible bad address line at offset 0x00000001716d01a8.
...
FAILURE: 0xffffffbffffffffe != 0xfffffffffffffffe at offset 0x00000001716d5b70.
FAILURE: 0x0000001300000001 != 0x0000000000000001 at offset 0x00000001716d5c28.
FAILURE: 0xffffffdffffffffe != 0xfffffffffffffffe at offset 0x00000001716d5c30.
FAILURE: 0x000000d000000001 != 0x0000000000000001 at offset 0x00000001716d5c68.
FAILURE: 0xffffffeffffffffe != 0xfffffffffffffffe at offset 0x00000001716d5c70.
FAILURE: 0x000000eb00000001 != 0x0000000000000001 at offset 0x00000001716d5f28.
FAILURE: 0x000000f300000001 != 0x0000000000000001 at offset 0x00000001716d5f68.
FAILURE: 0xffffffdffffffffe != 0xfffffffffffffffe at offset 0x00000001716d5f70.
FAILURE: 0x0000008400000001 != 0x0000000000000001 at offset 0x00000001716d6028.
FAILURE: 0xffffffdffffffffe != 0xfffffffffffffffe at offset 0x00000001716d6030.
FAILURE: 0x000000c300000001 != 0x0000000000000001 at offset 0x00000001716d6068.
FAILURE: 0x0000007200000001 != 0x0000000000000001 at offset 0x00000001716d6328.
FAILURE: 0xffffffeffffffffe != 0xfffffffffffffffe at offset 0x00000001716d6330.
FAILURE: 0x000000a000000001 != 0x0000000000000001 at offset 0x00000001716d6368.
FAILURE: 0xffffffbffffffffe != 0xfffffffffffffffe at offset 0x00000001716d6370.
FAILURE: 0x000000a100000001 != 0x0000000000000001 at offset 0x00000001716d6428.
FAILURE: 0xffffffbffffffffe != 0xfffffffffffffffe at offset 0x00000001716d6430.
FAILURE: 0x0000006000000001 != 0x0000000000000001 at offset 0x00000001716d6468.
FAILURE: 0xffffffeffffffffe != 0xfffffffffffffffe at offset 0x00000001716d6470.
FAILURE: 0x000000ef00000001 != 0x0000000000000001 at offset 0x00000001716d6728.
FAILURE: 0x000000d600000001 != 0x0000000000000001 at offset 0x00000001716d6768.
FAILURE: 0xffffffaffffffffe != 0xfffffffffffffffe at offset 0x00000001716d6830.
FAILURE: 0x0000002d00000001 != 0x0000000000000001 at offset 0x00000001716d6868.
FAILURE: 0xffffff7ffffffffe != 0xfffffffffffffffe at offset 0x00000001716d6870.
FAILURE: 0x0000000600000001 != 0x0000000000000001 at offset 0x00000001716d6b28.
FAILURE: 0xffffffbffffffffe != 0xfffffffffffffffe at offset 0x00000001716d6b30.
FAILURE: 0x0000002800000001 != 0x0000000000000001 at offset 0x00000001716d6b68.
FAILURE: 0xffffffaffffffffe != 0xfffffffffffffffe at offset 0x00000001716d6b70.
FAILURE: 0x0000003200000001 != 0x0000000000000001 at offset 0x00000001716d6c28.
FAILURE: 0xffffff3ffffffffe != 0xfffffffffffffffe at offset 0x00000001716d6c30.
FAILURE: 0x0000009100000001 != 0x0000000000000001 at offset 0x00000001716d6c68.
FAILURE: 0xffffff8ffffffffe != 0xfffffffffffffffe at offset 0x00000001716d6c70.
FAILURE: 0x000000e000000001 != 0x0000000000000001 at offset 0x00000001716d6f28.
FAILURE: 0xffffffdffffffffe != 0xfffffffffffffffe at offset 0x00000001716d6f30.
FAILURE: 0x0000007200000001 != 0x0000000000000001 at offset 0x00000001716d6f68.
FAILURE: 0xffffff5ffffffffe != 0xfffffffffffffffe at offset 0x00000001716d6f70.
FAILURE: 0x0000000100000001 != 0x0000000000000001 at offset 0x00000001716d7128.
FAILURE: 0xffffffeffffffffe != 0xfffffffffffffffe at offset 0x00000001716d7130.
FAILURE: 0x0000001800000001 != 0x0000000000000001 at offset 0x00000001716d7168.
FAILURE: 0x0000000c00000001 != 0x0000000000000001 at offset 0x00000001716d7228.
FAILURE: 0xffffff8ffffffffe != 0xfffffffffffffffe at offset 0x00000001716d7230.
FAILURE: 0x0000007d00000001 != 0x0000000000000001 at offset 0x00000001716d7268.
FAILURE: 0xffffffeffffffffe != 0xfffffffffffffffe at offset 0x00000001716d7270.
FAILURE: 0x0000008a00000001 != 0x0000000000000001 at offset 0x00000001716d7528.
FAILURE: 0xffffffbffffffffe != 0xfffffffffffffffe at offset 0x00000001716d7530.
FAILURE: 0x0000004000000001 != 0x0000000000000001 at offset 0x00000001716d7568.
FAILURE: 0xffffffdffffffffe != 0xfffffffffffffffe at offset 0x00000001716d7570.
FAILURE: 0x000000bf00000001 != 0x0000000000000001 at offset 0x00000001716d7628.
FAILURE: 0xffffffeffffffffe != 0xfffffffffffffffe at offset 0x00000001716d7630.
FAILURE: 0x000000c000000001 != 0x0000000000000001 at offset 0x00000001716d7668.
FAILURE: 0xffffffcffffffffe != 0xfffffffffffffffe at offset 0x00000001716d7670.
FAILURE: 0x0000008900000001 != 0x0000000000000001 at offset 0x00000001716d7928.
FAILURE: 0xffffff9ffffffffe != 0xfffffffffffffffe at offset 0x00000001716d7930.
FAILURE: 0x000000ad00000001 != 0x0000000000000001 at offset 0x00000001716d7968.
FAILURE: 0xffffffbffffffffe != 0xfffffffffffffffe at offset 0x00000001716d7970.
FAILURE: 0x0000008200000001 != 0x0000000000000001 at offset 0x00000001716d7a28.
FAILURE: 0xffffffaffffffffe != 0xfffffffffffffffe at offset 0x00000001716d7a30.
FAILURE: 0x000000b200000001 != 0x0000000000000001 at offset 0x00000001716d7a68.
FAILURE: 0x0000003800000001 != 0x0000000000000001 at offset 0x00000001716d7d28.
FAILURE: 0x0000005d00000001 != 0x0000000000000001 at offset 0x00000001716d7d68.
FAILURE: 0xffffff5ffffffffe != 0xfffffffffffffffe at offset 0x00000001716d7d70.
FAILURE: 0x0000004a00000001 != 0x0000000000000001 at offset 0x00000001716d7e28.
FAILURE: 0xffffff6ffffffffe != 0xfffffffffffffffe at offset 0x00000001716d7e30.
FAILURE: 0x0000000400000001 != 0x0000000000000001 at offset 0x00000001716d7e68.
FAILURE: 0xffffff0ffffffffe != 0xfffffffffffffffe at offset 0x00000001716d7e70.
...
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back