I have 2 nodes PVE cluster, B is using a disk in A via NFS to backup VMs (X and Y). Each time during a backup job, X always fail and Y always success. The backup logs are as followed:
It's quite disturbing me with these conflicting logs, does anyone have the solution?
Bash:
INFO: starting new backup job: vzdump 102 105 102 105 --quiet 1 --storage ext-hdd --notes-template '{{guestname}}' --prune-backups 'keep-daily=7,keep-monthly=1,keep-weekly=1' --compress zstd --mailnotification failure --mode snapshot --node B
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2023-10-14 04:00:07
INFO: status = running
INFO: VM Name: X
INFO: include disk 'sata0' 'local:102/vm-102-disk-1.qcow2' 32G
INFO: include disk 'efidisk0' 'local:102/vm-102-disk-0.qcow2' 128K
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/ext-hdd/dump/vzdump-qemu-102-2023_10_14-04_00_06.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'f4ca0813-c38d-418c-a42f-3dc21cc11b39'
INFO: resuming VM again
INFO: 1% (442.2 MiB of 32.0 GiB) in 3s, read: 147.4 MiB/s, write: 124.0 MiB/s
INFO: 2% (913.5 MiB of 32.0 GiB) in 6s, read: 157.1 MiB/s, write: 116.9 MiB/s
INFO: 3% (1.3 GiB of 32.0 GiB) in 9s, read: 128.1 MiB/s, write: 127.8 MiB/s
INFO: 5% (1.7 GiB of 32.0 GiB) in 12s, read: 160.0 MiB/s, write: 119.9 MiB/s
INFO: 7% (2.3 GiB of 32.0 GiB) in 15s, read: 186.8 MiB/s, write: 130.8 MiB/s
INFO: 8% (2.8 GiB of 32.0 GiB) in 18s, read: 164.3 MiB/s, write: 133.1 MiB/s
INFO: 9% (3.2 GiB of 32.0 GiB) in 21s, read: 139.0 MiB/s, write: 128.6 MiB/s
INFO: 11% (3.6 GiB of 32.0 GiB) in 24s, read: 148.6 MiB/s, write: 148.6 MiB/s
INFO: 12% (3.9 GiB of 32.0 GiB) in 26s, read: 138.1 MiB/s, write: 137.8 MiB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 102 failed - job failed with err -5 - Input/output error
INFO: Failed at 2023-10-14 04:02:00
INFO: Starting Backup of VM 102 (qemu)
INFO: Backup started at 2023-10-14 04:02:00
INFO: status = running
INFO: VM Name: X
INFO: include disk 'sata0' 'local:102/vm-102-disk-1.qcow2' 32G
INFO: include disk 'efidisk0' 'local:102/vm-102-disk-0.qcow2' 128K
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/ext-hdd/dump/vzdump-qemu-102-2023_10_14-04_02_00.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '24039f07-d339-4c44-8a84-2fc4faf41133'
INFO: resuming VM again
INFO: 1% (431.4 MiB of 32.0 GiB) in 3s, read: 143.8 MiB/s, write: 120.3 MiB/s
INFO: 2% (938.9 MiB of 32.0 GiB) in 6s, read: 169.2 MiB/s, write: 128.9 MiB/s
INFO: 4% (1.3 GiB of 32.0 GiB) in 9s, read: 146.2 MiB/s, write: 143.2 MiB/s
INFO: 5% (1.9 GiB of 32.0 GiB) in 12s, read: 187.3 MiB/s, write: 144.0 MiB/s
INFO: 7% (2.5 GiB of 32.0 GiB) in 15s, read: 194.2 MiB/s, write: 129.4 MiB/s
INFO: 9% (2.9 GiB of 32.0 GiB) in 18s, read: 162.0 MiB/s, write: 135.4 MiB/s
INFO: 10% (3.3 GiB of 32.0 GiB) in 21s, read: 135.6 MiB/s, write: 135.3 MiB/s
INFO: 11% (3.8 GiB of 32.0 GiB) in 24s, read: 152.2 MiB/s, write: 152.2 MiB/s
INFO: 12% (3.9 GiB of 32.0 GiB) in 25s, read: 98.1 MiB/s, write: 97.5 MiB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 102 failed - job failed with err -5 - Input/output error
INFO: Failed at 2023-10-14 04:03:59
INFO: Starting Backup of VM 105 (qemu)
INFO: Backup started at 2023-10-14 04:03:59
INFO: status = running
INFO: VM Name: Y
INFO: include disk 'scsi0' 'local:105/vm-105-disk-1.raw' 32G
INFO: include disk 'efidisk0' 'local:105/vm-105-disk-0.raw' 528K
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/ext-hdd/dump/vzdump-qemu-105-2023_10_14-04_03_59.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '9bc1ad51-66a5-433b-9a00-27ce4399d6b9'
INFO: resuming VM again
INFO: 1% (464.0 MiB of 32.0 GiB) in 3s, read: 154.7 MiB/s, write: 117.9 MiB/s
INFO: 3% (996.9 MiB of 32.0 GiB) in 6s, read: 177.6 MiB/s, write: 117.2 MiB/s
INFO: 4% (1.4 GiB of 32.0 GiB) in 9s, read: 160.7 MiB/s, write: 158.1 MiB/s
INFO: 6% (1.9 GiB of 32.0 GiB) in 12s, read: 169.0 MiB/s, write: 161.9 MiB/s
INFO: 9% (3.0 GiB of 32.0 GiB) in 15s, read: 368.8 MiB/s, write: 143.0 MiB/s
INFO: 10% (3.4 GiB of 32.0 GiB) in 18s, read: 126.1 MiB/s, write: 125.9 MiB/s
INFO: 11% (3.8 GiB of 32.0 GiB) in 21s, read: 142.0 MiB/s, write: 136.6 MiB/s
INFO: 13% (4.3 GiB of 32.0 GiB) in 24s, read: 182.5 MiB/s, write: 181.5 MiB/s
INFO: 15% (5.0 GiB of 32.0 GiB) in 27s, read: 237.9 MiB/s, write: 214.3 MiB/s
INFO: 17% (5.6 GiB of 32.0 GiB) in 30s, read: 197.1 MiB/s, write: 190.9 MiB/s
INFO: 21% (6.9 GiB of 32.0 GiB) in 33s, read: 425.3 MiB/s, write: 63.7 MiB/s
INFO: 22% (7.2 GiB of 32.0 GiB) in 36s, read: 128.1 MiB/s, write: 127.6 MiB/s
INFO: 27% (8.7 GiB of 32.0 GiB) in 39s, read: 493.0 MiB/s, write: 108.4 MiB/s
INFO: 28% (9.1 GiB of 32.0 GiB) in 42s, read: 151.0 MiB/s, write: 136.8 MiB/s
INFO: 29% (9.4 GiB of 32.0 GiB) in 45s, read: 78.5 MiB/s, write: 78.4 MiB/s
INFO: 33% (10.9 GiB of 32.0 GiB) in 48s, read: 518.4 MiB/s, write: 85.2 MiB/s
INFO: 34% (11.0 GiB of 32.0 GiB) in 51s, read: 54.4 MiB/s, write: 47.3 MiB/s
INFO: 35% (11.2 GiB of 32.0 GiB) in 55s, read: 45.3 MiB/s, write: 42.3 MiB/s
INFO: 37% (12.1 GiB of 32.0 GiB) in 1m, read: 175.4 MiB/s, write: 45.5 MiB/s
INFO: 39% (12.6 GiB of 32.0 GiB) in 1m 3s, read: 185.3 MiB/s, write: 17.7 MiB/s
INFO: 40% (12.9 GiB of 32.0 GiB) in 1m 7s, read: 68.5 MiB/s, write: 45.9 MiB/s
INFO: 41% (13.1 GiB of 32.0 GiB) in 1m 11s, read: 68.7 MiB/s, write: 68.7 MiB/s
INFO: 42% (13.5 GiB of 32.0 GiB) in 1m 16s, read: 66.8 MiB/s, write: 64.0 MiB/s
INFO: 43% (13.8 GiB of 32.0 GiB) in 1m 25s, read: 37.0 MiB/s, write: 33.6 MiB/s
INFO: 44% (14.1 GiB of 32.0 GiB) in 1m 41s, read: 18.4 MiB/s, write: 18.2 MiB/s
INFO: 45% (14.4 GiB of 32.0 GiB) in 2m, read: 17.7 MiB/s, write: 17.6 MiB/s
INFO: 46% (14.8 GiB of 32.0 GiB) in 2m 15s, read: 28.0 MiB/s, write: 22.4 MiB/s
INFO: 47% (15.1 GiB of 32.0 GiB) in 2m 21s, read: 42.3 MiB/s, write: 42.2 MiB/s
INFO: 48% (15.4 GiB of 32.0 GiB) in 2m 26s, read: 66.0 MiB/s, write: 65.7 MiB/s
INFO: 51% (16.6 GiB of 32.0 GiB) in 2m 29s, read: 424.1 MiB/s, write: 82.7 MiB/s
INFO: 53% (17.1 GiB of 32.0 GiB) in 2m 32s, read: 161.9 MiB/s, write: 116.6 MiB/s
INFO: 64% (20.5 GiB of 32.0 GiB) in 2m 35s, read: 1.1 GiB/s, write: 214.7 KiB/s
INFO: 65% (20.9 GiB of 32.0 GiB) in 2m 38s, read: 120.8 MiB/s, write: 31.8 MiB/s
INFO: 66% (21.1 GiB of 32.0 GiB) in 2m 51s, read: 17.6 MiB/s, write: 17.6 MiB/s
INFO: 67% (21.4 GiB of 32.0 GiB) in 3m 7s, read: 20.4 MiB/s, write: 19.4 MiB/s
INFO: 71% (23.0 GiB of 32.0 GiB) in 3m 22s, read: 107.3 MiB/s, write: 12.5 MiB/s
INFO: 83% (26.8 GiB of 32.0 GiB) in 3m 25s, read: 1.3 GiB/s, write: 2.7 KiB/s
INFO: 95% (30.5 GiB of 32.0 GiB) in 3m 28s, read: 1.2 GiB/s, write: 1.3 KiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 3m 31s, read: 502.2 MiB/s, write: 63.8 MiB/s
INFO: backup is sparse: 21.00 GiB (65%) total zero data
INFO: transferred 32.00 GiB in 211 seconds (155.3 MiB/s)
INFO: archive file size: 3.50GB
INFO: adding notes to backup
INFO: prune older backups with retention: keep-daily=7, keep-monthly=1, keep-weekly=1
INFO: removing backup 'ext-hdd:backup/vzdump-qemu-105-2023_10_07-04_09_46.vma.zst'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 105 (00:05:41)
INFO: Backup finished at 2023-10-14 04:09:40
INFO: Starting Backup of VM 105 (qemu)
INFO: Backup started at 2023-10-14 04:09:40
INFO: status = running
INFO: VM Name: Y
INFO: include disk 'scsi0' 'local:105/vm-105-disk-1.raw' 32G
INFO: include disk 'efidisk0' 'local:105/vm-105-disk-0.raw' 528K
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/ext-hdd/dump/vzdump-qemu-105-2023_10_14-04_09_40.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'f81df8fc-d8fb-457a-b81b-85cefdee32e3'
INFO: resuming VM again
INFO: 1% (482.1 MiB of 32.0 GiB) in 3s, read: 160.7 MiB/s, write: 123.9 MiB/s
INFO: 3% (1007.8 MiB of 32.0 GiB) in 6s, read: 175.2 MiB/s, write: 114.8 MiB/s
INFO: 4% (1.5 GiB of 32.0 GiB) in 9s, read: 161.9 MiB/s, write: 159.3 MiB/s
INFO: 6% (2.0 GiB of 32.0 GiB) in 13s, read: 133.5 MiB/s, write: 126.7 MiB/s
INFO: 9% (3.0 GiB of 32.0 GiB) in 16s, read: 362.1 MiB/s, write: 138.1 MiB/s
INFO: 10% (3.4 GiB of 32.0 GiB) in 19s, read: 122.8 MiB/s, write: 122.6 MiB/s
INFO: 12% (3.8 GiB of 32.0 GiB) in 22s, read: 151.6 MiB/s, write: 146.2 MiB/s
INFO: 13% (4.4 GiB of 32.0 GiB) in 25s, read: 197.0 MiB/s, write: 196.0 MiB/s
INFO: 15% (5.1 GiB of 32.0 GiB) in 28s, read: 226.0 MiB/s, write: 202.3 MiB/s
INFO: 17% (5.7 GiB of 32.0 GiB) in 31s, read: 194.5 MiB/s, write: 188.3 MiB/s
INFO: 21% (7.0 GiB of 32.0 GiB) in 34s, read: 451.9 MiB/s, write: 90.3 MiB/s
INFO: 23% (7.4 GiB of 32.0 GiB) in 37s, read: 154.7 MiB/s, write: 152.5 MiB/s
INFO: 27% (8.8 GiB of 32.0 GiB) in 40s, read: 481.7 MiB/s, write: 84.9 MiB/s
INFO: 28% (9.2 GiB of 32.0 GiB) in 43s, read: 118.6 MiB/s, write: 118.5 MiB/s
INFO: 31% (10.0 GiB of 32.0 GiB) in 46s, read: 269.6 MiB/s, write: 82.1 MiB/s
INFO: 34% (11.0 GiB of 32.0 GiB) in 49s, read: 337.1 MiB/s, write: 87.7 MiB/s
INFO: 35% (11.2 GiB of 32.0 GiB) in 54s, read: 52.2 MiB/s, write: 47.8 MiB/s
INFO: 39% (12.6 GiB of 32.0 GiB) in 1m 5s, read: 124.2 MiB/s, write: 19.4 MiB/s
INFO: 40% (12.9 GiB of 32.0 GiB) in 1m 8s, read: 101.8 MiB/s, write: 71.6 MiB/s
INFO: 41% (13.2 GiB of 32.0 GiB) in 1m 12s, read: 87.9 MiB/s, write: 87.3 MiB/s
INFO: 42% (13.5 GiB of 32.0 GiB) in 1m 16s, read: 71.6 MiB/s, write: 68.6 MiB/s
INFO: 43% (13.8 GiB of 32.0 GiB) in 1m 30s, read: 21.0 MiB/s, write: 18.8 MiB/s
INFO: 44% (14.1 GiB of 32.0 GiB) in 1m 41s, read: 29.7 MiB/s, write: 29.3 MiB/s
INFO: 45% (14.4 GiB of 32.0 GiB) in 2m 10s, read: 12.1 MiB/s, write: 12.0 MiB/s
INFO: 46% (14.8 GiB of 32.0 GiB) in 2m 15s, read: 86.3 MiB/s, write: 69.4 MiB/s
INFO: 47% (15.1 GiB of 32.0 GiB) in 2m 21s, read: 48.9 MiB/s, write: 48.8 MiB/s
INFO: 48% (15.4 GiB of 32.0 GiB) in 2m 25s, read: 65.2 MiB/s, write: 65.0 MiB/s
INFO: 51% (16.6 GiB of 32.0 GiB) in 2m 28s, read: 403.6 MiB/s, write: 62.2 MiB/s
INFO: 53% (17.0 GiB of 32.0 GiB) in 2m 31s, read: 136.5 MiB/s, write: 99.0 MiB/s
INFO: 57% (18.3 GiB of 32.0 GiB) in 2m 38s, read: 199.9 MiB/s, write: 17.0 MiB/s
INFO: 65% (20.9 GiB of 32.0 GiB) in 2m 41s, read: 871.2 MiB/s, write: 26.8 MiB/s
INFO: 66% (21.1 GiB of 32.0 GiB) in 2m 56s, read: 16.7 MiB/s, write: 16.7 MiB/s
INFO: 67% (21.5 GiB of 32.0 GiB) in 3m 12s, read: 20.6 MiB/s, write: 19.6 MiB/s
INFO: 72% (23.3 GiB of 32.0 GiB) in 3m 21s, read: 215.1 MiB/s, write: 19.7 MiB/s
INFO: 84% (27.0 GiB of 32.0 GiB) in 3m 24s, read: 1.2 GiB/s, write: 2.7 KiB/s
INFO: 99% (31.8 GiB of 32.0 GiB) in 3m 27s, read: 1.6 GiB/s, write: 353.3 KiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 3m 29s, read: 99.3 MiB/s, write: 95.2 MiB/s
INFO: backup is sparse: 21.00 GiB (65%) total zero data
INFO: transferred 32.00 GiB in 209 seconds (156.8 MiB/s)
INFO: archive file size: 3.50GB
INFO: adding notes to backup
INFO: prune older backups with retention: keep-daily=7, keep-monthly=1, keep-weekly=1
INFO: removing backup 'ext-hdd:backup/vzdump-qemu-105-2023_10_14-04_03_59.vma.zst'
INFO: pruned 1 backup(s) not covered by keep-retention policy
INFO: Finished Backup of VM 105 (00:05:43)
INFO: Backup finished at 2023-10-14 04:15:23
INFO: Backup job finished with errors
TASK ERROR: job errorsdmesg -w shows:
Bash:
[ 83.124103] BTRFS warning (device sda3): csum failed root 256 ino 111237 off 4157607936 csum 0x15aac573 expected csum 0x9e44ca3b mirror 1
[ 83.124115] BTRFS error (device sda3): bdev /dev/sda3 errs: wr 0, rd 0, flush 0, corrupt 47, gen 0badblocks -v /dev/sda3 and btrfs check --force /dev/sda3 shows no error found.It's quite disturbing me with these conflicting logs, does anyone have the solution?
