[SOLVED] Proxmox 6.4 ZFS some ZSTD backups sometimes failing with exit status 1

[Chiaki]

Hello everyone,

I have several VMs on my Proxmox 6.4 VE and sometimes some of the backups are failing.

The target is a CIFS backup storage which was connected via the Proxmox GUI.

Example of a fail:

820: 2021-06-20 04:45:14 INFO: Starting Backup of VM 820 (qemu)

820: 2021-06-20 04:45:14 INFO: status = running

820: 2021-06-20 04:45:14 INFO: VM Name: MyVM

820: 2021-06-20 04:45:14 INFO: include disk 'scsi0' 'local-zfs:vm-820-disk-0' 100G

820: 2021-06-20 04:45:14 INFO: backup mode: snapshot

820: 2021-06-20 04:45:14 INFO: ionice priority: 7

820: 2021-06-20 04:45:28 INFO: creating vzdump archive '/mnt/pve/CIFS_Storage/dump/vzdump-qemu-820-2021_06_20-04_45_14.vma.zst'

820: 2021-06-20 04:45:29 INFO: started backup task '1cb5b660-4238-401a-a383-ffbb8002f7f2'

820: 2021-06-20 04:45:29 INFO: resuming VM again

820: 2021-06-20 04:45:32 INFO:   0% (864.6 MiB of 100.0 GiB) in 3s, read: 288.2 MiB/s, write: 239.5 MiB/s

820: 2021-06-20 04:45:35 INFO:   1% (1.6 GiB of 100.0 GiB) in 6s, read: 268.8 MiB/s, write: 252.0 MiB/s

820: 2021-06-20 04:45:38 INFO:   2% (2.5 GiB of 100.0 GiB) in 9s, read: 280.3 MiB/s, write: 214.8 MiB/s

820: 2021-06-20 04:45:41 INFO:   3% (3.1 GiB of 100.0 GiB) in 12s, read: 235.3 MiB/s, write: 233.3 MiB/s

820: 2021-06-20 04:45:45 INFO:   4% (4.1 GiB of 100.0 GiB) in 16s, read: 240.4 MiB/s, write: 229.6 MiB/s

820: 2021-06-20 04:45:49 INFO:   5% (5.2 GiB of 100.0 GiB) in 20s, read: 278.2 MiB/s, write: 243.9 MiB/s

820: 2021-06-20 04:45:53 INFO:   6% (6.2 GiB of 100.0 GiB) in 24s, read: 262.8 MiB/s, write: 235.3 MiB/s

820: 2021-06-20 04:45:57 INFO:   7% (7.0 GiB of 100.0 GiB) in 28s, read: 215.7 MiB/s, write: 211.8 MiB/s

820: 2021-06-20 04:46:01 INFO:   8% (8.1 GiB of 100.0 GiB) in 32s, read: 271.0 MiB/s, write: 252.6 MiB/s

820: 2021-06-20 04:46:05 INFO:   9% (9.1 GiB of 100.0 GiB) in 36s, read: 255.6 MiB/s, write: 251.8 MiB/s

820: 2021-06-20 04:46:09 INFO:  10% (10.2 GiB of 100.0 GiB) in 40s, read: 285.6 MiB/s, write: 258.0 MiB/s

820: 2021-06-20 04:46:13 INFO:  11% (11.2 GiB of 100.0 GiB) in 44s, read: 255.4 MiB/s, write: 242.8 MiB/s

820: 2021-06-20 04:46:17 INFO:  12% (12.3 GiB of 100.0 GiB) in 48s, read: 270.1 MiB/s, write: 237.3 MiB/s

820: 2021-06-20 04:46:20 INFO:  13% (13.0 GiB of 100.0 GiB) in 51s, read: 253.8 MiB/s, write: 253.1 MiB/s

820: 2021-06-20 04:46:24 INFO:  14% (14.1 GiB of 100.0 GiB) in 55s, read: 279.1 MiB/s, write: 254.6 MiB/s

820: 2021-06-20 04:46:28 INFO:  15% (15.3 GiB of 100.0 GiB) in 59s, read: 310.3 MiB/s, write: 267.5 MiB/s

820: 2021-06-20 04:46:31 INFO:  16% (16.3 GiB of 100.0 GiB) in 1m 2s, read: 322.5 MiB/s, write: 259.8 MiB/s

820: 2021-06-20 04:46:34 INFO:  17% (17.1 GiB of 100.0 GiB) in 1m 5s, read: 301.7 MiB/s, write: 276.6 MiB/s

820: 2021-06-20 04:46:38 INFO:  18% (18.3 GiB of 100.0 GiB) in 1m 9s, read: 302.1 MiB/s, write: 261.2 MiB/s

820: 2021-06-20 04:46:41 INFO:  19% (19.1 GiB of 100.0 GiB) in 1m 12s, read: 279.9 MiB/s, write: 252.9 MiB/s

820: 2021-06-20 04:46:45 INFO:  20% (20.3 GiB of 100.0 GiB) in 1m 16s, read: 294.5 MiB/s, write: 240.8 MiB/s

820: 2021-06-20 04:46:48 INFO:  21% (21.1 GiB of 100.0 GiB) in 1m 19s, read: 282.8 MiB/s, write: 254.6 MiB/s

820: 2021-06-20 04:46:52 INFO:  22% (22.3 GiB of 100.0 GiB) in 1m 23s, read: 311.8 MiB/s, write: 256.0 MiB/s

820: 2021-06-20 04:46:55 INFO:  23% (23.2 GiB of 100.0 GiB) in 1m 26s, read: 284.5 MiB/s, write: 262.4 MiB/s

820: 2021-06-20 04:46:59 INFO:  24% (24.3 GiB of 100.0 GiB) in 1m 30s, read: 277.8 MiB/s, write: 252.9 MiB/s

820: 2021-06-20 04:47:03 INFO:  25% (25.2 GiB of 100.0 GiB) in 1m 34s, read: 246.5 MiB/s, write: 244.4 MiB/s

820: 2021-06-20 04:47:07 INFO:  26% (26.3 GiB of 100.0 GiB) in 1m 38s, read: 268.2 MiB/s, write: 240.2 MiB/s

820: 2021-06-20 04:47:10 INFO:  27% (27.0 GiB of 100.0 GiB) in 1m 41s, read: 259.4 MiB/s, write: 258.2 MiB/s

820: 2021-06-20 04:47:14 INFO:  28% (28.1 GiB of 100.0 GiB) in 1m 45s, read: 265.9 MiB/s, write: 257.4 MiB/s

820: 2021-06-20 04:47:18 INFO:  29% (29.1 GiB of 100.0 GiB) in 1m 49s, read: 254.7 MiB/s, write: 250.4 MiB/s

820: 2021-06-20 04:47:22 INFO:  30% (30.0 GiB of 100.0 GiB) in 1m 53s, read: 244.3 MiB/s, write: 244.2 MiB/s

820: 2021-06-20 04:47:28 INFO:  31% (31.0 GiB of 100.0 GiB) in 1m 59s, read: 173.0 MiB/s, write: 166.6 MiB/s

820: 2021-06-20 04:47:39 INFO:  32% (32.2 GiB of 100.0 GiB) in 2m 10s, read: 108.1 MiB/s, write: 97.6 MiB/s

820: 2021-06-20 04:47:44 INFO:  33% (33.1 GiB of 100.0 GiB) in 2m 15s, read: 188.5 MiB/s, write: 187.4 MiB/s

820: 2021-06-20 04:47:56 INFO:  34% (34.0 GiB of 100.0 GiB) in 2m 27s, read: 79.6 MiB/s, write: 77.6 MiB/s

820: 2021-06-20 04:48:06 INFO:  35% (35.2 GiB of 100.0 GiB) in 2m 37s, read: 117.9 MiB/s, write: 116.1 MiB/s

820: 2021-06-20 04:48:10 INFO:  36% (36.2 GiB of 100.0 GiB) in 2m 41s, read: 260.4 MiB/s, write: 251.3 MiB/s

820: 2021-06-20 04:48:15 INFO:  37% (37.1 GiB of 100.0 GiB) in 2m 46s, read: 188.2 MiB/s, write: 187.9 MiB/s

820: 2021-06-20 04:48:23 INFO:  38% (38.2 GiB of 100.0 GiB) in 2m 54s, read: 138.7 MiB/s, write: 129.5 MiB/s

820: 2021-06-20 04:48:29 INFO:  39% (39.1 GiB of 100.0 GiB) in 3m, read: 145.6 MiB/s, write: 145.1 MiB/s

820: 2021-06-20 04:48:37 INFO:  40% (40.1 GiB of 100.0 GiB) in 3m 8s, read: 127.2 MiB/s, write: 123.3 MiB/s

820: 2021-06-20 04:48:43 INFO:  41% (41.1 GiB of 100.0 GiB) in 3m 14s, read: 183.7 MiB/s, write: 182.1 MiB/s

820: 2021-06-20 04:48:50 INFO:  42% (42.1 GiB of 100.0 GiB) in 3m 21s, read: 142.5 MiB/s, write: 137.6 MiB/s

820: 2021-06-20 04:48:54 INFO:  43% (43.0 GiB of 100.0 GiB) in 3m 25s, read: 232.3 MiB/s, write: 226.1 MiB/s

820: 2021-06-20 04:49:02 INFO:  44% (44.2 GiB of 100.0 GiB) in 3m 33s, read: 151.7 MiB/s, write: 129.7 MiB/s

820: 2021-06-20 04:49:08 INFO:  45% (45.1 GiB of 100.0 GiB) in 3m 39s, read: 150.4 MiB/s, write: 140.5 MiB/s

820: 2021-06-20 04:49:15 INFO:  46% (46.0 GiB of 100.0 GiB) in 3m 46s, read: 134.6 MiB/s, write: 122.4 MiB/s

820: 2021-06-20 04:49:21 INFO:  47% (47.0 GiB of 100.0 GiB) in 3m 52s, read: 174.0 MiB/s, write: 140.8 MiB/s

820: 2021-06-20 04:49:28 INFO:  48% (48.0 GiB of 100.0 GiB) in 3m 59s, read: 144.5 MiB/s, write: 131.4 MiB/s

820: 2021-06-20 04:49:32 INFO:  49% (49.2 GiB of 100.0 GiB) in 4m 3s, read: 302.7 MiB/s, write: 270.7 MiB/s

820: 2021-06-20 04:49:38 INFO:  50% (50.2 GiB of 100.0 GiB) in 4m 9s, read: 165.1 MiB/s, write: 144.8 MiB/s

820: 2021-06-20 04:49:43 INFO:  51% (51.1 GiB of 100.0 GiB) in 4m 14s, read: 183.2 MiB/s, write: 176.4 MiB/s

820: 2021-06-20 04:49:50 INFO:  52% (52.0 GiB of 100.0 GiB) in 4m 21s, read: 140.9 MiB/s, write: 132.4 MiB/s

820: 2021-06-20 04:49:56 INFO:  53% (53.1 GiB of 100.0 GiB) in 4m 27s, read: 185.7 MiB/s, write: 160.3 MiB/s

820: 2021-06-20 04:50:03 INFO:  54% (54.1 GiB of 100.0 GiB) in 4m 34s, read: 146.6 MiB/s, write: 133.4 MiB/s

820: 2021-06-20 04:50:09 INFO:  55% (55.0 GiB of 100.0 GiB) in 4m 40s, read: 160.4 MiB/s, write: 151.2 MiB/s

820: 2021-06-20 04:50:17 INFO:  56% (56.2 GiB of 100.0 GiB) in 4m 48s, read: 148.7 MiB/s, write: 132.1 MiB/s

820: 2021-06-20 04:50:23 INFO:  57% (57.0 GiB of 100.0 GiB) in 4m 54s, read: 140.8 MiB/s, write: 134.0 MiB/s

820: 2021-06-20 04:50:30 INFO:  58% (58.0 GiB of 100.0 GiB) in 5m 1s, read: 142.9 MiB/s, write: 132.7 MiB/s

820: 2021-06-20 04:50:37 INFO:  59% (59.1 GiB of 100.0 GiB) in 5m 8s, read: 152.7 MiB/s, write: 138.5 MiB/s

820: 2021-06-20 04:50:45 INFO:  60% (60.2 GiB of 100.0 GiB) in 5m 16s, read: 145.9 MiB/s, write: 131.8 MiB/s

820: 2021-06-20 04:50:51 INFO:  61% (61.1 GiB of 100.0 GiB) in 5m 22s, read: 158.3 MiB/s, write: 151.6 MiB/s

820: 2021-06-20 04:50:58 INFO:  62% (62.2 GiB of 100.0 GiB) in 5m 29s, read: 156.0 MiB/s, write: 141.9 MiB/s

820: 2021-06-20 04:51:05 INFO:  63% (63.1 GiB of 100.0 GiB) in 5m 36s, read: 133.9 MiB/s, write: 131.1 MiB/s

820: 2021-06-20 04:51:12 INFO:  64% (64.1 GiB of 100.0 GiB) in 5m 43s, read: 149.1 MiB/s, write: 133.3 MiB/s

820: 2021-06-20 04:51:19 INFO:  65% (65.1 GiB of 100.0 GiB) in 5m 50s, read: 145.0 MiB/s, write: 137.6 MiB/s

820: 2021-06-20 04:51:25 INFO:  66% (66.1 GiB of 100.0 GiB) in 5m 56s, read: 166.8 MiB/s, write: 151.0 MiB/s

820: 2021-06-20 04:51:32 INFO:  67% (67.1 GiB of 100.0 GiB) in 6m 3s, read: 152.2 MiB/s, write: 144.7 MiB/s

820: 2021-06-20 04:51:39 INFO:  68% (68.1 GiB of 100.0 GiB) in 6m 10s, read: 148.1 MiB/s, write: 133.7 MiB/s

820: 2021-06-20 04:51:44 INFO:  69% (69.0 GiB of 100.0 GiB) in 6m 15s, read: 175.4 MiB/s, write: 164.4 MiB/s

820: 2021-06-20 04:51:52 INFO:  70% (70.2 GiB of 100.0 GiB) in 6m 23s, read: 155.9 MiB/s, write: 139.4 MiB/s

820: 2021-06-20 04:51:57 INFO:  71% (71.1 GiB of 100.0 GiB) in 6m 28s, read: 170.3 MiB/s, write: 163.8 MiB/s

820: 2021-06-20 04:52:04 INFO:  72% (72.1 GiB of 100.0 GiB) in 6m 35s, read: 155.9 MiB/s, write: 143.3 MiB/s

820: 2021-06-20 04:52:09 INFO:  73% (73.1 GiB of 100.0 GiB) in 6m 40s, read: 197.9 MiB/s, write: 194.1 MiB/s

820: 2021-06-20 04:52:17 INFO:  74% (74.1 GiB of 100.0 GiB) in 6m 48s, read: 135.9 MiB/s, write: 121.7 MiB/s

820: 2021-06-20 04:52:23 INFO:  75% (75.1 GiB of 100.0 GiB) in 6m 54s, read: 157.1 MiB/s, write: 152.0 MiB/s

820: 2021-06-20 04:52:30 INFO:  76% (76.0 GiB of 100.0 GiB) in 7m 1s, read: 138.8 MiB/s, write: 126.7 MiB/s

820: 2021-06-20 04:52:36 INFO:  77% (77.1 GiB of 100.0 GiB) in 7m 7s, read: 177.6 MiB/s, write: 166.7 MiB/s

820: 2021-06-20 04:52:43 INFO:  78% (78.0 GiB of 100.0 GiB) in 7m 14s, read: 138.8 MiB/s, write: 128.9 MiB/s

820: 2021-06-20 04:52:49 INFO:  79% (79.0 GiB of 100.0 GiB) in 7m 20s, read: 170.4 MiB/s, write: 155.1 MiB/s

820: 2021-06-20 04:52:56 INFO:  80% (80.0 GiB of 100.0 GiB) in 7m 27s, read: 146.0 MiB/s, write: 132.9 MiB/s

820: 2021-06-20 04:53:02 INFO:  81% (81.0 GiB of 100.0 GiB) in 7m 33s, read: 172.2 MiB/s, write: 158.1 MiB/s

820: 2021-06-20 04:53:09 INFO:  82% (82.2 GiB of 100.0 GiB) in 7m 40s, read: 180.7 MiB/s, write: 153.3 MiB/s

820: 2021-06-20 04:53:15 INFO:  83% (83.2 GiB of 100.0 GiB) in 7m 46s, read: 157.7 MiB/s, write: 152.4 MiB/s

820: 2021-06-20 04:53:19 INFO:  84% (84.1 GiB of 100.0 GiB) in 7m 50s, read: 233.5 MiB/s, write: 221.9 MiB/s

820: 2021-06-20 04:53:25 INFO:  85% (85.1 GiB of 100.0 GiB) in 7m 56s, read: 179.6 MiB/s, write: 177.8 MiB/s

820: 2021-06-20 04:53:29 INFO:  86% (86.2 GiB of 100.0 GiB) in 8m, read: 264.6 MiB/s, write: 255.7 MiB/s

820: 2021-06-20 04:53:34 INFO:  87% (87.2 GiB of 100.0 GiB) in 8m 5s, read: 218.9 MiB/s, write: 218.1 MiB/s

820: 2021-06-20 04:53:39 INFO:  88% (88.2 GiB of 100.0 GiB) in 8m 10s, read: 195.0 MiB/s, write: 188.1 MiB/s

820: 2021-06-20 04:53:44 INFO:  89% (89.1 GiB of 100.0 GiB) in 8m 15s, read: 189.9 MiB/s, write: 189.3 MiB/s

820: 2021-06-20 04:53:50 INFO:  90% (90.1 GiB of 100.0 GiB) in 8m 21s, read: 166.1 MiB/s, write: 158.7 MiB/s

820: 2021-06-20 04:53:56 INFO:  91% (91.0 GiB of 100.0 GiB) in 8m 27s, read: 156.5 MiB/s, write: 154.5 MiB/s

820: 2021-06-20 04:54:03 INFO:  92% (92.2 GiB of 100.0 GiB) in 8m 34s, read: 167.8 MiB/s, write: 161.4 MiB/s

820: 2021-06-20 04:54:14 INFO:  93% (93.1 GiB of 100.0 GiB) in 8m 45s, read: 88.0 MiB/s, write: 86.8 MiB/s

820: 2021-06-20 04:54:19 INFO:  94% (94.1 GiB of 100.0 GiB) in 8m 50s, read: 211.0 MiB/s, write: 195.3 MiB/s

820: 2021-06-20 04:54:24 INFO:  95% (95.1 GiB of 100.0 GiB) in 8m 55s, read: 200.8 MiB/s, write: 199.1 MiB/s

820: 2021-06-20 04:54:29 INFO:  96% (96.1 GiB of 100.0 GiB) in 9m, read: 207.3 MiB/s, write: 188.1 MiB/s

820: 2021-06-20 04:54:34 INFO:  97% (97.1 GiB of 100.0 GiB) in 9m 5s, read: 190.6 MiB/s, write: 186.3 MiB/s

820: 2021-06-20 04:54:41 INFO:  98% (98.2 GiB of 100.0 GiB) in 9m 12s, read: 161.1 MiB/s, write: 152.3 MiB/s

820: 2021-06-20 04:54:48 INFO:  99% (99.1 GiB of 100.0 GiB) in 9m 19s, read: 135.2 MiB/s, write: 131.0 MiB/s

820: 2021-06-20 04:54:55 INFO: 100% (100.0 GiB of 100.0 GiB) in 9m 26s, read: 134.8 MiB/s, write: 120.1 MiB/s

820: 2021-06-20 04:54:55 INFO: backup is sparse: 7.09 GiB (7%) total zero data

820: 2021-06-20 04:54:55 INFO: transferred 100.00 GiB in 566 seconds (180.9 MiB/s)

820: 2021-06-20 04:57:21 ERROR: Backup of VM 820 failed - zstd --rsyncable --threads=1 failed - wrong exit status 1

The other fails look the same. It runs up until 100% but then says exit status 1.
About 30% of the VMs have failed the yesterday backup.

I just ran a backup manually on this VM 820 and it went through fine; so this must be a sporadic issue.

Could it be because of the CIFS target? What could I do to improve the stability in that case?

 

[Chiaki]

While running manual backups now I encountered the error first time myself. The above log is from an email, the below partial log is from the web GUI:

INFO: starting new backup job: vzdump 410 --storage MyStorage --compress zstd --node thor --mode snapshot --remove 0
INFO: Starting Backup of VM 410 (qemu)
INFO: Backup started at 2021-06-21 09:37:20
INFO: status = running
INFO: VM Name: MyVM
INFO: include disk 'scsi0' 'local-zfs:vm-410-disk-0' 600G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/pve/MyStorage/dump/vzdump-qemu-410-2021_06_21-09_37_20.vma.zst'
INFO: started backup task '345fcd7f-21c2-4b3a-bc98-5a98e61f33a9'
INFO: resuming VM again
INFO:   0% (873.2 MiB of 600.0 GiB) in 3s, read: 291.1 MiB/s, write: 241.9 MiB/s
INFO:   1% (6.1 GiB of 600.0 GiB) in 24s, read: 255.4 MiB/s, write: 214.2 MiB/s
INFO:   2% (12.1 GiB of 600.0 GiB) in 45s, read: 295.3 MiB/s, write: 251.3 MiB/s
INFO:   3% (18.3 GiB of 600.0 GiB) in 1m 6s, read: 301.2 MiB/s, write: 252.0 MiB/s
INFO:   4% (24.1 GiB of 600.0 GiB) in 1m 35s, read: 203.9 MiB/s, write: 179.5 MiB/s
INFO:   5% (30.2 GiB of 600.0 GiB) in 2m 18s, read: 146.0 MiB/s, write: 123.6 MiB/s
INFO:   6% (36.0 GiB of 600.0 GiB) in 3m, read: 140.7 MiB/s, write: 122.5 MiB/s
INFO:   7% (42.0 GiB of 600.0 GiB) in 3m 38s, read: 162.3 MiB/s, write: 151.0 MiB/s
INFO:   8% (48.2 GiB of 600.0 GiB) in 4m 11s, read: 191.9 MiB/s, write: 180.9 MiB/s
INFO:   9% (54.0 GiB of 600.0 GiB) in 4m 47s, read: 164.8 MiB/s, write: 155.0 MiB/s
INFO:  10% (60.0 GiB of 600.0 GiB) in 5m 33s, read: 134.2 MiB/s, write: 127.3 MiB/s
INFO:  11% (66.1 GiB of 600.0 GiB) in 6m 17s, read: 140.8 MiB/s, write: 129.6 MiB/s
INFO:  12% (72.1 GiB of 600.0 GiB) in 7m, read: 142.7 MiB/s, write: 134.9 MiB/s
INFO:  13% (78.1 GiB of 600.0 GiB) in 7m 31s, read: 200.6 MiB/s, write: 189.8 MiB/s
INFO:  14% (84.1 GiB of 600.0 GiB) in 8m 7s, read: 169.9 MiB/s, write: 165.2 MiB/s
INFO:  15% (90.0 GiB of 600.0 GiB) in 8m 46s, read: 155.0 MiB/s, write: 150.8 MiB/s
INFO:  16% (96.0 GiB of 600.0 GiB) in 9m 27s, read: 150.0 MiB/s, write: 146.5 MiB/s
INFO:  17% (102.0 GiB of 600.0 GiB) in 10m 12s, read: 136.6 MiB/s, write: 133.9 MiB/s
INFO:  18% (108.1 GiB of 600.0 GiB) in 11m 2s, read: 124.4 MiB/s, write: 121.7 MiB/s
INFO:  19% (114.0 GiB of 600.0 GiB) in 11m 50s, read: 125.8 MiB/s, write: 123.3 MiB/s
INFO:  20% (120.0 GiB of 600.0 GiB) in 12m 37s, read: 130.9 MiB/s, write: 127.8 MiB/s
INFO:  21% (126.0 GiB of 600.0 GiB) in 13m 22s, read: 136.3 MiB/s, write: 133.7 MiB/s
INFO:  22% (132.1 GiB of 600.0 GiB) in 14m 9s, read: 133.4 MiB/s, write: 127.7 MiB/s
INFO:  23% (138.0 GiB of 600.0 GiB) in 14m 52s, read: 140.8 MiB/s, write: 133.3 MiB/s
INFO:  24% (144.0 GiB of 600.0 GiB) in 15m 29s, read: 166.4 MiB/s, write: 155.4 MiB/s
INFO:  25% (150.1 GiB of 600.0 GiB) in 16m 14s, read: 137.5 MiB/s, write: 129.5 MiB/s
INFO:  26% (156.0 GiB of 600.0 GiB) in 17m, read: 132.3 MiB/s, write: 127.9 MiB/s
INFO:  27% (162.0 GiB of 600.0 GiB) in 17m 51s, read: 120.4 MiB/s, write: 118.3 MiB/s
INFO:  28% (168.1 GiB of 600.0 GiB) in 18m 42s, read: 121.6 MiB/s, write: 117.9 MiB/s
INFO:  29% (174.1 GiB of 600.0 GiB) in 19m 30s, read: 128.2 MiB/s, write: 124.2 MiB/s
INFO:  30% (180.1 GiB of 600.0 GiB) in 20m 11s, read: 150.6 MiB/s, write: 142.9 MiB/s
INFO:  31% (186.1 GiB of 600.0 GiB) in 20m 53s, read: 146.3 MiB/s, write: 139.1 MiB/s
INFO:  32% (192.1 GiB of 600.0 GiB) in 21m 21s, read: 217.4 MiB/s, write: 207.5 MiB/s
INFO:  33% (198.1 GiB of 600.0 GiB) in 21m 47s, read: 239.4 MiB/s, write: 231.5 MiB/s
INFO:  34% (204.2 GiB of 600.0 GiB) in 22m 17s, read: 206.9 MiB/s, write: 197.6 MiB/s
INFO:  35% (210.1 GiB of 600.0 GiB) in 22m 46s, read: 207.1 MiB/s, write: 196.7 MiB/s
INFO:  36% (216.0 GiB of 600.0 GiB) in 23m 29s, read: 141.3 MiB/s, write: 138.3 MiB/s
INFO:  37% (222.1 GiB of 600.0 GiB) in 24m 21s, read: 119.4 MiB/s, write: 117.4 MiB/s
INFO:  38% (228.0 GiB of 600.0 GiB) in 25m 16s, read: 111.2 MiB/s, write: 108.8 MiB/s
INFO:  39% (234.2 GiB of 600.0 GiB) in 25m 56s, read: 157.0 MiB/s, write: 151.9 MiB/s
INFO:  40% (240.0 GiB of 600.0 GiB) in 26m 37s, read: 146.5 MiB/s, write: 142.4 MiB/s
INFO:  41% (246.1 GiB of 600.0 GiB) in 27m 30s, read: 116.5 MiB/s, write: 114.1 MiB/s
INFO:  42% (252.0 GiB of 600.0 GiB) in 28m 23s, read: 115.1 MiB/s, write: 113.3 MiB/s
INFO:  43% (258.1 GiB of 600.0 GiB) in 29m 17s, read: 114.7 MiB/s, write: 112.6 MiB/s
INFO:  44% (264.1 GiB of 600.0 GiB) in 30m 10s, read: 116.5 MiB/s, write: 114.5 MiB/s
INFO:  45% (270.0 GiB of 600.0 GiB) in 30m 53s, read: 141.4 MiB/s, write: 136.4 MiB/s
INFO:  46% (276.1 GiB of 600.0 GiB) in 31m 23s, read: 206.5 MiB/s, write: 197.2 MiB/s
INFO:  47% (282.1 GiB of 600.0 GiB) in 31m 59s, read: 170.9 MiB/s, write: 162.5 MiB/s
INFO:  48% (288.2 GiB of 600.0 GiB) in 32m 28s, read: 214.9 MiB/s, write: 205.2 MiB/s
INFO:  49% (294.2 GiB of 600.0 GiB) in 33m, read: 193.1 MiB/s, write: 180.1 MiB/s
INFO:  50% (300.1 GiB of 600.0 GiB) in 33m 22s, read: 272.9 MiB/s, write: 258.4 MiB/s
INFO:  51% (306.0 GiB of 600.0 GiB) in 33m 54s, read: 189.4 MiB/s, write: 181.2 MiB/s
INFO:  52% (312.1 GiB of 600.0 GiB) in 34m 16s, read: 283.2 MiB/s, write: 272.4 MiB/s
INFO:  53% (318.2 GiB of 600.0 GiB) in 34m 41s, read: 249.6 MiB/s, write: 233.6 MiB/s
INFO:  54% (324.0 GiB of 600.0 GiB) in 35m 15s, read: 176.1 MiB/s, write: 167.7 MiB/s
INFO:  55% (330.2 GiB of 600.0 GiB) in 35m 38s, read: 273.4 MiB/s, write: 255.0 MiB/s
INFO:  56% (336.3 GiB of 600.0 GiB) in 36m 1s, read: 272.5 MiB/s, write: 258.1 MiB/s
INFO:  57% (342.2 GiB of 600.0 GiB) in 36m 33s, read: 189.4 MiB/s, write: 180.4 MiB/s
INFO:  58% (348.1 GiB of 600.0 GiB) in 37m 5s, read: 189.1 MiB/s, write: 178.6 MiB/s
INFO:  59% (354.2 GiB of 600.0 GiB) in 37m 51s, read: 136.3 MiB/s, write: 127.6 MiB/s
INFO:  60% (360.1 GiB of 600.0 GiB) in 38m 13s, read: 272.5 MiB/s, write: 261.5 MiB/s
INFO:  61% (366.2 GiB of 600.0 GiB) in 38m 35s, read: 286.0 MiB/s, write: 273.0 MiB/s
INFO:  62% (372.0 GiB of 600.0 GiB) in 39m 8s, read: 179.6 MiB/s, write: 170.8 MiB/s
INFO:  63% (378.1 GiB of 600.0 GiB) in 39m 40s, read: 193.0 MiB/s, write: 184.2 MiB/s
INFO:  64% (384.0 GiB of 600.0 GiB) in 40m 2s, read: 277.5 MiB/s, write: 268.5 MiB/s
INFO:  65% (390.2 GiB of 600.0 GiB) in 40m 25s, read: 272.8 MiB/s, write: 253.5 MiB/s
INFO:  66% (396.2 GiB of 600.0 GiB) in 41m 9s, read: 141.6 MiB/s, write: 132.8 MiB/s
INFO:  67% (402.2 GiB of 600.0 GiB) in 41m 55s, read: 132.7 MiB/s, write: 124.4 MiB/s
INFO:  68% (408.2 GiB of 600.0 GiB) in 42m 43s, read: 127.4 MiB/s, write: 119.5 MiB/s
INFO:  69% (414.2 GiB of 600.0 GiB) in 43m 28s, read: 136.3 MiB/s, write: 127.2 MiB/s
INFO:  70% (420.2 GiB of 600.0 GiB) in 44m 10s, read: 147.2 MiB/s, write: 135.7 MiB/s
INFO:  71% (426.5 GiB of 600.0 GiB) in 44m 54s, read: 146.8 MiB/s, write: 116.7 MiB/s
INFO:  72% (432.1 GiB of 600.0 GiB) in 44m 57s, read: 1.9 GiB/s, write: 4.0 KiB/s
INFO:  73% (438.4 GiB of 600.0 GiB) in 45m 1s, read: 1.6 GiB/s, write: 3.0 KiB/s
INFO:  74% (445.6 GiB of 600.0 GiB) in 45m 5s, read: 1.8 GiB/s, write: 3.0 KiB/s
INFO:  75% (450.1 GiB of 600.0 GiB) in 45m 8s, read: 1.5 GiB/s, write: 4.0 KiB/s
INFO:  76% (457.0 GiB of 600.0 GiB) in 45m 12s, read: 1.7 GiB/s, write: 3.0 KiB/s
INFO:  77% (463.3 GiB of 600.0 GiB) in 45m 16s, read: 1.6 GiB/s, write: 3.0 KiB/s
INFO:  78% (468.3 GiB of 600.0 GiB) in 45m 19s, read: 1.7 GiB/s, write: 4.0 KiB/s
INFO:  79% (475.9 GiB of 600.0 GiB) in 45m 23s, read: 1.9 GiB/s, write: 3.0 KiB/s
INFO:  80% (481.5 GiB of 600.0 GiB) in 45m 26s, read: 1.9 GiB/s, write: 4.0 KiB/s
INFO:  81% (486.5 GiB of 600.0 GiB) in 45m 29s, read: 1.7 GiB/s, write: 4.0 KiB/s
INFO:  82% (492.2 GiB of 600.0 GiB) in 45m 32s, read: 1.9 GiB/s, write: 4.0 KiB/s
INFO:  83% (499.5 GiB of 600.0 GiB) in 45m 36s, read: 1.8 GiB/s, write: 3.0 KiB/s
INFO:  84% (505.3 GiB of 600.0 GiB) in 45m 39s, read: 1.9 GiB/s, write: 4.0 KiB/s
INFO:  85% (510.8 GiB of 600.0 GiB) in 45m 42s, read: 1.8 GiB/s, write: 4.0 KiB/s
INFO:  86% (517.6 GiB of 600.0 GiB) in 45m 46s, read: 1.7 GiB/s, write: 3.0 KiB/s
INFO:  87% (522.7 GiB of 600.0 GiB) in 45m 49s, read: 1.7 GiB/s, write: 4.0 KiB/s
INFO:  88% (528.3 GiB of 600.0 GiB) in 45m 52s, read: 1.9 GiB/s, write: 4.0 KiB/s
INFO:  89% (534.8 GiB of 600.0 GiB) in 45m 56s, read: 1.6 GiB/s, write: 3.0 KiB/s
INFO:  90% (540.2 GiB of 600.0 GiB) in 45m 59s, read: 1.8 GiB/s, write: 4.0 KiB/s
INFO:  91% (547.4 GiB of 600.0 GiB) in 46m 3s, read: 1.8 GiB/s, write: 3.0 KiB/s
INFO:  92% (552.1 GiB of 600.0 GiB) in 46m 6s, read: 1.6 GiB/s, write: 4.0 KiB/s
INFO:  93% (558.8 GiB of 600.0 GiB) in 46m 10s, read: 1.7 GiB/s, write: 3.0 KiB/s
INFO:  94% (565.4 GiB of 600.0 GiB) in 46m 14s, read: 1.6 GiB/s, write: 3.0 KiB/s
INFO:  95% (570.3 GiB of 600.0 GiB) in 46m 17s, read: 1.6 GiB/s, write: 4.0 KiB/s
INFO:  96% (577.2 GiB of 600.0 GiB) in 46m 21s, read: 1.7 GiB/s, write: 3.0 KiB/s
INFO:  97% (582.7 GiB of 600.0 GiB) in 46m 24s, read: 1.8 GiB/s, write: 4.0 KiB/s
INFO:  98% (588.0 GiB of 600.0 GiB) in 46m 27s, read: 1.8 GiB/s, write: 4.0 KiB/s
INFO:  99% (595.9 GiB of 600.0 GiB) in 46m 31s, read: 2.0 GiB/s, write: 3.0 KiB/s
zstd: /*stdout*\: Input/output error
INFO: 100% (600.0 GiB of 600.0 GiB) in 46m 34s, read: 1.4 GiB/s, write: 4.0 KiB/s
INFO: backup is sparse: 196.86 GiB (32%) total zero data
INFO: transferred 600.00 GiB in 2794 seconds (219.9 MiB/s)
ERROR: Backup of VM 410 failed - zstd --rsyncable --threads=1 failed - wrong exit status 1
INFO: Failed at 2021-06-21 10:26:42
INFO: Backup job finished with errors
TASK ERROR: job errors

Does anyone have an idea how I could look further into the reason for `zstd: /*stdout*\: Input/output error` ?

 

[Chiaki]

Just to not be that guy who says "Solved it" and leaves everyone else hanging dry:

Especially the big VMs had issues like this and the reason for that seems to be the CIFS caching.

I deactivated the caching by passing `cache=none` with the CIFS mount options.

Now I have no issues anymore.

Maybe Proxmox should allow setting this via Web GUI when adding CIFS storage...


Nice side-effect: You see the actual real bandwidth which is less than what CIFS with caching makes you believe it to be. And when a file is transferred it is actually already there on the shared storage and not still being in transit from cache.

 

[strawman]

Dear Chiaki,

Thanks for your report. I think I am having the same issue. I cannot figure out how to pass `cache=none` to the CIFS mount options. At https://pve.proxmox.com/wiki/Storage:_CIFS I do not see that listed anywhere.

Thanks in advance!

 

[RolandK]

unfortuntatly, it's not possible yet, see https://bugzilla.proxmox.com/show_bug.cgi?id=2920