zstd: error 25 : Write error : Input/output error (cannot write compressed block)

J

jsalas424

Active Member
Jul 5, 2020
142
3
38
34
I'm trying to send a VM backup to a S3 bucket that has been mounted to a local file. I have succesfully backed up to this mounted share directly from proxmox with a ~25GB disk, but fails on a ~125GB disk.

Code: 
INFO: starting new backup job: vzdump 900 --notes-template '{{guestname}} baseline nextcloud backup' --node Server --compress zstd --mode stop --storage S3_Backups --remove 0
INFO: Starting Backup of VM 900 (qemu)
INFO: Backup started at 2023-05-28 23:35:51
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: Nextcloud
INFO: include disk 'scsi0' 'new.ssd.dir:900/vm-900-disk-0.qcow2' 35G
INFO: include disk 'scsi1' 'new.ssd.dir:900/vm-900-disk-1.qcow2' 205G
INFO: creating vzdump archive '/mnt/s3-bucket/dump/vzdump-qemu-900-2023_05_28-23_35_50.vma.zst'
INFO: starting kvm to execute backup task
INFO: started backup task '28740446-3204-4fdb-8740-a236a358aa61'
INFO:   0% (126.9 MiB of 240.0 GiB) in 3s, read: 42.3 MiB/s, write: 42.0 MiB/s
INFO:   1% (2.4 GiB of 240.0 GiB) in 40s, read: 63.0 MiB/s, write: 63.0 MiB/s
INFO:   2% (4.9 GiB of 240.0 GiB) in 1m 21s, read: 61.3 MiB/s, write: 61.0 MiB/s
INFO:   3% (7.2 GiB of 240.0 GiB) in 5m 41s, read: 9.4 MiB/s, write: 9.4 MiB/s
INFO:   4% (9.6 GiB of 240.0 GiB) in 6m 18s, read: 65.9 MiB/s, write: 65.7 MiB/s
zstd: error 25 : Write error : Input/output error (cannot write compressed block)

I followed the top comment on this thread as a guide: https://www.reddit.com/r/homelab/comments/5m4fna/backing_up_proxmox_to_s3_or_glacier_what_is_the/

Here's the successful attempt:
Code: 
vzdump 700 --notes-template '{{guestname}} baseline time machine backups' --node Server --mode stop --compress zstd --storage S3_Backups --remove 0


700: 2023-05-28 22:57:00 INFO: Starting Backup of VM 700 (qemu)
700: 2023-05-28 22:57:00 INFO: status = stopped
700: 2023-05-28 22:57:00 INFO: backup mode: stop
700: 2023-05-28 22:57:00 INFO: ionice priority: 7
700: 2023-05-28 22:57:00 INFO: VM Name: Backups.Server
700: 2023-05-28 22:57:00 INFO: include disk 'scsi0' 'new.ssd.dir:700/vm-700-disk-0.qcow2' 25G
700: 2023-05-28 22:57:00 INFO: exclude disk 'scsi1' 'Storage.1:vm-700-disk-0' (backup=no)
700: 2023-05-28 22:57:00 INFO: creating vzdump archive '/mnt/s3-bucket/dump/vzdump-qemu-700-2023_05_28-22_56_59.vma.zst'
700: 2023-05-28 22:57:00 INFO: starting kvm to execute backup task
700: 2023-05-28 22:57:03 INFO: started backup task '25fd9479-985f-4714-8e78-9c9dc8a15cb3'
700: 2023-05-28 22:57:06 INFO:   0% (203.0 MiB of 25.0 GiB) in 3s, read: 67.7 MiB/s, write: 36.3 MiB/s
700: 2023-05-28 22:57:09 INFO:   1% (471.2 MiB of 25.0 GiB) in 6s, read: 89.4 MiB/s, write: 48.9 MiB/s
700: 2023-05-28 22:57:12 INFO:   3% (812.0 MiB of 25.0 GiB) in 9s, read: 113.6 MiB/s, write: 50.2 MiB/s
700: 2023-05-28 22:57:16 INFO:   4% (1.0 GiB of 25.0 GiB) in 13s, read: 63.4 MiB/s, write: 26.9 MiB/s
700: 2023-05-28 22:57:19 INFO:   5% (1.4 GiB of 25.0 GiB) in 16s, read: 132.9 MiB/s, write: 35.5 MiB/s
700: 2023-05-28 22:57:22 INFO:   6% (1.7 GiB of 25.0 GiB) in 19s, read: 102.7 MiB/s, write: 47.6 MiB/s
700: 2023-05-28 22:57:25 INFO:   7% (1.9 GiB of 25.0 GiB) in 22s, read: 49.5 MiB/s, write: 22.5 MiB/s
700: 2023-05-28 22:57:28 INFO:   8% (2.1 GiB of 25.0 GiB) in 25s, read: 82.2 MiB/s, write: 29.6 MiB/s
700: 2023-05-28 22:57:31 INFO:   9% (2.3 GiB of 25.0 GiB) in 28s, read: 54.4 MiB/s, write: 24.9 MiB/s
700: 2023-05-28 22:57:34 INFO:  10% (2.5 GiB of 25.0 GiB) in 31s, read: 90.6 MiB/s, write: 43.3 MiB/s
700: 2023-05-28 22:57:38 INFO:  11% (2.8 GiB of 25.0 GiB) in 35s, read: 67.1 MiB/s, write: 47.7 MiB/s
700: 2023-05-28 22:57:42 INFO:  12% (3.0 GiB of 25.0 GiB) in 39s, read: 58.9 MiB/s, write: 47.9 MiB/s
700: 2023-05-28 22:57:46 INFO:  13% (3.3 GiB of 25.0 GiB) in 43s, read: 70.7 MiB/s, write: 58.5 MiB/s
700: 2023-05-28 22:57:49 INFO:  14% (3.5 GiB of 25.0 GiB) in 46s, read: 73.7 MiB/s, write: 59.6 MiB/s
700: 2023-05-28 22:57:53 INFO:  15% (3.8 GiB of 25.0 GiB) in 50s, read: 72.5 MiB/s, write: 59.5 MiB/s
700: 2023-05-28 22:57:56 INFO:  16% (4.1 GiB of 25.0 GiB) in 53s, read: 113.6 MiB/s, write: 39.2 MiB/s
700: 2023-05-28 22:57:59 INFO:  17% (4.4 GiB of 25.0 GiB) in 56s, read: 88.2 MiB/s, write: 37.4 MiB/s
700: 2023-05-28 22:58:02 INFO:  18% (4.7 GiB of 25.0 GiB) in 59s, read: 103.9 MiB/s, write: 39.5 MiB/s
700: 2023-05-28 22:58:05 INFO:  21% (5.4 GiB of 25.0 GiB) in 1m 2s, read: 229.6 MiB/s, write: 34.6 MiB/s
700: 2023-05-28 22:58:08 INFO:  24% (6.1 GiB of 25.0 GiB) in 1m 5s, read: 232.0 MiB/s, write: 29.5 MiB/s
700: 2023-05-28 22:58:11 INFO:  25% (6.4 GiB of 25.0 GiB) in 1m 8s, read: 112.4 MiB/s, write: 27.9 MiB/s
700: 2023-05-28 22:58:14 INFO:  26% (6.6 GiB of 25.0 GiB) in 1m 11s, read: 60.4 MiB/s, write: 41.1 MiB/s
700: 2023-05-28 22:58:17 INFO:  27% (6.8 GiB of 25.0 GiB) in 1m 14s, read: 89.4 MiB/s, write: 24.4 MiB/s
700: 2023-05-28 22:58:20 INFO:  28% (7.0 GiB of 25.0 GiB) in 1m 17s, read: 62.8 MiB/s, write: 49.1 MiB/s
700: 2023-05-28 22:58:24 INFO:  29% (7.3 GiB of 25.0 GiB) in 1m 21s, read: 63.4 MiB/s, write: 28.6 MiB/s
700: 2023-05-28 22:58:27 INFO:  30% (7.6 GiB of 25.0 GiB) in 1m 24s, read: 100.3 MiB/s, write: 17.3 MiB/s
700: 2023-05-28 22:58:31 INFO:  31% (7.8 GiB of 25.0 GiB) in 1m 28s, read: 66.2 MiB/s, write: 38.3 MiB/s
700: 2023-05-28 22:58:35 INFO:  32% (8.1 GiB of 25.0 GiB) in 1m 32s, read: 77.9 MiB/s, write: 37.9 MiB/s
700: 2023-05-28 22:58:38 INFO:  33% (8.3 GiB of 25.0 GiB) in 1m 35s, read: 73.7 MiB/s, write: 28.4 MiB/s
700: 2023-05-28 22:58:41 INFO:  34% (8.7 GiB of 25.0 GiB) in 1m 38s, read: 142.6 MiB/s, write: 41.6 MiB/s
700: 2023-05-28 22:58:44 INFO:  35% (8.9 GiB of 25.0 GiB) in 1m 41s, read: 53.2 MiB/s, write: 43.9 MiB/s
700: 2023-05-28 22:58:47 INFO:  36% (9.1 GiB of 25.0 GiB) in 1m 44s, read: 58.0 MiB/s, write: 35.5 MiB/s
700: 2023-05-28 22:58:50 INFO:  37% (9.4 GiB of 25.0 GiB) in 1m 47s, read: 110.0 MiB/s, write: 41.8 MiB/s
700: 2023-05-28 22:58:53 INFO:  38% (9.6 GiB of 25.0 GiB) in 1m 50s, read: 77.3 MiB/s, write: 52.7 MiB/s
700: 2023-05-28 22:58:56 INFO:  39% (9.8 GiB of 25.0 GiB) in 1m 53s, read: 50.8 MiB/s, write: 39.6 MiB/s
700: 2023-05-28 22:59:00 INFO:  40% (10.1 GiB of 25.0 GiB) in 1m 57s, read: 90.6 MiB/s, write: 41.1 MiB/s
700: 2023-05-28 22:59:04 INFO:  41% (10.3 GiB of 25.0 GiB) in 2m 1s, read: 49.8 MiB/s, write: 22.8 MiB/s
700: 2023-05-28 22:59:07 INFO:  42% (10.5 GiB of 25.0 GiB) in 2m 4s, read: 78.5 MiB/s, write: 32.5 MiB/s
700: 2023-05-28 22:59:12 INFO:  43% (10.8 GiB of 25.0 GiB) in 2m 9s, read: 44.9 MiB/s, write: 34.8 MiB/s
700: 2023-05-28 22:59:17 INFO:  44% (11.0 GiB of 25.0 GiB) in 2m 14s, read: 52.9 MiB/s, write: 23.0 MiB/s
700: 2023-05-28 22:59:21 INFO:  45% (11.3 GiB of 25.0 GiB) in 2m 18s, read: 60.7 MiB/s, write: 30.5 MiB/s
700: 2023-05-28 22:59:27 INFO:  46% (11.6 GiB of 25.0 GiB) in 2m 24s, read: 64.6 MiB/s, write: 34.6 MiB/s
700: 2023-05-28 22:59:30 INFO:  47% (12.0 GiB of 25.0 GiB) in 2m 27s, read: 112.4 MiB/s, write: 26.4 MiB/s
700: 2023-05-28 22:59:33 INFO:  48% (12.2 GiB of 25.0 GiB) in 2m 30s, read: 94.2 MiB/s, write: 54.7 MiB/s
700: 2023-05-28 22:59:36 INFO:  49% (12.4 GiB of 25.0 GiB) in 2m 33s, read: 55.6 MiB/s, write: 20.6 MiB/s
700: 2023-05-28 22:59:39 INFO:  50% (12.7 GiB of 25.0 GiB) in 2m 36s, read: 84.6 MiB/s, write: 24.1 MiB/s
700: 2023-05-28 22:59:42 INFO:  51% (12.8 GiB of 25.0 GiB) in 2m 39s, read: 32.6 MiB/s, write: 25.8 MiB/s
700: 2023-05-28 22:59:46 INFO:  52% (13.0 GiB of 25.0 GiB) in 2m 43s, read: 65.2 MiB/s, write: 13.1 MiB/s
700: 2023-05-28 22:59:55 INFO:  53% (13.3 GiB of 25.0 GiB) in 2m 52s, read: 29.0 MiB/s, write: 24.1 MiB/s
700: 2023-05-28 22:59:58 INFO:  54% (13.6 GiB of 25.0 GiB) in 2m 55s, read: 103.9 MiB/s, write: 28.3 MiB/s
700: 2023-05-28 23:00:01 INFO:  55% (13.8 GiB of 25.0 GiB) in 2m 58s, read: 64.0 MiB/s, write: 31.1 MiB/s
700: 2023-05-28 23:00:10 INFO:  56% (14.1 GiB of 25.0 GiB) in 3m 7s, read: 42.7 MiB/s, write: 16.2 MiB/s
700: 2023-05-28 23:00:17 INFO:  57% (14.3 GiB of 25.0 GiB) in 3m 14s, read: 23.8 MiB/s, write: 21.7 MiB/s
700: 2023-05-28 23:00:22 INFO:  58% (14.5 GiB of 25.0 GiB) in 3m 19s, read: 46.4 MiB/s, write: 9.9 MiB/s
700: 2023-05-28 23:00:27 INFO:  59% (14.8 GiB of 25.0 GiB) in 3m 24s, read: 58.0 MiB/s, write: 10.3 MiB/s
700: 2023-05-28 23:00:36 INFO:  60% (15.0 GiB of 25.0 GiB) in 3m 33s, read: 22.6 MiB/s, write: 11.7 MiB/s
700: 2023-05-28 23:00:47 INFO:  61% (15.3 GiB of 25.0 GiB) in 3m 44s, read: 23.1 MiB/s, write: 17.1 MiB/s
700: 2023-05-28 23:00:53 INFO:  62% (15.5 GiB of 25.0 GiB) in 3m 50s, read: 45.3 MiB/s, write: 38.4 MiB/s
700: 2023-05-28 23:00:58 INFO:  63% (15.8 GiB of 25.0 GiB) in 3m 55s, read: 60.2 MiB/s, write: 39.6 MiB/s
700: 2023-05-28 23:01:03 INFO:  64% (16.0 GiB of 25.0 GiB) in 4m, read: 47.1 MiB/s, write: 35.2 MiB/s
700: 2023-05-28 23:01:06 INFO:  65% (16.3 GiB of 25.0 GiB) in 4m 3s, read: 72.5 MiB/s, write: 31.8 MiB/s
700: 2023-05-28 23:01:12 INFO:  66% (16.5 GiB of 25.0 GiB) in 4m 9s, read: 47.7 MiB/s, write: 24.9 MiB/s
700: 2023-05-28 23:01:17 INFO:  67% (16.8 GiB of 25.0 GiB) in 4m 14s, read: 52.2 MiB/s, write: 39.7 MiB/s
700: 2023-05-28 23:01:24 INFO:  68% (17.0 GiB of 25.0 GiB) in 4m 21s, read: 37.8 MiB/s, write: 26.5 MiB/s
700: 2023-05-28 23:01:28 INFO:  69% (17.3 GiB of 25.0 GiB) in 4m 25s, read: 54.4 MiB/s, write: 27.7 MiB/s
700: 2023-05-28 23:01:37 INFO:  70% (17.5 GiB of 25.0 GiB) in 4m 34s, read: 28.2 MiB/s, write: 19.3 MiB/s
700: 2023-05-28 23:01:41 INFO:  71% (17.8 GiB of 25.0 GiB) in 4m 38s, read: 68.0 MiB/s, write: 33.5 MiB/s
700: 2023-05-28 23:01:46 INFO:  72% (18.1 GiB of 25.0 GiB) in 4m 43s, read: 58.0 MiB/s, write: 29.2 MiB/s
700: 2023-05-28 23:01:50 INFO:  73% (18.3 GiB of 25.0 GiB) in 4m 47s, read: 56.2 MiB/s, write: 40.8 MiB/s
700: 2023-05-28 23:01:53 INFO:  74% (18.6 GiB of 25.0 GiB) in 4m 50s, read: 102.7 MiB/s, write: 41.7 MiB/s
700: 2023-05-28 23:01:56 INFO:  76% (19.1 GiB of 25.0 GiB) in 4m 53s, read: 188.5 MiB/s, write: 17.7 MiB/s
700: 2023-05-28 23:01:59 INFO:  77% (19.4 GiB of 25.0 GiB) in 4m 56s, read: 81.0 MiB/s, write: 25.2 MiB/s
700: 2023-05-28 23:02:02 INFO:  78% (19.5 GiB of 25.0 GiB) in 4m 59s, read: 56.8 MiB/s, write: 13.7 MiB/s
700: 2023-05-28 23:02:10 INFO:  79% (19.8 GiB of 25.0 GiB) in 5m 7s, read: 32.6 MiB/s, write: 20.2 MiB/s
700: 2023-05-28 23:02:16 INFO:  80% (20.0 GiB of 25.0 GiB) in 5m 13s, read: 41.7 MiB/s, write: 29.6 MiB/s
700: 2023-05-28 23:02:19 INFO:  81% (20.3 GiB of 25.0 GiB) in 5m 16s, read: 105.1 MiB/s, write: 73.8 MiB/s
700: 2023-05-28 23:02:22 INFO:  82% (20.5 GiB of 25.0 GiB) in 5m 19s, read: 64.0 MiB/s, write: 43.3 MiB/s
700: 2023-05-28 23:02:32 INFO:  83% (20.8 GiB of 25.0 GiB) in 5m 29s, read: 24.3 MiB/s, write: 20.2 MiB/s
700: 2023-05-28 23:02:37 INFO:  84% (21.0 GiB of 25.0 GiB) in 5m 34s, read: 51.5 MiB/s, write: 27.8 MiB/s
700: 2023-05-28 23:02:40 INFO:  85% (21.4 GiB of 25.0 GiB) in 5m 37s, read: 132.9 MiB/s, write: 22.6 MiB/s
700: 2023-05-28 23:02:44 INFO:  86% (21.6 GiB of 25.0 GiB) in 5m 41s, read: 58.9 MiB/s, write: 9.6 MiB/s
700: 2023-05-28 23:02:47 INFO:  87% (21.8 GiB of 25.0 GiB) in 5m 44s, read: 55.6 MiB/s, write: 24.5 MiB/s
700: 2023-05-28 23:02:53 INFO:  88% (22.0 GiB of 25.0 GiB) in 5m 50s, read: 39.9 MiB/s, write: 30.1 MiB/s
700: 2023-05-28 23:03:00 INFO:  89% (22.3 GiB of 25.0 GiB) in 5m 57s, read: 32.6 MiB/s, write: 15.6 MiB/s
700: 2023-05-28 23:03:10 INFO:  90% (22.5 GiB of 25.0 GiB) in 6m 7s, read: 26.1 MiB/s, write: 10.7 MiB/s
700: 2023-05-28 23:08:15 INFO:  91% (22.8 GiB of 25.0 GiB) in 11m 12s, read: 851.9 KiB/s, write: 480.2 KiB/s
700: 2023-05-28 23:10:53 INFO:  92% (23.0 GiB of 25.0 GiB) in 13m 50s, read: 1.6 MiB/s, write: 1.3 MiB/s
700: 2023-05-28 23:14:09 INFO:  93% (23.4 GiB of 25.0 GiB) in 17m 6s, read: 1.9 MiB/s, write: 959.9 KiB/s
700: 2023-05-28 23:14:14 INFO:  94% (23.6 GiB of 25.0 GiB) in 17m 11s, read: 54.4 MiB/s, write: 6.8 MiB/s
700: 2023-05-28 23:16:43 INFO:  95% (23.8 GiB of 25.0 GiB) in 19m 40s, read: 946.7 KiB/s, write: 313.9 KiB/s
700: 2023-05-28 23:16:47 INFO:  96% (24.1 GiB of 25.0 GiB) in 19m 44s, read: 94.2 MiB/s, write: 3.5 MiB/s
700: 2023-05-28 23:17:01 INFO:  97% (24.3 GiB of 25.0 GiB) in 19m 58s, read: 9.6 MiB/s, write: 4.4 MiB/s
700: 2023-05-28 23:28:01 INFO:  98% (24.6 GiB of 25.0 GiB) in 30m 58s, read: 483.7 KiB/s, write: 105.4 KiB/s
700: 2023-05-28 23:28:04 INFO: 100% (25.0 GiB of 25.0 GiB) in 31m 1s, read: 147.5 MiB/s, write: 13.6 MiB/s
700: 2023-05-28 23:28:04 INFO: backup is sparse: 13.60 GiB (54%) total zero data
700: 2023-05-28 23:28:04 INFO: transferred 25.00 GiB in 1861 seconds (13.8 MiB/s)
700: 2023-05-28 23:31:37 INFO: stopping kvm after backup task
700: 2023-05-28 23:32:03 INFO: archive file size: 4.39GB
700: 2023-05-28 23:32:03 INFO: adding notes to backup
700: 2023-05-28 23:32:03 INFO: Finished Backup of VM 700 (00:35:04)
 
Last edited:
I have the same kind of issue, does someone have the solution?

INFO: starting new backup job: vzdump 100 --remove 0 --notes-template '{{guestname}}' --mode snapshot --storage s3-swissbackup --compress zstd --node apps INFO: Starting Backup of VM 100 (qemu) INFO: Backup started at 2023-10-04 09:32:14 INFO: status = running INFO: VM Name: VM-APPS INFO: include disk 'scsi0' 'local-lvm:vm-100-disk-0' 2T INFO: include disk 'efidisk0' 'local-lvm:vm-100-disk-1' 4M INFO: backup mode: snapshot INFO: ionice priority: 7 INFO: creating vzdump archive '/mnt/s3-swissbackup/dump/vzdump-qemu-100-2023_10_04-09_32_10.vma.zst' INFO: started backup task '1e8693e3-9c32-4aae-8177-497c8681cca8' INFO: resuming VM again INFO: 0% (786.6 MiB of 2.0 TiB) in 3s, read: 262.2 MiB/s, write: 53.7 MiB/s INFO: 1% (20.5 GiB of 2.0 TiB) in 4m 38s, read: 73.5 MiB/s, write: 47.7 MiB/s zstd: error 25 : Write error : Input/output error (cannot write compressed block) INFO: 1% (24.4 GiB of 2.0 TiB) in 9m 17s, read: 14.2 MiB/s, write: 13.3 MiB/s Warning: unable to close filehandle GEN8894 properly: Input/output error at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 811. ERROR: vma_queue_write: write error - Broken pipe INFO: aborting backup job INFO: resuming VM again ERROR: Backup of VM 100 failed - vma_queue_write: write error - Broken pipe INFO: Failed at 2023-10-04 09:44:04 INFO: Backup job finished with errors TASK ERROR: job errors
 
Input/output errors might imply hardware issues. Check the output of dmesg for any messages about the disk.
 
Does your destination storage have enough free space?
 
We do not officially support Amazon S3 storage. You are probably using something like s3fs-fuse, so it might help to check the logs of that.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom