Why did my vzdump fail to a broken pipe?

[Alvks]

Here is the log from my Proxmox node:

vzdump 100 102 --mailto root --mode snapshot --quiet 1 --compress lzo --mailnotification always --storage vm-backups


100: 2019-06-14 01:00:01 INFO: Starting Backup of VM 100 (qemu)
100: 2019-06-14 01:00:01 INFO: status = running
100: 2019-06-14 01:00:02 INFO: update VM 100: -lock backup
100: 2019-06-14 01:00:02 INFO: VM Name: entei
100: 2019-06-14 01:00:02 INFO: include disk 'sata0' 'hdd1:vm-100-disk-0' 3260G
100: 2019-06-14 01:00:02 INFO: include disk 'sata1' 'local-lvm:vm-100-disk-1' 50G
100: 2019-06-14 01:00:02 INFO: backup mode: snapshot
100: 2019-06-14 01:00:02 INFO: ionice priority: 7
100: 2019-06-14 01:00:02 INFO: pending configuration changes found (not included into backup)
100: 2019-06-14 01:00:02 INFO: creating archive '/srv/pve-vm-backups/dump/vzdump-qemu-100-2019_06_14-01_00_01.vma.lzo'
100: 2019-06-14 01:00:05 INFO: started backup task '7f4a5a6d-9988-42d5-b387-7ebfaf7d7f73'
100: 2019-06-14 01:00:08 INFO: status: 0% (490012672/3554085437440), sparse 0% (72097792), duration 3, read/write 163/139 MB/s
100: 2019-06-14 01:08:25 INFO: status: 1% (35559178240/3554085437440), sparse 0% (6890741760), duration 500, read/write 70/56 MB/s
100: 2019-06-14 01:17:19 INFO: status: 2% (71152828416/3554085437440), sparse 0% (13100220416), duration 1034, read/write 66/55 MB/s
100: 2019-06-14 01:24:26 INFO: status: 3% (106628120576/3554085437440), sparse 0% (15488667648), duration 1461, read/write 83/77 MB/s
100: 2019-06-14 01:29:26 INFO: status: 4% (142217707520/3554085437440), sparse 0% (15488667648), duration 1761, read/write 118/118 MB/s
100: 2019-06-14 01:34:25 INFO: status: 5% (177805459456/3554085437440), sparse 0% (15488667648), duration 2060, read/write 119/119 MB/s
100: 2019-06-14 01:39:25 INFO: status: 6% (213274329088/3554085437440), sparse 0% (18245079040), duration 2360, read/write 118/109 MB/s
100: 2019-06-14 01:47:27 INFO: status: 7% (248861032448/3554085437440), sparse 0% (18245079040), duration 2842, read/write 73/73 MB/s
100: 2019-06-14 01:52:43 INFO: status: 8% (284410642432/3554085437440), sparse 0% (18245079040), duration 3158, read/write 112/112 MB/s
100: 2019-06-14 01:57:55 INFO: status: 9% (319982731264/3554085437440), sparse 0% (18245079040), duration 3470, read/write 114/114 MB/s
100: 2019-06-14 02:03:08 INFO: status: 10% (355420733440/3554085437440), sparse 0% (18616782848), duration 3783, read/write 113/112 MB/s
100: 2019-06-14 02:08:52 INFO: status: 11% (390963331072/3554085437440), sparse 0% (18616782848), duration 4127, read/write 103/103 MB/s
100: 2019-06-14 02:14:40 INFO: status: 12% (426607509504/3554085437440), sparse 0% (18616782848), duration 4475, read/write 102/102 MB/s
100: 2019-06-14 02:20:00 INFO: status: 13% (462048854016/3554085437440), sparse 0% (18616782848), duration 4795, read/write 110/110 MB/s
100: 2019-06-14 02:25:43 INFO: status: 14% (497657446400/3554085437440), sparse 0% (18616782848), duration 5138, read/write 103/103 MB/s
100: 2019-06-14 02:31:21 INFO: status: 15% (533177434112/3554085437440), sparse 0% (18616782848), duration 5476, read/write 105/105 MB/s
100: 2019-06-14 02:37:01 INFO: status: 16% (568745132032/3554085437440), sparse 0% (18616782848), duration 5816, read/write 104/104 MB/s
100: 2019-06-14 02:42:35 INFO: status: 17% (604270362624/3554085437440), sparse 0% (18694754304), duration 6150, read/write 106/106 MB/s
100: 2019-06-14 02:48:08 INFO: status: 18% (639774425088/3554085437440), sparse 0% (18851127296), duration 6483, read/write 106/106 MB/s
100: 2019-06-14 02:53:45 INFO: status: 19% (675343106048/3554085437440), sparse 0% (19007983616), duration 6820, read/write 105/105 MB/s
100: 2019-06-14 02:59:22 INFO: status: 20% (710833733632/3554085437440), sparse 0% (19186581504), duration 7157, read/write 105/104 MB/s
100: 2019-06-14 03:05:04 INFO: status: 21% (746466574336/3554085437440), sparse 0% (19361800192), duration 7499, read/write 104/103 MB/s
100: 2019-06-14 03:10:49 INFO: status: 22% (781940817920/3554085437440), sparse 0% (19535155200), duration 7844, read/write 102/102 MB/s
100: 2019-06-14 03:16:41 INFO: status: 23% (817530404864/3554085437440), sparse 0% (19691941888), duration 8196, read/write 101/100 MB/s
100: 2019-06-14 03:22:32 INFO: status: 24% (852998094848/3554085437440), sparse 0% (19872370688), duration 8547, read/write 101/100 MB/s
100: 2019-06-14 03:28:34 INFO: status: 25% (888608587776/3554085437440), sparse 0% (20043321344), duration 8909, read/write 98/97 MB/s
100: 2019-06-14 03:34:40 INFO: status: 26% (924071100416/3554085437440), sparse 0% (20198240256), duration 9275, read/write 96/96 MB/s
100: 2019-06-14 03:40:53 INFO: status: 27% (959683493888/3554085437440), sparse 0% (20364861440), duration 9648, read/write 95/95 MB/s
100: 2019-06-14 03:47:07 INFO: status: 28% (995209576448/3554085437440), sparse 0% (20528205824), duration 10022, read/write 94/94 MB/s
100: 2019-06-14 03:53:26 INFO: status: 29% (1030693847040/3554085437440), sparse 0% (20682833920), duration 10401, read/write 93/93 MB/s
100: 2019-06-14 03:59:48 INFO: status: 30% (1066230808576/3554085437440), sparse 0% (20847558656), duration 10783, read/write 93/92 MB/s
100: 2019-06-14 04:06:16 INFO: status: 31% (1101847986176/3554085437440), sparse 0% (21001953280), duration 11171, read/write 91/91 MB/s
100: 2019-06-14 04:12:53 INFO: status: 32% (1137334943744/3554085437440), sparse 0% (21157699584), duration 11568, read/write 89/88 MB/s
100: 2019-06-14 04:19:40 INFO: status: 33% (1172890320896/3554085437440), sparse 0% (21320167424), duration 11975, read/write 87/86 MB/s
100: 2019-06-14 04:26:33 INFO: status: 34% (1208458215424/3554085437440), sparse 0% (21484761088), duration 12388, read/write 86/85 MB/s
100: 2019-06-14 04:33:40 INFO: status: 35% (1243994783744/3554085437440), sparse 0% (21640769536), duration 12815, read/write 83/82 MB/s
100: 2019-06-14 04:40:48 INFO: status: 36% (1279533645824/3554085437440), sparse 0% (21808959488), duration 13243, read/write 83/82 MB/s
100: 2019-06-14 04:48:55 INFO: status: 37% (1315028205568/3554085437440), sparse 0% (21970759680), duration 13730, read/write 72/72 MB/s
100: 2019-06-14 04:55:58 INFO: status: 38% (1350556975104/3554085437440), sparse 0% (22142111744), duration 14153, read/write 83/83 MB/s
100: 2019-06-14 05:01:08 INFO: status: 38% (1375579537408/3554085437440), sparse 0% (22251888640), duration 14463, read/write 80/80 MB/s
100: 2019-06-14 05:01:08 ERROR: vma_queue_write: write error - Broken pipe
100: 2019-06-14 05:01:08 INFO: aborting backup job
100: 2019-06-14 05:01:35 ERROR: Backup of VM 100 failed - vma_queue_write: write error - Broken pipe

I have a hunch though but just want confirmation. One of the disks used by the VM is twice as large as the external hard drive I am using for backups.

100: 2019-06-14 01:00:02 INFO: include disk 'sata0' 'hdd1:vm-100-disk-0' 3260G

That is a nearly 4TB disk that I was hoping could be compressed and backed up to a 2TB external hard drive. Looks like the backup job aborted after it ran out of space. Thoughts? I was hoping LZO or GZIP would do a better job compressing these snapshots but I guess I was wrong!

 

[Alvks]

Thanks mods! Bumping this for exposure.

 

[Faris Raouf]

I don't think you'll get 2:1 compression unless your disk image only contains plain text. There are too many things that are pre-compressed, at least to an extent. e.g. videos and images. Sure, plain text might be 2:1 or better. Having said that, if it fails at 38% I'm not sure what kind of compression you'r getting.

 

[Alvks]

I don't think you'll get 2:1 compression unless your disk image only contains plain text. There are too many things that are pre-compressed, at least to an extent. e.g. videos and images. Sure, plain text might be 2:1 or better. Having said that, if it fails at 38% I'm not sure what kind of compression you'r getting.

Honestly, that’s okay. The disk has a bunch of media on it but I have that all copied to an external that is passed through to the VM. How can I tell vzdump to ignore that 4TB disk and just backup the boot disk?

 

[sb-jw]

How can I tell vzdump to ignore that 4TB disk and just backup the boot disk?

If you assigned this as an virtual disk, then you can enable the checkbox "no backup". But if it is pass-through, I think that's not possible, but you can backup your VM from inside with Veeam for free and Incrementel.

 

[Alvks]

If you assigned this as an virtual disk, then you can enable the checkbox "no backup". But if it is pass-through, I think that's not possible, but you can backup your VM from inside with Veeam for free and Incrementel.

Thanks for the suggestion! Another question. For snapshot backups through Proxmox, which compression method do you recommend? LZO or GZIP? GZIP seems to compress a tad bit more than LZO.

 

[oguz]

For snapshot backups through Proxmox, which compression method do you recommend? LZO or GZIP? GZIP seems to compress a tad bit more than LZO.

depends on what you want. lzo is a fast compressor/decompressor but achieves less compression, gzip is slower but you get more compression.

 

[alexskysilk]

where is
/srv/pve-vm-backups
and what type of mount is it?

a broken pipe is pretty self explanatory. your target storage disappeared while you were backing up.