Why did my vzdump fail to a broken pipe?

Discussion in 'Proxmox VE: Installation and configuration' started by Alvks, Jun 14, 2019.

  1. Alvks

    Alvks New Member

    Joined:
    Jun 1, 2019
    Messages:
    17
    Likes Received:
    0
    Here is the log from my Proxmox node:

    Code:
    vzdump 100 102 --mailto root --mode snapshot --quiet 1 --compress lzo --mailnotification always --storage vm-backups
    
    
    100: 2019-06-14 01:00:01 INFO: Starting Backup of VM 100 (qemu)
    100: 2019-06-14 01:00:01 INFO: status = running
    100: 2019-06-14 01:00:02 INFO: update VM 100: -lock backup
    100: 2019-06-14 01:00:02 INFO: VM Name: entei
    100: 2019-06-14 01:00:02 INFO: include disk 'sata0' 'hdd1:vm-100-disk-0' 3260G
    100: 2019-06-14 01:00:02 INFO: include disk 'sata1' 'local-lvm:vm-100-disk-1' 50G
    100: 2019-06-14 01:00:02 INFO: backup mode: snapshot
    100: 2019-06-14 01:00:02 INFO: ionice priority: 7
    100: 2019-06-14 01:00:02 INFO: pending configuration changes found (not included into backup)
    100: 2019-06-14 01:00:02 INFO: creating archive '/srv/pve-vm-backups/dump/vzdump-qemu-100-2019_06_14-01_00_01.vma.lzo'
    100: 2019-06-14 01:00:05 INFO: started backup task '7f4a5a6d-9988-42d5-b387-7ebfaf7d7f73'
    100: 2019-06-14 01:00:08 INFO: status: 0% (490012672/3554085437440), sparse 0% (72097792), duration 3, read/write 163/139 MB/s
    100: 2019-06-14 01:08:25 INFO: status: 1% (35559178240/3554085437440), sparse 0% (6890741760), duration 500, read/write 70/56 MB/s
    100: 2019-06-14 01:17:19 INFO: status: 2% (71152828416/3554085437440), sparse 0% (13100220416), duration 1034, read/write 66/55 MB/s
    100: 2019-06-14 01:24:26 INFO: status: 3% (106628120576/3554085437440), sparse 0% (15488667648), duration 1461, read/write 83/77 MB/s
    100: 2019-06-14 01:29:26 INFO: status: 4% (142217707520/3554085437440), sparse 0% (15488667648), duration 1761, read/write 118/118 MB/s
    100: 2019-06-14 01:34:25 INFO: status: 5% (177805459456/3554085437440), sparse 0% (15488667648), duration 2060, read/write 119/119 MB/s
    100: 2019-06-14 01:39:25 INFO: status: 6% (213274329088/3554085437440), sparse 0% (18245079040), duration 2360, read/write 118/109 MB/s
    100: 2019-06-14 01:47:27 INFO: status: 7% (248861032448/3554085437440), sparse 0% (18245079040), duration 2842, read/write 73/73 MB/s
    100: 2019-06-14 01:52:43 INFO: status: 8% (284410642432/3554085437440), sparse 0% (18245079040), duration 3158, read/write 112/112 MB/s
    100: 2019-06-14 01:57:55 INFO: status: 9% (319982731264/3554085437440), sparse 0% (18245079040), duration 3470, read/write 114/114 MB/s
    100: 2019-06-14 02:03:08 INFO: status: 10% (355420733440/3554085437440), sparse 0% (18616782848), duration 3783, read/write 113/112 MB/s
    100: 2019-06-14 02:08:52 INFO: status: 11% (390963331072/3554085437440), sparse 0% (18616782848), duration 4127, read/write 103/103 MB/s
    100: 2019-06-14 02:14:40 INFO: status: 12% (426607509504/3554085437440), sparse 0% (18616782848), duration 4475, read/write 102/102 MB/s
    100: 2019-06-14 02:20:00 INFO: status: 13% (462048854016/3554085437440), sparse 0% (18616782848), duration 4795, read/write 110/110 MB/s
    100: 2019-06-14 02:25:43 INFO: status: 14% (497657446400/3554085437440), sparse 0% (18616782848), duration 5138, read/write 103/103 MB/s
    100: 2019-06-14 02:31:21 INFO: status: 15% (533177434112/3554085437440), sparse 0% (18616782848), duration 5476, read/write 105/105 MB/s
    100: 2019-06-14 02:37:01 INFO: status: 16% (568745132032/3554085437440), sparse 0% (18616782848), duration 5816, read/write 104/104 MB/s
    100: 2019-06-14 02:42:35 INFO: status: 17% (604270362624/3554085437440), sparse 0% (18694754304), duration 6150, read/write 106/106 MB/s
    100: 2019-06-14 02:48:08 INFO: status: 18% (639774425088/3554085437440), sparse 0% (18851127296), duration 6483, read/write 106/106 MB/s
    100: 2019-06-14 02:53:45 INFO: status: 19% (675343106048/3554085437440), sparse 0% (19007983616), duration 6820, read/write 105/105 MB/s
    100: 2019-06-14 02:59:22 INFO: status: 20% (710833733632/3554085437440), sparse 0% (19186581504), duration 7157, read/write 105/104 MB/s
    100: 2019-06-14 03:05:04 INFO: status: 21% (746466574336/3554085437440), sparse 0% (19361800192), duration 7499, read/write 104/103 MB/s
    100: 2019-06-14 03:10:49 INFO: status: 22% (781940817920/3554085437440), sparse 0% (19535155200), duration 7844, read/write 102/102 MB/s
    100: 2019-06-14 03:16:41 INFO: status: 23% (817530404864/3554085437440), sparse 0% (19691941888), duration 8196, read/write 101/100 MB/s
    100: 2019-06-14 03:22:32 INFO: status: 24% (852998094848/3554085437440), sparse 0% (19872370688), duration 8547, read/write 101/100 MB/s
    100: 2019-06-14 03:28:34 INFO: status: 25% (888608587776/3554085437440), sparse 0% (20043321344), duration 8909, read/write 98/97 MB/s
    100: 2019-06-14 03:34:40 INFO: status: 26% (924071100416/3554085437440), sparse 0% (20198240256), duration 9275, read/write 96/96 MB/s
    100: 2019-06-14 03:40:53 INFO: status: 27% (959683493888/3554085437440), sparse 0% (20364861440), duration 9648, read/write 95/95 MB/s
    100: 2019-06-14 03:47:07 INFO: status: 28% (995209576448/3554085437440), sparse 0% (20528205824), duration 10022, read/write 94/94 MB/s
    100: 2019-06-14 03:53:26 INFO: status: 29% (1030693847040/3554085437440), sparse 0% (20682833920), duration 10401, read/write 93/93 MB/s
    100: 2019-06-14 03:59:48 INFO: status: 30% (1066230808576/3554085437440), sparse 0% (20847558656), duration 10783, read/write 93/92 MB/s
    100: 2019-06-14 04:06:16 INFO: status: 31% (1101847986176/3554085437440), sparse 0% (21001953280), duration 11171, read/write 91/91 MB/s
    100: 2019-06-14 04:12:53 INFO: status: 32% (1137334943744/3554085437440), sparse 0% (21157699584), duration 11568, read/write 89/88 MB/s
    100: 2019-06-14 04:19:40 INFO: status: 33% (1172890320896/3554085437440), sparse 0% (21320167424), duration 11975, read/write 87/86 MB/s
    100: 2019-06-14 04:26:33 INFO: status: 34% (1208458215424/3554085437440), sparse 0% (21484761088), duration 12388, read/write 86/85 MB/s
    100: 2019-06-14 04:33:40 INFO: status: 35% (1243994783744/3554085437440), sparse 0% (21640769536), duration 12815, read/write 83/82 MB/s
    100: 2019-06-14 04:40:48 INFO: status: 36% (1279533645824/3554085437440), sparse 0% (21808959488), duration 13243, read/write 83/82 MB/s
    100: 2019-06-14 04:48:55 INFO: status: 37% (1315028205568/3554085437440), sparse 0% (21970759680), duration 13730, read/write 72/72 MB/s
    100: 2019-06-14 04:55:58 INFO: status: 38% (1350556975104/3554085437440), sparse 0% (22142111744), duration 14153, read/write 83/83 MB/s
    100: 2019-06-14 05:01:08 INFO: status: 38% (1375579537408/3554085437440), sparse 0% (22251888640), duration 14463, read/write 80/80 MB/s
    100: 2019-06-14 05:01:08 ERROR: vma_queue_write: write error - Broken pipe
    100: 2019-06-14 05:01:08 INFO: aborting backup job
    100: 2019-06-14 05:01:35 ERROR: Backup of VM 100 failed - vma_queue_write: write error - Broken pipe
    I have a hunch though but just want confirmation. One of the disks used by the VM is twice as large as the external hard drive I am using for backups.

    100: 2019-06-14 01:00:02 INFO: include disk 'sata0' 'hdd1:vm-100-disk-0' 3260G

    That is a nearly 4TB disk that I was hoping could be compressed and backed up to a 2TB external hard drive. Looks like the backup job aborted after it ran out of space. Thoughts? I was hoping LZO or GZIP would do a better job compressing these snapshots but I guess I was wrong!
     
  2. Alvks

    Alvks New Member

    Joined:
    Jun 1, 2019
    Messages:
    17
    Likes Received:
    0
    Thanks mods! Bumping this for exposure.
     
  3. Faris Raouf

    Faris Raouf Member

    Joined:
    Mar 19, 2018
    Messages:
    95
    Likes Received:
    7
    I don't think you'll get 2:1 compression unless your disk image only contains plain text. There are too many things that are pre-compressed, at least to an extent. e.g. videos and images. Sure, plain text might be 2:1 or better. Having said that, if it fails at 38% I'm not sure what kind of compression you'r getting.
     
  4. Alvks

    Alvks New Member

    Joined:
    Jun 1, 2019
    Messages:
    17
    Likes Received:
    0
    Honestly, that’s okay. The disk has a bunch of media on it but I have that all copied to an external that is passed through to the VM. How can I tell vzdump to ignore that 4TB disk and just backup the boot disk?
     
  5. sb-jw

    sb-jw Active Member

    Joined:
    Jan 23, 2018
    Messages:
    547
    Likes Received:
    47
    If you assigned this as an virtual disk, then you can enable the checkbox "no backup". But if it is pass-through, I think that's not possible, but you can backup your VM from inside with Veeam for free and Incrementel.
     
  6. Alvks

    Alvks New Member

    Joined:
    Jun 1, 2019
    Messages:
    17
    Likes Received:
    0
    Thanks for the suggestion! Another question. For snapshot backups through Proxmox, which compression method do you recommend? LZO or GZIP? GZIP seems to compress a tad bit more than LZO.
     
  7. oguz

    oguz Proxmox Staff Member
    Staff Member

    Joined:
    Nov 19, 2018
    Messages:
    584
    Likes Received:
    62
    depends on what you want. lzo is a fast compressor/decompressor but achieves less compression, gzip is slower but you get more compression.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
    Alvks likes this.
  8. alexskysilk

    alexskysilk Active Member

    Joined:
    Oct 16, 2015
    Messages:
    559
    Likes Received:
    59
    where is
    /srv/pve-vm-backups
    and what type of mount is it?

    a broken pipe is pretty self explanatory. your target storage disappeared while you were backing up.
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice