ERROR: vma_queue_write: write error - Broken pipe

Discussion in 'Proxmox VE: Installation and configuration' started by Ivan Gonzalez, Sep 16, 2018.

  1. Ivan Gonzalez

    Ivan Gonzalez Member

    Joined:
    Jan 20, 2014
    Messages:
    68
    Likes Received:
    0
    i get this on only 1 VM while backing up, the others dont happen not sure why


    INFO: starting new backup job: vzdump 110 --mode snapshot --compress lzo --mailnotification always --quiet 1 --mailto ivanjr@nocroom.com --storage nfs-105
    INFO: Starting Backup of VM 110 (qemu)
    INFO: status = running
    INFO: update VM 110: -lock backup
    INFO: VM Name: CID203-PBX
    INFO: include disk 'sata0' 'local-lvm:vm-110-disk-1' 2100G
    INFO: backup mode: snapshot
    INFO: ionice priority: 7
    INFO: creating archive '/mnt/pve/nfs-105/dump/vzdump-qemu-110-2018_09_16-02_00_01.vma.lzo'
    INFO: started backup task '759e97b9-1dcd-410f-a313-86951c65c01d'
    INFO: status: 0% (456130560/2254857830400), sparse 0% (149807104), duration 3, 152/102 MB/s
    INFO: status: 1% (22677291008/2254857830400), sparse 0% (889499648), duration 233, 96/93 MB/s
    INFO: status: 2% (45145063424/2254857830400), sparse 0% (1554042880), duration 465, 96/93 MB/s
    INFO: status: 3% (67730145280/2254857830400), sparse 0% (2498838528), duration 740, 82/78 MB/s
    INFO: status: 4% (90363265024/2254857830400), sparse 0% (3444244480), duration 1015, 82/78 MB/s
    INFO: status: 5% (112789684224/2254857830400), sparse 0% (4272078848), duration 1300, 78/75 MB/s
    INFO: status: 6% (135428964352/2254857830400), sparse 0% (5270265856), duration 1555, 88/84 MB/s
    INFO: status: 7% (157885792256/2254857830400), sparse 0% (6380662784), duration 1796, 93/88 MB/s
    INFO: status: 8% (180540276736/2254857830400), sparse 0% (7352872960), duration 2050, 89/85 MB/s
    INFO: status: 9% (203042717696/2254857830400), sparse 0% (8446140416), duration 2324, 82/78 MB/s
    INFO: status: 10% (225590771712/2254857830400), sparse 0% (9560981504), duration 2568, 92/87 MB/s
    INFO: status: 11% (248078008320/2254857830400), sparse 0% (10660569088), duration 2849, 80/76 MB/s
    INFO: status: 12% (270617411584/2254857830400), sparse 0% (11770339328), duration 3124, 81/77 MB/s
    INFO: status: 13% (293280546816/2254857830400), sparse 0% (13026332672), duration 3383, 87/82 MB/s
    INFO: status: 14% (315763982336/2254857830400), sparse 0% (14308524032), duration 3646, 85/80 MB/s
    INFO: status: 15% (338274025472/2254857830400), sparse 0% (15679827968), duration 3890, 92/86 MB/s
    INFO: status: 16% (360928444416/2254857830400), sparse 0% (16969453568), duration 4131, 94/88 MB/s
    INFO: status: 17% (383430950912/2254857830400), sparse 0% (18351190016), duration 4378, 91/85 MB/s
    INFO: status: 18% (405906784256/2254857830400), sparse 0% (19581796352), duration 4634, 87/82 MB/s
    INFO: status: 19% (428503531520/2254857830400), sparse 0% (20943134720), duration 4898, 85/80 MB/s
    INFO: status: 20% (451086516224/2254857830400), sparse 0% (22199062528), duration 5137, 94/89 MB/s
    INFO: status: 21% (473642172416/2254857830400), sparse 1% (23526105088), duration 5416, 80/76 MB/s
    INFO: status: 22% (496087269376/2254857830400), sparse 1% (24745619456), duration 5658, 92/87 MB/s
    INFO: status: 23% (518736052224/2254857830400), sparse 1% (26194481152), duration 5895, 95/89 MB/s
    INFO: status: 24% (541297999872/2254857830400), sparse 1% (27523837952), duration 6146, 89/84 MB/s
    INFO: status: 25% (563845791744/2254857830400), sparse 1% (28822945792), duration 6396, 90/84 MB/s
    INFO: status: 26% (586410688512/2254857830400), sparse 1% (30131793920), duration 6650, 88/83 MB/s
    INFO: status: 27% (608922894336/2254857830400), sparse 1% (31624286208), duration 6899, 90/84 MB/s
    INFO: status: 28% (631448141824/2254857830400), sparse 1% (33053663232), duration 7136, 95/89 MB/s
    INFO: status: 29% (653930987520/2254857830400), sparse 1% (34393911296), duration 7390, 88/83 MB/s
    INFO: status: 30% (676564762624/2254857830400), sparse 1% (35703549952), duration 7627, 95/89 MB/s
    INFO: status: 31% (699159740416/2254857830400), sparse 1% (37194743808), duration 7862, 96/89 MB/s
    INFO: status: 32% (721583341568/2254857830400), sparse 1% (38620278784), duration 8104, 92/86 MB/s
    INFO: status: 33% (744268234752/2254857830400), sparse 1% (40092438528), duration 8354, 90/84 MB/s
    INFO: status: 34% (766728273920/2254857830400), sparse 1% (41524240384), duration 8623, 83/78 MB/s
    lzop: No space left on device: <stdout>
    INFO: status: 34% (769408696320/2254857830400), sparse 1% (41778450432), duration 8661, 70/63 MB/s
    ERROR: vma_queue_write: write error - Broken pipe
    INFO: aborting backup job
    ERROR: Backup of VM 110 failed - vma_queue_write: write error - Broken pipe
    INFO: Backup job finished with errors
    TASK ERROR: job errors
     
  2. Ivan Gonzalez

    Ivan Gonzalez Member

    Joined:
    Jan 20, 2014
    Messages:
    68
    Likes Received:
    0
    and it has a lot of space left
     
  3. dietmar

    dietmar Proxmox Staff Member
    Staff Member

    Joined:
    Apr 28, 2005
    Messages:
    16,257
    Likes Received:
    276
    But the error message says: "lzop: No space left on device" .
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  4. Ivan Gonzalez

    Ivan Gonzalez Member

    Joined:
    Jan 20, 2014
    Messages:
    68
    Likes Received:
    0
    I know, which is why I'm very confused.
     
  5. sahostking

    sahostking Member

    Joined:
    Oct 6, 2015
    Messages:
    305
    Likes Received:
    6
    Whenever I received that error it was always diskspace related. Either actuall disk space or inodes, etc.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  6. Ivan Gonzalez

    Ivan Gonzalez Member

    Joined:
    Jan 20, 2014
    Messages:
    68
    Likes Received:
    0
    I see, maybe node what you do to fix ?
     
  7. Ivan Gonzalez

    Ivan Gonzalez Member

    Joined:
    Jan 20, 2014
    Messages:
    68
    Likes Received:
    0
    I have all the backups going to NFS
     
  8. sahostking

    sahostking Member

    Joined:
    Oct 6, 2015
    Messages:
    305
    Likes Received:
    6
    You clear the space usually. Have you checked your nfs location for actual diskspace used aswell as your local disks on your node? one of them surely has run out of space.

    try loading on both servers.

    df -i

    and

    df -h

    to see.
     
    Stop hovering to collapse... Click to collapse... Hover to expand... Click to expand...
  9. Ivan Gonzalez

    Ivan Gonzalez Member

    Joined:
    Jan 20, 2014
    Messages:
    68
    Likes Received:
    0
    main proxmox df -i
     
  10. Ivan Gonzalez

    Ivan Gonzalez Member

    Joined:
    Jan 20, 2014
    Messages:
    68
    Likes Received:
    0
  11. Ivan Gonzalez

    Ivan Gonzalez Member

    Joined:
    Jan 20, 2014
    Messages:
    68
    Likes Received:
    0
    bump, still getting this

    INFO: starting new backup job: vzdump 110 --mailto ivanjr@xx --compress lzo --mode snapshot --mailnotification always --storage nfs-105 --quiet 1
    INFO: Starting Backup of VM 110 (qemu)
    INFO: status = running
    INFO: update VM 110: -lock backup
    INFO: VM Name: CID203-PBX
    INFO: include disk 'sata0' 'local-lvm:vm-110-disk-1' 2100G
    INFO: backup mode: snapshot
    INFO: ionice priority: 7
    INFO: creating archive '/mnt/pve/nfs-105/dump/vzdump-qemu-110-2018_11_04-02_00_01.vma.lzo'
    INFO: started backup task 'b03921cf-ecd6-4408-926a-e8e2e503bf68'
    INFO: status: 0% (444727296/2254857830400), sparse 0% (146264064), duration 3, 148/99 MB/s
    INFO: status: 1% (22574661632/2254857830400), sparse 0% (762900480), duration 265, 84/82 MB/s
    INFO: status: 2% (45187334144/2254857830400), sparse 0% (1421217792), duration 519, 89/86 MB/s
    INFO: status: 3% (67737288704/2254857830400), sparse 0% (2347225088), duration 799, 80/77 MB/s
    INFO: status: 4% (90202898432/2254857830400), sparse 0% (3234537472), duration 1067, 83/80 MB/s
    INFO: status: 5% (112785883136/2254857830400), sparse 0% (4081397760), duration 1335, 84/81 MB/s
    INFO: status: 6% (135345340416/2254857830400), sparse 0% (4986425344), duration 1602, 84/81 MB/s
    INFO: status: 7% (157866786816/2254857830400), sparse 0% (5889912832), duration 1870, 84/80 MB/s
    INFO: status: 8% (180417003520/2254857830400), sparse 0% (6817251328), duration 2131, 86/82 MB/s
    INFO: status: 9% (202996908032/2254857830400), sparse 0% (7881228288), duration 2411, 80/76 MB/s
    INFO: status: 10% (225577271296/2254857830400), sparse 0% (8989962240), duration 2670, 87/82 MB/s
    INFO: status: 11% (248130306048/2254857830400), sparse 0% (10083733504), duration 2940, 83/79 MB/s
    INFO: status: 12% (270619312128/2254857830400), sparse 0% (11182759936), duration 3215, 81/77 MB/s
    INFO: status: 13% (293142593536/2254857830400), sparse 0% (12417196032), duration 3463, 90/85 MB/s
    INFO: status: 14% (315725971456/2254857830400), sparse 0% (13701107712), duration 3722, 87/82 MB/s
    INFO: status: 15% (338239815680/2254857830400), sparse 0% (15056183296), duration 3987, 84/79 MB/s
    INFO: status: 16% (360924708864/2254857830400), sparse 0% (16299286528), duration 4256, 84/79 MB/s
    INFO: status: 17% (383453757440/2254857830400), sparse 0% (17685323776), duration 4515, 86/81 MB/s
    INFO: status: 18% (405876375552/2254857830400), sparse 0% (18889469952), duration 4785, 83/78 MB/s
    INFO: status: 19% (428468928512/2254857830400), sparse 0% (20221808640), duration 5040, 88/83 MB/s
    INFO: status: 20% (450972483584/2254857830400), sparse 0% (21437218816), duration 5295, 88/83 MB/s
    INFO: status: 21% (473569951744/2254857830400), sparse 1% (22753460224), duration 5565, 83/78 MB/s
    INFO: status: 22% (496106602496/2254857830400), sparse 1% (23932452864), duration 5824, 87/82 MB/s
    INFO: status: 23% (518716981248/2254857830400), sparse 1% (25340907520), duration 6075, 90/84 MB/s
    INFO: status: 24% (541240721408/2254857830400), sparse 1% (26649223168), duration 6342, 84/79 MB/s
    INFO: status: 25% (563800178688/2254857830400), sparse 1% (27915780096), duration 6602, 86/81 MB/s
    INFO: status: 26% (586354524160/2254857830400), sparse 1% (29179871232), duration 6874, 82/78 MB/s
    INFO: status: 27% (608966344704/2254857830400), sparse 1% (30597656576), duration 7138, 85/80 MB/s
    INFO: status: 28% (631364714496/2254857830400), sparse 1% (31964680192), duration 7394, 87/82 MB/s
    INFO: status: 29% (653942980608/2254857830400), sparse 1% (33277960192), duration 7659, 85/80 MB/s
    INFO: status: 30% (676554014720/2254857830400), sparse 1% (34556461056), duration 7943, 79/75 MB/s
    INFO: status: 31% (699084701696/2254857830400), sparse 1% (35999842304), duration 8205, 85/80 MB/s
    lzop: No space left on device: <stdout>
    INFO: status: 31% (700681224192/2254857830400), sparse 1% (36133847040), duration 8225, 79/73 MB/s
    ERROR: vma_queue_write: write error - Broken pipe
    INFO: aborting backup job
    ERROR: Backup of VM 110 failed - vma_queue_write: write error - Broken pipe
    INFO: Backup job finished with errors
    TASK ERROR: job errors
     
  12. Ivan Gonzalez

    Ivan Gonzalez Member

    Joined:
    Jan 20, 2014
    Messages:
    68
    Likes Received:
    0
    says "lzop: No space left on device: <stdout>" but is on 24TB storage device and all other backups are fine, is just 1 KVM that just wont backup!

    any help will be great
     

    Attached Files:

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice