Backup VM failed error "vma_queue_write: write error - Broken pipe"

V

vabraham

New Member
May 11, 2020
18
1
3
Need some help to fix this issue below :

172: 2020-05-30 02:00:03 INFO: Starting Backup of VM 172 (qemu)
172: 2020-05-30 02:00:03 INFO: status = running
172: 2020-05-30 02:00:04 INFO: update VM 172: -lock backup
172: 2020-05-30 02:00:04 INFO: VM Name: Jenkinsdev
172: 2020-05-30 02:00:04 INFO: include disk 'scsi0' 'data:vm-172-disk-1' 600G
172: 2020-05-30 02:00:04 INFO: backup mode: snapshot
172: 2020-05-30 02:00:04 INFO: ionice priority: 7
172: 2020-05-30 02:00:04 INFO: creating archive '/MasterBackup/dump/vzdump-qemu-172-2020_05_30-02_00_02.vma.lzo'
172: 2020-05-30 02:00:04 INFO: started backup task '3fe5f738-d166-4a55-ae65-34408f1d474f'
172: 2020-05-30 02:00:07 INFO: status: 0% (73465856/644245094400), sparse 0% (31756288), duration 3, read/write 24/13 MB/s
172: 2020-05-30 02:00:57 INFO: status: 1% (6482034688/644245094400), sparse 0% (730742784), duration 53, read/write 128/114 MB/s
172: 2020-05-30 02:02:15 INFO: status: 2% (12933398528/644245094400), sparse 0% (895664128), duration 131, read/write 82/80 MB/s
172: 2020-05-30 02:03:03 INFO: status: 3% (19356712960/644245094400), sparse 0% (990130176), duration 179, read/write 133/131 MB/s
172: 2020-05-30 02:04:20 INFO: status: 4% (25809125376/644245094400), sparse 0% (1144229888), duration 256, read/write 83/81 MB/s
172: 2020-05-30 02:05:26 INFO: status: 5% (32277790720/644245094400), sparse 0% (1278926848), duration 322, read/write 98/95 MB/s
172: 2020-05-30 02:06:30 INFO: status: 6% (38657261568/644245094400), sparse 0% (1429331968), duration 386, read/write 99/97 MB/s
172: 2020-05-30 02:07:29 INFO: status: 7% (45119700992/644245094400), sparse 0% (1587965952), duration 445, read/write 109/106 MB/s
172: 2020-05-30 02:08:42 INFO: status: 8% (51649183744/644245094400), sparse 0% (1687429120), duration 518, read/write 89/88 MB/s
172: 2020-05-30 02:10:25 INFO: status: 9% (58012008448/644245094400), sparse 0% (1725890560), duration 621, read/write 61/61 MB/s
172: 2020-05-30 02:11:51 INFO: status: 10% (64479625216/644245094400), sparse 0% (1787146240), duration 707, read/write 75/74 MB/s
172: 2020-05-30 02:13:38 INFO: status: 11% (70934462464/644245094400), sparse 0% (1854943232), duration 814, read/write 60/59 MB/s
172: 2020-05-30 02:16:52 INFO: status: 12% (77317144576/644245094400), sparse 0% (2228572160), duration 1008, read/write 32/30 MB/s
172: 2020-05-30 03:46:41 INFO: status: 13% (83760775168/644245094400), sparse 0% (2383495168), duration 6397, read/write 1/1 MB/s
172: 2020-05-30 03:54:50 INFO: status: 14% (90226425856/644245094400), sparse 0% (2514554880), duration 6886, read/write 13/12 MB/s
172: 2020-05-30 04:02:43 INFO: status: 15% (96646463488/644245094400), sparse 0% (2649239552), duration 7359, read/write 13/13 MB/s
172: 2020-05-30 04:11:02 INFO: status: 16% (103081705472/644245094400), sparse 0% (2755014656), duration 7858, read/write 12/12 MB/s
172: 2020-05-30 04:19:43 INFO: status: 17% (109528350720/644245094400), sparse 0% (2974728192), duration 8379, read/write 12/11 MB/s
172: 2020-05-30 04:27:41 INFO: status: 18% (115967393792/644245094400), sparse 0% (3079852032), duration 8857, read/write 13/13 MB/s
172: 2020-05-30 04:36:28 INFO: status: 19% (122410237952/644245094400), sparse 0% (3304427520), duration 9384, read/write 12/11 MB/s
172: 2020-05-30 04:45:02 INFO: status: 20% (128853082112/644245094400), sparse 0% (3434033152), duration 9898, read/write 12/12 MB/s
172: 2020-05-30 04:53:39 INFO: status: 21% (135311130624/644245094400), sparse 0% (3573010432), duration 10415, read/write 12/12 MB/s
172: 2020-05-30 05:01:48 INFO: status: 22% (141738770432/644245094400), sparse 0% (3706630144), duration 10904, read/write 13/12 MB/s
172: 2020-05-30 05:10:17 INFO: status: 23% (148189216768/644245094400), sparse 0% (3850149888), duration 11413, read/write 12/12 MB/s
172: 2020-05-30 05:16:39 INFO: status: 23% (153256067072/644245094400), sparse 0% (3911684096), duration 11795, read/write 13/13 MB/s
172: 2020-05-30 05:16:39 ERROR: vma_queue_write: write error - Broken pipe
172: 2020-05-30 05:16:39 INFO: aborting backup job
172: 2020-05-30 05:16:44 ERROR: Backup of VM 172 failed - vma_queue_write: write error - Broken pipe
 
hi,

maybe you're running out of space? check df -hl
 
current space configuration :
root@lscproxmoxmaster:~# df -hl
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 171M 6.2G 3% /run
/dev/mapper/pve-root 94G 41G 49G 46% /
tmpfs 32G 66M 32G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/sda2 253M 290K 252M 1% /boot/efi
vmpool 1.8T 128K 1.8T 1% /vmpool
/dev/fuse 30M 40K 30M 1% /etc/pve
tmpfs 6.3G 0 6.3G 0% /run/user/0
s3fs 256T 0 256T 0% /MasterBackup
root@lscproxmoxmaster:~#

I am saving the backup on AWS bucket , my space is 256 T.
 
vabraham said:
I am saving the backup on AWS bucket , my space is 256 T.
Click to expand...

in that case maybe your connection is not reliable? that could cause the broken pipe.

can you try taking a local backup to see if it works?
 
oguz said:
in that case maybe your connection is not reliable? that could cause the broken pipe.

can you try taking a local backup to see if it works?
Click to expand...
I tried for others VM , The backup worked without any issue.
100: 2020-06-04 02:00:12 INFO: Starting Backup of VM 100 (qemu)
100: 2020-06-04 02:00:12 INFO: status = running
100: 2020-06-04 02:00:13 INFO: update VM 100: -lock backup
100: 2020-06-04 02:00:13 INFO: VM Name: LSCGATEWAY
100: 2020-06-04 02:00:13 INFO: include disk 'scsi0' 'data:vm-100-disk-1' 50G
100: 2020-06-04 02:00:13 INFO: backup mode: snapshot
100: 2020-06-04 02:00:13 INFO: ionice priority: 7
100: 2020-06-04 02:00:13 INFO: creating archive '/MasterBackup/dump/vzdump-qemu-100-2020_06_04-02_00_05.vma.lzo'
100: 2020-06-04 02:01:54 ERROR: got timeout
100: 2020-06-04 02:01:54 INFO: aborting backup job
100: 2020-06-04 02:09:27 ERROR: Backup of VM 100 failed - got timeout

109: 2020-06-04 02:09:28 INFO: Starting Backup of VM 109 (qemu)
109: 2020-06-04 02:09:28 INFO: status = running
109: 2020-06-04 02:09:28 INFO: update VM 109: -lock backup
109: 2020-06-04 02:09:28 INFO: VM Name: DomainControlerLSC
109: 2020-06-04 02:09:28 INFO: include disk 'scsi0' 'data:vm-109-disk-1' 20G
109: 2020-06-04 02:09:28 INFO: backup mode: snapshot
109: 2020-06-04 02:09:28 INFO: ionice priority: 7
109: 2020-06-04 02:09:29 INFO: creating archive '/MasterBackup/dump/vzdump-qemu-109-2020_06_04-02_09_28.vma.lzo'
109: 2020-06-04 02:09:29 INFO: started backup task '649a01b6-6ba4-4acf-882e-980b296390ed'
109: 2020-06-04 02:09:32 INFO: status: 3% (665780224/21474836480), sparse 1% (416653312), duration 3, read/write 221/83 MB/s
109: 2020-06-04 02:09:35 INFO: status: 4% (878051328/21474836480), sparse 1% (419758080), duration 6, read/write 70/69 MB/s
109: 2020-06-04 02:09:38 INFO: status: 6% (1330380800/21474836480), sparse 2% (434556928), duration 9, read/write 150/145 MB/s
109: 2020-06-04 02:09:41 INFO: status: 8% (1761607680/21474836480), sparse 2% (448561152), duration 12, read/write 143/139 MB/s
109: 2020-06-04 02:09:44 INFO: status: 10% (2153185280/21474836480), sparse 2% (449171456), duration 15, read/write 130/130 MB/s
109: 2020-06-04 02:09:47 INFO: status: 12% (2599944192/21474836480), sparse 2% (487194624), duration 18, read/write 148/136 MB/s
109: 2020-06-04 02:09:50 INFO: status: 13% (2967404544/21474836480), sparse 2% (602394624), duration 21, read/write 122/84 MB/s
109: 2020-06-04 02:09:55 INFO: status: 14% (3026518016/21474836480), sparse 2% (602595328), duration 26, read/write 11/11 MB/s
109: 2020-06-04 02:10:00 INFO: status: 15% (3314548736/21474836480), sparse 2% (603041792), duration 31, read/write 57/57 MB/s
109: 2020-06-04 02:10:03 INFO: status: 17% (3698458624/21474836480), sparse 2% (605155328), duration 34, read/write 127/127 MB/s
109: 2020-06-04 02:10:06 INFO: status: 18% (4036624384/21474836480), sparse 2% (620888064), duration 37, read/write 112/107 MB/s
109: 2020-06-04 02:10:09 INFO: status: 19% (4112777216/21474836480), sparse 2% (634458112), duration 40, read/write 25/20 MB/s
109: 2020-06-04 02:10:12 INFO: status: 21% (4531879936/21474836480), sparse 2% (639131648), duration 43, read/write 139/138 MB/s
109: 2020-06-04 02:10:15 INFO: status: 23% (5071699968/21474836480), sparse 3% (766627840), duration 46, read/write 179/137 MB/s
109: 2020-06-04 02:10:19 INFO: status: 24% (5160239104/21474836480), sparse 3% (766976000), duration 50, read/write 22/22 MB/s
109: 2020-06-04 02:10:30 INFO: status: 25% (5374738432/21474836480), sparse 3% (767258624), duration 61, read/write 19/19 MB/s
109: 2020-06-04 02:10:34 INFO: status: 26% (5686427648/21474836480), sparse 3% (767692800), duration 65, read/write 77/77 MB/s
109: 2020-06-04 02:10:37 INFO: status: 28% (6100746240/21474836480), sparse 3% (768045056), duration 68, read/write 138/137 MB/s
109: 2020-06-04 02:10:40 INFO: status: 30% (6516834304/21474836480), sparse 3% (768610304), duration 71, read/write 138/138 MB/s
109: 2020-06-04 02:10:43 INFO: status: 31% (6739525632/21474836480), sparse 3% (768815104), duration 74, read/write 74/74 MB/s
109: 2020-06-04 02:10:52 INFO: status: 32% (6948388864/21474836480), sparse 3% (768995328), duration 83, read/write 23/23 MB/s
109: 2020-06-04 02:10:55 INFO: status: 34% (7472939008/21474836480), sparse 4% (882950144), duration 86, read/write 174/136 MB/s
109: 2020-06-04 02:10:58 INFO: status: 36% (7860191232/21474836480), sparse 4% (883318784), duration 89, read/write 129/128 MB/s
109: 2020-06-04 02:11:01 INFO: status: 38% (8276410368/21474836480), sparse 4% (883699712), duration 92, read/write 138/138 MB/s
109: 2020-06-04 02:11:04 INFO: status: 40% (8672968704/21474836480), sparse 4% (884408320), duration 95, read/write 132/131 MB/s
109: 2020-06-04 02:11:07 INFO: status: 42% (9080799232/21474836480), sparse 4% (884838400), duration 98, read/write 135/135 MB/s
109: 2020-06-04 02:11:10 INFO: status: 43% (9276620800/21474836480), sparse 4% (1007779840), duration 101, read/write 65/24 MB/s
109: 2020-06-04 02:11:18 INFO: status: 44% (9483714560/21474836480), sparse 4% (1007808512), duration 109, read/write 25/25 MB/s
109: 2020-06-04 02:11:21 INFO: status: 46% (9898098688/21474836480), sparse 4% (1008177152), duration 112, read/write 138/138 MB/s
109: 2020-06-04 02:11:24 INFO: status: 47% (10304749568/21474836480), sparse 4% (1008631808), duration 115, read/write 135/135 MB/s
109: 2020-06-04 02:11:27 INFO: status: 49% (10719068160/21474836480), sparse 4% (1009053696), duration 118, read/write 138/137 MB/s
109: 2020-06-04 02:11:30 INFO: status: 51% (11102978048/21474836480), sparse 4% (1009414144), duration 121, read/write 127/127 MB/s
109: 2020-06-04 02:11:33 INFO: status: 54% (11635130368/21474836480), sparse 5% (1137782784), duration 124, read/write 177/134 MB/s
109: 2020-06-04 02:11:36 INFO: status: 56% (12049448960/21474836480), sparse 5% (1138184192), duration 127, read/write 138/137 MB/s
109: 2020-06-04 02:11:39 INFO: status: 57% (12437159936/21474836480), sparse 5% (1138548736), duration 130, read/write 129/129 MB/s
109: 2020-06-04 02:11:42 INFO: status: 59% (12840075264/21474836480), sparse 5% (1146011648), duration 133, read/write 134/131 MB/s
109: 2020-06-04 02:11:45 INFO: status: 61% (13269008384/21474836480), sparse 5% (1164513280), duration 136, read/write 142/136 MB/s
109: 2020-06-04 02:11:48 INFO: status: 63% (13675397120/21474836480), sparse 6% (1291055104), duration 139, read/write 135/93 MB/s
109: 2020-06-04 02:11:51 INFO: status: 65% (14033616896/21474836480), sparse 6% (1305870336), duration 142, read/write 119/114 MB/s
109: 2020-06-04 02:11:54 INFO: status: 73% (15721299968/21474836480), sparse 13% (2850664448), duration 145, read/write 562/47 MB/s
109: 2020-06-04 02:11:57 INFO: status: 74% (16043409408/21474836480), sparse 13% (2850779136), duration 148, read/write 107/107 MB/s
109: 2020-06-04 02:12:06 INFO: status: 75% (16117989376/21474836480), sparse 13% (2850836480), duration 157, read/write 8/8 MB/s
109: 2020-06-04 02:12:39 INFO: status: 78% (16941907968/21474836480), sparse 16% (3493425152), duration 190, read/write 24/5 MB/s
109: 2020-06-04 02:12:42 INFO: status: 90% (19370475520/21474836480), sparse 27% (5879508992), duration 193, read/write 809/14 MB/s
109: 2020-06-04 02:12:45 INFO: status: 92% (19929104384/21474836480), sparse 27% (5880864768), duration 196, read/write 186/185 MB/s
109: 2020-06-04 02:12:48 INFO: status: 95% (20433928192/21474836480), sparse 27% (5883670528), duration 199, read/write 168/167 MB/s
109: 2020-06-04 02:12:56 INFO: status: 96% (20628373504/21474836480), sparse 27% (5891100672), duration 207, read/write 24/23 MB/s
109: 2020-06-04 02:13:05 INFO: status: 97% (20853751808/21474836480), sparse 27% (5894750208), duration 216, read/write 25/24 MB/s
109: 2020-06-04 02:13:14 INFO: status: 98% (21055406080/21474836480), sparse 27% (5905711104), duration 225, read/write 22/21 MB/s
109: 2020-06-04 02:13:20 INFO: status: 99% (21308571648/21474836480), sparse 27% (5922500608), duration 231, read/write 42/39 MB/s
109: 2020-06-04 02:13:22 INFO: status: 100% (21474836480/21474836480), sparse 27% (5942800384), duration 233, read/write 83/72 MB/s
109: 2020-06-04 02:13:22 INFO: transferred 21474 MB in 233 seconds (92 MB/s)
109: 2020-06-04 02:24:56 INFO: archive file size: 7.29GB
109: 2020-06-04 02:24:59 INFO: Finished Backup of VM 109 (00:15:31)

The command used to mount the Bucket directory is :
s3fs proxmoxbackup -o use_cache=/var/tmp -o allow_other -o uid=1001 -o mp_umask=002 -o multireq_max=5 /MasterBackup
 
vabraham said:
I tried for others VM , The backup worked without any issue.
Click to expand...

but vm 100 failed to backup in the output you sent. are you sure?

can you try a local backup on a local storage with the affected VM (172) so we can be sure if the problem is the bucket or the VM
 
oguz said:
but vm 100 failed to backup in the output you sent. are you sure?

can you try a local backup on a local storage with the affected VM (172) so we can be sure if the problem is the bucket or the VM
Click to expand...

VM size is 600 G , i have no space to backup this VM locally , please check the reason of backup failed below :

INFO: status: 24% (154620657664/644245094400), sparse 0% (5938814976), duration 11779, read/write 13/12 MB/s
lzop: Input/output error: <stdout>
INFO: status: 24% (158265901056/644245094400), sparse 0% (6117949440), duration 12060, read/write 12/12 MB/s
Warning: unable to close filehandle GEN72 properly: Input/output error at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 453.
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
ERROR: Backup of VM 172 failed - vma_queue_write: write error - Broken pipe
INFO: Backup job finished with errors
TASK ERROR: job errors
 
Hello,

The version of s3fs available for installation using the systems package manager does not support files larger than 10GB. It is therefore recommended to compile a version, including the reqired corrections, from the s3fs source code repository. This tutorial will guide you through that process. Please note that even with the source code compiled version of s3fs, there is a maximum file size of 128GB when using s3fs with Scaleway Object Storage.

See https://www.scaleway.com/en/docs/tutorials/object-storage-s3fs/ for the solution.

David.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom