Backup of VM 102 failed - vma_queue_write: write error - Broken pipe

[sgala]

Hello,

I'm running proxmox 4.1 and I'm experiencing since the beginning of the setup of this system an issue every night during the backups.

Randomly but every day, some backups fail, i have only 3VM and I'm backupping to a remote NFS.

Here the last day:
INFO: starting new backup job: vzdump 102 --mailto 'mail@mail...' --compress lzo --storage dataset1_proxmoxbackup1 --mode snapshot --quiet 1 --mailnotification always
INFO: Starting Backup of VM 102 (qemu)
INFO: status = running
INFO: update VM 102: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/pve/dataset1_proxmoxbackup1/dump/vzdump-qemu-102-2016_07_26-23_00_02.vma.lzo'
INFO: started backup task 'd1c0fcb1-2827-4c4b-beb9-0b95867c2a79'
INFO: status: 0% (304939008/322122547200), sparse 0% (272789504), duration 4, 76/8 MB/s
lzop: Stale file handle: <stdout>
INFO: status: 0% (1638727680/322122547200), sparse 0% (632692736), duration 32, 47/34 MB/s
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
ERROR: Backup of VM 102 failed - vma_queue_write: write error - Broken pipe
INFO: Backup job finished with errors
TASK ERROR: job errors


cat /etc/pve/storage.cfg
dir: local
path /var/lib/vz
maxfiles 0
content vztmpl,images,backup,iso,rootdir

zfspool: zfspool1
pool zfspool1
content rootdir,images
sparse

dir: backup
path /zfspool1/backups
maxfiles 1
content backup,images

nfs: dataset1_proxmoxbackup1
server 192.168.0.75
export /dataset1_proxmoxbackup1
path /mnt/pve/dataset1_proxmoxbackup1
options vers=3
content backup
maxfiles 8

I have free space:
192.168.0.75:/dataset1_proxmoxbackup1 7.3T 4.1T 3.3T 56% /mnt/pve/dataset1_proxmoxbackup1

No logs error found. The NFS / network connectivity is local and good. I tried to divide in multiple job, delete/create jobs, change start date, nothing, randomly some job fail with that error...

Any hint?

Thanks

Matteo

 

[dietmar]

Maybe a permissions problem - is root allowed to write to the NFS share?

 

[sgala]

Maybe a permissions problem - is root allowed to write to the NFS share?

The permissions are ok, sometimes the jobs are completed successfully so is not a "permanent" problem i think... should not be a random problem if are there permission issues, right?

 

[dietmar]

should not be a random problem if are there permission issues, right?

right. Maybe a problem with the NFS server then - maybe high network load the those errors occur?

 

[sgala]

right. Maybe a problem with the NFS server then - maybe high network load the those errors occur?

the NFS server is used only by PVE for this backups, so the whole traffic is only of the current backup that is running (so the traffic is near 0 or "something" during the backups)
I can't see any logs/server/issues on the NFS server.
Can I check something particular fot debug/detect about this error? thanks!

M.

 

[ioanv]

Hi everybody

Having the exact same problem. As you can see only VM 112 fails. The next day another machine fails or maybe none will fail.

110cassiopeaOK00:06:154.22GB/mnt/pve/prox4bkp_vault/dump/vzdump-qemu-110-2016_08_23-23_00_01.vma.lzo
112volansFAILED00:29:26vma_queue_write: write error - Broken pipe113octansOK00:10:474.28GB/mnt/pve/prox4bkp_vault/dump/vzdump-qemu-113-2016_08_23-23_36_00.vma.lzo
114crux-el6.6-masterOK00:03:422.81GB/mnt/pve/prox4bkp_vault/dump/vzdump-qemu-114-2016_08_23-23_46_50.vma.lzo
115coronaOK00:07:065.57GB/mnt/pve/prox4bkp_vault/dump/vzdump-qemu-115-2016_08_23-23_50_39.vma.lzo
116cassiopea2OK00:03:255.76GB/mnt/pve/prox4bkp_vault/dump/vzdump-qemu-116-2016_08_23-23_57_45.vma.lzo
118octans2OK00:02:085.35GB/mnt/pve/prox4bkp_vault/dump/vzdump-qemu-118-2016_08_24-00_01_10.vma.lzo

TOTAL 01:03:18 27.99GB

and the same on a different node:

108 aquila FAILED 00:37:02 vma_queue_write: write error - Broken pipe
109 canes OK 00:19:39 10.82GB /mnt/pve/prox4bkp_vault/dump/vzdump-qemu-109-2016_08_23-23_37_04.vma.lzo
111 leo OK 00:13:03 4.44GB /mnt/pve/prox4bkp_vault/dump/vzdump-qemu-111-2016_08_23-23_56_43.vma.lzo
119 aquila2 OK 00:53:17 5.67GB /mnt/pve/prox4bkp_vault/dump/vzdump-qemu-119-2016_08_24-00_09_46.vma.lzo
120 leo2 OK 00:31:19 5.99GB /mnt/pve/prox4bkp_vault/dump/vzdump-qemu-120-2016_08_24-01_03_03.vma.lzo
121 canes2 OK 00:59:21 7.28GB /mnt/pve/prox4bkp_vault/dump/vzdump-qemu-121-2016_08_24-01_34_22.vma.lzo

any ideas? thanks
Vlad

 

[Shotgun Teddy]

The first time I attempted a backup to USB stick I attempted the compression method and i failed in the identical manner. Afterwards I deleted the screwed backup and did a backup with no compression and the backup completed successfully.

 

[sgala]

I did many tests.
I have a backup host where is running BackupPC and also is the target of the data of vzdump mounted via NFS.
ALL times that vzdump try to backup on that target and BackupPC is also running, vzdump fail.
But the backup server is not "freezed/blocked". No problems on the backup server/target of vzdump, is only a bit slow as it have lot of I/O(the io wait is like 50% of average).
This situation is quite normal and should not crash vzdump!
Any hints?

Thanks

M.