vzdump fails on sshfs mounted folder

[HerrWeltschaft]

hi forum,
i mounted to our pve server via sshfs a remote folder via:

sshfs -o reconnect,allow_other,IdentityFile=/root/.ssh/id_rsa -p 222 backupon@78.4x.xxx.xxx:/usr/home/backupon/proxmoxbackup /sshfs;

in pve i set the folder /sshfs as backup folder with vzdump feature.

but the backup of some vm's ( 250 GB + > ) dont work and end up with this:

Warning: unable to close filehandle GEN58 properly: Input/output error at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 474.                                                                                                               
ERROR: VM 104 qmp command 'query-backup' failed - got timeout
INFO: aborting backup job
ERROR: Backup of VM 104 failed - VM 104 qmp command 'query-backup' failed - got timeout
INFO: Failed at 2019-12-22 14:24:28
INFO: Backup job finished with errors
job errors

any ideas or hints what causes this? i tried several mount options on sshfs but without luck.
any help would be nice

kind regards
marko horn

 

[aaron]

What version of Proxmox VE are you running?

pveversion -v

 

[HerrWeltschaft]

What version of Proxmox VE are you running?

pveversion -v

output of pveversion -v:


proxmox-ve: 6.1-2 (running kernel: 5.3.13-1-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.0: 6.0-11
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-4-pve: 5.0.21-9
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-15
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2

 

[HerrWeltschaft]

What version of Proxmox VE are you running?

pveversion -v

now we give a try with "suspend" and get this:


INFO: status: 89% (468259897344/526133493760), sparse 12% (68030521344), duration 5633, read/write 75/72 MB/s
INFO: status: 90% (473524404224/526133493760), sparse 12% (68093423616), duration 5706, read/write 72/71 MB/s
INFO: status: 91% (478830723072/526133493760), sparse 12% (68188176384), duration 5769, read/write 84/82 MB/s
lzop: Input/output error: <stdout>
INFO: status: 91% (479271583744/526133493760), sparse 12% (68188176384), duration 5773, read/write 110/110 MB/s
Warning: unable to close filehandle GEN72 properly: Input/output error at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 474.
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
INFO: resume vm
INFO: guest is online again after 5778 seconds
ERROR: Backup of VM 104 failed - vma_queue_write: write error - Broken pipe
INFO: Failed at 2020-01-08 02:36:35
INFO: Backup job finished with errors

TASK ERROR: job errors

 

[aaron]

Well, you have the same error messages in both:

 Warning: unable to close filehandle GEN72 properly: Input/output error at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 474.

This looks like the sshfs is not working as expected.

I found an issue on the BorgBackup tracker that sounds very much like your problem. Some people seem to have found a combination of mount options that work. https://github.com/borgbackup/borg/issues/583#issuecomment-173524751

 

[HerrWeltschaft]

Well, you have the same error messages in both:

 Warning: unable to close filehandle GEN72 properly: Input/output error at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 474.

This looks like the sshfs is not working as expected.

I found an issue on the BorgBackup tracker that sounds very much like your problem. Some people seem to have found a combination of mount options that work. https://github.com/borgbackup/borg/issues/583#issuecomment-173524751

hello,
when i set the cache options i get this on backup mode snapshot:

VM 104 qmp command 'change' failed - unable to connect to VM 104 qmp socket - timeout after 600 retries
TASK ERROR: Failed to run vncproxy.