vzdump fails on sshfs mounted folder

HerrWeltschaft

New Member
Dec 22, 2019
5
0
1
44
hi forum,
i mounted to our pve server via sshfs a remote folder via:

Bash:
sshfs -o reconnect,allow_other,IdentityFile=/root/.ssh/id_rsa -p 222 backupon@78.4x.xxx.xxx:/usr/home/backupon/proxmoxbackup /sshfs;
in pve i set the folder /sshfs as backup folder with vzdump feature.

but the backup of some vm's ( 250 GB + > ) dont work and end up with this:

Bash:
Warning: unable to close filehandle GEN58 properly: Input/output error at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 474.                                                                                                               
ERROR: VM 104 qmp command 'query-backup' failed - got timeout
INFO: aborting backup job
ERROR: Backup of VM 104 failed - VM 104 qmp command 'query-backup' failed - got timeout
INFO: Failed at 2019-12-22 14:24:28
INFO: Backup job finished with errors
job errors
any ideas or hints what causes this? i tried several mount options on sshfs but without luck.
any help would be nice

kind regards
marko horn
 

aaron

Proxmox Staff Member
Staff member
Jun 3, 2019
519
31
28
What version of Proxmox VE are you running?
Code:
pveversion -v
 

HerrWeltschaft

New Member
Dec 22, 2019
5
0
1
44
What version of Proxmox VE are you running?
Code:
pveversion -v
output of pveversion -v:


proxmox-ve: 6.1-2 (running kernel: 5.3.13-1-pve)
pve-manager: 6.1-5 (running version: 6.1-5/9bf06119)
pve-kernel-5.3: 6.1-1
pve-kernel-helper: 6.1-1
pve-kernel-5.0: 6.0-11
pve-kernel-5.3.13-1-pve: 5.3.13-1
pve-kernel-5.0.21-5-pve: 5.0.21-10
pve-kernel-5.0.21-4-pve: 5.0.21-9
ceph-fuse: 12.2.11+dfsg1-2.1+b1
corosync: 3.0.2-pve4
criu: 3.11-3
glusterfs-client: 5.5-3
ifupdown: 0.8.35+pve1
libjs-extjs: 6.0.1-10
libknet1: 1.13-pve1
libpve-access-control: 6.0-5
libpve-apiclient-perl: 3.0-2
libpve-common-perl: 6.0-9
libpve-guest-common-perl: 3.0-3
libpve-http-server-perl: 3.0-3
libpve-storage-perl: 6.1-3
libqb0: 1.0.5-1
libspice-server1: 0.14.2-4~pve6+1
lvm2: 2.03.02-pve3
lxc-pve: 3.2.1-1
lxcfs: 3.0.3-pve60
novnc-pve: 1.1.0-1
proxmox-mini-journalreader: 1.1-1
proxmox-widget-toolkit: 2.1-1
pve-cluster: 6.1-2
pve-container: 3.0-15
pve-docs: 6.1-3
pve-edk2-firmware: 2.20191127-1
pve-firewall: 4.0-9
pve-firmware: 3.0-4
pve-ha-manager: 3.0-8
pve-i18n: 2.0-3
pve-qemu-kvm: 4.1.1-2
pve-xtermjs: 3.13.2-1
qemu-server: 6.1-4
smartmontools: 7.0-pve2
spiceterm: 3.1-1
vncterm: 1.6-1
zfsutils-linux: 0.8.2-pve2
 

HerrWeltschaft

New Member
Dec 22, 2019
5
0
1
44
What version of Proxmox VE are you running?
Code:
pveversion -v
now we give a try with "suspend" and get this:


INFO: status: 89% (468259897344/526133493760), sparse 12% (68030521344), duration 5633, read/write 75/72 MB/s
INFO: status: 90% (473524404224/526133493760), sparse 12% (68093423616), duration 5706, read/write 72/71 MB/s
INFO: status: 91% (478830723072/526133493760), sparse 12% (68188176384), duration 5769, read/write 84/82 MB/s
lzop: Input/output error: <stdout>
INFO: status: 91% (479271583744/526133493760), sparse 12% (68188176384), duration 5773, read/write 110/110 MB/s
Warning: unable to close filehandle GEN72 properly: Input/output error at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 474.
ERROR: vma_queue_write: write error - Broken pipe
INFO: aborting backup job
INFO: resume vm
INFO: guest is online again after 5778 seconds
ERROR: Backup of VM 104 failed - vma_queue_write: write error - Broken pipe
INFO: Failed at 2020-01-08 02:36:35
INFO: Backup job finished with errors

TASK ERROR: job errors
 

aaron

Proxmox Staff Member
Staff member
Jun 3, 2019
519
31
28
Well, you have the same error messages in both:
Code:
 Warning: unable to close filehandle GEN72 properly: Input/output error at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 474.
This looks like the sshfs is not working as expected.

I found an issue on the BorgBackup tracker that sounds very much like your problem. Some people seem to have found a combination of mount options that work. https://github.com/borgbackup/borg/issues/583#issuecomment-173524751
 

HerrWeltschaft

New Member
Dec 22, 2019
5
0
1
44
Well, you have the same error messages in both:
Code:
 Warning: unable to close filehandle GEN72 properly: Input/output error at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 474.
This looks like the sshfs is not working as expected.

I found an issue on the BorgBackup tracker that sounds very much like your problem. Some people seem to have found a combination of mount options that work. https://github.com/borgbackup/borg/issues/583#issuecomment-173524751
hello,
when i set the cache options i get this on backup mode snapshot:

VM 104 qmp command 'change' failed - unable to connect to VM 104 qmp socket - timeout after 600 retries
TASK ERROR: Failed to run vncproxy.
 

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!