I have a strange issue while trying to backup one of my proxmox servers. Depending on how the backup is started I will get the error message:
ERROR: vma_queue_write: write error - Invalid argument
Backup target is a cifs (autofs) mounted filesystems in another data center. If I try to backup anything (like the backup scheduler does) the backup fails (see log below) but if I run a single backup either from the gui or command line the backup job runs fine (the second log). Even more strange, another proxmox server in the same data center with the same backup target running the same proxmox version (but without ZFS storage) work fine all the time.
------- trying to backup all nodes at once -------
vzdump --mailnotification failure --all 1 --storage vmfs-backup --mode snapshot --mailto bitbucket@example.com
INFO: starting new backup job: vzdump --mode snapshot --storage vmfs-backup --mailnotification failure --mailto bitbucket@example.com --all 1
INFO: Starting Backup of VM 6201 (qemu)
INFO: status = running
INFO: update VM 6201: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/vmfs-backup/dump/vzdump-qemu-6201-2016_02_01-11_28_11.vma'
INFO: started backup task '429b612b-bb90-4d4a-a4ca-110c50b093a3'
INFO: status: 0% (3801088/68719476736), sparse 0% (995328), duration 1, 3/2 MB/s
ERROR: vma_queue_write: write error - Invalid argument
INFO: aborting backup job
ERROR: Backup of VM 6201 failed - vma_queue_write: write error - Invalid argument
INFO: Starting Backup of VM 7206 (qemu)
INFO: status = running
INFO: update VM 7206: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/vmfs-backup/dump/vzdump-qemu-7206-2016_02_01-11_28_13.vma'
INFO: started backup task 'bd47a050-8339-4891-bac3-01da623e1983'
INFO: status: 0% (3801088/68719476736), sparse 0% (995328), duration 1, 3/2 MB/s
ERROR: vma_queue_write: write error - Invalid argument
INFO: aborting backup job
ERROR: Backup of VM 7206 failed - vma_queue_write: write error - Invalid argument
INFO: Starting Backup of VM 7219 (qemu)
INFO: status = running
INFO: update VM 7219: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/vmfs-backup/dump/vzdump-qemu-7219-2016_02_01-11_28_15.vma'
INFO: started backup task '37ff81dd-00f3-481c-a42d-14537b65b905'
INFO: status: 0% (3801088/34359738368), sparse 0% (479232), duration 0, 3/3 MB/s
ERROR: vma_queue_write: write error - Invalid argument
INFO: aborting backup job
ERROR: Backup of VM 7219 failed - vma_queue_write: write error - Invalid argument
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp24217 for temporary files
INFO: Starting Backup of VM 6209 (lxc)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/mnt/vmfs-backup/dump/vzdump-lxc-6209-2016_02_01-11_28_16.tar'
INFO: Total bytes written: 2685583360 (2.6GiB, 119MiB/s)
INFO: archive file size: 2.50GB
INFO: delete old backup '/mnt/vmfs-backup/dump/vzdump-lxc-6209-2016_01_30-11_04_33.tar.lzo'
INFO: Finished Backup of VM 6209 (00:02:25)
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp24217 for temporary files
INFO: Starting Backup of VM 7211 (lxc)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot snapshot
INFO: creating archive '/mnt/vmfs-backup/dump/vzdump-lxc-7211-2016_02_01-11_30_41.tar'
INFO: Total bytes written: 2236682240 (2.1GiB, 86MiB/s)
INFO: archive file size: 2.08GB
INFO: delete old backup '/mnt/vmfs-backup/dump/vzdump-lxc-7211-2016_01_30-10_58_10.tar.lzo'
INFO: remove vzdump snapshot
INFO: Finished Backup of VM 7211 (00:02:24)
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp24217 for temporary files
INFO: Starting Backup of VM 7216 (lxc)
INFO: status = running
INFO: found old vzdump snapshot (force removal)
ERROR: Backup of VM 7216 failed - VM is locked (snapshot)
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp24217 for temporary files
INFO: Starting Backup of VM 7217 (lxc)
INFO: status = running
INFO: found old vzdump snapshot (force removal)
ERROR: Backup of VM 7217 failed - VM is locked (snapshot)
INFO: Backup job finished with errors
job errors
------- this is just one node -------
root@pobbel:~# vzdump 6201 --remove 0 --mode snapshot --compress lzo --node pobbel --storage vmfs-backup
INFO: starting new backup job: vzdump 6201 --remove 0 --storage vmfs-backup --node pobbel --compress lzo --mode snapshot
INFO: Starting Backup of VM 6201 (qemu)
INFO: status = running
INFO: update VM 6201: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/vmfs-backup/dump/vzdump-qemu-6201-2016_02_01-11_51_24.vma.lzo'
INFO: started backup task 'f8edcedf-79b3-4fbe-b09c-f0a9a89198b1'
INFO: status: 4% (2755788800/68719476736), sparse 3% (2287820800), duration 3, 918/155 MB/s
INFO: status: 7% (4849598464/68719476736), sparse 5% (3933478912), duration 6, 697/149 MB/s
INFO: status: 11% (7590707200/68719476736), sparse 9% (6441545728), duration 9, 913/77 MB/s
INFO: status: 15% (10865147904/68719476736), sparse 13% (9438945280), duration 12, 1091/92 MB/s
INFO: status: 20% (14258143232/68719476736), sparse 18% (12777816064), duration 15, 1130/18 MB/s
INFO: status: 24% (16527130624/68719476736), sparse 20% (14396887040), duration 18, 756/216 MB/s
INFO: status: 26% (18009554944/68719476736), sparse 21% (14595854336), duration 21, 494/427 MB/s
INFO: status: 28% (19746652160/68719476736), sparse 21% (14817193984), duration 24, 579/505 MB/s
INFO: status: 31% (21362966528/68719476736), sparse 21% (14890201088), duration 27, 538/514 MB/s
INFO: status: 33% (22981378048/68719476736), sparse 22% (15222378496), duration 30, 539/428 MB/s
INFO: status: 35% (24352456704/68719476736), sparse 22% (15500443648), duration 33, 457/364 MB/s
INFO: status: 37% (25729564672/68719476736), sparse 22% (15537704960), duration 36, 459/446 MB/s
INFO: status: 39% (27373469696/68719476736), sparse 22% (15751028736), duration 39, 547/476 MB/s
INFO: status: 42% (29044113408/68719476736), sparse 23% (15973728256), duration 42, 556/482 MB/s
INFO: status: 45% (31217025024/68719476736), sparse 24% (16854978560), duration 45, 724/430 MB/s
INFO: status: 49% (34148974592/68719476736), sparse 28% (19362459648), duration 48, 977/141 MB/s
INFO: status: 50% (35042230272/68719476736), sparse 28% (19560448000), duration 51, 297/231 MB/s
INFO: status: 51% (35582902272/68719476736), sparse 28% (19571449856), duration 54, 180/176 MB/s
INFO: status: 52% (36136943616/68719476736), sparse 28% (19579826176), duration 57, 184/181 MB/s
INFO: status: 53% (36840144896/68719476736), sparse 28% (19713064960), duration 60, 234/189 MB/s
INFO: status: 54% (37512937472/68719476736), sparse 28% (19730862080), duration 63, 224/218 MB/s
INFO: status: 55% (38193332224/68719476736), sparse 28% (19747803136), duration 66, 226/221 MB/s
INFO: status: 56% (38972555264/68719476736), sparse 28% (19886342144), duration 69, 259/213 MB/s
INFO: status: 57% (39679557632/68719476736), sparse 28% (19901652992), duration 72, 235/230 MB/s
INFO: status: 58% (40405565440/68719476736), sparse 28% (19908349952), duration 75, 242/239 MB/s
INFO: status: 59% (41230401536/68719476736), sparse 29% (20061188096), duration 78, 274/223 MB/s
INFO: status: 61% (42024828928/68719476736), sparse 29% (20121309184), duration 81, 264/244 MB/s
INFO: status: 62% (42895278080/68719476736), sparse 29% (20261969920), duration 84, 290/243 MB/s
INFO: status: 63% (43894964224/68719476736), sparse 30% (20650815488), duration 87, 333/203 MB/s
INFO: status: 64% (44522143744/68719476736), sparse 30% (20650815488), duration 90, 209/209 MB/s
INFO: status: 65% (45325090816/68719476736), sparse 30% (20835684352), duration 93, 267/206 MB/s
INFO: status: 71% (48989405184/68719476736), sparse 35% (24388026368), duration 96, 1221/37 MB/s
INFO: status: 77% (53123678208/68719476736), sparse 41% (28391985152), duration 99, 1378/43 MB/s
INFO: status: 84% (58337787904/68719476736), sparse 48% (33606082560), duration 102, 1738/0 MB/s
INFO: status: 92% (63446319104/68719476736), sparse 56% (38704201728), duration 105, 1702/3 MB/s
INFO: status: 97% (67032186880/68719476736), sparse 61% (42054598656), duration 108, 1195/78 MB/s
INFO: status: 98% (68009066496/68719476736), sparse 61% (42174369792), duration 111, 325/285 MB/s
INFO: status: 100% (68719476736/68719476736), sparse 61% (42295078912), duration 114, 236/196 MB/s
INFO: transferred 68719 MB in 114 seconds (602 MB/s)
INFO: archive file size: 7.72GB
INFO: Finished Backup of VM 6201 (00:07:52)
INFO: Backup job finished successfully
pveversion -v
proxmox-ve: 4.1-34 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-31
qemu-server: 4.0-49
pve-firmware: 1.1-7
libpve-common-perl: 4.0-45
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-3
pve-container: 1.0-39
pve-firewall: 2.0-15
pve-ha-manager: 1.0-19
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-6
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
openvswitch-switch: 2.3.2-2
ERROR: vma_queue_write: write error - Invalid argument
Backup target is a cifs (autofs) mounted filesystems in another data center. If I try to backup anything (like the backup scheduler does) the backup fails (see log below) but if I run a single backup either from the gui or command line the backup job runs fine (the second log). Even more strange, another proxmox server in the same data center with the same backup target running the same proxmox version (but without ZFS storage) work fine all the time.
------- trying to backup all nodes at once -------
vzdump --mailnotification failure --all 1 --storage vmfs-backup --mode snapshot --mailto bitbucket@example.com
INFO: starting new backup job: vzdump --mode snapshot --storage vmfs-backup --mailnotification failure --mailto bitbucket@example.com --all 1
INFO: Starting Backup of VM 6201 (qemu)
INFO: status = running
INFO: update VM 6201: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/vmfs-backup/dump/vzdump-qemu-6201-2016_02_01-11_28_11.vma'
INFO: started backup task '429b612b-bb90-4d4a-a4ca-110c50b093a3'
INFO: status: 0% (3801088/68719476736), sparse 0% (995328), duration 1, 3/2 MB/s
ERROR: vma_queue_write: write error - Invalid argument
INFO: aborting backup job
ERROR: Backup of VM 6201 failed - vma_queue_write: write error - Invalid argument
INFO: Starting Backup of VM 7206 (qemu)
INFO: status = running
INFO: update VM 7206: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/vmfs-backup/dump/vzdump-qemu-7206-2016_02_01-11_28_13.vma'
INFO: started backup task 'bd47a050-8339-4891-bac3-01da623e1983'
INFO: status: 0% (3801088/68719476736), sparse 0% (995328), duration 1, 3/2 MB/s
ERROR: vma_queue_write: write error - Invalid argument
INFO: aborting backup job
ERROR: Backup of VM 7206 failed - vma_queue_write: write error - Invalid argument
INFO: Starting Backup of VM 7219 (qemu)
INFO: status = running
INFO: update VM 7219: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/vmfs-backup/dump/vzdump-qemu-7219-2016_02_01-11_28_15.vma'
INFO: started backup task '37ff81dd-00f3-481c-a42d-14537b65b905'
INFO: status: 0% (3801088/34359738368), sparse 0% (479232), duration 0, 3/3 MB/s
ERROR: vma_queue_write: write error - Invalid argument
INFO: aborting backup job
ERROR: Backup of VM 7219 failed - vma_queue_write: write error - Invalid argument
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp24217 for temporary files
INFO: Starting Backup of VM 6209 (lxc)
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/mnt/vmfs-backup/dump/vzdump-lxc-6209-2016_02_01-11_28_16.tar'
INFO: Total bytes written: 2685583360 (2.6GiB, 119MiB/s)
INFO: archive file size: 2.50GB
INFO: delete old backup '/mnt/vmfs-backup/dump/vzdump-lxc-6209-2016_01_30-11_04_33.tar.lzo'
INFO: Finished Backup of VM 6209 (00:02:25)
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp24217 for temporary files
INFO: Starting Backup of VM 7211 (lxc)
INFO: status = running
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot snapshot
INFO: creating archive '/mnt/vmfs-backup/dump/vzdump-lxc-7211-2016_02_01-11_30_41.tar'
INFO: Total bytes written: 2236682240 (2.1GiB, 86MiB/s)
INFO: archive file size: 2.08GB
INFO: delete old backup '/mnt/vmfs-backup/dump/vzdump-lxc-7211-2016_01_30-10_58_10.tar.lzo'
INFO: remove vzdump snapshot
INFO: Finished Backup of VM 7211 (00:02:24)
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp24217 for temporary files
INFO: Starting Backup of VM 7216 (lxc)
INFO: status = running
INFO: found old vzdump snapshot (force removal)
ERROR: Backup of VM 7216 failed - VM is locked (snapshot)
INFO: filesystem type on dumpdir is 'cifs' -using /var/tmp/vzdumptmp24217 for temporary files
INFO: Starting Backup of VM 7217 (lxc)
INFO: status = running
INFO: found old vzdump snapshot (force removal)
ERROR: Backup of VM 7217 failed - VM is locked (snapshot)
INFO: Backup job finished with errors
job errors
------- this is just one node -------
root@pobbel:~# vzdump 6201 --remove 0 --mode snapshot --compress lzo --node pobbel --storage vmfs-backup
INFO: starting new backup job: vzdump 6201 --remove 0 --storage vmfs-backup --node pobbel --compress lzo --mode snapshot
INFO: Starting Backup of VM 6201 (qemu)
INFO: status = running
INFO: update VM 6201: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/vmfs-backup/dump/vzdump-qemu-6201-2016_02_01-11_51_24.vma.lzo'
INFO: started backup task 'f8edcedf-79b3-4fbe-b09c-f0a9a89198b1'
INFO: status: 4% (2755788800/68719476736), sparse 3% (2287820800), duration 3, 918/155 MB/s
INFO: status: 7% (4849598464/68719476736), sparse 5% (3933478912), duration 6, 697/149 MB/s
INFO: status: 11% (7590707200/68719476736), sparse 9% (6441545728), duration 9, 913/77 MB/s
INFO: status: 15% (10865147904/68719476736), sparse 13% (9438945280), duration 12, 1091/92 MB/s
INFO: status: 20% (14258143232/68719476736), sparse 18% (12777816064), duration 15, 1130/18 MB/s
INFO: status: 24% (16527130624/68719476736), sparse 20% (14396887040), duration 18, 756/216 MB/s
INFO: status: 26% (18009554944/68719476736), sparse 21% (14595854336), duration 21, 494/427 MB/s
INFO: status: 28% (19746652160/68719476736), sparse 21% (14817193984), duration 24, 579/505 MB/s
INFO: status: 31% (21362966528/68719476736), sparse 21% (14890201088), duration 27, 538/514 MB/s
INFO: status: 33% (22981378048/68719476736), sparse 22% (15222378496), duration 30, 539/428 MB/s
INFO: status: 35% (24352456704/68719476736), sparse 22% (15500443648), duration 33, 457/364 MB/s
INFO: status: 37% (25729564672/68719476736), sparse 22% (15537704960), duration 36, 459/446 MB/s
INFO: status: 39% (27373469696/68719476736), sparse 22% (15751028736), duration 39, 547/476 MB/s
INFO: status: 42% (29044113408/68719476736), sparse 23% (15973728256), duration 42, 556/482 MB/s
INFO: status: 45% (31217025024/68719476736), sparse 24% (16854978560), duration 45, 724/430 MB/s
INFO: status: 49% (34148974592/68719476736), sparse 28% (19362459648), duration 48, 977/141 MB/s
INFO: status: 50% (35042230272/68719476736), sparse 28% (19560448000), duration 51, 297/231 MB/s
INFO: status: 51% (35582902272/68719476736), sparse 28% (19571449856), duration 54, 180/176 MB/s
INFO: status: 52% (36136943616/68719476736), sparse 28% (19579826176), duration 57, 184/181 MB/s
INFO: status: 53% (36840144896/68719476736), sparse 28% (19713064960), duration 60, 234/189 MB/s
INFO: status: 54% (37512937472/68719476736), sparse 28% (19730862080), duration 63, 224/218 MB/s
INFO: status: 55% (38193332224/68719476736), sparse 28% (19747803136), duration 66, 226/221 MB/s
INFO: status: 56% (38972555264/68719476736), sparse 28% (19886342144), duration 69, 259/213 MB/s
INFO: status: 57% (39679557632/68719476736), sparse 28% (19901652992), duration 72, 235/230 MB/s
INFO: status: 58% (40405565440/68719476736), sparse 28% (19908349952), duration 75, 242/239 MB/s
INFO: status: 59% (41230401536/68719476736), sparse 29% (20061188096), duration 78, 274/223 MB/s
INFO: status: 61% (42024828928/68719476736), sparse 29% (20121309184), duration 81, 264/244 MB/s
INFO: status: 62% (42895278080/68719476736), sparse 29% (20261969920), duration 84, 290/243 MB/s
INFO: status: 63% (43894964224/68719476736), sparse 30% (20650815488), duration 87, 333/203 MB/s
INFO: status: 64% (44522143744/68719476736), sparse 30% (20650815488), duration 90, 209/209 MB/s
INFO: status: 65% (45325090816/68719476736), sparse 30% (20835684352), duration 93, 267/206 MB/s
INFO: status: 71% (48989405184/68719476736), sparse 35% (24388026368), duration 96, 1221/37 MB/s
INFO: status: 77% (53123678208/68719476736), sparse 41% (28391985152), duration 99, 1378/43 MB/s
INFO: status: 84% (58337787904/68719476736), sparse 48% (33606082560), duration 102, 1738/0 MB/s
INFO: status: 92% (63446319104/68719476736), sparse 56% (38704201728), duration 105, 1702/3 MB/s
INFO: status: 97% (67032186880/68719476736), sparse 61% (42054598656), duration 108, 1195/78 MB/s
INFO: status: 98% (68009066496/68719476736), sparse 61% (42174369792), duration 111, 325/285 MB/s
INFO: status: 100% (68719476736/68719476736), sparse 61% (42295078912), duration 114, 236/196 MB/s
INFO: transferred 68719 MB in 114 seconds (602 MB/s)
INFO: archive file size: 7.72GB
INFO: Finished Backup of VM 6201 (00:07:52)
INFO: Backup job finished successfully
pveversion -v
proxmox-ve: 4.1-34 (running kernel: 4.2.6-1-pve)
pve-manager: 4.1-5 (running version: 4.1-5/f910ef5c)
pve-kernel-4.2.6-1-pve: 4.2.6-34
pve-kernel-4.2.2-1-pve: 4.2.2-16
pve-kernel-4.2.3-2-pve: 4.2.3-22
lvm2: 2.02.116-pve2
corosync-pve: 2.3.5-2
libqb0: 0.17.2-1
pve-cluster: 4.0-31
qemu-server: 4.0-49
pve-firmware: 1.1-7
libpve-common-perl: 4.0-45
libpve-access-control: 4.0-11
libpve-storage-perl: 4.0-38
pve-libspice-server1: 0.12.5-2
vncterm: 1.2-1
pve-qemu-kvm: 2.5-3
pve-container: 1.0-39
pve-firewall: 2.0-15
pve-ha-manager: 1.0-19
ksm-control-daemon: 1.2-1
glusterfs-client: 3.5.2-2+deb8u1
lxc-pve: 1.1.5-6
lxcfs: 0.13-pve3
cgmanager: 0.39-pve1
criu: 1.6.0-1
zfsutils: 0.6.5-pve7~jessie
openvswitch-switch: 2.3.2-2
