VM on NFS - Very Slow Backups

L

longhair

Member
Jun 27, 2014
113
0
16
I have a couple of VMs on the local drive and on a NFS (Synology 415+). There are 2x RAID 1 volumes (vol 1 for VMs and vol 2 for backups).

The following is for a Windows 7 backup - clean install & cloned so everything is exactly the same.

When I run a backup for VM on the local drive:

Code: 
104: Mar 07 18:21:02 INFO: Starting Backup of VM 104 (qemu)
104: Mar 07 18:21:02 INFO: status = stopped
104: Mar 07 18:21:02 INFO: update VM 104: -lock backup
104: Mar 07 18:21:02 INFO: backup mode: stop
104: Mar 07 18:21:02 INFO: bandwidth limit: 230400 KB/s
104: Mar 07 18:21:02 INFO: ionice priority: 7
104: Mar 07 18:21:02 INFO: creating archive '/mnt/pve/aspa-sds2/dump/vzdump-qemu-104-2015_03_07-18_21_02.vma.lzo'
104: Mar 07 18:21:02 INFO: starting kvm to execute backup task
104: Mar 07 18:21:03 INFO: started backup task 'e4b756c7-8acf-47e7-aea5-04131c3069db'
104: Mar 07 18:21:06 INFO: status: 1% (578289664/34359738368), sparse 1% (533106688), duration 3, 192/15 MB/s
104: Mar 07 18:21:09 INFO: status: 2% (859045888/34359738368), sparse 1% (598077440), duration 6, 93/71 MB/s
104: Mar 07 18:21:12 INFO: status: 3% (1087111168/34359738368), sparse 1% (621256704), duration 9, 76/68 MB/s
104: Mar 07 18:21:16 INFO: status: 4% (1377501184/34359738368), sparse 1% (621719552), duration 13, 72/72 MB/s
104: Mar 07 18:21:21 INFO: status: 5% (1737097216/34359738368), sparse 1% (625463296), duration 18, 71/71 MB/s
...
104: Mar 07 18:29:55 INFO: status: 91% (31446269952/34359738368), sparse 70% (24113029120), duration 532, 236/0 MB/s
104: Mar 07 18:29:58 INFO: status: 93% (32156024832/34359738368), sparse 72% (24822784000), duration 535, 236/0 MB/s
104: Mar 07 18:30:01 INFO: status: 95% (32865779712/34359738368), sparse 74% (25532538880), duration 538, 236/0 MB/s
104: Mar 07 18:30:04 INFO: status: 97% (33575534592/34359738368), sparse 76% (26242293760), duration 541, 236/0 MB/s
104: Mar 07 18:30:07 INFO: status: 99% (34285289472/34359738368), sparse 78% (26952048640), duration 544, 236/0 MB/s
104: Mar 07 18:30:08 INFO: status: 100% (34359738368/34359738368), sparse 78% (27026493440), duration 545, 74/0 MB/s
104: Mar 07 18:30:08 INFO: transferred 34359 MB in 545 seconds (63 MB/s)
104: Mar 07 18:30:08 INFO: stopping kvm after backup task
104: Mar 07 18:30:09 INFO: archive file size: 3.98GB
104: Mar 07 18:30:09 INFO: delete old backup '/mnt/pve/aspa-sds2/dump/vzdump-qemu-104-2015_03_07-17_48_01.vma.lzo'
104: Mar 07 18:30:09 INFO: Finished Backup of VM 104 (00:09:07)

When I run a backup for VM on the NFS:

Code: 
105: Mar 07 18:30:09 INFO: Starting Backup of VM 105 (qemu)
105: Mar 07 18:30:09 INFO: status = stopped
105: Mar 07 18:30:10 INFO: update VM 105: -lock backup
105: Mar 07 18:30:10 INFO: backup mode: stop
105: Mar 07 18:30:10 INFO: bandwidth limit: 230400 KB/s
105: Mar 07 18:30:10 INFO: ionice priority: 7
105: Mar 07 18:30:10 INFO: creating archive '/mnt/pve/aspa-sds2/dump/vzdump-qemu-105-2015_03_07-18_30_09.vma.lzo'
105: Mar 07 18:30:10 INFO: starting kvm to execute backup task
105: Mar 07 18:30:10 INFO: started backup task '0633cc5e-791c-4df1-8e32-116b9b350166'
105: Mar 07 18:30:13 INFO: status: 0% (35323904/34359738368), sparse 0% (18206720), duration 3, 11/5 MB/s
105: Mar 07 18:30:40 INFO: status: 1% (353435648/34359738368), sparse 0% (305528832), duration 30, 11/1 MB/s
105: Mar 07 18:31:09 INFO: status: 2% (693108736/34359738368), sparse 1% (595189760), duration 59, 11/1 MB/s
105: Mar 07 18:31:47 INFO: status: 3% (1037565952/34359738368), sparse 1% (616513536), duration 97, 9/8 MB/s
105: Mar 07 18:32:36 INFO: status: 4% (1379598336/34359738368), sparse 1% (617504768), duration 146, 6/6 MB/s
...
105: Mar 07 19:22:19 INFO: status: 95% (32642826240/34359738368), sparse 73% (25304965120), duration 3129, 11/0 MB/s
105: Mar 07 19:22:49 INFO: status: 96% (32996261888/34359738368), sparse 74% (25658400768), duration 3159, 11/0 MB/s
105: Mar 07 19:23:18 INFO: status: 97% (33337769984/34359738368), sparse 75% (25999908864), duration 3188, 11/0 MB/s
105: Mar 07 19:23:48 INFO: status: 98% (33679474688/34359738368), sparse 76% (26341613568), duration 3218, 11/0 MB/s
105: Mar 07 19:24:17 INFO: status: 99% (34020982784/34359738368), sparse 77% (26683121664), duration 3247, 11/0 MB/s
105: Mar 07 19:24:46 INFO: status: 100% (34359738368/34359738368), sparse 78% (27021873152), duration 3276, 11/0 MB/s
105: Mar 07 19:24:46 INFO: transferred 34359 MB in 3276 seconds (10 MB/s)
105: Mar 07 19:24:46 INFO: stopping kvm after backup task
105: Mar 07 19:24:47 INFO: archive file size: 3.99GB
105: Mar 07 19:24:47 INFO: Finished Backup of VM 105 (00:54:38)

What can I do to speed everything up on the NFS?
 
After trying different CAT6 cables, swapped out the switch and that improved things.

VM on the local drive:

Code: 
transferred 34359 MB in 545 seconds (63 MB/s)
Finished Backup of VM 104 (00:09:07)

transferred 34359 MB in 253 seconds (135 MB/s)
Finished Backup of VM 104 (00:04:15)

VM on the NFS:

Code: 
transferred 34359 MB in 3276 seconds (10 MB/s)
Finished Backup of VM 105 (00:54:38)

transferred 34359 MB in 452 seconds (76 MB/s)
Finished Backup of VM 105 (00:07:35)

Local: 4:52 minutes faster.
NFS: 47:03 minutes faster.

There is still a difference of 3:20 minutes between the Local and NFS. I'm sure that is because Proxmox has to read & write to the NFS at the same time.

Is there a way to automatically store all the .lzo on a local drive until the last VM is backed up, then move all the .lzo files to the NFS?

BTW, this did not change the speeds: https://dev-eth0.de/increase-performance-proxmox-backups-nfs-target/
 
Add the following to /etc/vzdump.conf
tmpdir: /path/to/tmpdir

Sorry, missed seeing your link.

If that does not help there is nothing you can do. The bad performance is cause either by slow server hosting your nfs share or because the disks in the server hosting your nfs share is slow. Network quality is also a factor.

The server hosting your share is running RAID1. This effectively reduce the write speed by 50%.
 
Last edited:
Everything is on a Gigabit LAN. Both backups (cloned) are going to the same NFS.

VM 104 - local drive to NFS drive 135 MB/s.
VM 105 - NFS drive to NFS drive 76 MB/s.

NFS --> NFS 56% slower then local --> NFS.
 
Hi Longhair,

I too have struggled with backup speeds to my synology NAS 1513+

Currently i found that i have ALLOT of discarded packets..

I was using MTU 9000 and when set that back to default 1500 that encreased my speeds.

I too am running with default settings and two switches in stack everything is Gb speeds..

Also one reason that could have encreased my speeds was that the bond on my storage synology one of the interfaces was runing at 100Mbit and not 1Gb.. so perhaps you should check your bonds to make sure the right speed has been negotiated..

currently my backup speed is at 50MBytes/s which about half of what i should see with 1Gb network.. still investigating..

My next step is to firmware update both boxes..
 
you can also try to increase nfs windows size

,rsize=65536,wsize=65536

Here my nfs options

options rw,noatime,nodiratime,noacl,vers=4,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600

(i'm using nfsv4 over tcp)
 
Spirit

those setting would have to be set on both the synology nas and the proxmox node.. right? or is it enough to just do it on the proxmox node?
 
I made a change to the /etc/vzdump.conf

Code: 
tmpdir: /tmp
to
tmpdir: /var/tmp

Results VM105 (NFS --> NFS):

Code: 
transferred 34359 MB in 195 seconds (176 MB/s)

The transfer rate is a lot faster than it was previously.

I run a backup job for all the VMs (100 & 101 local, 102 & 103 nfs):

Code: 
100: Mar 09 09:02:00 INFO: transferred 21474 MB in 110 seconds (195 MB/s)
100: Mar 09 09:02:01 INFO: archive file size: 899MB
100: Mar 09 09:02:01 INFO: vm is online again after 119 seconds
100: Mar 09 09:02:01 INFO: Finished Backup of VM 100 (00:02:00)

101: Mar 09 09:05:00 INFO: transferred 21474 MB in 178 seconds (120 MB/s)
101: Mar 09 09:05:01 INFO: stopping kvm after backup task
101: Mar 09 09:05:02 INFO: archive file size: 2.32GB
101: Mar 09 09:05:02 INFO: delete old backup '....vma.lzo'
101: Mar 09 09:05:02 INFO: Finished Backup of VM 101 (00:03:01)

102: Mar 09 10:19:40 INFO: transferred 377957 MB in 4477 seconds (84 MB/s)
102: Mar 09 10:19:40 INFO: stopping kvm after backup task
102: Mar 09 10:19:41 INFO: archive file size: 12.83GB
102: Mar 09 10:19:41 INFO: delete old backup '....vma.lzo'
102: Mar 09 10:19:43 INFO: Finished Backup of VM 102 (01:14:41)

103: Mar 09 12:28:19 INFO: transferred 171798 MB in 7715 seconds (22 MB/s)
103: Mar 09 12:28:19 INFO: stopping kvm after backup task
103: Mar 09 12:28:20 INFO: archive file size: 15.11GB
103: Mar 09 12:28:20 INFO: delete old backup '....vma.lzo'
103: Mar 09 12:28:21 INFO: Finished Backup of VM 103 (02:08:38)

The transfer kept getting slower each time a backup of a VM started.

VM 101: 195MB/s
VM 102: 120 MB/s - 75 MB/s slower than previous backup.
VM 103: 84 MB/s - 36 MB/s slower than previous backup.
VM 104: 22 MB/s - 62 MB/s slower than previous backup.

I decided to try to back up each machine individually with a 5 minute wait time between the end and start of

the next backup. I am editing a single backup job instead of creating 4 individual ones.

Code: 
100: Mar 09 13:24:07 INFO: transferred 21474 MB in 117 seconds (183 MB/s)
100: Mar 09 13:24:07 INFO: archive file size: 899MB
100: Mar 09 13:24:07 INFO: delete old backup '....vma.lzo'
100: Mar 09 13:24:08 INFO: vm is online again after 127 seconds
100: Mar 09 13:24:08 INFO: Finished Backup of VM 100 (00:02:07)

101: Mar 09 13:31:50 INFO: transferred 21474 MB in 168 seconds (127 MB/s)
101: Mar 09 13:31:50 INFO: stopping kvm after backup task
101: Mar 09 13:31:51 INFO: archive file size: 2.32GB
101: Mar 09 13:31:51 INFO: delete old backup '....vma.lzo'
101: Mar 09 13:31:52 INFO: Finished Backup of VM 101 (00:02:51)

102: Mar 09 14:51:38 INFO: transferred 377957 MB in 4476 seconds (84 MB/s)
102: Mar 09 14:51:38 INFO: stopping kvm after backup task
102: Mar 09 14:51:39 INFO: archive file size: 12.83GB
102: Mar 09 14:51:39 INFO: delete old backup '....vma.lzo'
102: Mar 09 14:51:41 INFO: Finished Backup of VM 102 (01:14:40)

I stopped 103 because it was going 22MB/s again.

/var/tmp is emptied once the backup job has finished or stopped.
 
longhair said:
I made a change to the /etc/vzdump.conf

Code: 
tmpdir: /tmp
to
tmpdir: /var/tmp

Results VM105 (NFS --> NFS):

Code: 
transferred 34359 MB in 195 seconds (176 MB/s)

The transfer rate is a lot faster than it was previously.

I run a backup job for all the VMs (100 & 101 local, 102 & 103 nfs):

Code: 
100: Mar 09 09:02:00 INFO: transferred 21474 MB in 110 seconds (195 MB/s)
100: Mar 09 09:02:01 INFO: archive file size: 899MB
100: Mar 09 09:02:01 INFO: vm is online again after 119 seconds
100: Mar 09 09:02:01 INFO: Finished Backup of VM 100 (00:02:00)

101: Mar 09 09:05:00 INFO: transferred 21474 MB in 178 seconds (120 MB/s)
101: Mar 09 09:05:01 INFO: stopping kvm after backup task
101: Mar 09 09:05:02 INFO: archive file size: 2.32GB
101: Mar 09 09:05:02 INFO: delete old backup '....vma.lzo'
101: Mar 09 09:05:02 INFO: Finished Backup of VM 101 (00:03:01)

102: Mar 09 10:19:40 INFO: transferred 377957 MB in 4477 seconds (84 MB/s)
102: Mar 09 10:19:40 INFO: stopping kvm after backup task
102: Mar 09 10:19:41 INFO: archive file size: 12.83GB
102: Mar 09 10:19:41 INFO: delete old backup '....vma.lzo'
102: Mar 09 10:19:43 INFO: Finished Backup of VM 102 (01:14:41)

103: Mar 09 12:28:19 INFO: transferred 171798 MB in 7715 seconds (22 MB/s)
103: Mar 09 12:28:19 INFO: stopping kvm after backup task
103: Mar 09 12:28:20 INFO: archive file size: 15.11GB
103: Mar 09 12:28:20 INFO: delete old backup '....vma.lzo'
103: Mar 09 12:28:21 INFO: Finished Backup of VM 103 (02:08:38)

The transfer kept getting slower each time a backup of a VM started.

VM 101: 195MB/s
VM 102: 120 MB/s - 75 MB/s slower than previous backup.
VM 103: 84 MB/s - 36 MB/s slower than previous backup.
VM 104: 22 MB/s - 62 MB/s slower than previous backup.

I decided to try to back up each machine individually with a 5 minute wait time between the end and start of

the next backup. I am editing a single backup job instead of creating 4 individual ones.

Code: 
100: Mar 09 13:24:07 INFO: transferred 21474 MB in 117 seconds (183 MB/s)
100: Mar 09 13:24:07 INFO: archive file size: 899MB
100: Mar 09 13:24:07 INFO: delete old backup '....vma.lzo'
100: Mar 09 13:24:08 INFO: vm is online again after 127 seconds
100: Mar 09 13:24:08 INFO: Finished Backup of VM 100 (00:02:07)

101: Mar 09 13:31:50 INFO: transferred 21474 MB in 168 seconds (127 MB/s)
101: Mar 09 13:31:50 INFO: stopping kvm after backup task
101: Mar 09 13:31:51 INFO: archive file size: 2.32GB
101: Mar 09 13:31:51 INFO: delete old backup '....vma.lzo'
101: Mar 09 13:31:52 INFO: Finished Backup of VM 101 (00:02:51)

102: Mar 09 14:51:38 INFO: transferred 377957 MB in 4476 seconds (84 MB/s)
102: Mar 09 14:51:38 INFO: stopping kvm after backup task
102: Mar 09 14:51:39 INFO: archive file size: 12.83GB
102: Mar 09 14:51:39 INFO: delete old backup '....vma.lzo'
102: Mar 09 14:51:41 INFO: Finished Backup of VM 102 (01:14:40)

I stopped 103 because it was going 22MB/s again.

/var/tmp is emptied once the backup job has finished or stopped.
Click to expand...

is it the entire backup it stores in tmp when it does the backup or does it delete for each VM it completes? assuming you have setup one job?
 
I didn't check when running multiple backups on 1 job.

What I cannot figure out is why the backups are fast on VM #100 and very slow on VM #103. I even rebooted Proxmox, started a backup job for #103 and it was the same slow 22 MB/s speed.

New CAT6 cables didn't change anything either :/
 
offerlam - I did a multiple backup job (2 VMs) on 1 job. There was only 1 tmp file and the file name did not change when VM1 finished and VM2 started. I don't know what the file size was though because I forgot to check.
 
Something is very wrong.

I am getting about 50 MB/s with LACP properly set up (Netgear GS716Tv3).

Synology DS415+ Network Status: 2000 Mbps, Full duplex, MTU 1500.

Proxmox 3.4-1/3f2d890e

/etc/network/interfaces

Code: 
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

auto bond0
iface bond0 inet static
            address 192.168.1.3
            netmask 255.255.255.0
            slaves eth1 eth2
            bond_miimon 100
            bond_mode 802.3ad

auto vmbr0
iface vmbr0 inet static
           address 192.168.1.2
           netmask 255.255.255.0
           gatway 192.168.1.1
           bridge_ports eth0
           bridge_stp off
           bridge_fd 0
 
Change the subnet for the bond0 and have the NFS on the same subnet.

/etc/network/interfaces

Code: 
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

iface eth1 inet manual

iface eth2 inet manual

auto bond0
iface bond0 inet static
            address 192.168.2.3
            netmask 255.255.255.0
            slaves eth1 eth2
            bond_miimon 100
            bond_mode 802.3ad

auto vmbr0
iface vmbr0 inet static
           address 192.168.1.2
           netmask 255.255.255.0
           gatway 192.168.1.1
           bridge_ports eth0
           bridge_stp off
           bridge_fd 0

Still seeing
INFO: transferred 171798 MB in 3104 seconds (55 MB/s)
 
spirit said:
you can also try to increase nfs windows size

,rsize=65536,wsize=65536

Here my nfs options

options rw,noatime,nodiratime,noacl,vers=4,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600

(i'm using nfsv4 over tcp)
Click to expand...

I changed the Synology to NFSv4 and added your options to /etc/pve/storage.cfg but no change in speeds.
 
One thing,
when the log show

"103: Mar 09 12:28:19 INFO: transferred 171798 MB in 7715 seconds (22 MB/s)"

The transferred size is the total block size, including zero blocks.

That mean than a same total size, with or without zero block will have a big difference of speed.
Can you check in your backup task logs, the sparse % counter value ?
 
I just added a local hard drive (15k RPM SAS) to Proxmox, pointed the backup to that drive and still no increase in speed (55 MB/s).
 
Here is my vmid.config for the backup speed of 55 MB/s - Windows 7.

Code: 
balloon: 4096
bootdisk: ide0
cores: 2
ide0: NFS1:103/vm-103-disk-1.qcow2,format=qcow2,size=160G
ide1: local:iso/virtio-win-0.1-100.iso,media=cdrom,size=68314K
ide2: local:iso/Windows_7.iso,media=cdrom
memory: 8192
name: win7
net0: virtio=4E:8A:2E:64:3D:D0,bridge=vmbr0
numa: 0
ostype: win7
smbios1: uuid=5d9e809f-932b-4ae7-a35c-b997a0122fd9
sockets: 1
vga: qxl

The qemu disk throttle feature is default.

I am doing a backup of a SBS2011 VM and the speeds are over doubled - 120 MB/s. Here is the VMID config.

Code: 
balloon: 12288
bootdisk: ide0
cores: 2
cpuunits: 50000
ide0: NFS1:102/vm-102-disk-1.qcow2,format=qcow2,cache=writeback,size=320G
ide1: NFS1:102/vm-102-disk-2.qcow2,format=qcow2,cache=writeback,size=160G
ide2: none,media=cdrom
ide3: NFS:102/vm-102-disk-3.qcow2,format=qcow2,cache=writeback,backup=no,size=320G
lock: backup
memory: 24576
name: sbs2011
net0: virtio=C6:CA:E2:9C:56:5F,bridge=vmbr0
numa: 0
onboot: 1
ostype: win7
smbios1: uuid=c857bf1e-b3f1-4f63-940b-5d57bd481afe
sockets: 2
startup: order=2,up=120,down=600
vga: qxl

Writeback cache did not make a difference with the Win 7 VM because it was enabled before I removed the VM and re-created it in hopes that would solve the problem.
 
Last edited:
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back