MTU vs vzdump speed

[dr-evil]

I have a 3-node cluster, with IBM v3700 storage system connected via iSCSI over 10Gbit ethernet.

To efficiently use 10Gbit we decided to set mtu from 1500 to 9000, and noticied that backup speed dropped significantly - from 80MB/S
to 1MB/S

I've changed MTU back to 1500 and speed gone back to 80MB/s

root@virt08:~# pveversion
pve-manager/3.2-30/1d095287 (running kernel: 2.6.32-32-pve)

What to do to have fast backup and big mtu?


With mtu 9000:

root@virt08:~# vzdump 156
INFO: starting new backup job: vzdump 156
INFO: Starting Backup of VM 156 (qemu)
INFO: status = running
INFO: update VM 156: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-156-2014_09_11-12_21_29.vma'
INFO: started backup task 'aa067e7b-66de-41d0-8b02-8fede56a9a49'
INFO: status: 0% (26411008/64424509440), sparse 0% (10145792), duration 3, 8/5 MB/s
INFO: status: 1% (687079424/64424509440), sparse 0% (618266624), duration 193, 3/0 MB/s

With mtu 1500:

root@virt08:~# vzdump 156
INFO: starting new backup job: vzdump 156
INFO: Starting Backup of VM 156 (qemu)
INFO: status = running
INFO: update VM 156: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-156-2014_09_11-12_10_08.vma'
INFO: started backup task 'd1803f54-2d14-456a-aece-bb9be5ea29a2'
INFO: status: 0% (144834560/64424509440), sparse 0% (77807616), duration 3, 48/22 MB/s
INFO: status: 1% (706281472/64424509440), sparse 0% (638521344), duration 13, 56/0 MB/s
INFO: status: 2% (1344274432/64424509440), sparse 1% (1276448768), duration 21, 79/0 MB/s
INFO: status: 3% (1947533312/64424509440), sparse 2% (1879703552), duration 29, 75/0 MB/s
INFO: status: 4% (2579300352/64424509440), sparse 3% (2511470592), duration 37, 78/0 MB/s

 

[Mr.Holmes]

Hello dr-evil

I have a 3-node cluster, with IBM v3700 storage system connected via iSCSI over 10Gbit ethernet.

To efficiently use 10Gbit we decided to set mtu from 1500 to 9000, and noticied that backup speed dropped significantly - from 80MB/S
to 1MB/S

I've changed MTU back to 1500 and speed gone back to 80MB/s

root@virt08:~# pveversion
pve-manager/3.2-30/1d095287 (running kernel: 2.6.32-32-pve)

What to do to have fast backup and big mtu?

Nothing, I think - the problem is rather that your environment does not really support jumbo mtus.

Probably it is not possible to accelerate the backup process by improving the network speed - the main effort is extracting data and compress them. I usually have (physical, that means sparse amount subtracted) rates around 25MB/s - with local storage which provides a data transfer rate around 200MB/s.

Recommend to keep mtu=1500

Kind regards

Mr.Holmes

 

[dr-evil]

Hello dr-evil
Nothing, I think - the problem is rather that your environment does not really support jumbo mtus.

Really? Why?

I've made a tests - so if I grow up mtu to 9000 - I have over 300mb/s transfer in guests.
With 1500 - it only 90-100.

Probably it is not possible to accelerate the backup process by improving the network speed - the main effort is extracting data and compress them. I usually have (physical, that means sparse amount subtracted) rates around 25MB/s - with local storage which provides a data transfer rate around 200MB/s.

Recommend to keep mtu=1500

Kind regards

Mr.Holmes

The question is why backup (and only backup) speed dropped down with raising mtu to 9000.

 

[Mr.Holmes]

Strange, difficult to say (without deeper investigation) - but the following workaround can help:

#create the backup file in a local storage
vzdump 100 -dumpdir /temp/
# move to your network storage
mv /temp/*.vma /netstorage/dump/.

 

[dr-evil]

Strange, difficult to say (without deeper investigation) - but the following workaround can help:

#create the backup file in a local storage
vzdump 100 -dumpdir /temp/
# move to your network storage
mv /temp/*.vma /netstorage/dump/.

I've tryed this, with no success

 

[dietmar]

I guess some component (switch) inside your network does not support mtu 9000, leading to packet fragmentation.
You should be able to seed that with tcpdump.

 

[Mr.Holmes]

I've tryed this, with no success

What does it mean? What kind of problems you had (with vzdump or mv)?

 

[spirit]

when you setup MTU 9000, do you have configured it everywhere ? (proxmox - switch - san).

If not, you'll have fragmentation and bad performance

 

[dr-evil]

ok. Let's start once more.

The only performance problem with MTU 9000 - is slow vzdump.

Guest r/w performance is good.
Host to SAN is good.


Switch configured to max mtu to 9216

 

[dr-evil]

any suggestions ?

 

[Mr.Holmes]

Hello dr-evil

any suggestions ?

I posted already any:

the following workaround can help:

#create the backup file in a local storage
vzdump 100 -dumpdir /temp/
# move to your network storage
mv /temp/*.vma /netstorage/dump/.

you answered

I've tryed this, with no success

but did not specify what in particular went wrong. You wrote

Host to SAN is good

and backup to local storage too? If "Yes" this workaround must work, or ....?

Kind regards

Mr.Holmes

 

[dr-evil]

Sorry.
This workaround as slow as main process, nothing went wrong except speed.

Backup to local storage and backup to NFS have no difference in speed.