Disk throttle and backup speed

[robhost]

Hi,

is disk throttling (kvm) also throttling the backup speed (mode: snapshot)? We're seeing extremely slow backups from VMs where iops_rd and iops_wr is set to 100 each and we have no other explanation for this behavior.

Example:

INFO: status: 0% (20381696/751619276800), sparse 0% (16015360), duration 3, 6/1 MB/s
INFO: status: 1% (7520321536/751619276800), sparse 0% (2376753152), duration 1179, 6/4 MB/s
INFO: status: 2% (15037497344/751619276800), sparse 1% (9448251392), duration 2332, 6/0 MB/s
INFO: status: 3% (22549299200/751619276800), sparse 2% (16550060032), duration 3482, 6/0 MB/s
INFO: status: 4% (30069882880/751619276800), sparse 3% (23609221120), duration 4639, 6/0 MB/s
INFO: status: 5% (37584437248/751619276800), sparse 4% (30883770368), duration 5802, 6/0 MB/s
INFO: status: 6% (45103120384/751619276800), sparse 5% (37972566016), duration 6954, 6/0 MB/s
INFO: status: 7% (52615839744/751619276800), sparse 6% (45224361984), duration 8105, 6/0 MB/s
INFO: status: 8% (60134522880/751619276800), sparse 6% (52379807744), duration 9261, 6/0 MB/s
INFO: status: 9% (67650846720/751619276800), sparse 7% (59618160640), duration 10419, 6/0 MB/s
INFO: status: 10% (75166318592/751619276800), sparse 8% (66947981312), duration 11569, 6/0 MB/s

The whole backups takes almot 24hours. Other lager VMs only need 30 minutes (same host and enviroment).
This VM is only around 60 Gbytes.

PVE is the latest version.

Thanks!

 

[robhost]

Ok, it's definitly the IOPS limit, without the backup is fast.
As mentioned here this seems to be "normal": We found an explanation here: https://forum.proxmox.com/threads/disk-throttling-effects-backup-speed.23895/

Is there any way to prevent this? Would be nice to have backup with full speed, especially when the backup is written to an external storage where no IO sharing exists.

 

[robhost]

If anyone is interested:

We've fixed the issue for us by adding a hook-script to /etc/vzdump.conf which removes the limits for each VM when its backup starts and sets it again when finished. Works fine here and backup runs with full speed. While backup runs the VM has no IO limits, but thats ok for us.

Some large VMs are now backing uo in 3 instead of 48 hours

 

[elmemis]

Hi @robhost ,

Can you shared the script?. I have the same problem with backups (kvm and lxc).

Thanks,
Best regards

 

[robhost]

Hi @robhost ,

Can you shared the script?. I have the same problem with backups (kvm and lxc).

Thanks,
Best regards

We just uploaded the script: https://github.com/robhost/proxmox-scripts
Tested with PVE 4.2, but should also work on other versions.

 

[elmemis]

Hi, thanks for the script!.
Tested with one QEMU VM offline and not work. I have NFS Storage (SAS - 10000 rpm, 2.5) with 10 Gbps connection. Last week running, but this week stopped walking well.

I use Proxmox VE 4.2 up to date.
My vzdump.conf is correct?.

noc:~# cat /etc/vzdump.conf
# vzdump default settings

#tmpdir: DIR
#dumpdir: DIR
#storage: STORAGE_ID
#mode: snapshot|suspend|stop
#bwlimit: KBPS
#ionice: PRI
#lockwait: MINUTES
#stopwait: MINUTES
#size: MB
#stdexcludes: BOOLEAN
#mailto: ADDRESSLIST
#maxfiles: N
script: /opt/scripts/vzdump_mod.sh
#exclude-path: PATHLIST
#pigz: N:

Thanks.

 

[robhost]

Can you provide the output of the backup task? Is your script "/opt/scripts/vzdump_mod.sh" executable?
There should appear something like this:

INFO: update VM 663: -scsi0 ceph0:vm-663-disk-1,discard=on,size=250G -skiplock 1
INFO: started backup task '97bcb7b7-8312-4a65-a48a-4bd883543a1a'
[...]
INFO: update VM 663: -scsi0 ceph0:vm-663-disk-1,discard=on,iops_rd=250,iops_rd_max=1000,iops_wr=150,iops_wr_max=500,size=250G -skiplock 1

 

[elmemis]

noc:~# ll /opt/scripts/vzdump_mod.sh
-rwxr-xr-x 1 root root 4650 Aug 30 16:59 /opt/scripts/vzdump_mod.sh

Task viewer: Backup

OutputStatus

Stop
INFO: starting new backup job: vzdump 113 --mode snapshot --storage BackupsVMS --remove 0 --node noc --compress lzo
INFO: Starting Backup of VM 113 (qemu)
INFO: status = stopped
INFO: update VM 113: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: gabrielbrahim
INFO: creating archive '/mnt/pve/BackupsVMS/dump/vzdump-qemu-113-2016_08_30-21_48_20.vma.lzo'
INFO: starting kvm to execute backup task
INFO: started backup task '199247a1-a19c-43b7-9318-593c79c0d46f'
INFO: status: 0% (3080192/161061273600), sparse 0% (1007616), duration 3, 1/0 MB/s
INFO: status: 1% (1612382208/161061273600), sparse 0% (399462400), duration 1412, 1/0 MB/s
INFO: status: 2% (3222929408/161061273600), sparse 1% (2010009600), duration 2507, 1/0 MB/s
INFO: status: 3% (4832624640/161061273600), sparse 2% (3619704832), duration 3613, 1/0 MB/s
INFO: status: 4% (6442516480/161061273600), sparse 3% (5229596672), duration 4879, 1/0 MB/s
INFO: status: 5% (8053260288/161061273600), sparse 4% (6840340480), duration 6158, 1/0 MB/s
INFO: status: 6% (9664724992/161061273600), sparse 4% (7372591104), duration 7167, 1/1 MB/s
INFO: status: 7% (11274551296/161061273600), sparse 5% (8500621312), duration 8305, 1/0 MB/s

From and destination is a Storage NAS.
Best regards

 

[robhost]

Strange. Did you modify our script because its named "vzdump_mod.sh"?
Do you really have IOPS limits in place? Please provide the config of the VM (located at etc/pve/qemu-server/113.conf).

 

[elmemis]

Strange. Did you modify our script because its named "vzdump_mod.sh"?
Do you really have IOPS limits in place? Please provide the config of the VM (located at etc/pve/qemu-server/113.conf).

No, I not modified, only changed the name.

noc:~# cat /etc/pve/qemu-server/113.conf
balloon: 3072
bootdisk: virtio0
cores: 1
memory: 4096
name: gabrielbrahim
net0: bridge=vmbr2,virtio=66:36:65:31:30:65,tag=10
numa: 0
ostype: l26
smbios1: uuid=3a98f903-9f66-4a09-a46d-3c0d3f72d1df
sockets: 4
virtio0: NAS-VMs1:113/vm-113-disk-1.qcow2,size=150G

This is an older backup log:

112: Jul 18 05:13:19 INFO: Starting Backup of VM 112 (qemu)

112: Jul 18 05:13:19 INFO: status = running

112: Jul 18 05:13:19 INFO: update VM 112: -lock backup

112: Jul 18 05:13:19 INFO: backup mode: snapshot

112: Jul 18 05:13:19 INFO: ionice priority: 7

112: Jul 18 05:13:19 INFO: creating archive '/mnt/pve/BackupsVMS/dump/vzdump-qemu-112-2016_07_18-05_13_19.vma.lzo'

112: Jul 18 05:13:19 INFO: started backup task '6b9d8ad6-4a02-4144-93c7-0b64a02a7a40'

112: Jul 18 05:13:22 INFO: status: 0% (263127040/161061273600), sparse 0% (109371392), duration 3, 87/51 MB/s

112: Jul 18 05:13:39 INFO: status: 1% (1656422400/161061273600), sparse 0% (968679424), duration 20, 81/31 MB/s

112: Jul 18 05:13:52 INFO: status: 2% (3295739904/161061273600), sparse 1% (2606952448), duration 33, 126/0 MB/s

112: Jul 18 05:14:32 INFO: status: 3% (4897439744/161061273600), sparse 1% (3061510144), duration 73, 40/28 MB/s

112: Jul 18 05:15:11 INFO: status: 4% (6459686912/161061273600), sparse 1% (3061510144), duration 112, 40/40 MB/s

112: Jul 18 05:15:42 INFO: status: 5% (8085307392/161061273600), sparse 1% (3061510144), duration 143, 52/52 MB/s

112: Jul 18 05:16:10 INFO: status: 6% (9669115904/161061273600), sparse 1% (3061510144), duration 171, 56/56 MB/s

112: Jul 18 05:16:29 INFO: status: 7% (11310530560/161061273600), sparse 1% (3185754112), duration 190, 86/79 MB/s

112: Jul 18 05:16:47 INFO: status: 8% (12999589888/161061273600), sparse 3% (4874551296), duration 208, 93/0 MB/s

112: Jul 18 05:16:59 INFO: status: 9% (14608760832/161061273600), sparse 4% (6483722240), duration 220, 134/0 MB/s

112: Jul 18 05:17:11 INFO: status: 10% (16194404352/161061273600), sparse 5% (8069038080), duration 232, 132/0 MB/s

112: Jul 18 05:17:23 INFO: status: 11% (17773297664/161061273600), sparse 5% (9647341568), duration 244, 131/0 MB/s

112: Jul 18 05:17:35 INFO: status: 12% (19370016768/161061273600), sparse 6% (11244060672), duration 256, 133/0 MB/s

112: Jul 18 05:17:47 INFO: status: 13% (20958019584/161061273600), sparse 7% (12831539200), duration 268, 132/0 MB/s

112: Jul 18 05:18:00 INFO: status: 14% (22572433408/161061273600), sparse 8% (14445559808), duration 281, 124/0 MB/s

112: Jul 18 05:18:12 INFO: status: 15% (24160960512/161061273600), sparse 9% (16034086912), duration 293, 132/0 MB/s

112: Jul 18 05:18:25 INFO: status: 16% (25896222720/161061273600), sparse 11% (17768960000), duration 306, 133/0 MB/s

112: Jul 18 05:18:36 INFO: status: 17% (27382841344/161061273600), sparse 11% (19255578624), duration 317, 135/0 MB/s

112: Jul 18 05:18:49 INFO: status: 18% (29110173696/161061273600), sparse 13% (20982648832), duration 330, 132/0 MB/s

112: Jul 18 05:19:01 INFO: status: 19% (30719803392/161061273600), sparse 14% (22591889408), duration 342, 134/0 MB/s

112: Jul 18 05:19:13 INFO: status: 20% (32304201728/161061273600), sparse 15% (24176222208), duration 354, 132/0 MB/s

112: Jul 18 05:19:25 INFO: status: 21% (33905049600/161061273600), sparse 16% (25777070080), duration 366, 133/0 MB/s

112: Jul 18 05:19:44 INFO: status: 22% (35457925120/161061273600), sparse 16% (26900553728), duration 385, 81/22 MB/s

112: Jul 18 05:20:29 INFO: status: 23% (37048221696/161061273600), sparse 16% (26900566016), duration 430, 35/35 MB/s

112: Jul 18 05:21:04 INFO: status: 24% (38746849280/161061273600), sparse 16% (26900582400), duration 465, 48/48 MB/s

112: Jul 18 05:21:37 INFO: status: 25% (40287731712/161061273600), sparse 16% (26900582400), duration 498, 46/46 MB/s

112: Jul 18 05:22:02 INFO: status: 26% (41910075392/161061273600), sparse 16% (26900582400), duration 523, 64/64 MB/s

112: Jul 18 05:22:35 INFO: status: 27% (43536416768/161061273600), sparse 16% (26900586496), duration 556, 49/49 MB/s

112: Jul 18 05:22:56 INFO: status: 28% (45175799808/161061273600), sparse 16% (27297800192), duration 577, 78/59 MB/s

112: Jul 18 05:23:11 INFO: status: 29% (46765441024/161061273600), sparse 17% (28887441408), duration 592, 105/0 MB/s

112: Jul 18 05:23:24 INFO: status: 30% (48435298304/161061273600), sparse 18% (30557167616), duration 605, 128/0 MB/s

112: Jul 18 05:23:36 INFO: status: 31% (49991843840/161061273600), sparse 19% (32113713152), duration 617, 129/0 MB/s

 

[robhost]

As you have no thottling set, our script can't speedup your backup, sorry.

 

[elmemis]

Ok, thanks for your help like. I will continue investigating what happens.
I started a ticket at OVH to analyze latency storage but still got no answer.

Best regards