Slow VM's when backups are running

[weblike]

Hi,
I have this topology from bellow:
The nodes has 10GBe link with a Qnap Nas and I have a backup server pluged in a 1GB switch. The nodes also are pluged in the 1GB switch.
Live production is between nodes and Qnap on 10GB, but when the backups are running the VM's are very slow, almost freezing.

Any advice will be much appreciated. Thank you

 

[weblike]

I have to mention that almost all of my VM's are Linux.

 

[aderumier]

when backup is running, if a new write is coming on a block not yet backuped, the block is first written to the backup storage before acking the new write. (it's only occur at first write on this block).

if you have a lot of writes, on differents block, and you have a slow backup storage or overloaded network link , it can slowdown vm.

 

[weblike]

The problem is that all VM's are running slow when a backup is running. That's the big problem.

I have 10 VM's -> If Vm1 is scheduled for backup-> than al the remaing 9 are freezing, unresponsive, slow, until the backup is finished.
I think that's not normal.

 

[weblike]

Based on my picture can you describe how the traffic is flowing when a backup is made?
The VM hardisk is on "storage qnap".
thank you

 

[robhost]

Do you have IOPS limits via Proxmox in place? If yes, this also slows down your backups!

 

[weblike]

No. It's default configuration. Do you have a tutorial on setting this parameter?

 

[robhost]

No. It's default configuration. Do you have a tutorial on setting this parameter?

If its not set, then you're fine.

 

[weblike]

If its not set, then you're fine.

But, I'm not...
My cluster is slow (

 

[robhost]

But, I'm not...
My cluster is slow (

Hard to say why, but you could try to limit the bandwidth using "bwlimit" in /etc/vzdump.conf (see https://pve.proxmox.com/pve-docs/vzdump.1.html, it is in KBytes per second) i.e. for testing.

 

[GadgetPig]

I would check network stats from each node, to check for network card errors:
#ip -s link
Also do you have another 1GB switch and/or Cat6 cables to switch out and retest?
Is the switch using VLAN and/or is there any QOS or bandwidth throttling enabled?

 

[weblike]

Hi again,

No errors on network cards...only dropped packets.

I have changed the old HP-switch with a CISCO SG-500.
On the switch I have made LAG (link aggregation group) and on each node I have made a bond0 (eth1 eth2 eth3)-balance-rr

The results are worst than before....the speed is 30-40 MB/s