Backups Causing VMs to Run Slow?

[ArcticEmile]

I have a VM running Grafana that, when running a backup on any other VM on any of the Proxmox nodes in my cluster, sends alerts stating that the InfluxDB on Grafana has locked up. I'm not sure why it would seemingly only affect this one VM but I'm also worried that it's having unseen effects on other VMs.

Bearing in mind that Grafana and InfluxDB are on the same VM so there should be no latency.

 

[LnxBil]

Sure ... every action even observation alters the underlying system as Heisenberg already showed us.

Backup is reading data, so everything else that wants to read will be slower. Depending on the quality of hardware, the impact can be none, noticeable or break everything if the disk is completely stalled.

 

[ArcticEmile]

Sure ... every action even observation alters the underlying system as Heisenberg already showed us.

Backup is reading data, so everything else that wants to read will be slower. Depending on the quality of hardware, the impact can be none, noticeable or break everything if the disk is completely stalled.

This seems like a bit of a non-answer...we have another cluster on older hardware with a slower network that does not have this problem. Is there a way to diagnose and stop it from happening?

I don't see how VMs grinding to a halt when backing up another VM is a feature...

 

[uzumo]

Changing your approach might be worth a try.

Would slowing down the network (bandwidth limitations for that processing, etc.) fix it? Maybe the disk I/O could keep up.

 

[powersupport]

Hi,

Backups can cause high disk I/O on the node, and Grafana/InfluxDB is sensitive to that. Even though both run on the same VM, the storage is shared, so the backup load can slow it down and trigger alerts.

You can try limiting backup speed, checking I/O usage, or moving the VM to faster storage.