Backup loses vm data

Proximate

Member
Feb 13, 2022
222
13
23
65
Hi,

Running the backups is causing hosts across the cluster to lose data, presumably because the backups are heavy, causing latency.
Other than trying to stagger the backups, are there any other tricks someone could share?
 
Hey,

what do you mean with loosing data? Some things to consider
- the cluster needs its own network
- backups target storage should not be the same as VM storage
- if you use ceph, it should also have its own network. You really don't want high network load impacting your VMs IO performance, of course also not the other way around
- more generally, if you use network storage for your VMs, that should not share the network with normal traffic. Same reason as for ceph, just more important for ceph.
 
  • Like
Reactions: LnxBil
Hi,

Not using ceph.

What I mean is that there is real time data streaming into the network 24/7 and during the backups, we've noticed that some data is being lost. Some devices sending data, some of it goes missing during the backups.

Each host has its own local storage that the vms use/run on.
Cluster backups are sent to a TrueNAS host on the LAN but yes, on the same network.

Even if I were to use another interface for backups on each host, that traffic would still be flowing in and out of the host.
I used to have separate interfaces for management and vms on vmware but this setup is not doing that.

One idea might be to enable a 1GbE interface for backups which might force backups to run a little slower while the cluster keeps using its own 10GbE network but still on the same host/LAN.

Maybe I'm missing something here.