Vzdump on NFS slows live VM down

[robhost]

We observed a strange issue, when using scheduled backups with vzdump (from local ssd storage, snapshot, lzo).

It seems when the NFS storage is under heavy network load or the uplink to the NFS node is "full", vzdump blocks the running VM to death.

We can see lots of "[sda] abort" messages inside the VM and the IO utilization and wait goes to 100%, always when the storage-networks seems to be saturated - loadavg inside the VM then goes very high and it becomes unusable.

Is this a bug, or normal behavior with vzdump on NFS? We tried "bwlimit" in vzdump.conf, but the problem still occours when the NFS uplink is full.

The same backups on local storage are running without problemlos, so this definitly a network/nfs issue here, and this should not bring the VM down which is backed up.

 

[Thomas Gutzmann]

Did you see this thread in the Ubuntu Forums?

large NFS copy locks up/hang client with large files (again) (lucid)

There seem to be some good ideas, especially #23 and #72, possibly #55.

There is also a reference to "Troubleshooting NFS".

As you don't see these problems with local storage, you may have to deal with an NFS and not a Proxmox problem here.

 

[robhost]

The hanging NFS ist one problem, but that should imho not bringing the running VM from which the backup is made in an unuseable state :/

 

[Thomas Gutzmann]

Well, Proxmox is just Linux and the VM to be backed up a number of processes within the server, so I can very well imagine that a backup of the client (which in any case is a heavy operation) can bring the client down or at least make it hang.

I'm not a friend of hot VM backups, by the way. With hot backups, too much can go wrong. I prefer cold backups after substantial changes (like a major OS upgrade or the installation/upgrade of subsystems, e.g. the database software). For regular (daily, hourly) backups I'm using rsnaphot which has the additional benefit of providing versioning.

 

[robhost]

Well, Proxmox is just Linux and the VM to be backed up a number of processes within the server, so I can very well imagine that a backup of the client (which in any case is a heavy operation) can bring the client down or at least make it hang.

I'm not a friend of hot VM backups, by the way. With hot backups, too much can go wrong. I prefer cold backups after substantial changes (like a major OS upgrade or the installation/upgrade of subsystems, e.g. the database software). For regular (daily, hourly) backups I'm using rsnaphot which has the additional benefit of providing versioning.

For daily backups we also use rsync and Bareos, but to have a VMA image by the hand is good to know in case of hardware emergency