Backup Problem. Way to gracefully stop backup cron job.

[vcp_ai]

Hi, I have a Server with 6 WinXP machines and 2 debian (trunkey core), all them KVM

I have three mounts for our NFS Server (Openfiler)
-1 x isos
-1 x backups2 ( keep two copies)
-1 x backups4 ( keep four copies)

I have an scheduled backup every two days for this 8 machines, running against backups4 mount.
Usually backup time for each machine (10 Gb raw disk) is 3-4 minutes.

Server Web is on a 100Mb network
NFS is on a 1 Gb independent network

But some times do not why, I find that backups take form 30 to 40 minutes each machine.

During this process I can find in SYSLOG:

Jun 19 11:03:00 servidor176 pvestatd[729170]: status update time (6.139 seconds)
Jun 19 11:03:06 servidor176 pvestatd[729170]: WARNING: command 'df -P -B 1 /mnt/pve/Of.201.20.isos' failed: got timeout
Jun 19 11:03:08 servidor176 pvestatd[729170]: WARNING: command 'df -P -B 1 /mnt/pve/Of.201.20.backups2' failed: got timeout
Jun 19 11:03:10 servidor176 pvestatd[729170]: WARNING: command 'df -P -B 1 /mnt/pve/Of.201.20.backups4' failed: got timeout
Jun 19 11:03:10 servidor176 pvestatd[729170]: status update time (6.145 seconds)
Jun 19 11:03:16 servidor176 pvestatd[729170]: WARNING: command 'df -P -B 1 /mnt/pve/Of.201.20.isos' failed: got timeout
Jun 19 11:03:18 servidor176 pvestatd[729170]: WARNING: command 'df -P -B 1 /mnt/pve/Of.201.20.backups2' failed: got timeout
Jun 19 11:03:20 servidor176 pvestatd[729170]: WARNING: command 'df -P -B 1 /mnt/pve/Of.201.20.backups4' failed: got timeout
Jun 19 11:03:20 servidor176 pvestatd[729170]: status update time (6.149 seconds)
Jun 19 11:03:30 servidor176 pvestatd[729170]: WARNING: command 'df -P -B 1 /mnt/pve/Of.201.20.isos' failed: got timeout
Jun 19 11:03:32 servidor176 pvestatd[729170]: WARNING: command 'df -P -B 1 /mnt/pve/Of.201.20.backups2' failed: got timeout
Jun 19 11:03:34 servidor176 pvestatd[729170]: WARNING: command 'df -P -B 1 /mnt/pve/Of.201.20.backups4' failed: got timeout
Jun 19 11:03:34 servidor176 pvestatd[729170]: status update time (9.573 seconds)
Jun 19 11:03:36 servidor176 pvestatd[729170]: WARNING: command 'df -P -B 1 /mnt/pve/Of.201.20.isos' failed: got timeout

It seems that vzdump program is constantly doing 'df -P -B 1 /mnt ...... ' to check for space on ALL mounts on the server (backup is executing against /mnt/pve/Of.201.20.backups4) .
, df gets a timeout, and it 'gets in some kind of loop' that makes backup last for a very long time .

On ssh df takes about 8 to 10 seconds during this process.


May be increasing df timeout OR reducing periodicity of it, OR doing df only against the destination of the backup, would help ....

root@servidor176:~# pveversion -v
pve-manager: 2.1-1 (pve-manager/2.1/f9b0f63a)
running kernel: 2.6.32-12-pve
proxmox-ve-2.6.32: 2.1-68
pve-kernel-2.6.32-12-pve: 2.6.32-68
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.3-1
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.8-3
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.7-2
pve-cluster: 1.0-26
qemu-server: 2.0-39
pve-firmware: 1.0-16
libpve-common-perl: 1.0-27
libpve-access-control: 1.0-21
libpve-storage-perl: 2.0-18
vncterm: 1.0-2
vzctl: 3.0.30-2pve5
vzprocps: 2.0.11-2
vzquota: 3.0.12-3
pve-qemu-kvm: 1.0-9
ksm-control-daemon: 1.1-1
root@servidor176:~#

During this time Server GUI is not partially responsive (status, grafs, do not work)

So will like to have a way to gracefully stop the complete backup schedule (from ssh).

Regards

 

[dietmar]

On ssh df takes about 8 to 10 seconds during this process.


May be increasing df timeout OR reducing periodicity of it, OR doing df only against the destination of the backup, would help ....

This just shows that there is too much load on your NFS server. I suggest you limit backup speed.

 

[rengiared]

started an extra thread for my problem
pls delete this post