High server load during backup creation

Discussion in 'Proxmox VE: Installation and configuration' started by Nico Haase, Nov 4, 2013.

  1. cesarpk

    cesarpk Member

    Joined:
    Mar 31, 2012
    Messages:
    770
    Likes Received:
    2
    #81 cesarpk, Jan 27, 2014
    Last edited: Jan 27, 2014
  2. Nico Haase

    Nico Haase Member

    Joined:
    Feb 27, 2013
    Messages:
    31
    Likes Received:
    1
    So, even after having updated to Proxmox v3.2 and a reboot for all changes to take place, running a backup to a NFS share still causes high load (~3.0) within a virtual machine. I have not monitored that machine for more detailled problems occuring within it, but even the high load should not occur. Is there any way to help you further to find the problem?

    Code:
    root@proxmox:~# pveversion -v
    proxmox-ve-2.6.32: 3.2-121 (running kernel: 2.6.32-27-pve)
    pve-manager: 3.2-1 (running version: 3.2-1/1933730b)
    pve-kernel-2.6.32-27-pve: 2.6.32-121
    pve-kernel-2.6.32-25-pve: 2.6.32-113
    pve-kernel-2.6.32-26-pve: 2.6.32-114
    lvm2: 2.02.98-pve4
    clvm: 2.02.98-pve4
    corosync-pve: 1.4.5-1
    openais-pve: 1.1.4-3
    libqb0: 0.11.1-2
    redhat-cluster-pve: 3.2.0-2
    resource-agents-pve: 3.9.2-4
    fence-agents-pve: 4.0.5-1
    pve-cluster: 3.0-12
    qemu-server: 3.1-15
    pve-firmware: 1.1-2
    libpve-common-perl: 3.0-14
    libpve-access-control: 3.0-11
    libpve-storage-perl: 3.0-19
    pve-libspice-server1: 0.12.4-3
    vncterm: 1.1-6
    vzctl: 4.0-1pve5
    vzprocps: 2.0.11-2
    vzquota: 3.1-2
    pve-qemu-kvm: 1.7-4
    ksm-control-daemon: 1.1-1
    glusterfs-client: 3.4.2-1
     
  3. e100

    e100 Active Member
    Proxmox Subscriber

    Joined:
    Nov 6, 2010
    Messages:
    1,235
    Likes Received:
    24
    Your original post said load went over 6 and now you observe 3. Seems like an improvement.

    Backups have an impact on disk IO, the size of that impact depends on your systems IO capabilities. Your IO subsystem can only go so fast and when you add a backup process to that IO you should expect higher load average. There is no free lunch.

    Be careful, looks like backups are causing VMs to crash now, KVM fails with a segmentation fault.
    http://forum.proxmox.com/threads/18069-After-update-to-3-2-VM-crashing-during-backup
     
  4. Nico Haase

    Nico Haase Member

    Joined:
    Feb 27, 2013
    Messages:
    31
    Likes Received:
    1
    As I stated some times: IO that happens on the VM host system should never, never affect the IO or load of the guest system. My server should be able to handle this without problems. Data by pveperf:
    Code:
    root@proxmox:~# pveperf
    CPU BOGOMIPS:      31997.08
    REGEX/SECOND:      839138
    HD SIZE:           94.49 GB (/dev/mapper/pve-root)
    BUFFERED READS:    111.14 MB/sec
    AVERAGE SEEK TIME: 7.90 ms
    FSYNCS/SECOND:     2278.53
    DNS EXT:           18.31 ms
    DNS INT:           0.96 ms (my.domain)
    The CPU is an Intel Xeon E5504, VMs are stored on a RAID 5 with Hot Spare, the backup is done using NFS onto an external server, according to the backup log with a speed of 50 MB/s average. The load is larger than 1.5 during the first three of five hours of the backup execution, while there is no other extraordinary usage on the server. During the last weeks, where we had the backup disabled, the load was never higher than 1.1, and even that is only reached very rarely...
     
  5. e100

    e100 Active Member
    Proxmox Subscriber

    Joined:
    Nov 6, 2010
    Messages:
    1,235
    Likes Received:
    24
    The totality of the disk IO will cause the load average of the host and the VMs to go up. What you think 'should never' happen is a wrong assumption on your part.
    You have a finite amount of IO to/from your VM disk.
    When the backup process is reading from your VM disk, that uses some of the total IO available to your VM disk.

    Your pveperf says that you can read at 111MB/sec, if your VM is reading 60MB/sec and the backup is reading at 51MB/sec you are using 100% of your total IO.
    If either the backup or the VM wants to (can) read more than that, then load average goes up, there is no way to avoid this unless your have infinite IO available.

    The load average is just one metric that gives you an idea of what is going on, it is ok for it to go up and is perfectly normal for it to go up especially when IO intensive tasks such as backups are taking place.
    Are you having a performance problem or just see a higher load average an 'think' there is a problem?

    If the backup is causing issues, because it is using too much IO, then you need to limit how fast the backup process reads OR you need to get a faster IO system.

    All that being said, I am not happy with the recent changes to the backups in Proxmox either. I have not put the latest version of Proxmox into production yet so I do not know, first hand, if there is an issue in 3.2. I did test KVM 1.7 ( whats in the latest version of Proxmox ) awhile back and it had greatly improved the performance of disk IO inside the VM while backups were taking place. You can see my benchmarks on the pve-devel mailing list archives here.
     
  6. cesarpk

    cesarpk Member

    Joined:
    Mar 31, 2012
    Messages:
    770
    Likes Received:
    2
    Hi e100

    Only for know, do you have tested the same test of I/O without a KVM Live backup in progress for compare the benchmark?
    If the answer is positive, can you post the results?

    Best regards
    Cesar
     
  7. e100

    e100 Active Member
    Proxmox Subscriber

    Joined:
    Nov 6, 2010
    Messages:
    1,235
    Likes Received:
    24
    Cesar, I have done lots of tests with IO inside the VM when backups are not being performed. The limitations, in my experience, is not the speed only IOPS. I can easily get 2000MB/sec sequential ( buss speed of my RAID card ) but when doing random IO the operations per second are limited by the speed of the CPU. KVM only has a single thread for IO, that's the limitation. If my memory is correct most of my systems top out at 30k IOPS even on the servers with 22 SSDs that can do 60k+ IOPS.

    Eric
     
  8. cesarpk

    cesarpk Member

    Joined:
    Mar 31, 2012
    Messages:
    770
    Likes Received:
    2
    Wow e100!!! (You are a Master and very noble for help to much people)
    Many thanks for your answer

    Best regards
    Cesar
     
  9. Nico Haase

    Nico Haase Member

    Joined:
    Feb 27, 2013
    Messages:
    31
    Likes Received:
    1
    Even if I don't monitor my VM all the time, I cannot think of anything happening within that causes a stable load of 60 MB/s on the I/O over three hours. As this happens during the night, there should be anything running that hard. But let's see, I pulled the handbrake down to 35MB/s.....
     
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice