Results 1 to 11 of 11

Thread: New kernel backup again crashes

  1. #1
    Join Date
    Jun 2010
    Posts
    101

    Default New kernel backup again crashes

    Hi there,

    Again with the new kernel the backup will stall, but this time the compute module will not responde so the HA KVM will migrate to other servers. So far so good, but it will crash one one backup of a KVM HA we will get failed startup of the HA process because the VM is locked by the backup process.
    So the auto start on a other server will not work!

    This is a real problem!

    Code:
    INFO: starting new backup job: vzdump --quiet 1 --mailto proxmox2@xxxxxxxxx.nl --mode snapshot --compress lzo --maxfiles 2 --storage backup_day --all 1
    INFO: Starting Backup of VM 103 (openvz)
    INFO: CTID 103 exist unmounted down
    INFO: status = stopped
    INFO: backup mode: stop
    INFO: ionice priority: 7
    INFO: creating archive '/mnt/pve/backup_day/dump/vzdump-openvz-103-2012_06_01-03_45_01.tar.lzo'
    INFO: Total bytes written: 1199616000 (1.2GiB, 7.7MiB/s)
    INFO: archive file size: 484MB
    INFO: delete old backup '/mnt/pve/backup_day/dump/vzdump-openvz-103-2012_05_28-03_55_49.tar.lzo'
    INFO: Finished Backup of VM 103 (00:02:52)
    INFO: Starting Backup of VM 104 (qemu)
    INFO: status = stopped
    INFO: backup mode: stop
    INFO: ionice priority: 7
    INFO: creating archive '/mnt/pve/backup_day/dump/vzdump-qemu-104-2012_06_01-03_47_53.tar.lzo'
    INFO: adding '/mnt/pve/backup_day/dump/vzdump-qemu-104-2012_06_01-03_47_53.tmp/qemu-server.conf' to archive ('qemu-server.conf')
    INFO: adding '/dev/vmdisks/vm-104-disk-1' to archive ('vm-disk-ide0.raw')
    INFO: Total bytes written: 34359740928 (14.58 MiB/s)
    INFO: archive file size: 7.48GB
    INFO: delete old backup '/mnt/pve/backup_day/dump/vzdump-qemu-104-2012_05_28-04_16_33.tar.lzo'
    INFO: Finished Backup of VM 104 (00:39:15)
    INFO: Starting Backup of VM 200 (qemu)
    INFO: status = running
    INFO: backup mode: snapshot
    INFO: ionice priority: 7
    INFO:   Logical volume "vzsnap-sint-0" created
    INFO: creating archive '/mnt/pve/backup_day/dump/vzdump-qemu-200-2012_06_01-04_27_08.tar.lzo'
    INFO: adding '/mnt/pve/backup_day/dump/vzdump-qemu-200-2012_06_01-04_27_08.tmp/qemu-server.conf' to archive ('qemu-server.conf')
    INFO: adding '/dev/vmdisks/vzsnap-sint-0' to archive ('vm-disk-virtio0.raw')
    Then it is unresponsive an then we will get:
    Code:
    task started by HA resource agent
    TASK ERROR: VM is locked (backup)
    multiple times on all our hosts.

    We have to do manualy:
    Code:
    qm unlock 200
    and then it will start, but then the HA is useless.

    Also is it possible to downgrade the kernel? to the "old" of 2.1, that was really stable?

    It crashes on multiple IMS compute modules as host and the are in short:
    Code:
    RAM usageTotal: 23.48GB
    Used: 6.20GB
    CPUs
    16 x Intel(R) Xeon(R) CPU L5630 @ 2.13GHz
    PVE Manager version
    pve-manager/2.1-1/f9b0f63a
    Kernel version
    Linux 2.6.32-12-pve #1 SMP Tue May 15 06:02:20 CEST 2012
    It crashes always on a HA KVM with storage in the LVM pool.


    the only thing in /var/log/message is:
    sint kernel: igb_rx:HBO bit set..
    and that is the gigabit driver

    root@sint:~# pveversion -vpve-manager: 2.1-1 (pve-manager/2.1/f9b0f63a)
    running kernel: 2.6.32-12-pve
    proxmox-ve-2.6.32: 2.1-68
    pve-kernel-2.6.32-12-pve: 2.6.32-68
    pve-kernel-2.6.32-7-pve: 2.6.32-60
    lvm2: 2.02.95-1pve2
    clvm: 2.02.95-1pve2
    corosync-pve: 1.4.3-1
    openais-pve: 1.1.4-2
    libqb: 0.10.1-2
    redhat-cluster-pve: 3.1.8-3
    resource-agents-pve: 3.9.2-3
    fence-agents-pve: 3.1.7-2
    pve-cluster: 1.0-26
    qemu-server: 2.0-39
    pve-firmware: 1.0-16
    libpve-common-perl: 1.0-27
    libpve-access-control: 1.0-21
    libpve-storage-perl: 2.0-18
    vncterm: 1.0-2
    vzctl: 3.0.30-2pve5
    vzprocps: 2.0.11-2
    vzquota: 3.0.12-3
    pve-qemu-kvm: 1.0-9
    ksm-control-daemon: 1.1-1
    we have the problem on multiple hosts...
    Last edited by bazzi; 06-01-2012 at 08:51 AM. Reason: added info

  2. #2
    Join Date
    Aug 2006
    Posts
    9,759

    Default Re: New kernel backup again crashes

    Quote Originally Posted by bazzi View Post
    Also is it possible to downgrade the kernel? to the "old" of 2.1, that was really stable?
    ...
    yes, you can just boot the old kernel (as far as I see you still have pve-kernel-2.6.32-7-pve installed, so the bootloader will still offer this one).
    Best regards,
    Tom

    Do you have already a Commercial Support Subscription? - If not, Buy now

  3. #3
    Join Date
    Jun 2010
    Posts
    101

    Default Re: New kernel backup again crashes

    Quote Originally Posted by tom View Post
    yes, you can just boot the old kernel (as far as I see you still have pve-kernel-2.6.32-7-pve installed, so the bootloader will still offer this one).
    I just installed 2.6.32.11 and rebooted and the backup still runs, I will post the result.
    The
    sint kernel: igb_rx:HBO bit set..
    is gone from /var/log/messages

    But the lock for HA KVM is a real problem because it makes the HA useless if there was a problem during a backup...
    Last edited by bazzi; 06-01-2012 at 09:29 AM.

  4. #4
    Join Date
    Jun 2010
    Posts
    101

    Default Re: New kernel backup again crashes

    Ok the backups finish without a problem. So the New kernel crashes kvm backups on a IMS.

    Also the HA feature is broken if its go wrong during a backup. So 2 major problems, but the work around is use the old 2.6.32.11 kernel!

  5. #5
    Join Date
    Jun 2010
    Posts
    101

    Default Re: New kernel backup again crashes

    Ok multiple backup's alter I can confirm it runs again smooth. So there is really a problem with the 2.6.32.12 kernel on IMS.

  6. #6
    Join Date
    Aug 2006
    Posts
    9,759

    Default Re: New kernel backup again crashes

    can you tell all detail about your IMS (including firmware version)?

    also other reported problems with the latest intel driver (see http://sourceforge.net/p/e1000/bugs/341/)
    Best regards,
    Tom

    Do you have already a Commercial Support Subscription? - If not, Buy now

  7. #7
    Join Date
    Jun 2010
    Posts
    101

    Default Re: New kernel backup again crashes

    Offcourse!

    ----------------------------------------------------------------
    Firmware

    Current Build Version: 10.4.100.20110602.29753

    Firmware Inventory


    Component Subsystem Status Current Version
    Server 1 BMC Firmware ok 1.26.3
    BMC Boot ok 0.28
    BIOS ok S5500.86B.01.20.0055.050420111308
    Server 2 BMC Firmware ok 1.26.3
    BMC Boot ok 0.28
    BIOS ok S5500.86B.01.20.0055.050420111308
    Server 3 BMC Firmware ok 1.26.3
    BMC Boot ok 0.28
    BIOS ok S5500.86B.01.20.0055.050420111308
    Server 4 BMC Firmware not present --
    BMC Boot not present --
    BIOS not present --
    Server 5 BMC Firmware not present --
    BMC Boot not present --
    BIOS not present --
    Server 6 BMC Firmware not present --
    BMC Boot not present --
    BIOS not present --
    Switch 1 Firmware ok 1.0.0.27
    Boot ok 1.0.0.6
    Switch 2 Firmware not present --
    Boot not present --
    Storage Control Module 1 Firmware ok 3.08.0140.08
    Storage Control Module 2 Firmware not present --
    System Fan 1 Firmware ok 1.2
    Boot ok 1.2
    System Fan 2 Firmware ok 1.2
    Boot ok 1.2
    I/O Fan Firmware ok 1.2
    Boot ok 1.2
    Power Supply 1 Firmware not applicable --
    Boot not applicable --
    Power Supply 2 Firmware not applicable --
    Boot not applicable --
    Power Supply 3 Firmware not applicable --
    Boot not applicable --
    Power Supply Blank 4 Firmware ok 1.2
    Boot ok 1.2

  8. #8
    Join Date
    Jun 2010
    Posts
    101

    Default Re: New kernel backup again crashes

    As you can see it is a MFSYS25V2 with 3 compute modules (MFS5520VI, with each 2x L5630 AND 24GB RAM).
    There are 7x 900GB (ST9900805SS) and those provide a LVM shared disk, just like one your wiki.

  9. #9
    Join Date
    Jun 2010
    Posts
    101

    Default Re: New kernel backup again crashes

    the hint in the bug report that is an issue with a fiber/serdes device, dus I don't have a mezzanine card in the modules:
    root@sint:~# lspci |grep Ether
    01:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
    01:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)

  10. #10
    Join Date
    Aug 2006
    Posts
    9,759

    Default Re: New kernel backup again crashes

    pls report your specific issue in a new bug report (http://sourceforge.net/p/e1000/bugs/)
    Best regards,
    Tom

    Do you have already a Commercial Support Subscription? - If not, Buy now

  11. #11
    Join Date
    Jun 2010
    Posts
    101

    Default Re: New kernel backup again crashes


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •