After update to 3.2: VM crashing during backup
- Thread starter jens.kuespert
- Start date
Code:
root@pve-node3:~# /usr/bin/kvm -id 1080 -chardev socket,id=qmp,path=/var/run/qemu-server/1080.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/1080.vnc,x509,password -pidfile /var/run/qemu-server/1080.pid -name ftp -smp sockets=1,cores=2 -nodefaults -boot menu=on -vga cirrus -cpu kvm64,+lahf_lm,+x2apic,+sep -k en-us -m 1024 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -drive if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100 -drive file=iscsi://172.16.99.15/iqn.1986-03.com.sun:02:stor2.xxxxxx/1,if=none,id=drive-virtio0,aio=native,cache=none -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=200 -netdev type=tap,id=net0,ifname=tap1080i0,script=/var/lib/qemu-server/pve-bridge,vhost=on -device virtio-net-pci,romfile=,mac=52:39:70:DD:CF:86,netdev=net0,bus=pci.0,addr=0x12,id=net0
main-loop: WARNING: I/O thread spun for 1000 iterations
Warning: no scancode found for keysym 3
Warning: no scancode found for keysym 3
Warning: no scancode found for keysym 3
Segmentation faultTo create a load I used "dbench 128". VM crash very quickly.
How I could get more verbose messages?
Last edited:
Clearly there is a serious flaw in KVM or a bunch of you suddenly have bad RAM.
Thanks for the test johnplv
Do other people also get the same errors when performing this test?
Code:
main-loop: WARNING: I/O thread spun for 1000 iterations
Warning: no scancode found for keysym 3
Warning: no scancode found for keysym 3
Warning: no scancode found for keysym 3
[B]Segmentation fault[/B]Thanks for the test johnplv
Do other people also get the same errors when performing this test?
Last edited:
Code:
$ sudo qm showcmd 302
/usr/bin/kvm -id 302 -chardev socket,id=qmp,path=/var/run/qemu-server/302.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/302.vnc,x509,password -pidfile /var/run/qemu-server/302.pid -daemonize -name monit
-smp sockets=1,cores=2 -nodefaults -boot menu=on -vga cirrus -cpu kvm64,+lahf_lm,+x2apic,+sep -k en-us -m 2048 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -drive if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200
-drive file=/dev/vg0/vm-302-disk-1,if=none,id=drive-virtio0,cache=none,aio=native -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap302i0,script=/var/lib/qemu-server/pve-bridge,vhost=on
-device virtio-net-pci,mac=A2:B6:45:66:C6:EC,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300
Code:
$ sudo vzdump 302 --mode snapshot --compress gzip
INFO: starting new backup job: vzdump 302 --mode snapshot --compress gzip
INFO: Starting Backup of VM 302 (qemu)
INFO: status = [COLOR=#ff0000]stopped[/COLOR]
INFO: update VM 302: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/srv/dump/vzdump-qemu-302-2014_03_21-22_26_29.vma.gz'
INFO: starting kvm to execute backup task
INFO: started backup task 'c8b6576d-cff1-4d22-9e52-f9f2b064b4a1'
INFO: status: 1% (239534080/16110321664), sparse 1% (182738944), duration 3, 79/18 MB/s
...
INFO: status: 100% (16110321664/16110321664), sparse 20% (3283845120), duration 489, 0/0 MB/s
INFO: transferred 16110 MB in 489 seconds (32 MB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 6.25GB
INFO: delete old backup '/srv/dump/vzdump-qemu-302-2014_03_13-00_08_14.vma.gz'
INFO: Finished Backup of VM 302 (00:08:12)
INFO: Backup job finished successfully
Code:
$ sudo vzdump 302 --compress gzip
INFO: starting new backup job: vzdump 302 --compress gzip
INFO: Starting Backup of VM 302 (qemu)
INFO: status = [COLOR=#ff0000]running[/COLOR]
INFO: update VM 302: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/srv/dump/vzdump-qemu-302-2014_03_21-22_44_55.vma.gz'
INFO: started backup task '748b0226-12b9-4d9b-893e-3e3ae0c5b15a'
INFO: status: 1% (290914304/16110321664), sparse 1% (234041344), duration 3, 96/18 MB/s
INFO: status: 7% (1197342720/16110321664), sparse 6% (1043898368), duration 6, 302/32 MB/s
INFO: status: 8% (1315176448/16110321664), sparse 6% (1081188352), duration 9, 39/26 MB/s
INFO: status: 9% (1467219968/16110321664), sparse 6% (1082793984), duration 16, 21/21 MB/s
ERROR: VM 302 not running
INFO: aborting backup job
ERROR: VM 302 not running
ERROR: Backup of VM 302 failed - VM 302 not running
INFO: Backup job finished with errors
job errorsInfo from gdb:
Code:
Program received signal SIGSEGV, Segmentation fault.
0x00007f8bf77ce61a in ?? ()
(gdb) backtrace
#0 0x00007f8bf77ce61a in ?? ()
#1 0x00007f8bf77ced75 in ?? ()
#2 0x00007f8bf77ce4f1 in ?? ()
#3 0x00007f8bf77ce888 in ?? ()
#4 0x00007f8bf781cc4d in ?? ()
#5 0x00007f8bf76b2e8a in ?? ()
#6 0x00007f8bf146b020 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x00007fff9bd03a40 in ?? ()
#8 0x0000000000000000 in ?? ()
(gdb) info frame
Stack level 6, frame at 0x7f8b9e4009b8:
rip = 0x7f8bf146b020 in __start_context; saved rip 0x7fff9bd03a40
called by frame at 0x7f8b9e4009c0, caller of frame at 0x7f8b9e4009b0
Arglist at 0x7f8b9e4009a8, args:
Locals at 0x7f8b9e4009a8, Previous frame's sp is 0x7f8b9e4009b8
Saved registers:
rip at 0x7f8b9e4009b0
(gdb) info registers
rax 0x0 0
rbx 0x7f8b9e4009b0 140237632178608
rcx 0x0 0
rdx 0x17680000000040 6588273673633856
rsi 0x0 0
rdi 0x7f8bb2b01000 140237975064576
rbp 0x7f8bf766ee80 0x7f8bf766ee80
rsp 0x7f8b9e4009b0 0x7f8b9e4009b0
r8 0x7f8b9e400890 140237632178320
r9 0x7f8b9aa5e558 140237571745112
r10 0x0 0
r11 0x0 0
r12 0x18 24
r13 0xc7b608 13088264
r14 0xf 15
r15 0x7fff9bd042d0 140735807505104
rip 0x7f8bf146b020 0x7f8bf146b020 <__start_context>
eflags 0x10246 [ PF ZF IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
(gdb) info threads
Id Target Id Frame
4 Thread 0x7f8be88da700 (LWP 818073) 0x00007f8bf14fd137 in ioctl () at ../sysdeps/unix/syscall-template.S:82
3 Thread 0x7f8be7ed9700 (LWP 818074) 0x00007f8bf14fd137 in ioctl () at ../sysdeps/unix/syscall-template.S:82
2 Thread 0x7f8ba63ff700 (LWP 818076) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
* 1 Thread 0x7f8bf748c900 (LWP 818057) 0x00007f8bf77ce61a in ?? ()
Hi. I have it too now. First I thought the problem is related to QXL but on weekend it happened to a machine with standard graphics. One backup a night with nfs storage. I never had problem before. Mar 24 02:00:01 node1 /USR/SBIN/CRON[84892]: (root) CMD (vzdump 114 --quiet 1 --mode snapshot --mailto XXX --compress lzo --storage NFS-NAS) Mar 24 02:00:02 node1 vzdump[84892]: starting task UPID:node1:00014B9E:0B0D9120:532F8392:vzdump::root@pam: Mar 24 02:00:02 node1 vzdump[84894]: INFO: starting new backup job: vzdump 114 --quiet 1 --mailto XXX --mode snapshot --compress lzo --storage NFS-NAS Mar 24 02:00:02 node1 vzdump[84894]: INFO: Starting Backup of VM 114 (qemu) Mar 24 02:00:03 node1 qm[84899]: update VM 114: -lock backup Mar 24 02:02:54 node1 pvestatd[316944]: status update time (6.893 seconds) Mar 24 02:04:00 node1 vzdump[84894]: VM 114 qmp command failed - VM 114 not running Mar 24 02:04:00 node1 vzdump[84894]: VM 114 qmp command failed - VM 114 not running Mar 24 02:04:01 node1 kernel: vmbr0: port 2(tap114i0) entering disabled state Mar 24 02:04:01 node1 kernel: vmbr0: port 2(tap114i0) entering disabled state Mar 24 02:04:01 node1 vzdump[84894]: ERROR: Backup of VM 114 failed - VM 114 not running Mar 24 02:04:01 node1 vzdump[84894]: INFO: Backup job finished with errors Mar 24 02:04:01 node1 vzdump[84894]: job errors pveversion -v proxmox-ve-2.6.32: 3.2-121 (running kernel: 2.6.32-27-pve) pve-manager: 3.2-1 (running version: 3.2-1/1933730b) pve-kernel-2.6.32-27-pve: 2.6.32-121 pve-kernel-2.6.32-24-pve: 2.6.32-111 pve-kernel-2.6.32-26-pve: 2.6.32-114 lvm2: 2.02.98-pve4 clvm: 2.02.98-pve4 corosync-pve: 1.4.5-1 openais-pve: 1.1.4-3 libqb0: 0.11.1-2 redhat-cluster-pve: 3.2.0-2 resource-agents-pve: 3.9.2-4 fence-agents-pve: 4.0.5-1 pve-cluster: 3.0-12 qemu-server: 3.1-15 pve-firmware: 1.1-2 libpve-common-perl: 3.0-14 libpve-access-control: 3.0-11 libpve-storage-perl: 3.0-19 pve-libspice-server1: 0.12.4-3 vncterm: 1.1-6 vzctl: 4.0-1pve5 vzprocps: 2.0.11-2 vzquota: 3.1-2 pve-qemu-kvm: 1.7-4 ksm-control-daemon: 1.1-1 glusterfs-client: 3.4.2-1 cat /etc/pve/qemu-server/114.conf boot: c cores: 4 cpuunits: 25000 freeze: 0 ide1: none,media=cdrom kvm: 1 memory: 16384 name: XXX net0: virtio=9A:3B:BC
8:C7:E5,bridge=vmbr0 onboot: 1 ostype: win7 scsihw: virtio-scsi-pci sockets: 1 tablet: 0 virtio0: LVM-VM-Disks:vm-114-disk-1,cache=none virtio1: LVM-VM-Disks:vm-114-disk-2,cache=none virtio2: LVM-VM-Disks2:vm-114-disk-1,cache=none,backup=no,size=200G
8:C7:E5,bridge=vmbr0 onboot: 1 ostype: win7 scsihw: virtio-scsi-pci sockets: 1 tablet: 0 virtio0: LVM-VM-Disks:vm-114-disk-1,cache=none virtio1: LVM-VM-Disks:vm-114-disk-2,cache=none virtio2: LVM-VM-Disks2:vm-114-disk-1,cache=none,backup=no,size=200G
Last edited:
NexentaStor 3.1.3.5 via ZFS plugin (12x2Tb SAS + SSD for ZIL and ARC)
10Gb network
backup to local SATA disk 3Tb
Can you reproduce the crash if you backup to /dev/null
# vzdump <VMID> -stdout >/dev/null
Compression as in compressing the backup package or as in cluster wide compression (migration_unsecure: 1). I have migration_unsecure: 1 and my backups works without flaws even to a slow Qnap via NFS.
Hello mir,
when I disable compression, the backup in snapshot mode works fine.
I test 10 backups, all 10 backups completes with no error.
But this is not a solution for the problem, just a hint in the right direction.
The backupfile is threetimes bigger than with compression.
Best regards Stephan
My storage is lvm for images anf nfs for backup. I changed /etc/storage to lvm: LVM-VM-Disks vgname vmdisks shared content images nfs: NFS-NAS path /mnt/pve/NFS-NAS server 192.148.150.9 export /Sicherung options vers=3,nolock -> added this content backup maxfiles 2 Lets see what happens.
I did a lot of tests. To create a load using various utilities (stress, fio, bonnie++, dbench)
and in our case, the best was "dbench -s 128" inside VM (Key "-s" very important)
and start backup process with GZIP !!!, without compression or LZO - VM don't crush
Where from does not matter (iSCSI, local, NFS, /dev/null, etc..)
Last edited: