After update to 3.2: VM crashing during backup

e100 said:
We need a way to reproduce this if we want it fixed.
Maybe more information will lead to a way to reliably reproduce the problem.

pveversion -v
proxmox-ve-2.6.32: 3.2-121 (running kernel: 2.6.32-27-pve)
pve-manager: 3.2-1 (running version: 3.2-1/1933730b)
pve-kernel-2.6.32-27-pve: 2.6.32-121
pve-kernel-2.6.32-28-pve: 2.6.32-123
pve-kernel-2.6.32-26-pve: 2.6.32-114
lvm2: 2.02.98-pve4
clvm: 2.02.98-pve4
corosync-pve: 1.4.5-1
openais-pve: 1.1.4-3
libqb0: 0.11.1-2
redhat-cluster-pve: 3.2.0-2
resource-agents-pve: 3.9.2-4
fence-agents-pve: 4.0.5-1
pve-cluster: 3.0-12
qemu-server: 3.1-15
pve-firmware: 1.1-2
libpve-common-perl: 3.0-14
libpve-access-control: 3.0-11
libpve-storage-perl: 3.0-19
pve-libspice-server1: 0.12.4-3
vncterm: 1.1-6
vzctl: 4.0-1pve5
vzprocps: 2.0.11-2
vzquota: 3.1-2
pve-qemu-kvm: 1.7-4
ksm-control-daemon: 1.1-1
glusterfs-client: 3.4.2-1



qm showcmd 110
/usr/bin/kvm -id 110 -chardev socket,id=qmp,path=/var/run/qemu-server/110.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/110.vnc,x509,password -pidfile /var/run/qemu-server/110.pid -daemonize -name Zabbix -smp sockets=2,cores=1 -nodefaults -boot menu=on -vga qxl -cpu kvm64,+lahf_lm,+x2apic,+sep -k de -spice tls-port=61002,addr=127.0.0.1,tls-ciphers=DES-CBC3-SHA,seamless-migration=on -device virtio-serial,id=spice,bus=pci.0,addr=0x9 -chardev spicevmc,id=vdagent,name=vdagent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -m 2048 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -drive if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 -drive file=/var/lib/vz/images/110/vm-110-disk-1.qcow2,if=none,id=drive-virtio0,format=qcow2,aio=native,cache=none -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap110i0,script=/var/lib/qemu-server/pve-bridge,vhost=on -device virtio-net-pci,mac=00:50:56:00:40:54,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300



VM 110 stopped
vzdump 110 --mode snapshot --compress gzip --storage backup
INFO: starting new backup job: vzdump 110 --mode snapshot --compress gzip --storage backup
INFO: Starting Backup of VM 110 (qemu)
INFO: status = stopped
INFO: update VM 110: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/mnt/backup-server/iBackOffice/dump/vzdump-qemu-110-2014_03_21-20_00_21.vma.gz'
INFO: starting kvm to execute backup task
INFO: started backup task '36b1ef9a-ebb0-43c2-89b1-34c62f3038f4'
INFO: status: 0% (125435904/53687091200), sparse 0% (25870336), duration 3, 41/33 MB/s
INFO: status: 1% (775421952/53687091200), sparse 1% (591761408), duration 6, 216/28 MB/s
INFO: status: 2% (1106116608/53687091200), sparse 1% (681570304), duration 11, 66/48 MB/s
INFO: status: 3% (1619263488/53687091200), sparse 1% (739471360), duration 39, 18/16 MB/s
INFO: status: 4% (2151415808/53687091200), sparse 1% (748359680), duration 71, 16/16 MB/s
INFO: status: 5% (2831679488/53687091200), sparse 1% (883367936), duration 106, 19/15 MB/s
INFO: status: 6% (3230924800/53687091200), sparse 1% (884117504), duration 131, 15/15 MB/s
INFO: status: 7% (3763077120/53687091200), sparse 1% (913559552), duration 162, 17/16 MB/s
INFO: status: 8% (4295229440/53687091200), sparse 1% (932290560), duration 197, 15/14 MB/s
INFO: status: 9% (4983226368/53687091200), sparse 2% (1084465152), duration 232, 19/15 MB/s
INFO: status: 10% (5382340608/53687091200), sparse 2% (1088126976), duration 256, 16/16 MB/s
INFO: status: 11% (5922095104/53687091200), sparse 2% (1102012416), duration 287, 17/16 MB/s
INFO: status: 12% (6442844160/53687091200), sparse 2% (1162391552), duration 308, 24/21 MB/s
INFO: status: 13% (6982598656/53687091200), sparse 2% (1238347776), duration 333, 21/18 MB/s
INFO: status: 14% (7564820480/53687091200), sparse 2% (1407479808), duration 346, 44/31 MB/s
INFO: status: 15% (8069709824/53687091200), sparse 2% (1424343040), duration 368, 22/22 MB/s
INFO: status: 16% (8594259968/53687091200), sparse 2% (1432915968), duration 396, 18/18 MB/s
INFO: status: 17% (9268887552/53687091200), sparse 2% (1571651584), duration 424, 24/19 MB/s
INFO: status: 18% (9681371136/53687091200), sparse 2% (1576980480), duration 445, 19/19 MB/s
INFO: status: 19% (10202120192/53687091200), sparse 2% (1585278976), duration 472, 19/18 MB/s
INFO: status: 20% (10749083648/53687091200), sparse 2% (1592909824), duration 505, 16/16 MB/s
INFO: status: 21% (11429871616/53687091200), sparse 3% (1745719296), duration 533, 24/18 MB/s
INFO: status: 22% (11824791552/53687091200), sparse 3% (1756463104), duration 553, 19/19 MB/s
INFO: status: 23% (12357337088/53687091200), sparse 3% (1774313472), duration 580, 19/19 MB/s
INFO: status: 24% (12908494848/53687091200), sparse 3% (1802145792), duration 607, 20/19 MB/s
INFO: status: 25% (13581287424/53687091200), sparse 3% (1968594944), duration 633, 25/19 MB/s
INFO: status: 26% (13976600576/53687091200), sparse 3% (1975304192), duration 653, 19/19 MB/s
INFO: status: 27% (14516355072/53687091200), sparse 3% (1991131136), duration 680, 19/19 MB/s
INFO: status: 28% (15048507392/53687091200), sparse 3% (2015571968), duration 705, 21/20 MB/s
INFO: status: 29% (15721299968/53687091200), sparse 4% (2205073408), duration 730, 26/19 MB/s
INFO: status: 30% (16120414208/53687091200), sparse 4% (2253504512), duration 748, 22/19 MB/s
INFO: status: 31% (16652566528/53687091200), sparse 4% (2331492352), duration 772, 22/18 MB/s
INFO: status: 32% (17188519936/53687091200), sparse 4% (2442821632), duration 795, 23/18 MB/s
INFO: status: 33% (17980325888/53687091200), sparse 5% (3100598272), duration 800, 158/26 MB/s
INFO: status: 45% (24421990400/53687091200), sparse 17% (9370972160), duration 803, 2147/57 MB/s
INFO: status: 53% (28644999168/53687091200), sparse 25% (13498294272), duration 806, 1407/31 MB/s
INFO: status: 57% (30750801920/53687091200), sparse 28% (15377985536), duration 815, 233/25 MB/s
INFO: status: 58% (31567839232/53687091200), sparse 30% (16130191360), duration 818, 272/21 MB/s
INFO: status: 61% (32982040576/53687091200), sparse 32% (17430663168), duration 821, 471/37 MB/s
INFO: status: 65% (35072638976/53687091200), sparse 36% (19424493568), duration 824, 696/32 MB/s
INFO: status: 69% (37212651520/53687091200), sparse 39% (21412802560), duration 829, 428/30 MB/s
INFO: status: 73% (39345061888/53687091200), sparse 43% (23374131200), duration 835, 355/28 MB/s
INFO: status: 81% (43636490240/53687091200), sparse 51% (27489861632), duration 842, 613/25 MB/s
INFO: status: 100% (53687091200/53687091200), sparse 69% (37537918976), duration 844, 5025/1 MB/s
INFO: transferred 53687 MB in 844 seconds (63 MB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 5.07GB
INFO: delete old backup '/mnt/backup-server/iBackOffice/dump/vzdump-qemu-110-2014_03_05-00_41_27.vma.gz'
INFO: Finished Backup of VM 110 (00:14:06)
INFO: Backup job finished successfully



VM 110 running
vzdump 110 --mode snapshot --compress gzip --storage backup
INFO: starting new backup job: vzdump 110 --mode snapshot --compress gzip --storage backup
INFO: Starting Backup of VM 110 (qemu)
INFO: status = running
INFO: update VM 110: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/backup-server/iBackOffice/dump/vzdump-qemu-110-2014_03_21-20_23_16.vma.gz'
INFO: started backup task '2feb8937-2f90-43bb-8ef8-8f354320050b'
INFO: status: 0% (136839168/53687091200), sparse 0% (27357184), duration 3, 45/36 MB/s
INFO: status: 1% (789970944/53687091200), sparse 1% (591691776), duration 6, 217/29 MB/s
INFO: status: 2% (1109917696/53687091200), sparse 1% (681500672), duration 10, 79/57 MB/s
ERROR: VM 110 not running
INFO: aborting backup job
ERROR: VM 110 not running
ERROR: Backup of VM 110 failed - VM 110 not running
INFO: Backup job finished with errors
job errors


Results
backup mode: snapshot -> backup crash the VM
backup mode: stopped -> backup runnning fine

Best regards Stephan
Click to expand...
 
shartenauer said:
e100 said:
We need a way to reproduce this if we want it fixed.
Maybe more information will lead to a way to reliably reproduce the problem.

Hello e100,

OK here I started the VM with the command output of 'qm showcmd 110'

vzdump 110 --mode snapshot --compress gzip --storage backup
INFO: starting new backup job: vzdump 110 --mode snapshot --compress gzip --storage backup
INFO: Starting Backup of VM 110 (qemu)
INFO: status = running
INFO: update VM 110: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/mnt/backup-server/iBackOffice/dump/vzdump-qemu-110-2014_03_21-20_55_38.vma.gz'
INFO: started backup task '93447892-56e9-4c41-9007-f1aba15c8f7b'
INFO: status: 0% (136839168/53687091200), sparse 0% (27291648), duration 3, 45/36 MB/s
INFO: status: 1% (779223040/53687091200), sparse 1% (591990784), duration 6, 214/25 MB/s
INFO: status: 2% (1090912256/53687091200), sparse 1% (683790336), duration 10, 77/54 MB/s
INFO: status: 3% (1619263488/53687091200), sparse 1% (744960000), duration 36, 20/17 MB/s
INFO: status: 4% (2151415808/53687091200), sparse 1% (756506624), duration 67, 17/16 MB/s
INFO: status: 5% (2698838016/53687091200), sparse 1% (760414208), duration 101, 16/15 MB/s
INFO: status: 6% (3234725888/53687091200), sparse 1% (895967232), duration 126, 21/16 MB/s
INFO: status: 7% (3766878208/53687091200), sparse 1% (927076352), duration 154, 19/17 MB/s
INFO: status: 8% (4302831616/53687091200), sparse 1% (946515968), duration 188, 15/15 MB/s
INFO: status: 9% (4846387200/53687091200), sparse 1% (955404288), duration 222, 15/15 MB/s
INFO: status: 10% (5378539520/53687091200), sparse 2% (1104945152), duration 245, 23/16 MB/s
ERROR: VM 110 not running
INFO: aborting backup job
ERROR: VM 110 not running
ERROR: Backup of VM 110 failed - VM 110 not running
INFO: Backup job finished with errors

The problem is reproduceable.

Best regards Stephan
Click to expand...
Click to expand...
 
Code: 
root@pve-node3:~# /usr/bin/kvm -id 1080 -chardev socket,id=qmp,path=/var/run/qemu-server/1080.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/1080.vnc,x509,password -pidfile /var/run/qemu-server/1080.pid -name ftp -smp sockets=1,cores=2 -nodefaults -boot menu=on -vga cirrus -cpu kvm64,+lahf_lm,+x2apic,+sep -k en-us -m 1024 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -drive if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100 -drive file=iscsi://172.16.99.15/iqn.1986-03.com.sun:02:stor2.xxxxxx/1,if=none,id=drive-virtio0,aio=native,cache=none -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=200 -netdev type=tap,id=net0,ifname=tap1080i0,script=/var/lib/qemu-server/pve-bridge,vhost=on -device virtio-net-pci,romfile=,mac=52:39:70:DD:CF:86,netdev=net0,bus=pci.0,addr=0x12,id=net0
main-loop: WARNING: I/O thread spun for 1000 iterations
Warning: no scancode found for keysym 3
Warning: no scancode found for keysym 3
Warning: no scancode found for keysym 3
Segmentation fault

To create a load I used "dbench 128". VM crash very quickly.
How I could get more verbose messages?
 
Last edited:
  • Like
Reactions: 1 person
Clearly there is a serious flaw in KVM or a bunch of you suddenly have bad RAM.

Code: 
main-loop: WARNING: I/O thread spun for 1000 iterations
Warning: no scancode found for keysym 3
Warning: no scancode found for keysym 3
Warning: no scancode found for keysym 3
[B]Segmentation fault[/B]

Thanks for the test johnplv

Do other people also get the same errors when performing this test?
 
Last edited:
Code: 
$ sudo qm showcmd 302
/usr/bin/kvm -id 302 -chardev socket,id=qmp,path=/var/run/qemu-server/302.qmp,server,nowait -mon chardev=qmp,mode=control -vnc unix:/var/run/qemu-server/302.vnc,x509,password -pidfile /var/run/qemu-server/302.pid -daemonize -name monit 
-smp sockets=1,cores=2 -nodefaults -boot menu=on -vga cirrus -cpu kvm64,+lahf_lm,+x2apic,+sep -k en-us -m 2048 -device piix3-usb-uhci,id=uhci,bus=pci.0,addr=0x1.0x2 -device usb-tablet,id=tablet,bus=uhci.0,port=1 
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -drive if=none,id=drive-ide2,media=cdrom,aio=native -device ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200 
-drive file=/dev/vg0/vm-302-disk-1,if=none,id=drive-virtio0,cache=none,aio=native -device virtio-blk-pci,drive=drive-virtio0,id=virtio0,bus=pci.0,addr=0xa,bootindex=100 -netdev type=tap,id=net0,ifname=tap302i0,script=/var/lib/qemu-server/pve-bridge,vhost=on 
-device virtio-net-pci,mac=A2:B6:45:66:C6:EC,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300

Code: 
$ sudo vzdump 302 --mode snapshot --compress gzip 
INFO: starting new backup job: vzdump 302 --mode snapshot --compress gzip
INFO: Starting Backup of VM 302 (qemu)
INFO: status = [COLOR=#ff0000]stopped[/COLOR]
INFO: update VM 302: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: creating archive '/srv/dump/vzdump-qemu-302-2014_03_21-22_26_29.vma.gz'
INFO: starting kvm to execute backup task
INFO: started backup task 'c8b6576d-cff1-4d22-9e52-f9f2b064b4a1'
INFO: status: 1% (239534080/16110321664), sparse 1% (182738944), duration 3, 79/18 MB/s
...
INFO: status: 100% (16110321664/16110321664), sparse 20% (3283845120), duration 489, 0/0 MB/s
INFO: transferred 16110 MB in 489 seconds (32 MB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 6.25GB
INFO: delete old backup '/srv/dump/vzdump-qemu-302-2014_03_13-00_08_14.vma.gz'
INFO: Finished Backup of VM 302 (00:08:12)
INFO: Backup job finished successfully

Code: 
$ sudo vzdump 302 --compress gzip 
INFO: starting new backup job: vzdump 302 --compress gzip
INFO: Starting Backup of VM 302 (qemu)
INFO: status = [COLOR=#ff0000]running[/COLOR]
INFO: update VM 302: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/srv/dump/vzdump-qemu-302-2014_03_21-22_44_55.vma.gz'
INFO: started backup task '748b0226-12b9-4d9b-893e-3e3ae0c5b15a'
INFO: status: 1% (290914304/16110321664), sparse 1% (234041344), duration 3, 96/18 MB/s
INFO: status: 7% (1197342720/16110321664), sparse 6% (1043898368), duration 6, 302/32 MB/s
INFO: status: 8% (1315176448/16110321664), sparse 6% (1081188352), duration 9, 39/26 MB/s
INFO: status: 9% (1467219968/16110321664), sparse 6% (1082793984), duration 16, 21/21 MB/s
ERROR: VM 302 not running
INFO: aborting backup job
ERROR: VM 302 not running
ERROR: Backup of VM 302 failed - VM 302 not running
INFO: Backup job finished with errors
job errors
 
Info from gdb:

Code: 
Program received signal SIGSEGV, Segmentation fault.
0x00007f8bf77ce61a in ?? ()
(gdb) backtrace
#0  0x00007f8bf77ce61a in ?? ()
#1  0x00007f8bf77ced75 in ?? ()
#2  0x00007f8bf77ce4f1 in ?? ()
#3  0x00007f8bf77ce888 in ?? ()
#4  0x00007f8bf781cc4d in ?? ()
#5  0x00007f8bf76b2e8a in ?? ()
#6  0x00007f8bf146b020 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
#7  0x00007fff9bd03a40 in ?? ()
#8  0x0000000000000000 in ?? ()


(gdb) info frame
Stack level 6, frame at 0x7f8b9e4009b8:
 rip = 0x7f8bf146b020 in __start_context; saved rip 0x7fff9bd03a40
 called by frame at 0x7f8b9e4009c0, caller of frame at 0x7f8b9e4009b0
 Arglist at 0x7f8b9e4009a8, args:
 Locals at 0x7f8b9e4009a8, Previous frame's sp is 0x7f8b9e4009b8
 Saved registers:
  rip at 0x7f8b9e4009b0


(gdb) info registers
rax            0x0      0
rbx            0x7f8b9e4009b0   140237632178608
rcx            0x0      0
rdx            0x17680000000040 6588273673633856
rsi            0x0      0
rdi            0x7f8bb2b01000   140237975064576
rbp            0x7f8bf766ee80   0x7f8bf766ee80
rsp            0x7f8b9e4009b0   0x7f8b9e4009b0
r8             0x7f8b9e400890   140237632178320
r9             0x7f8b9aa5e558   140237571745112
r10            0x0      0
r11            0x0      0
r12            0x18     24
r13            0xc7b608 13088264
r14            0xf      15
r15            0x7fff9bd042d0   140735807505104
rip            0x7f8bf146b020   0x7f8bf146b020 <__start_context>
eflags         0x10246  [ PF ZF IF RF ]
cs             0x33     51
ss             0x2b     43
ds             0x0      0
es             0x0      0
fs             0x0      0
gs             0x0      0


(gdb) info threads
  Id   Target Id         Frame
  4    Thread 0x7f8be88da700 (LWP 818073) 0x00007f8bf14fd137 in ioctl () at ../sysdeps/unix/syscall-template.S:82
  3    Thread 0x7f8be7ed9700 (LWP 818074) 0x00007f8bf14fd137 in ioctl () at ../sysdeps/unix/syscall-template.S:82
  2    Thread 0x7f8ba63ff700 (LWP 818076) pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:162
* 1    Thread 0x7f8bf748c900 (LWP 818057) 0x00007f8bf77ce61a in ?? ()
 
Dear developers! You were able to reproduce this error? Will you fix this bug?
Maybe I can help you more than to localize a problem?
Maybe we can go back to the previous algorithm where the backup did not have this bug?


Thank you.
 
Hi. I have it too now. First I thought the problem is related to QXL but on weekend it happened to a machine with standard graphics. One backup a night with nfs storage. I never had problem before. Mar 24 02:00:01 node1 /USR/SBIN/CRON[84892]: (root) CMD (vzdump 114 --quiet 1 --mode snapshot --mailto XXX --compress lzo --storage NFS-NAS) Mar 24 02:00:02 node1 vzdump[84892]: starting task UPID:node1:00014B9E:0B0D9120:532F8392:vzdump::root@pam: Mar 24 02:00:02 node1 vzdump[84894]: INFO: starting new backup job: vzdump 114 --quiet 1 --mailto XXX --mode snapshot --compress lzo --storage NFS-NAS Mar 24 02:00:02 node1 vzdump[84894]: INFO: Starting Backup of VM 114 (qemu) Mar 24 02:00:03 node1 qm[84899]: update VM 114: -lock backup Mar 24 02:02:54 node1 pvestatd[316944]: status update time (6.893 seconds) Mar 24 02:04:00 node1 vzdump[84894]: VM 114 qmp command failed - VM 114 not running Mar 24 02:04:00 node1 vzdump[84894]: VM 114 qmp command failed - VM 114 not running Mar 24 02:04:01 node1 kernel: vmbr0: port 2(tap114i0) entering disabled state Mar 24 02:04:01 node1 kernel: vmbr0: port 2(tap114i0) entering disabled state Mar 24 02:04:01 node1 vzdump[84894]: ERROR: Backup of VM 114 failed - VM 114 not running Mar 24 02:04:01 node1 vzdump[84894]: INFO: Backup job finished with errors Mar 24 02:04:01 node1 vzdump[84894]: job errors pveversion -v proxmox-ve-2.6.32: 3.2-121 (running kernel: 2.6.32-27-pve) pve-manager: 3.2-1 (running version: 3.2-1/1933730b) pve-kernel-2.6.32-27-pve: 2.6.32-121 pve-kernel-2.6.32-24-pve: 2.6.32-111 pve-kernel-2.6.32-26-pve: 2.6.32-114 lvm2: 2.02.98-pve4 clvm: 2.02.98-pve4 corosync-pve: 1.4.5-1 openais-pve: 1.1.4-3 libqb0: 0.11.1-2 redhat-cluster-pve: 3.2.0-2 resource-agents-pve: 3.9.2-4 fence-agents-pve: 4.0.5-1 pve-cluster: 3.0-12 qemu-server: 3.1-15 pve-firmware: 1.1-2 libpve-common-perl: 3.0-14 libpve-access-control: 3.0-11 libpve-storage-perl: 3.0-19 pve-libspice-server1: 0.12.4-3 vncterm: 1.1-6 vzctl: 4.0-1pve5 vzprocps: 2.0.11-2 vzquota: 3.1-2 pve-qemu-kvm: 1.7-4 ksm-control-daemon: 1.1-1 glusterfs-client: 3.4.2-1 cat /etc/pve/qemu-server/114.conf boot: c cores: 4 cpuunits: 25000 freeze: 0 ide1: none,media=cdrom kvm: 1 memory: 16384 name: XXX net0: virtio=9A:3B:BC:D8:C7:E5,bridge=vmbr0 onboot: 1 ostype: win7 scsihw: virtio-scsi-pci sockets: 1 tablet: 0 virtio0: LVM-VM-Disks:vm-114-disk-1,cache=none virtio1: LVM-VM-Disks:vm-114-disk-2,cache=none virtio2: LVM-VM-Disks2:vm-114-disk-1,cache=none,backup=no,size=200G
 
Last edited:
spirit said:
Hi Guys,

does it help if you disable compression ?
Click to expand...
Compression as in compressing the backup package or as in cluster wide compression (migration_unsecure: 1). I have migration_unsecure: 1 and my backups works without flaws even to a slow Qnap via NFS.
 
mir said:
Where do you run dbench while doing backup? in host, VM or storage
Click to expand...

Hello mir,

when I disable compression, the backup in snapshot mode works fine.
I test 10 backups, all 10 backups completes with no error.
But this is not a solution for the problem, just a hint in the right direction.
The backupfile is threetimes bigger than with compression.

Best regards Stephan
 
My storage is lvm for images anf nfs for backup. I changed /etc/storage to lvm: LVM-VM-Disks vgname vmdisks shared content images nfs: NFS-NAS path /mnt/pve/NFS-NAS server 192.148.150.9 export /Sicherung options vers=3,nolock -> added this content backup maxfiles 2 Lets see what happens.
 
dietmar said:
Can you reproduce the crash if you backup to /dev/null

# vzdump <VMID> -stdout >/dev/null
Click to expand...

mir said:
Where do you run dbench while doing backup? in host, VM or storage
Click to expand...

I did a lot of tests. To create a load using various utilities (stress, fio, bonnie++, dbench)
and in our case, the best was "dbench -s 128" inside VM (Key "-s" very important)
and start backup process with GZIP !!!, without compression or LZO - VM don't crush
Where from does not matter (iSCSI, local, NFS, /dev/null, etc..)
 
Last edited:
You must log in or register to reply here.
Top Bottom