Linux (Debian) VMs nach Backup down

[fabian]

yes, installed with "apt install pve-qemu-kvm-dbg" and also "apt install systemd-coredump"

root@pve4:~# systemctl enable systemd-coredump@


a more precise explanation how to enable coredump would be helpful...

Roman

that was correct, but it seems that does not work with Qemu.. two more things you could try:

• does the problem only occur with VMs with disks on PVE4ZFS? what kind of storage is that?
• you could temporarily downgrade to pve-qemu-kvm(-dbg) 2.9.0-2 (http://download.proxmox.com/debian/...n/binary-amd64/pve-qemu-kvm_2.9.0-2_amd64.deb and http://download.proxmox.com/debian/...nary-amd64/pve-qemu-kvm-dbg_2.9.0-2_amd64.deb), and check if the problem also existed back then.

 

[romanpoe]

good morning,

pve4ZFS is a 4TB RAID-Z2 Pool with 2 SSDs for log and cache
the problem also exists on an older Nodes with ext4 (unsupported) SoftRAID 10

Today I moved the Disk from local ZFS storage to a NFS Share and changed from RAW to qcow2 and tried the Backup (Backup and NFS share are both connected via 2x10 GBoE to the pve nodes)

INFO: starting new backup job: vzdump 555 --compress lzo --storage BackUp_LNXStorage --mode stop --node pve4 --remove 0
INFO: Starting Backup of VM 555 (qemu)
INFO: status = running
INFO: update VM 555: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: xxxxxx
INFO: include disk 'virtio0' 'Disks_LNXStorage:555/vm-555-disk-1.qcow2' 32G
INFO: stopping vm
INFO: creating archive '/mnt/pve/BackUp_LNXStorage/dump/vzdump-qemu-555-2017_08_25-08_43_21.vma.lzo'
INFO: starting kvm to execute backup task
INFO: started backup task '081b4ced-801a-4468-abbd-cb43066ad272'
INFO: resume VM
ERROR: VM 555 qmp command 'query-backup' failed - client closed connection
INFO: aborting backup job
ERROR: VM 555 not running
INFO: restarting vm
INFO: start failed: org.freedesktop.systemd1.UnitExists: Unit 555.scope already exists.
command 'qm start 555 --skiplock' failed: exit code 255
ERROR: Backup of VM 555 failed - VM 555 qmp command 'query-backup' failed - client closed connection
INFO: Backup job finished with errors
TASK ERROR: job errors

 

[romanpoe]

same machine (pve4) same backup target, other VM:

INFO: starting new backup job: vzdump 111 --mode stop --node pve4 --remove 0 --compress lzo --storage BackUp_LNXStorage
INFO: Starting Backup of VM 111 (qemu)
INFO: status = running
INFO: update VM 111: -lock backup
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: xxxxx
INFO: include disk 'ide0' 'pve4ZFS:111/vm-111-disk-1.raw' 8G
INFO: stopping vm
INFO: creating archive '/mnt/pve/BackUp_LNXStorage/dump/vzdump-qemu-111-2017_08_25-09_05_07.vma.lzo'
INFO: starting kvm to execute backup task
INFO: started backup task '479f68a4-c090-486d-939c-0f0b647f1c6c'
INFO: resume VM
INFO: status: 13% (1162346496/8589934592), sparse 10% (932225024), duration 3, read/write 387/76 MB/s
INFO: status: 40% (3473670144/8589934592), sparse 35% (3051909120), duration 6, read/write 770/63 MB/s
INFO: status: 43% (3742629888/8589934592), sparse 35% (3087261696), duration 9, read/write 89/77 MB/s
INFO: status: 48% (4158390272/8589934592), sparse 37% (3183546368), duration 13, read/write 103/79 MB/s
INFO: status: 69% (5982388224/8589934592), sparse 55% (4768882688), duration 16, read/write 607/79 MB/s
INFO: status: 100% (8589934592/8589934592), sparse 82% (7086403584), duration 19, read/write 869/96 MB/s
INFO: transferred 8589 MB in 19 seconds (452 MB/s)
INFO: archive file size: 973MB
INFO: vm is online again after 51 seconds
INFO: Finished Backup of VM 111 (00:00:52)
INFO: Backup job finished successfully
TASK OK

 

[fabian]

just noticed that there already is a bug filed for this, and that it seems to be triggered by virtio-blk. since those are linux VMs which should support virtio-scsi, could you try using that instead for now?

https://bugzilla.proxmox.com/show_bug.cgi?id=1420

 

[RedneckBob]

just noticed that there already is a bug filed for this, and that it seems to be triggered by virtio-blk. since those are linux VMs which should support virtio-scsi, could you try using that instead for now?

https://bugzilla.proxmox.com/show_bug.cgi?id=1420

Is there an ETA for a fix or a valid workaround? I've tried switching between SCSI and virtio, enabled/disabled writeback, and tried other suggestions in this forum still broken.

This starting to become an issue for us because now about 80% of our VMs won't restart after backup. I can provide additional data or run some tests if needed, just let me know how to proceed.

-RB

 

[fabian]

Is there an ETA for a fix or a valid workaround? I've tried switching between SCSI and virtio, enabled/disabled writeback, and tried other suggestions in this forum still broken.

This starting to become an issue for us because now about 80% of our VMs won't restart after backup. I can provide additional data or run some tests if needed, just let me know how to proceed.

-RB

the valid workaround is not using virtio-block (config prefix 'virtioN'), but use (virtio-)scsi (config prefix scsiN and scsihw set to virtio-scsi or virtio-scsi-single).

if you still experience problems, please double check that the affected VM is not (still) using virtio-block, and post the affected configuration here.

a patch which should fix this is currently under review, and fixed packages will hopefully be available next week.

(slightly OT: thanks for poking us regarding this issue, but please don't cross-post in multiple threads. answering the same questions multiple times or getting them answered by multiple people takes up valuable developer time which could be spent actually fixing the bugs!)

 

[romanpoe]

problem solved with

proxmox-ve: 5.0-21 (running kernel: 4.10.17-3-pve)

thank you all!

best regards,

roman