Snapshots backups causing timeouts on VM host

[gerdnl]

Hi,

Is it normal that a snapshot backup can cause timeouts on the running VM
I didnt notice this before, but it seem to happen lately and almost everytime i find rsync or the backup still running

i was just wondering if this is normal but i just never noticed it

 

[manu]

Please post your VM config and the exact timeout message on the VM.

 

[gerdnl]

is was giving me gateway timeouts on a nginx/uwsgi/django while the backup was running
its a 6 gig mem and 4 cores VM setup with qcow2 disk and a virtio network device.

 

[gerdnl]

i do see this in the node log

Nov 15 08:45:08 pve03 kernel: INFO: task lzop:71561 blocked for more than 120 seconds.
Nov 15 08:45:08 pve03 kernel: Not tainted 2.6.32-43-pve #1
Nov 15 08:45:08 pve03 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Nov 15 08:45:08 pve03 kernel: lzop D ffff88047b306600 0 71561 21715 0 0x00000000
Nov 15 08:45:08 pve03 kernel: ffff88013cdffa48 0000000000000086 0000000000000000 ffff88047ddc2d40
Nov 15 08:45:08 pve03 kernel: ffff88013cdffa08 ffffffff81009804 0000000000000000 0000000000000000
Nov 15 08:45:08 pve03 kernel: 000017e91d552f1c 0000001200000000 00000001018c3c17 000000000000032a

Nov 15 08:45:08 pve03 kernel: Call Trace:
Nov 15 08:45:08 pve03 kernel: [<ffffffff81009804>] ? __switch_to+0x1f4/0x310
Nov 15 08:45:08 pve03 kernel: [<ffffffffa0328600>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs]
Nov 15 08:45:08 pve03 kernel: [<ffffffff815a9763>] io_schedule+0x73/0xc0
Nov 15 08:45:08 pve03 kernel: [<ffffffffa032860e>] nfs_wait_bit_uninterruptible+0xe/0x20 [nfs]
Nov 15 08:45:08 pve03 kernel: [<ffffffff815aa770>] __wait_on_bit+0x60/0x90
Nov 15 08:45:08 pve03 kernel: [<ffffffffa0328600>] ? nfs_wait_bit_uninterruptible+0x0/0x20 [nfs]
Nov 15 08:45:08 pve03 kernel: [<ffffffff815aa81c>] out_of_line_wait_on_bit+0x7c/0x90
Nov 15 08:45:08 pve03 kernel: [<ffffffffa031d530>] ? nfs_wait_bit_killable+0x0/0x40 [nfs]
Nov 15 08:45:08 pve03 kernel: [<ffffffff810a9220>] ? wake_bit_function+0x0/0x50
Nov 15 08:45:08 pve03 kernel: [<ffffffffa0328b5f>] nfs_wait_on_request+0x2f/0x40 [nfs]
Nov 15 08:45:08 pve03 kernel: [<ffffffffa032f2b2>] nfs_updatepage+0x122/0x530 [nfs]
Nov 15 08:45:08 pve03 kernel: [<ffffffffa031cae8>] nfs_write_end+0x138/0x2e0 [nfs]
Nov 15 08:45:08 pve03 kernel: [<ffffffffa031bdaf>] ? nfs_write_begin+0x8f/0x1f0 [nfs]
Nov 15 08:45:08 pve03 kernel: [<ffffffff8113dcaa>] generic_file_buffered_write_iter+0x18a/0x2b0
Nov 15 08:45:08 pve03 kernel: [<ffffffff8113e1d5>] __generic_file_write_iter+0x215/0x3e0
Nov 15 08:45:08 pve03 kernel: [<ffffffff8113e423>] __generic_file_aio_write+0x83/0xa0
Nov 15 08:45:08 pve03 kernel: [<ffffffff8113e4c8>] generic_file_aio_write+0x88/0x100
Nov 15 08:45:08 pve03 kernel: [<ffffffffa031c3ce>] nfs_file_write+0x11e/0x220 [nfs]
Nov 15 08:45:08 pve03 kernel: [<ffffffff811b195c>] do_sync_write+0xec/0x140
Nov 15 08:45:08 pve03 kernel: [<ffffffff811b21b1>] vfs_write+0xa1/0x190
Nov 15 08:45:08 pve03 kernel: [<ffffffff811b39fe>] ? fget_light_pos+0x3e/0x50
Nov 15 08:45:08 pve03 kernel: [<ffffffff811b251a>] sys_write+0x4a/0xa0
Nov 15 08:45:08 pve03 kernel: [<ffffffff8100b1e2>] system_call_fastpath+0x16/0x1b
Nov 15 08:45:59 pve03 kernel: __ratelimit: 1 callbacks suppressed

 

[manu]

A backup in snapshot mode will slow down the I/O on the VM while the backup is running, but not to point that it should vause HTTP timeouts on web services running on it.

For this to debug, please send the VM config:

qm config <my_vmid>

Also have a look inside the VM /var/log/kern.log for messages converning tasks blocked, controller resets, lockups.

 

[gerdnl]

not sure if its related do

 

[gerdnl]

bootdisk: virtio0
cores: 4
ide2: local:iso/ubuntu-12.04.3-server-amd64.iso,media=cdrom
lock: backup
memory: 6144
name: Pobbchat.nl
net0: virtio=B2:47:4D:1F:5E:3F,bridge=vmbr159,firewall=1
onboot: 1
ostype: other
sockets: 1
virtio0: local:118/vm-118-disk-1.qcow2,format=qcow2,size=75G

 

[gerdnl]

kern.log:
http://paste.ubuntu.com/25966449/

nothing special there it seems

 

[gerdnl]

i do see some KVM running at high cpu on the node itself 202% 242% not causing any issues atm all running fine

 

[gerdnl]

i think it safe to assume it was a IO overload somehow