Windows Guest hangs during Backup

[poxin]

Attempting to backup to the local storage instead of NFS produced nearly the same results. The system took a slightly longer time to go unresponsive, but still occurred.

Reply from: bytes=32 time<1ms TTL=127
Reply from: bytes=32 time<1ms TTL=127
Reply from: bytes=32 time=2592ms TTL=127
Reply from: bytes=32 time<1ms TTL=127
Request timed out.
Request timed out.
Request timed out.
Reply from: bytes=32 time=995ms TTL=127
Request timed out.
Request timed out.
Reply from: bytes=32 time=1323ms TTL=127
Reply from: bytes=32 time<1ms TTL=127
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from: Destination host unreachable.

On this particular node:

~# cat /etc/pve/storage.cfg
zfspool: local-zfs
    pool rpool/data
    content images,rootdir
    sparse 1

~# zpool status
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0 in 0h7m with 0 errors on Thu Aug 24 09:22:23 2017
config:

    NAME        STATE     READ WRITE CKSUM
    rpool       ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        sda2    ONLINE       0     0     0
        sdb2    ONLINE       0     0     0
      mirror-1  ONLINE       0     0     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0
      mirror-2  ONLINE       0     0     0
        sde     ONLINE       0     0     0
        sdf     ONLINE       0     0     0
      mirror-3  ONLINE       0     0     0
        sdg     ONLINE       0     0     0
        sdh     ONLINE       0     0     0

errors: No known data errors

The local ZFS system has compression turned on, not sure how to accurately test the performance here.

# dd if=/dev/zero of=test1.img bs=20G count=1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 1.52511 s, 1.4 GB/s

And the NFS Share

# dd if=/dev/zero of=temp.dat bs=20G count=1
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 21.9257 s, 97.9 MB/s

 

[Alwin]

Your dd files need to be bigger then RAM, as it will cache the 20GB probably. Also the ARC of ZFS will cache too.

To have an idea, why I am asking for those measurements. The vzdump utility is doing backup through qemu. This means, to have consistent backup, qemu will write changing blocks first to the backup file and then to the VM disk. As a result the write speed of a VM will be always the write speed of the slowest storage to write too. With the measurements above, the write speed would drop from 1.4 GB/s to ~100 MB/s.

 

[poxin]

Here's the result with a file larger than RAM on the system:

# dd if=/dev/zero of=temp.dat bs=1G count=150
150+0 records in
150+0 records out
161061273600 bytes (161 GB, 150 GiB) copied, 73.0341 s, 2.2 GB/s

I would think 2.2 GB/s should be adequate for a backup write? It's still odd to me why the guest completely freezes. Even hypothetically at 100mb/s it should respond to ping - most desktop sata drives have a write speed around that.

From the local backup job:

INFO: starting new backup job: vzdump 106 --storage local --node vs100 --mode snapshot --compress 0 --remove 0
INFO: Starting Backup of VM 106 (qemu)
INFO: status = running
INFO: update VM 106: -lock backup
INFO: VM Name: wintest
INFO: include disk 'virtio0' 'local-zfs:vm-106-disk-1' 30G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-106-2017_08_29-09_53_18.vma'
ERROR: VM 106 qmp command 'guest-fsfreeze-thaw' failed - got timeout
INFO: started backup task 'ad639e7f-43c4-4812-b71e-6db0f2144008'

If backing up a Windows system, do you also get that Error? 'guest-fsfreeze-thaw failed - got timeout'. Again I really appreciate the time you've spend helping to assist me with this. I'd really love to be able to move windows systems over.

 

[Alwin]

My bet, I forgot to mention that zfs is zero length aware, it will compress a long run of zeros. So in the end the test will not show the real write performance. As your machine is powerful enough, we won't get far with dd, a better test is to go with the tool fio. Or you copy a large file (not sparse), bigger then 50% of your RAM to the zfs storage.

The timeout is, because the guest-fsfreeze-thaw doesn't return, so your machine is already gone. Another maybe faster option is, to try to backup the machine to local storage for test.

 

[poxin]

The first post on page two was the results from local storage. I've been writing locally instead of the NFS share to simplify things - the issue still occurs. I'll check out 'fio'.

 

[Alwin]

Does a other Windows VM show the same issues? Was it already converted in the past? Maybe there are some leftovers and that is why it is failing.

 

[poxin]

This is a brand new install from within proxmox itself. I've created about 3 different test systems with the same results. I'm going to bring another node online in the next few weeks and will test a windows system from there too. Stumped. I'm going to try a 2012 R2 system as well.

 

[poxin]

Unfortunately this still occurs. I've built a new node with SSD disks and have attempted both Server 2016 and 2012R2 with SCSI, VirtIO, IDE, Ballooning Enabled/Disabled, and QEMU Enabled/Disabled. No luck on stable performance during a backup with Windows. All works fine on Linux systems and Containers.

Reply from 10.100.0.200: bytes=32 time=4ms TTL=127
Reply from 10.100.0.200: bytes=32 time=137ms TTL=127
Reply from 10.100.0.200: bytes=32 time=51ms TTL=127
Reply from 10.100.0.200: bytes=32 time=38ms TTL=127
Reply from 10.100.0.200: bytes=32 time=82ms TTL=127
Reply from 10.100.0.200: bytes=32 time=581ms TTL=127
Reply from 10.100.0.200: bytes=32 time=396ms TTL=127
Request timed out.
Request timed out.
Reply from 10.100.0.200: bytes=32 time=3634ms TTL=127
Reply from 10.100.0.200: bytes=32 time=856ms TTL=127
Reply from 10.100.0.200: bytes=32 time=185ms TTL=127
Reply from 10.100.0.200: bytes=32 time=2883ms TTL=127
Reply from 10.100.0.200: bytes=32 time=75ms TTL=127
Reply from 10.100.0.200: bytes=32 time=379ms TTL=127
Reply from 10.100.0.200: bytes=32 time=389ms TTL=127
Reply from 10.100.0.200: bytes=32 time=35ms TTL=127
Reply from 10.100.0.200: bytes=32 time=58ms TTL=127
Reply from 10.100.0.200: bytes=32 time=709ms TTL=127
Reply from 10.100.0.200: bytes=32 time=427ms TTL=127
Reply from 10.100.0.200: bytes=32 time=106ms TTL=127
Request timed out.
Reply from 10.100.0.200: bytes=32 time=1452ms TTL=127
Reply from 10.100.0.200: bytes=32 time=41ms TTL=127
Reply from 10.100.0.200: bytes=32 time=255ms TTL=127
Reply from 10.100.0.200: bytes=32 time=86ms TTL=127
Reply from 10.100.0.200: bytes=32 time=225ms TTL=127
Reply from 10.100.0.200: bytes=32 time=340ms TTL=127

You can see the sporadic pings to the guest during backup, soon as it completes - back to < 1ms.

 

[Alwin]

There are one or two threads on the forum, that show a similar problem. Their network drops completely under high IO load, seen on linux. As far as we know, there is a bug in the virtio_net kernel driver, newer kernels are out on the pvetest repository. Maybe those will fix the issue. Also there is a 4.13 Kernel for testing too.

https://pve.proxmox.com/wiki/Package_Repositories
https://bugs.launchpad.net/ubuntu/+source/qemu-kvm/+bug/1423631

 

[poxin]

Thanks for the information, I'll have to take a look once those reach the subscription repo. I tested the fix from the end of the reported bug on launchpad, no luck. Setting txqueuelen higher at 10000 didn't resolve it - it was actually worse this backup

The freezing also occurs on the guest OS in the VNC console.

Request timed out.
Reply from 10.100.0.200: bytes=32 time=2967ms TTL=127
Reply from 10.100.0.200: bytes=32 time<1ms TTL=127
Reply from 10.100.0.200: bytes=32 time=676ms TTL=127
Reply from 10.100.0.200: bytes=32 time=742ms TTL=127
Reply from 10.100.0.200: bytes=32 time=1214ms TTL=127
Reply from 10.100.0.200: bytes=32 time<1ms TTL=127
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 10.100.0.200: bytes=32 time=433ms TTL=127
Request timed out.
Reply from 10.100.0.200: bytes=32 time=3ms TTL=127
Reply from 10.100.0.200: bytes=32 time=2925ms TTL=127
Request timed out.
Reply from 10.100.0.200: bytes=32 time=1627ms TTL=127
Reply from 10.100.0.200: bytes=32 time=1ms TTL=127
Request timed out.
Reply from 10.100.0.200: bytes=32 time=992ms TTL=127
Reply from 10.100.0.200: bytes=32 time<1ms TTL=127
Reply from 10.100.0.200: bytes=32 time=1519ms TTL=127
Reply from 10.100.0.200: bytes=32 time<1ms TTL=127
Reply from 10.100.0.200: bytes=32 time=3062ms TTL=127
Reply from 10.100.0.200: bytes=32 time=1ms TTL=127
Request timed out.
Reply from 10.100.0.200: bytes=32 time=1528ms TTL=127
Reply from 10.100.0.200: bytes=32 time<1ms TTL=127
Request timed out.
Request timed out.
Request timed out.

 

[aderumier]

maybe for your windows case (don't known which driver version do you use):

" Latest latest virtio driver (network) for Windows drops lots of packets"
https://bugzilla.redhat.com/show_bug.cgi?id=1451978


Peixiu Hou 2017-07-06 01:16:17 EDT
Reproduced this issue with virtio-win-prewhql-139, the result like comment#38, at least 10% of pings timed out.

Verified this issue with virtio-win-prewhql-140, the ping flood test, no timeout occurs.

 

[VGusev2007]

To have an idea, why I am asking for those measurements. The vzdump utility is doing backup through qemu. This means, to have consistent backup, qemu will write changing blocks first to the backup file and then to the VM disk. As a result the write speed of a VM will be always the write speed of the slowest storage to write too. With the measurements above, the write speed would drop from 1.4 GB/s to ~100 MB/s.

oh... I think you are right. I have problem with totally hangs of my vm during backup, with message of guest vm like: i/o timeout... I have so slow backup storage...

 

[lucius_the]

The vzdump utility is doing backup through qemu. This means, to have consistent backup, qemu will write changing blocks first to the backup file and then to the VM disk. As a result the write speed of a VM will be always the write speed of the slowest storage to write too. With the measurements above, the write speed would drop from 1.4 GB/s to ~100 MB/s.

Does this still apply for current PBS/PVE version ?
It doesn't make much sense that writes would first go to the backup file... Maybe I don't understand how snapshot mode backup works (coming from ESXi) ?

 

[fiona]

Hi,

Does this still apply for current PBS/PVE version ?

yes.

It doesn't make much sense that writes would first go to the backup file...

It does. Before the new data can be written to disk, the old data needs to be copied out of the way into the backup. Otherwise, the backup would become inconsistent, because it would not represent the state of the disks when the backup was started, but some mishmash between old and new data. That's why it's called "snapshot" even if it's a different mechanism technically.

Maybe I don't understand how snapshot mode backup works (coming from ESXi) ?

 

[lucius_the]

Hi,

yes.

It does. Before the new data can be written to disk, the old data needs to be copied out of the way into the backup. Otherwise, the backup would become inconsistent, because it would not represent the state of the disks when the backup was started, but some mishmash between old and new data. That's why it's called "snapshot" even if it's a different mechanism technically.

Hi there,
Thank you for the explanation. I understand what snapshot is and why it is important.

However, I don't understand how exactly the snapshot mechanism works during backup on proxmox. And why it has anything to do with the link speed to backup storage...

I do understand the "backup mode snapshot" is something entirely different from a normal volume snapshot. For instance, in ESXi a normal VMFS snapshot is being used also for backups, but here something different is going on -> I would like to understand what exactly happens. how does backup (in mode snapshot) work. Where can I find more info about that ?

 

[spirit]

Basicaly, it's use a technique "copy before write".
I have attached a small schema.

for example, your backup start at 00:00 (green blocks). you want to have the image of the disk at 00:00 , even if backup is running for 8h.
(That's why it's called snapshot, even if it don't use disk snapshot, as not all storage support them)


backup job begin to backup green block sequentially.

Now , your vm need to overwrite a not yet backuped green block with a red block.

to keep consistency, the backup job will backup first the green block before it can be overwrite the vm.
(That mean, than for the 1st overwrite of this block, if you backup storage is slow, it can impact vm performance).


Qemu have recently added the support to add some kind of cache called "fleecing backup", to be able to send green blocks first to a cache layer , but it's not yet implemented in proxmox backup
https://bugzilla.proxmox.com/show_bug.cgi?id=4136

 

[lucius_the]

So... basically a CoW system. It stops write IO in the guest, until it copies the block the guest is trying to write to. Pretty standard snapshotting, it seems. Only things is, does it copy that block on demand (I would actually bet that it does), or it waits until the backup jobs comes (sequentially) to the block in question... that would certainly not work well, I don't believe that's what anybody would design.

So, if backup target is slow and guest is write heavy (worst case would probably be many random writes - like a DB), I understand now how it can bring down write IOPS in the VM and easily hog the guest system. Thank you very much for the explanation !

I will look into QEMU backups more deeply to understand details about the new option with local caching of writes during backup - that sounds promising.

 

[spirit]

does it copy that block on demand (I would actually bet that it does), or it waits until the backup jobs comes (sequentially) to the block in question...

on demand of course