VZDUMP slow read over NFS

[hakcenter]

NAS VM, to local dump HDD

INFO: starting new backup job: vzdump 101 --storage dump --mode snapshot --remove 0 --node masamune --compress zstd
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2020-09-16 12:23:18
INFO: status = running
INFO: VM Name: Kotetsu
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating vzdump archive '/mnt/dump/vzdump/dump/vzdump-qemu-101-2020_09_16-12_23_18.vma.zst'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task '653c1b60-e474-4f45-9e63-918aee60a4a2'
INFO: resuming VM again
INFO:   0% (308.1 MiB of 32.0 GiB) in  3s, read: 102.7 MiB/s, write: 78.9 MiB/s
INFO:   1% (426.4 MiB of 32.0 GiB) in  6s, read: 39.4 MiB/s, write: 38.3 MiB/s
INFO:   2% (668.5 MiB of 32.0 GiB) in 12s, read: 40.3 MiB/s, write: 39.0 MiB/s
INFO:   3% (1000.5 MiB of 32.0 GiB) in 20s, read: 41.5 MiB/s, write: 39.4 MiB/s
[...]
INFO:  98% (31.4 GiB of 32.0 GiB) in 12m 23s, read: 42.0 MiB/s, write: 40.0 MiB/s
INFO:  99% (31.7 GiB of 32.0 GiB) in 12m 32s, read: 37.2 MiB/s, write: 36.3 MiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in 12m 39s, read: 41.7 MiB/s, write: 41.3 MiB/s
INFO: backup is sparse: 8.29 GiB (25%) total zero data
INFO: transferred 32.00 GiB in 759 seconds (43.2 MiB/s)
INFO: archive file size: 10.58GB
INFO: Finished Backup of VM 101 (00:12:46)
INFO: Backup finished at 2020-09-16 12:36:04
INFO: Backup job finished successfully
TASK OK

clone of NAS VM on dump drive, backup back to NAS

INFO: starting new backup job: vzdump 9999 --remove 0 --compress zstd --node masamune --storage NAS_Daily_Backup --mode snapshot
INFO: Starting Backup of VM 9999 (qemu)
INFO: Backup started at 2020-09-16 12:17:49
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: test
INFO: include disk 'virtio0' 'dump:9999/vm-9999-disk-0.qcow2' 32G
INFO: creating vzdump archive '/mnt/pve/NAS_Daily_Backup/dump/vzdump-qemu-9999-2020_09_16-12_17_49.vma.zst'
INFO: starting kvm to execute backup task
audio: Could not init `spice' audio driver
audio: warning: Using timer based audio emulation
INFO: started backup task '92fe89de-c9c1-4c58-a15f-d0d32e09b82e'
INFO:   0% (302.4 MiB of 32.0 GiB) in  3s, read: 100.8 MiB/s, write: 77.1 MiB/s
INFO:   1% (612.6 MiB of 32.0 GiB) in  6s, read: 103.4 MiB/s, write: 99.9 MiB/s
INFO:   2% (929.6 MiB of 32.0 GiB) in  9s, read: 105.7 MiB/s, write: 100.4 MiB/s
INFO:   3% (1.2 GiB of 32.0 GiB) in 12s, read: 107.0 MiB/s, write: 101.2 MiB/s
[...]
INFO:  98% (31.4 GiB of 32.0 GiB) in  4m 48s, read: 104.7 MiB/s, write: 99.4 MiB/s
INFO:  99% (31.7 GiB of 32.0 GiB) in  4m 51s, read: 105.5 MiB/s, write: 103.0 MiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in  4m 54s, read: 103.2 MiB/s, write: 100.5 MiB/s
INFO: backup is sparse: 8.31 GiB (25%) total zero data
INFO: transferred 32.00 GiB in 294 seconds (111.5 MiB/s)
INFO: stopping kvm after backup task
INFO: archive file size: 10.58GB
INFO: Finished Backup of VM 9999 (00:05:03)
INFO: Backup finished at 2020-09-16 12:22:52
INFO: Backup job finished successfully
TASK OK

Copy of a ISO on NAS to local dump

root@monado:/mnt/dump# time cp /mnt/pve/NAS/template/iso/windows_10_1909_x86_64.iso /mnt/dump/

real    0m52.035s
user    0m0.095s
sys     0m8.238s
root@monado:/mnt/dump# ls -la /mnt/pve/NAS/template/iso/ |grep windows_10
-rwxrwx--x 1 proxmox proxmox 4269735936 Aug 19 07:14 windows_10_1909_x86_64.iso

82.11 MB/sec

I don't know what to do here..

The NAS is TrueNAS beta2.1, with a NFS4 share, 2 vdev mirror, (raid10 basically) with sync disabled, and record size 128k.

/etc/pve/storage.conf

nfs: NAS
        export /mnt/DATA/proxmox
        path /mnt/pve/NAS
        server 192.168.1.11
        content iso,images
        options noatime,nodiratime,async,vers=4

mount

192.168.1.11:/mnt/DATA/proxmox on /mnt/pve/NAS type nfs4 (rw,noatime,nodiratime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.13,local_lock=none,addr=192.168.1.11)

 

[DANILO MONTAGNA]

Hi,

I saw the same bad results using CIFS as backend storage... very slow... using local disk as ZFS improve a lot the backup performance.. but some times there aren't other options to grow backup storage.. so, it's something that must be investigated/validated by proxmox team!!

 

[hakcenter]

I've cloned VM's to RAW / QCOW2, no difference.

It really seems there's an issue with "vzdump" itself. In VM performance is around 115MB/sec

 

[hakcenter]

I really wish someone could shed some light on this situation, it's extremely frustrating to backup vm's at these super slow speeds.

 

[DANILO MONTAGNA]

I really wish someone could shed some light on this situation, it's extremely frustrating to backup vm's at these super slow speeds.

Hi, NFS is a network based transfer protocol.. what is your network speed from backup server to nfs server ? 115mb's is a good performance if your network is 1GBps (not good, but it's the limit you can get with 1Gbps)... also keep in mind that the performance is not only related to network speed.. but also to iops/throughput of your nfs server...

So, in resume... if your network speed is 1gbps, you'll never get a better performance without upgrade to a 10Gbps network.. AND if you only have a couple of disks pairs running in yout NFS server.. 115mbps is also the limit of a SATA/SAS disk write performance.. (if it is the case of your setup)

how many disks do you have in nfs server where backup is being written? what is the disk technology?

 

[hakcenter]

Hi, NFS is a network based transfer protocol.. what is your network speed from backup server to nfs server ? 115mb's is a good performance if your network is 1GBps (not good, but it's the limit you can get with 1Gbps)... also keep in mind that the performance is not only related to network speed.. but also to iops/throughput of your nfs server...

So, in resume... if your network speed is 1gbps, you'll never get a better performance without upgrade to a 10Gbps network.. AND if you only have a couple of disks pairs running in yout NFS server.. 115mbps is also the limit of a SATA/SAS disk write performance.. (if it is the case of your setup)

how many disks do you have in nfs server where backup is being written? what is the disk technology?

I understand the limit of the 1Gbps link, but we're not even getting half only during backups, which is locks the VM down in our cluster, and making maintenance a total bear.

The NFS machine is a R710 TrueNASRC1 with a 310 in IT Mode, Raid10 via mirror + mirror stripe ZFS. And 2 SLOG SSD's. Disabling Sync writes doesn't change anything.

Like I said, in VM performance is 90+MB/sec, and file transfers to local LAN machines from the VM's proxmox even hosts, we get 115MB/sec transfer rates... so the NFS is performing at link speed in everything but backups.
Why ?

 

[DANILO MONTAGNA]

You said you are getting 115MB/sec during backups using a 1GBit network.. for me it's almost the maximum you will get for a gigabit network!!! Or I'm reading something wrong....

 

[hakcenter]

Backups of VMs running on the Shared storage, only get 35MB/sec

When reading / writing to the NFS machine is 115MB/sec
And when running a VM on a local drive backing it up to the NFS share is also 115MB/sec

The problem is VM's running on shared NFS storage, only backup @ 35MB/sec

 

[DANILO MONTAGNA]

OK, now I understood!!! maybe it's because the way vzdump executes the backup process.. it first copy the backup to NFS server target and zips it afterwards... this will result in too many small files to copy do NFS.. lowering the performance on copy process... When you use other type of storage.. like LOCAL-STORAGE.. the vzdump first rsync data to /tmp and than ZIP and copy it to the local-storage backup.

I think you can configure a temp-dir in /etc/vzdump.conf which is used for the rsync process to a local SSD disk or something else before copy it to NFS.. it may improve your backup speed!!

Please, see this post: https://www.dev-eth0.de/2017/01/14/proxmox-nfs-backups/

Just be in mind that y will need some space locally on SSD disks for the temp-copy process..

 

[hakcenter]

/etc/vzdump.conf

# vzdump default settings
tmpdir: /mnt/dump/vzdump

/mnt/dump is a local drive


/shrug Like I said, none of it makes any sense. When the systems were iSCSI they were just fine for VZDumps, we decided to migrate from iSCSI to NFS to keep it simple for HA here. But literally the worst backup performance I've ever seen.

 

[DANILO MONTAGNA]

iSCSI and NFS are two completely different things!! iSCSI uses LVM and NFS is a completely different protocol!!!

 

[hakcenter]

AND NFS PERFORMANCE IS LINE SPEED, SO WE CONVERTED!

In VM performance, gets LINE speed
Reading and Writing to NFS server gets LINE speed

VZDUMP is utter garbage over NFS

 

[DANILO MONTAGNA]

So, using NFS for vzdump probably is a bad idea as you tested!!! have you tried the PBS instead of native vzdump?

 

[Craig St George]

I have the same problem with the PBS my VM are backed by Ceph
The VM read write speed to Ceph is fine the network to the PBS is 10Gb and tested ok
The PBS write and read speed is not that great for the TLS test but at least 150 MiB/s,

But I never get much more than below the incremental's are fast because there is not much data to write but the initials are slow

I am also wondering if there is some local data been written maybe back to the ceph like in the vzdump ?


INFO: started backup task '2b24571c-6df7-4719-b963-af166baadef1'
INFO: 1% (580.0 MiB of 50.0 GiB) in 3s, read: 193.3 MiB/s, write: 0 B/s
INFO: 2% (1.0 GiB of 50.0 GiB) in 15s, read: 39.0 MiB/s, write: 0 B/s
INFO: 3% (1.5 GiB of 50.0 GiB) in 30s, read: 35.5 MiB/s, write: 0 B/s
INFO: 4% (2.0 GiB of 50.0 GiB) in 44s, read: 36.0 MiB/s, write: 0 B/s
INFO: 5% (2.5 GiB of 50.0 GiB) in 1m 0s, read: 31.8 MiB/s, write: 0 B/s
INFO: 6% (3.0 GiB of 50.0 GiB) in 1m 14s, read: 34.9 MiB/s, write: 0 B/s
INFO: 7% (3.5 GiB of 50.0 GiB) in 1m 31s, read: 31.5 MiB/s, write: 0 B/s