ZFS network send/recv very poor performance

[Gabor Szel]

I have a partially proxmox specific question.

We have two backup servers running PBS current version.
We make SMB (samba) backups are made to first server.
Second server is at a different location (geo redundant)
We use ZFS file system.
On the primary server every day is taken ZFS snapshot.
We send incremental snapshot to secondary server every day (ZFS send, recv) over SSH.
it's a completely basic ZFS backup system.

But ZFS sending/recv is very very slow!

server specifications:
- Dell R730xd
- 2x Xeon E5-2695
- 64 Gb RAM
- 8x12Tb HDD(primary), 8x14Tb HDD(secondary)
- 10Gbit network connection (network speed is okay)

ZFS specifications:
- RaidZ2
- LZ4 compression
- zfs_arc_max 48Gb
- no dedup!

We send 3 ZFS pool to secondary server:
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
xxxxbck2 12.6T 59.7T 0B 256K 0B 59.7T
xxxxbck2/samba/smbxxx1 12.6T 25.6T 11.3T 14.3T 0B 0B
xxxxbck2/samba/smbxxx2 12.6T 21.6T 5.04T 16.5T 0B 0B
xxxxbck2/samba/smbxxx3 12.6T 12.2T 1.67T 10.5T 0B 0B

Send/recv is very simple:
zfs send -R -i xxxxbck2/samba/smbxxx1@bcksend$current xxxxbck2/samba/smbxxx1@bcksend$new | ssh root@10.x.x.x zfs recv xxxxbck2/samba/smbxxx1
zfs send -R -i xxxxbck2/samba/smbxxx1@bcksend$current xxxxbck2/samba/smbxxx2@bcksend$new | ssh root@10.x.x.x zfs recv xxxxbck2/samba/smbxxx2
zfs send -R -i xxxxbck3/samba/smbxxx1@bcksend$current xxxxbck2/samba/smbxxx3@bcksend$new | ssh root@10.x.x.x zfs recv xxxxbck2/samba/smbxxx2
(there are other pools, we do not send them)

Problem:
- transfer of 3 pool snapshots takes 6-7 days! (previously it was 1-2 days)
- no high CPU usage, no high IO, no full memory usege
- network speed is 2-4Gbit only 5-6 sec, and then 10-50Mbit 15-30 sec

Why do you think it can be so slow?

 

[tcabernoch]

It sounds like you've considered all this carefully, so I'm sure you have your reasons but ... why aren't you using PBS sync jobs?
This is the perfect scenario. It's two PBS servers trading data. You know, like a sync job would do.

If you don't know about this, in PBS on the left hand navigation tree, click on Remotes.
Setup each PBS server as a Remote to the other.

Now click down on the datastore. Hit the Sync Jobs tab. Setup a Job.
PBS Sync Jobs are a bit weird. It's a pull, not a push. And its all controlled by filters. You can sync everything or ... nothing.

After you get into replicating data, you'll discover the need for Namespaces. You'll understand when you get that far.