CEPH with Proxmox?

[wahmed]

After weeks of unsuccessful attempts to increase overall backup speed of my Proxmox VMs, i am now leaning towards some other solution. I have tried FreeNAS, OmniOS+Napp-it, Yottabyte, nothing increased the backup speed to match Gigabit network. I also changed hardware to all genuine intel without any success. Since Proxmox does backup at good speed locally, i have to draw conclusion that the problem may be with ZFS. I want to give other solution a try just to see if there is any change. I noticed that Proxmox supports RBD option from CEPH. My question is has anybody used it with great success who can give me some pointer? Is it more of headache than what is worth? I do like the clustered storage idea. Any other options out there for open source storage cluster?
Thanks!!

 

[kchawk]

We have a three node proxmox cluster connected to a three node ceph cluster. So far so good. We run about 20 KVMs on it.
Do you have vzdump limit set?

 

[wahmed]

We have a three node proxmox cluster connected to a three node ceph cluster. So far so good. We run about 20 KVMs on it.
Do you have vzdump limit set?

I do not have vzdump limit set. During testing I didnt see much difference changing vzdump limit and i have also assigned dedicated NIC for backup only. I think that sort of takes away the need of setting limit since there are no other traffic other then during backup.

 

[mir]

But the problem is that vzdump is limited to 10000 Kbps by default and since pve uses ionize doing backup it would fit on a 100Mbps network. I have increased that to 100000Kbps giving me backup performance like 16-18 MB/s over gigabit network through NFS to a fairly slow Qnap NAS (that more or less 1 GB data per minut). See /etc/vzdump.conf