glusterfs, ceph or freenas

[sahostking]

Hi

We currently use Freenas with NFS for backups of proxmox servers. Now all works well besides the fact that at night or sometimes early mornings servers become a little slow as backups are run. Now when we had hyperv we never had this issue as we used Hyperoo to backup servers which did only changes every night.

Do you think Ceph or GlusterFS is better for this? We just need it for backups really - looking for fast but reliable backup system. We have OpenVZ, LXC and KVM vms.

Thoughts?

 

[mir]

How is this Freenas box configured, RAM and Network?

 

[alexskysilk]

Your preferred backup mechanism should mirror your primary storage choice.

If you're already using a ceph/gluster storage solution, you can integrate your backup mechanism into your cluster. I'm less experienced with gluster, but with ceph you can perform

rbd export-diff --from-snap snap1 pool/image@snap2 pool_image_snap1_to_snap2.diff

to create incremental updates. I imagine gluster has a similar function.

If your primary storage mechanism is ZFS based, you can zfs-send your incrementals to your freenas box; it doesnt have to be complete backups. Proxmox has a nice control tool for this (pve-zsync, see https://pve.proxmox.com/wiki/PVE-zsync)

If you're storage mechanism is LVM you can use something like rsnapshot to accomplish similar functionality but I have no experience with this.

your last and most granular resort is vzdump over NFS, but the drawbacks to this method are obvious.

 

[Q-wulf]

How "Fast" is considered "fast" by you ? can you give us an idea of how many MB/s you are currently writing Backups at to your FreeNas ?

Now all works well besides the fact that at night or sometimes early mornings servers become a little slow as backups are run

Which servers ? Proxmox or Freenas ?

Or do you have FreeNas on Proxmox as a VM ?
If this is the case, can you tell us how your Storage-Subsystem works ?

Do you think Ceph or GlusterFS is better for this? We just need it for backups really - looking for fast but reliable backup system. We have OpenVZ, LXC and KVM vms.

Both Ceph and GlusterFS require multiple "Backup Servers". They also have the habbit of Replicating over the net during writes. Which means that during your backups you do not only write a single Copy to a single Storage-Subsystem, but you write to 2 (more likely 3) Storage-Subsystems, which also need to have the network capacity to sustain that.

If you're already using a ceph/gluster storage solution, you can integrate your backup mechanism into your cluster. I'm less experienced with gluster, but with ceph you can perform

rbd export-diff --from-snap snap1 pool/image@snap2 pool_image_snap1_to_snap2.diff

Sry, but that is NOT a sane Backup strategy.
Snapshots ARE NOT Backups

Best practice is to maintain a 3-2-1 strategy at minimum. 3 Copies. 2 different media, one not local system.

 

[alexskysilk]

Sry, but that is NOT a sane Backup strategy.
Snapshots ARE NOT Backups

Best practice is to maintain a 3-2-1 strategy at minimum. 3 Copies. 2 different media, one not local system.

True, but not necessarily practical. If the delta is too big to process in the available time to process it, using snapshots to create a seperate datastream (which you can then backup at your leisure to slower medium) may be the most practical mechanism to get you where you need to be without having to rearchitect the primary storage/available backup mechanism. I'm not suggesting that its best practice, just a solution to the OP's stated problem.

 

[sahostking]

How is this Freenas box configured, RAM and Network?

Specs:

Intel Xeon E5-1620 3.5GHz
32GB ECC RAM
6 x 2TB Enterprise Disks in RAIDZ2
Back dedicated Network on 10.0.0.x range on 1Gb/s backlink between all servers.

Basically when servers die or customers need restores all works fine and great as Freenas shared storage is used to backup VMs, openvz and KVM.

Always looking for better options though. So with one dedicated backup server what do you think is best? Also note do not want to stop VMs when backing up so must be Live backups - we store max of 10 to 15 vps per server.

 

[Q-wulf]

Your Cpu, Ram and Disks are sufficient for the NAS.

Also note do not want to stop VMs when backing up so must be Live backups - we store max of 10 to 15 vps per server.

How much Backup-data do you generate during your backup cycle ??
How many Nodes in your proxmox Cluster ? So i can roughly grasp the total backup-amount during your backup-window.
Do you do backups from all nodes at the same time, or do you stagger them ? Same thing for VM's all at the same time or staggered ?
What does your IO look like (proxmox nodes) when you do backups at night and your proxmox-nodes slow down ?

Back dedicated Network on 10.0.0.x range on 1Gb/s backlink between all servers.

I assume that is 1G link on the Proxmox-Nodes dedicated to NAS access.
How many Nics does the Nas have ? If its 1G, remember that is "just" 125 MB/s
Is there a switch in-between Nodes / NAS ? If so, whats its resource utilisation during your slowdown ?

 

[sahostking]

Your Cpu, Ram and Disks are sufficient for the NAS.


How much Backup-data do you generate during your backup cycle ??
How many Nodes in your proxmox Cluster ? So i can roughly grasp the total backup-amount during your backup-window.
Do you do backups from all nodes at the same time, or do you stagger them ? Same thing for VM's all at the same time or staggered ?
What does your IO look like (proxmox nodes) when you do backups at night and your proxmox-nodes slow down ?



I assume that is 1G link on the Proxmox-Nodes dedicated to NAS access.
How many Nics does the Nas have ? If its 1G, remember that is "just" 125 MB/s
Is there a switch in-between Nodes / NAS ? If so, whats its resource utilisation during your slowdown ?

Around 1.4TB each night.
2 nodes
scatter them 1 node does it at 23:00 and the other at 03:00. I worked out where the one finishes and the other begins and gave around hour or so break in between.
well slows down but slightly. Unoticable almost.

Yes the 1Gb link is just for NAS

NIC has 2 cards 1 for public 1 x backup network.
yes there is a switched in between. Not sure about resource usuage on the switch as yet.

 

[Q-wulf]

okay, so you are trying to Back up 700GB of Data per Node via 125 MB/s connection. That should take best case 95 Minutes (not counting overhead).

How long does it ACTUALLY take ?

Do you back up all VM's with a single Backup rule ?
Or do you back up every VM with a different Backup rule at separate times ?

What does the "server-load" and IO-wait look like on the proxmox-node you are performing your backup on, once you are actually backing up ?? Whats the normal server-laod and IOwait when your not backing up ?
In order for this to mean anything i'd need to now what type of CPU use on the proxmox nodes and how many