[SOLVED] shared backup storage

[timfi]

I have a 5 node proxmox cluster with ceph storage running a mixture of lxc containers and qemu vm's. I have an extra server I want to configure as a backup storage and to mount extra partitions on the containers. (To have some extra storage for tar backups).

This server is connected to the cluster over bonded 10Gbe (802.3ad) and has centos 7 running.

When I try to run this over NFS, I get lockups and my nodes take ~10-15 minutes to reboot because of this.
If I use lvm over ISCSI, I have to mount it as directory to be able to backup there and somehow the connection to this mount is instable.

Ideally I'd like to get the NFS mount working, am I missing someting in my configuration? I just shared a folder on the storage server and added that through the proxmox GUI.

I also have a couple of big containers (biggest 750GB+) and the backup of those containers never seems to finish. What's the best way to get backups of big containers?

Thanks!

 

[LnxBil]

We backup a four-node cluster to NFS without any problems. What about the load and I/O response times on your backup storage? What are the specs of the backup server?

 

[timfi]

Thanks for your reply. This is an e5-2609v3 with 16GB ECC RAM. 8 7200RPM SAS disks on a raid card with 1G memory. (write-back with BBU) connected over bonded dual 10GBe. The hardware was previously used as an primary iscsi storage for vmware (40+ vm's). So hardly any load and i/o reponds are perfectly fine. Do you have any special configuration for your nfs server?

 

[LnxBil]

Do you have any special configuration for your nfs server?

No, oridinary setup without any mount arguments.

So hardly any load and i/o reponds are perfectly fine.

That is maybe the problem. If you hammer your server with 1 GB/s, your local disks could be the bottleneck. Do you have performance metrics for your server while backing up? Load and I/O wait times would be interesting.

Have you tried to switch from NFS to CIFS?

 

[janos]

Or as an alternatives you can save CEPH rbd images directly from ceph cluster to your backup storage. You can find many scripts for this orpuse, for example: https://github.com/magusnebula/ceph_backup_script

 

[timfi]

lnxbil: thx for your suggestion, The backup server was not yet in monitoring, so I'm lacking these metrics. I've decided to do a clean installation and try once more, just to be sure there wasn't a simple mistake in my configuration. I'll let you know if there is any difference.

janos: thanks for the suggestion, do you have any experience with this?

 

[janos]

Hi,

Yes, we do a full rbd export every month to a server with large storage.

 

[timfi]

hi janos, thanks, i will definitely check this out as an extra precaution!

 

[janos]

Check this: https://archive.fosdem.org/2018/schedule/event/backup_ceph_at_scale/

 

[timfi]

Ah, i wanted to go to that talk but missed it somehow

 

[timfi]

marked as solved since NFS doesn't seem to be the problem. I also have these problems to local ZFS and LVM.