Best Practices for Backup & Restore for LXC File Servers?

[sdet00]

Howdy folks. I have been using a Debian LXC with Cockpit with a handful of bind mounts for SMB file share duties, and I recently came to the realisation that my backup strategy is really not ideal now that I have over 20TB of data across various shares. I recently changed my Proxmox Backup Server storage and initiated a full backup, and it took over a week to complete. I initially had it running for 3 days before a power cut interrupted the backup, and then I had to start again, which took another 4 days. Naturally this is a worry, because if it takes this long to backup, it will probably take equally as long to restore. I am IO limited by my disks at the moment and upgrading to 30TB worth of SSDs is out of the budget at the moment. I have used the "Live Restore" feature of Proxmox Backup Server for VMs in the past which is great, but this feature is missing for LXCs as far as I can tell.

Out of the 20TB of data, maybe only 2TB is really important. I would like to make some changes so that if my server were to completely fall over, I'd be able to restore a recent backup in a much shorter timeframe.

What are some strategies that I can use to reduce the risk of very long backup and restore times? I have already split my various fileshares across 8 mount points. I was thinking of creating a handful of dummy LXCs and assigning the same mount points, and then backing up from there, which would allow me to target specific mount points first for restore, and then move to other mount points. Suggestions appreciated!

 

[tcabernoch]

Sure. You've created a monster that exceeds your management capacity.
Break it up into pieces you can handle. Seems like you've got that totally figured out.
When you are done, write a nice set of policies to prevent anyone from doing that again.

 

[sdet00]

Sure. You've created a monster that exceeds your management capacity.
Break it up into pieces you can handle. Seems like you've got that totally figured out.
When you are done, write a nice set of policies to prevent anyone from doing that again.

If I had it totally figured out I wouldn't be asking would I? I'm curious if anyone has been in this situation and how they approached it. I'm looking for three things really:

1. Is there some sort of alternative to Proxmox Backup Server live restores, but for LXCs?
2. If I am going "break it into pieces", how should I do this, exactly?
3. How would the restore process look like in this sort of situation?

 

[tcabernoch]

It sounded to me like you were going to break this octopus up into about 8 vms or containers.
And then you went on to describe the data locality and priority. You mentioned various shares.
And all that sounds totally logical. I would use those criteria to decide how you break it up.
After you have manageable machines, the challenges of backup and restore are greatly reduced.

...

Maybe breaking it up is not the best idea. It's sure not the only one.
Do you want something more like this? (this option is only available for container backups.)
https://pve.proxmox.com/pve-docs/chapter-vzdump.html#_file_exclusions
Using that, you could construct a set of backup jobs that prioritized whatever directories you need.
The restore would be interesting. I think you'd restore several containers and then reassemble them. There's probably a better way to do that bit.

 

[sdet00]

I don't think exclusions would work if I had just a single LXC. The way I'd break it up would be a bit different to overcome some of PBS shortcomings with LXCs.

LXC1 = Primary container, online. This would backup boot volume only and would have all mountpoints assigned. Backed up daily.
LXC2 = Data1, offline. Manually add critical data mountpoints used on LXC1. Backed up daily.
LXC3 = Data2, offline. Manually add critical data mountpoints used on LXC1. Backed up every 2 days.

If my server goes up in smoke and takes all drives with it, I'd be able to restore LXC1 and LXC2 within hours to get the most important things up and running. From there I can restore LXC3 over the next few days, but at least I'd have most of the important services running.

I just am not sure what the restore process would be like, whether someone has attempted this, pitfalls etc.