Help with setting up a good backup strategy on single server.

J

jim.bond.9862

Renowned Member
Apr 17, 2015
395
34
68
Hello everyone.
I have been using Proxmox for several month now and I am trying to learn as I go by searching the forums and Google for what I need. but I had a system failure couple of weeks back that I have been trying to recover from and that is what brings me here today. I had no real backup to speak of so recovering the setup was/is a top priority for me. now that I am back and can get to the data I want to do this properly but can not find the info I need so far.

this is a single node home server. it pulls a double duty as Proxmox Server AND as File server for other use.


here is my old setup: a Dual Opteron based system with 49GB RAM
Proxmox VE 5.1 on dual 120GB SSD in ZFS mirrored setup.

Code: 
 pool: rpool
 state: ONLINE
  scan: scrub repaired 0B in 0h4m with 0 errors on Sun Sep  9 00:28:32 2018
config:

    NAME                                                  STATE     READ WRITE CKSUM
    rpool                                                 ONLINE       0     0     0
      mirror-0                                            ONLINE       0     0     0
        ata-INTEL_SSDSC2BW120H6_CVTR52060450120AGN-part2  ONLINE       0     0     0
        ata-SanDisk_SDSSDA120G_161443400890-part2         ONLINE       0     0     0

also have an extra ZFS pool with dual 1TB spinners in zfs mirror for extra system local storage.



this is all related to system setup. in a gui main boot volume and the extra storage volume configured as local_storage.


my main data storage (user data) is on several ZFS pools I have configured manually after the system setup, at the time WebGUI did not support zfs management so CLI manual work.

all pools are ZFS mirror.

pool0 is 6x2TB drives --- holds all main folders for use with file server and other VM/LXI
pool1 is 4x3TB drives --- holds a back up of several main folders from pool0
pool2 is 2x4TB drives --- empty and ready for use


specific folders from each pool is bind-mount (BM) to containers for use.
as in , the pool0 "emblib" is BM into "/mnt/embylib" in emby container and than used like
{ rsync -av --recursive "/mnt/embylib/" "/mnt/embylibbk/" } where ""/mnt/embylibbk/"" is BM from pool1


after my setup went down. I upgraded the server with Dual Xenon MB and 128GB ram.
which I now can start on an older kernel (Linux pve2 4.15.17-2-pve #1 SMP PVE 4.15.17-10 )
any of 4.15.18- kernels still get stuck on empty screen with blinking cursor no matter what I try.

SO, what I am thinking is, if I can configure a good setup of the current system and figure out how to make a full working setup of everything. I can dump the config and re do the system from scratch without worying about loosing data or VMs.

can someone help me do the proper backup of everything.

my thoughts:

I have a 5TB external USB drive. it is GPT formated with a single EXT4 partition.

I had attached it to the server and mount by label into /media/extbk

also changed fstab to have line "LABEL=EXT_BK /media/extbk ext4 defaults 0 0"
to automount the disk on reboot by label.


all backups I want should be going there

I guess for main user data backup I can run "rsync -av --recursive source_pool#_folder target_folder "
to make a full copy of all data I have on my pools this is a one time run for now but I will want to setup a back up job later to have all data up to date.

but how do I backup the system and VM and containers so I can blow the whole system away and rebuild from scratch without loosing a working VM/LXC.

all LXC use zfs subvol as system disks how do I backup that?

Thank for all your help.
 
Hello Jim,

I don't have such a big setup but still have the same issues, good how to backup data.

What my approach is to heavily rely on zfs based tools and a general tool that can to backup the operating system.

1st
Make regular backups of your vm and LXC to another pool or at best externally with the integrated mechanism.

2nd
I recommend to zfs snapshot/send from one pool to another instead of rsync through a LXC.
Znapzend does a good job to configure and automate that on a dataset by dataset base if you like.
Even if you don't want to send them to another pool.

3rd
For the base os and its config a disaster recovery image (DR) in my opinion is the best choice.
In your case zfs snapshots of your rpool don't hurt.
Do it automatically via cronjob at best to a remote/external location.
USB Stick /HDD recommended.
I did use mondorescue for a while, but I don't know if it's still being maintained and the installation is a little messy.

Hope you find this useful.

Best regards
FleischKarussel
 
Check out borg for your backups assuming you don't have another node for ZFS send/receive. We are using it in production and it's awesome. This is one of the best option if you want compression/incremental backups.

The easiest way to go would be to use the Proxmox webui built-in backup option to backup the full LXC everyday then rsync the host to your USB drive. Since linux is mainly files you can use tools like borg or rsync

The way we do it is:
1- Create a ZFS snapshot to based the LXC backup from to prevent corrupted backup data
Code: 
zfs snapshot rpool/data/subvol-${CT_ID}-disk-1@borg
2- Launch borg from the ZFS snapshot. You can tweak the settings to your liking.
Code: 
   SRC_BACKUP_DIR="/rpool/data/subvol-${CT_ID}-disk-1/.zfs/snapshot/borg"
   TGT_BACKUP_DIR="whatevever"
   cd $SRC_BACKUP_DIR
   borg create                        \
        --compression zlib,5                \
        --exclude 'dev/*'                 \
        --exclude 'run/*'                 \
        --exclude 'mnt/*'                 \
        --exclude 'sys/*'                 \
        --exclude 'proc/*'                 \
        --exclude 'tmp/*'                 \
        --exclude 'media/*'                \
        ${TGT_BACKUP_DIR}::$(date +%Y-%m-%d-%H:%M)     \
        .
3- Destroy the snapshot
Code: 
    cd ~
    zfs destroy rpool/data/subvol-${CT_ID}-disk-1@borg
To backup the host we use the same procedure. You can also run borg prune after to only keep a certain amount of backups.

To restore:
1- Create a CT in the webui with the same name/specs
2- Make sure the CT is shutdown
3- Mount the borg repo
Code: 
borg mount /mnt/whatever/backup/path/ /mnt/tmp
4- Rsync to the ZFS dataset of the container
Code: 
rsync -av --delete /mnt/tmp/2018-02-25-23:00/* /rpool/data/subvol-104-disk-1/
5- Umount the borg repo
Code: 
borg umount /mnt/tmp
6- You can now start the container.

Make sure you run borg check often to check the integrity of your backups.

This is a brief overview of the way we do it. Let me know if you need more help.
 
  • Like
Reactions: James Crook
thanks FleischKarussel it sound like a good suggestions.
my problems are two fold,

I am not sure how to make the backup properly. I have the backup turned on and I can see it in the GUI but where is it to take ti offsite?
I understand using zfs snapshot or such to put extra backup on other ZFS volume. but I only have one server so if it goes down how will I get the backups to recover?
my external drive is not ZFS it is EXT4. can I send any thing to it with zfs tools? I think not. so that is a no go for me.

right this moment I am running rsync to at least back up my media and other data off the pools.

if I have to rebuild the VM containers from scratch I will do it. now that I have the server up and running in more or less stable fashion I can at least copy the configs to refer to.
I might even rethink the local storage configuration a little bit to.
 
Hello Jim,

you can add a nfs share or whatever local directory as a storage destination and enable backups for it and modify your backup job(s) to use this as your backup target.
So then you're able to backup to whereever you want it to.

I guess there is no zfs-tools way to directly send data to a ext4 destination, not that I am aware of.
Is there a need for ext4 for this disk?
Would make things a lot easier if not, as I pointed out in my post before. ;)

To rsync something off your server is in general a good idea, as long as its data consistent.
 
Ok so I understand this correctly, I need to create a directory and than set it as a local store with type backup. That point proxmox backup for each vm/container to that store.

What about the system config?
Will this work for vm/containers that use zfs volume as system disks?

I will try it though.

Also. Rsync works well. Got almost all of my data to external drive. Still cleaning up.
 
Hi,

I will tell you what I do in your case:

1. I will buy 2 Raspberry Pi (rpi)devices
2. I will conect 2 x usb hdd on each rpi(I know is working from forums, so if I am wrong ... corect me)
3. on each rpi I will create the desired storage using usb disks (samba, nfs, iscsi, whatever or maybe zfs fuse ...)
4. On the server side I will use rsnapshot (could be any backup tool), and save/copy the VM/CT data(pmx vzdump files) on this 2 rpi .
5. Using rsnapshot is an advantage because of this:
- you can access/restore any backup even without rsnapshot(any tool who can copy a file is ok)
- if your remote rpi is not online, rsnapshot will not make any backup(see do not create repository option)
- you can use pre/post backup custom scripts (like mount the remote storage and unmount at the end of backup)
- you can create a retention periods for your backups (like daily backups are hold for 4 days, the weekly ... monthly. .or whatever period.)

Also you can use wake-on-lan to power on your rpi and then after your backup is finish (post-backup rsnapshot script) you can power off the rpi and maybe send a success mail?
 
  • Like
Reactions: Jarvar
Why do I need raspberry pi setup?

What does it gain me over a simple usb disk for backup.

If you read my post here.
I have a 5tb usb disk that is connected and mounted on the host.
I run rsynck to drop everything there.
I also dupe all my user data on second zfs pool within the server.
My daya should be ok.
All I need now is info on how to backup my vm/containers and if possible my proxmox config. So if I have to rebuild proxmox a new I could install it and restore the setup.
Import my zfs pools and done.

I do not need automation beyond ,maybe, setting up a cron job to run rsynck. But mostly I can run it manually. My data does not change much.
 
Hi,

Think of this:
- any of your usb disks if are all connected to the same server could be broken all at the same time
- so is not important if you have backups if your PSU will burn your server and all your usb disk connected to the server
- my fault, I have the wrong ideea that you have several hdd-usb insted of only one
- you could use in this case only one rpi with your only one usb-hdd
- using a separate host (rpi) for backup is better then have all of your data on the same host (the pmx server)

The rest of what I allready say about rsnapshot is valid if you want to use only the pmx server or the better variant with rpi(you can power on only for backup with a script, so you are not forced to keep your usb hdd power on as for pmx only host)
 
CharlesErickT said:
Check out borg for your backups assuming you don't have another node for ZFS send/receive. We are using it in production and it's awesome. This is one of the best option if you want compression/incremental backups.

The easiest way to go would be to use the Proxmox webui built-in backup option to backup the full LXC everyday then rsync the host to your USB drive. Since linux is mainly files you can use tools like borg or rsync

The way we do it is:
1- Create a ZFS snapshot to based the LXC backup from to prevent corrupted backup data
Code: 
zfs snapshot rpool/data/subvol-${CT_ID}-disk-1@borg
2- Launch borg from the ZFS snapshot. You can tweak the settings to your liking.
Code: 
   SRC_BACKUP_DIR="/rpool/data/subvol-${CT_ID}-disk-1/.zfs/snapshot/borg"
   TGT_BACKUP_DIR="whatevever"
   cd $SRC_BACKUP_DIR
   borg create                        \
        --compression zlib,5                \
        --exclude 'dev/*'                 \
        --exclude 'run/*'                 \
        --exclude 'mnt/*'                 \
        --exclude 'sys/*'                 \
        --exclude 'proc/*'                 \
        --exclude 'tmp/*'                 \
        --exclude 'media/*'                \
        ${TGT_BACKUP_DIR}::$(date +%Y-%m-%d-%H:%M)     \
        .
3- Destroy the snapshot
Code: 
    cd ~
    zfs destroy rpool/data/subvol-${CT_ID}-disk-1@borg
To backup the host we use the same procedure. You can also run borg prune after to only keep a certain amount of backups.

To restore:
1- Create a CT in the webui with the same name/specs
2- Make sure the CT is shutdown
3- Mount the borg repo
Code: 
borg mount /mnt/whatever/backup/path/ /mnt/tmp
4- Rsync to the ZFS dataset of the container
Code: 
rsync -av --delete /mnt/tmp/2018-02-25-23:00/* /rpool/data/subvol-104-disk-1/
5- Umount the borg repo
Code: 
borg umount /mnt/tmp
6- You can now start the container.

Make sure you run borg check often to check the integrity of your backups.

This is a brief overview of the way we do it. Let me know if you need more help.
Click to expand...


Any pointers on how to do this with Virtual Machines ?
Every time i do a backup from a ZFS snapshot (i just called) of the machines the repository grows by the size of disk.

i.e. 20 GB drive, first backup takes 20GB, then the second backup (called minutes after the first one completes) takes up 20GB so when running du -h on the location i get 40GB used.....
 
James Crook said:
Any pointers on how to do this with Virtual Machines ?
Every time i do a backup from a ZFS snapshot (i just called) of the machines the repository grows by the size of disk.

i.e. 20 GB drive, first backup takes 20GB, then the second backup (called minutes after the first one completes) takes up 20GB so when running du -h on the location i get 40GB used.....
Click to expand...
Unfortunately, you don't have access to the files directly in the snapshot like an LXC container dataset. This will not work with borg since you are trying to backup 1 giant raw file (like an archive). Incremental will not work since the whole file as changed therefore it will be backed again. This is why every time you backup it add 20GB.

You could look into ZFS send/receive or use the built-in backup in the webui but it's not incremental.
 
CharlesErickT said:
Unfortunately, you don't have access to the files directly in the snapshot like an LXC container dataset. This will not work with borg since you are trying to backup 1 giant raw file (like an archive). Incremental will not work since the whole file as changed therefore it will be backed again. This is why every time you backup it add 20GB.

You could look into ZFS send/receive or use the built-in backup in the webui but it's not incremental.
Click to expand...


Already trying that....

zfs send $zfsvmdisk@borg | borg create -v --stats \
$borg_repository::$vm-'{now:%d-%m-%Y-%H:%M}' \
-
 
James Crook said:
Already trying that....

zfs send $zfsvmdisk@borg | borg create -v --stats \
$borg_repository::$vm-'{now:%d-%m-%Y-%H:%M}' \
-
Click to expand...
That's what I said. You will not be able to use borg with a dataset using raw files. When specifying zfs send/receive, I was talking about sending the dataset to another remote server using ZFS and ditching borg.

Borg is fantastic but in your case you won't be able to use it. You could try to run it directly inside the VM but depending on what is running on it you might have some issue with file consistency since they will change while you back them up (The whole point of doing a snapshot with LXC)
 
CharlesErickT said:
That's what I said. You will not be able to use borg with a dataset using raw files. When specifying zfs send/receive, I was talking about sending the dataset to another remote server using ZFS and ditching borg.

Borg is fantastic but in your case you won't be able to use it. You could try to run it directly inside the VM but depending on what is running on it you might have some issue with file consistency since they will change while you back them up (The whole point of doing a snapshot with LXC)
Click to expand...

Sorry misunderstood. Hmmmm Going to have to think of another plan for fat virtuals then....
 
CharlesErickT said:
That's what I said. You will not be able to use borg with a dataset using raw files. When specifying zfs send/receive, I was talking about sending the dataset to another remote server using ZFS and ditching borg.

Borg is fantastic but in your case you won't be able to use it. You could try to run it directly inside the VM but depending on what is running on it you might have some issue with file consistency since they will change while you back them up (The whole point of doing a snapshot with LXC)
Click to expand...

Sorry one last question, could i not mount the read only snapshot created by zfs snapshot and either point borg at that or pipe it in to borg ?
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back