How to make a backup of Proxmox configuration

[Imbean]

Hi all,

New here so I hope I am doing everything according to the rules. If not, please let me know.

I have a new server (HP Proliant DL 380 Gen 9) and installed Proxmox on it. I am (re)building VM's and Containers as replacements of my Synology (-> VM TrueNas) and Proliant ML310e. On the latter one I have also Proxmox installed which I have been using for 2 years now, I think. I am not a newbie to Proxmox but certainly no expert either.

What I was wondering is, is there a way to backup one's Proxmox configuration? In the sense that, if the Proxmox bootdrive should fail one day and/or I cannot boot from it, naturally, I could install Proxmox on a new drive but how to get the VM/LXC configs to get them configured in the freshly installed Proxmox?
- should I copy a couple of directories, like /etc/pve/* ...? Or is there a function with which you can create a backup?
- clone the disk periodically using the Linux DD command? By the way, can you create an ISO of that disk clone?
- isn't there an out-of-the-box function that you could export out the configuration and import in the new installed Proxmox?
- any other (backup) solution?

I am not using any other physical disk for storing the VM's and LXC's other than the boot drive as far as it concerns the root disks of the VM's and LXC's. Actual data like the TrueNas zpools are on other physical disks.

Ps. I searched for it but I couldn't find the answer yet. Admins, if there is already a thread explaining this, could you provide me the link, please?

Thanks in advance.

 

[Dunuin]

I could install Proxmox on a new drive but how to get the VM/LXC configs to get them configured in the freshly installed Proxmox?

VM/LXC backups contain the VM/LXC configs. Whats missing is other stuff like security groups, aliases, IP sets and so on that are on datacenter level. As well as all the host configs.

- should I copy a couple of directories, like /etc/pve/* ...? Or is there a function with which you can create a backup?

No official host backup or config export is available yet. But there are some unofficial config backup/restore scripts on github.

- clone the disk periodically using the Linux DD command?

Yes, but a bit annoying as this then should be done by booting into a live linux or clonezilla so the PVE system disk isn't in use when backing it up on block level. Without that you would need to work with snapshots for data consistency.

 

[Imbean]

VM/LXC backups contain the VM/LXC configs. Whats missing is other stuff like security groups, aliases, IP sets and so on that are on datacenter level. As well as all the host configs.


No official host backup or config export is available yet. But there are some unofficial config backup/restore scripts on github.


Yes, but a bit annoying as this then should be done by booting into a live linux or clonezilla so the PVE system disk isn't in use when backing it up on block level. Without that you would need to work with snapshots for data consistency.

Thank you, Dunuin!

My critical data is back-upped twice (local 2nd server and in the cloud) and it would be no biggie if I lost the other data so I could consider trying one of these unofficial config backup/restore scripts. Aside from that, cloning the disk every week or so is certainly a good thing, be it a bit annoying indeed. Nevertheless, I am convinced I'll be more than happy to have done that annoying cloning if I have run into serious issues.

 

[esi_y]

What I was wondering is, is there a way to backup one's Proxmox configuration? In the sense that, if the Proxmox bootdrive should fail one day and/or I cannot boot from it, naturally, I could install Proxmox on a new drive but how to get the VM/LXC configs to get them configured in the freshly installed Proxmox?
- should I copy a couple of directories, like /etc/pve/* ...? Or is there a function with which you can create a backup?

Basically, yes (you should) - because they do not provide (after 10+ years) anything that would do it for you, probably assuming everyone is running clusters and then a single node failure does not really matter if you had VMs backups.

- clone the disk periodically using the Linux DD command? By the way, can you create an ISO of that disk clone?

This is overkill, but should you insist on doing that, you could basically make a snapshot while the system is even running and dd that. This is possible with LVM thick but needs spare space in the VG and the snapshot needs to be destroyed after you are done with backup. A bit easier with ZFS where you can e.g. zfs send/receive (over mbuffer, probably). Again, you are on your own.

- isn't there an out-of-the-box function that you could export out the configuration and import in the new installed Proxmox?

Basically /etc/pve copy paste.

- any other (backup) solution?

I would focus on VMs (CTs) backups, be it ZFS zvols or LVM thin volumes, just keep those backed up and the /etc/pve, you may want to keep (parts of) /var should you want to check back some e.g. logs.

I am not using any other physical disk for storing the VM's and LXC's other than the boot drive as far as it concerns the root disks of the VM's and LXC's. Actual data like the TrueNas zpools are on other physical disks.

You do backup your VMs (CTs), right?

Ps. I searched for it but I couldn't find the answer yet. Admins, if there is already a thread explaining this, could you provide me the link, please?

You can also have a look at PBS by Proxmox, but it's not what you were asking about, I think. I never went beyond trying it out, it's useless to my ZFS backups as it does not make use of its snapshots.

 

[Imbean]

Basically, yes (you should) - because they do not provide (after 10+ years) anything that would do it for you, probably assuming everyone is running clusters and then a single node failure does not really matter if you had VMs backups.

First all of, thank you for your reply.

I have a spare drive where I could install Proxmox on, then copy the /etc/pve/* directory onto it, boot from that drive and see if it works?
Dunuin said that some settings on datacenter level are of course not in the /etc/pve-directory of the node, like permissions. I don't have specific permissions set up on datacenter level but maybe there are more settings I am (unaware of) using on that level.

This is overkill, but should you insist on doing that, you could basically make a snapshot while the system is even running and dd that. This is possible with LVM thick but needs spare space in the VG and the snapshot needs to be destroyed after you are done with backup. A bit easier with ZFS where you can e.g. zfs send/receive (over mbuffer, probably). Again, you are on your own.

I understand what you're saying. On the other hand, cloning the drive, I think, would be the safest because you also copy the settings or just anything else that is needed for my Proxmox to operate. However, I sure will try copying /pve-directory as described above.

I would focus on VMs (CTs) backups, be it ZFS zvols or LVM thin volumes, just keep those backed up and the /etc/pve, you may want to keep (parts of) /var should you want to check back some e.g. logs.

You do backup your VMs (CTs), right?

Yes, I do back-up my critical data.
However, it looks like I have lost my music collections (5TB) when I moved the drives from Synology to this new server. I wanted to reuse these 8 disks in my new server but Synology's SHR (their custom raid-system) is not compatible with ZFS. So I had to use my backup drives to copy the data back onto the new array I had created with these 8 disks I moved from Synology to the HP DL380 server.

Just two hours after I had created the new array (so no way back anymore and was completely depending on the backup drives) the zpool my music was on was unavailable. zpool import -f returns corrupted metadata but the pool and drives are ONLINE. Trying to repair with zpool import -fFX returns "one or more devices are unavailable" (?). Just my luck, right?
Luckily I don't need the disks in that server right away, so I can see if someone or a company here in The Netherlands can help me to bring the pool back online ... but that's another story.

As for the VM's with large data, like my own movies library (using Plex to stream them) I back-up just the data to another HP server I have. I do backup the LXC's, using Proxmox backup and copy these backups to another Proxmox instance on that other server.

You can also have a look at PBS by Proxmox, but it's not what you were asking about, I think. I never went beyond trying it out, it's useless to my ZFS backups as it does not make use of its snapshots.

I have thought about that indeed. Not sure if it meets my requirements but I'll have another look.

Again, thank you for your reply. Really appreaciated.

 

[patch]

is there a way to backup one's Proxmox configuration?

This thread discussed options fairly thoroughly https://forum.proxmox.com/threads/how-to-backup-proxmox-configuration-files.67789/

 

[Dunuin]

Dunuin said that some settings on datacenter level are of course not in the /etc/pve-directory of the node, like permissions. I don't have specific

They are in /etc/pve just not part of your VM/LXC backups. As they are part of the cluster.fw and not of the YourVmidHere.fw.

 

[esi_y]

I have a spare drive where I could install Proxmox on, then copy the /etc/pve/* directory onto it, boot from that drive and see if it works?

Yep, you should always test the backups.

I understand what you're saying. On the other hand, cloning the drive, I think, would be the safest because you also copy the settings or just anything else that is needed for my Proxmox to operate. However, I sure will try copying /pve-directory as described above.

I just realised despite it was talked about a lot, it was not in this thread - the /etc/pve is a mounted filesystem, it's virtual, created in RAM by pmxcfs and its basically all saved in config.db file in /var/lib/pve-cluster. So if you dd anything, you would essentially not get any of the files as mounted in /etc/pve (i.e. the most important part of the "host backup", but you will get the underlying DB). You can have a look at the file with any SQL tool, it's just sqlite.db, it's using write-ahead-log (not the default rollback-log) and I would argue very safe to copy out during runtime without even any snapshotting. The pve-cluster service goes on to do the mounting for you upon start, if all else is well (e.g. your freshly installed node has same hostname, etc.).

zpool my music was on was unavailable. zpool import -f returns corrupted metadata

Not that it changes anything now, but any idea why that might have happened? To avoid in the future...

but the pool and drives are ONLINE. Trying to repair with zpool import -fFX

I would be trying without -X at first. Also, did you try -m before?

returns "one or more devices are unavailable" (?). Just my luck, right?

I would wonder how you got into this (to avoid it in the future), was this over some RAID controller, etc?

Luckily I don't need the disks in that server right away, so I can see if someone or a company here in The Netherlands can help me to bring the pool back online ... but that's another story.

I didn't need this before, so can't endorse, but for Windows there's a tool called Klennet for ZFS recovery you can have a look at perhaps.

I do backup the LXC's, using Proxmox backup and copy these backups to another Proxmox instance on that other server.

These (instances) are not in a cluster, correct?

 

[Imbean]

They are in /etc/pve just not part of your VM/LXC backups. As they are part of the cluster.fw and not of the YourVmidHere.fw.

Thank you, I misunderstood that part then.

 

[Imbean]

This thread discussed options fairly thoroughly https://forum.proxmox.com/threads/how-to-backup-proxmox-configuration-files.67789/

Thank you, Patch. I guess I must have overlooked this when I searched if this topic was discussed before.

 

[Imbean]

Yep, you should always test the backups.



I just realised despite it was talked about a lot, it was not in this thread - the /etc/pve is a mounted filesystem, it's virtual, created in RAM by pmxcfs and its basically all saved in config.db file in /var/lib/pve-cluster. So if you dd anything, you would essentially not get any of the files as mounted in /etc/pve (i.e. the most important part of the "host backup", but you will get the underlying DB). You can have a look at the file with any SQL tool, it's just sqlite.db, it's using write-ahead-log (not the default rollback-log) and I would argue very safe to copy out during runtime without even any snapshotting. The pve-cluster service goes on to do the mounting for you upon start, if all else is well (e.g. your freshly installed node has same hostname, etc.).



Not that it changes anything now, but any idea why that might have happened? To avoid in the future...


I would be trying without -X at first. Also, did you try -m before?



I would wonder how you got into this (to avoid it in the future), was this over some RAID controller, etc?



I didn't need this before, so can't endorse, but for Windows there's a tool called Klennet for ZFS recovery you can have a look at perhaps.



These (instances) are not in a cluster, correct?

To be honest, I have no idea how I wound up with this. I shut down Truenas in the proper way and I noticed it took quite longer than normal but it did shut down in the end. After rebooting the server the pool was unavailable. I am using ECC memory, and the smart info of all disks is okay, I didn't get any errors in the days before, .. . I moved the disks the another HP Proliant m310e server I have to exclude it was a hardware failure. Alas, it wasn't; on the other server, the same fault occurred.

I am at the point where I'll risk 'anything' (up to a certain extent) to see if I can successfully mount the ZFS-pool with my music on it. So you know, I am a big music lover and audiophile; I have 6 TB of music (high res, dsd, ripped cd's) on that pool. It took me years to create this collection, little by little. If I would indeed lose this pool, it would take years to rebuild that again. And yes, I did have a (double!) backup but I needed all the drives, including the backup drives, to migrate from Synology to HP DL380 server.

I'll send the link to the TrueNas forum in a private message; I assume it's not done to post in this forum a link to another forum about a different topic and I'll also let you know what I tried.

Thanks for the tip, I sure will try Klennet!

No, these instances are not in a cluster.

 

[patch]

I am a big music lover and audiophile; I have 6 TB of music (high res, dsd, ripped cd's) on that pool. It took me years to create this collection, little by little. If I would indeed lose this pool, it would take years to rebuild that again. And yes, I did have a (double!) backup but I needed all the drives, including the backup drives, to migrate from Synology to HP DL380 server.

Ouch
That sound stressful. No doubt different decisions would be chosen for your 6TB data in hind sight. An off site disk is good to have going forwards.

 

[Imbean]

Ouch
That sound stressful. No doubt different decisions would be chosen for your 6TB data in hind sight. An off site disk is good to have going forwards.

It's no use crying over spilled milk but if I could turn back time, I would have taken 10TB of external storage, even if it were only during the migration.
Well, lessons learned the hard way; should I do this again, I will take external storage for my music collection.
I haven't given up on the lost pool yet, I am going to try the application Tempacc suggested.

 

[nmincone]

Basically /etc/pve copy paste.

This is great. I did a test using

cp -r /etc/pve /mnt/pve/DS1515/pve-backup

for my setup, writing the files to a NFS share and I'll setup a cron job to do this daily before PBS starts the PVE backups.