PVE LXC/VM failsafe storage location

[crispyduck]

Hi!

Don't judge me, I just switched some of my Lab servers from VMWare to PVE.
I really love it, took me just some hours to get a multi host cluster online. Easy VM/LC migration,...

What concerns me now a little bit is how are VMs and LXCs are stored.
The data disks/volumes,... are stored on my configurable storage location, ZFS, Ceph, SAN,... but what about the VM/LXC config?

This is all stored on the host disk, so if it fails all the VM configs are lost.


Okay, my Lab boot disks are RAID1, but also these can fail. Does this mean even if I have a shared storage I need to backup my host?

Why is there not stored at least a copy of the actual vm conf with the rootfs disk? Updated each boot/config cahnge?

I am still reading/learning all the backgrounds, but this is one thing I don't get.
My concept was always to have my VMs more or less independent from my hyperviser. If my host fails, I don't care to much, I can replace it or even spin up my VMs somewhere else with where my network bridges are available.

So why this concept to depend on cfg from the host?

And, what files/db need to be saved/backed up to be save even when a host in a single PVE environment with shared storage is lost?

Thanks,
cd

 

[Dunuin]

Hi!

Don't judge me, I just switched some of my Lab servers from VMWare to PVE.
I really love it, took me just some hours to get a multi host cluster online. Easy VM/LC migration,...

What concerns me now a little bit is how are VMs and LXCs are stored.
The data disks/volumes,... are stored on my configurable storage location, ZFS, Ceph, SAN,... but what about the VM/LXC config?

This is all stored on the host disk, so if it fails all the VM configs are lost.


Okay, my Lab boot disks are RAID1, but also these can fail. Does this mean even if I have a shared storage I need to backup my host?

Why is there not stored at least a copy of the actual vm conf with the rootfs disk? Updated each boot/config cahnge?

I am still reading/learning all the backgrounds, but this is one thing I don't get.
My concept was always to have my VMs more or less independent from my hyperviser. If my host fails, I don't care to much, I can replace it or even spin up my VMs somewhere else with where my network bridges are available.

So why this concept to depend on cfg from the host?

And, what files/db need to be saved/backed up to be save even when a host in a single PVE environment with shared storage is lost?

Thanks,
cd

The usual way is to backup your guests is using vzdump to a NAS/external disk or better to run a proxmox backup server (PBS). In both cases the VM/LXC configs are stored together with the virtual disks. You only would need to backup your hosts /etc folder (especially /etc/pve) if you need to reinstall your PVE OS or don't got backup but lost your PVE OS disk but not the VM/LXC storage. In that case its useful to have a backup of the VM/LXCs config files in /etc/pve/lxc and /etc/pve/qemu-server. Also keep in mind that the /etc/pve is no normal folder. Its the mountpoint of the pmxcfs filesystem and is empty if you try to access the data on that disk from a live linux or from another machine.
I personally also liked to use clonezilla to backup my PVE OS disks. But meanwhile I switched from vzdump to PBS and with that you can use the proxmox-backup-client to store a blocklevel backup of your PVE OS disks and your /etc folder on file level to your PBS server.

 

[crispyduck]

Hi Dunuin!

Thanks for your reply.
A PBS (what I will add for sure) will do the job, or mitigate my main concerns.

Backing up the files/path mentioned maybe bit more will help to be on the save side. But nevertheless at the moment the concept is a bit confusing to me.

Maybe it is just as I am used to ESXi, but what I loved and used there very much is that the VMs are from config point of view Hyperviser independent.

Simple shared Storage, (NFS, SAN,...) with multiple independent free ESXi, and you are able to spin up your VMs where ever you want.

Take the VM folder, spin it up on your workstation, or convert it and run it on a HyperV or VBox.

Why don't store the VM config or a copy of it with the VM disk files?

What is the reason to not keep all VM/LC files on one place?

For Labor personal use I will for sure find ways to backup,.... but I am also thinking about our business cases and at the moment this is a bigger problem in the way we use and deploy HW independent systems around the globe.

Regards,
cb

 

[Dunuin]

Why don't store the VM config or a copy of it with the VM disk files?
What is the reason to not keep all VM/LC files on one place?

All configs are stored in /etc/pve which is the mountpoint of a pmxcfs filesystem that is based on fuse. The configs you see in files in /etc/pve are actually configs stored in a SQLite DB that are mapped to files. And this will be synced every minute across all nodes of the cluster using corosync so HA is possible.

 

[crispyduck]

Thanks! Yes, think for a cluster this is good.

But beside my business case, e.g. a example how I am used to use our Lab environment:

-Central Storage, preferred NFS on top of ZFS or another snapshot/clone aware FS.

-Several ESXi servers, sometimes old HW, just added as it is available.

-no cluster,... one host is not aware of the other one.

-still able to boot a VM stored on the shared storage on any available host, just because the cfg is stored with the vm and can be regstered at any host having access to it.

 

[fabian]

because PVE supports storage types where storing files is not even possible, among other reasons it's also a plain, readable text file - so saving it/copying it to some other system is a simple scp away anyway..

 

[crispyduck]

Thanks! Sure, I expected that there are some other reasons behind. Just wanted to explain why I am asking for this and not just a backup strategy.
Would just be nice to have the VM config also on the shared Storage.

Anyway, it is as it is. Then back to backup; what files from my pve host do I need to backup:
1. to be able to quickly spin up my VMs on another host , or a nwe host that has access to the storage?
2. to be able to quickly restore also the host config after a new install?

Br,
cd

 

[fabian]

basically all of PVE-specific config is in /etc/pve (except for a settings file affecting local backup jobs /etc/vzdump.cfg). what else you'd want to backup depends on what customization you did - e.g., /etc/network/interfaces , /etc/hosts are possibly good candidates. many people simply backup ALL of /etc - it's only config files so doesn't take up much space anyway, and better to be safe than sorry.

giving multiple PVE clusters access to the same shared storage without some sort of namespacing (like different NFS exports / sub directories per cluster) is generally frowned upon - it can lead to unexpected behaviour if one cluster thinks files belong to itself that are actually used by some other cluster, e.g. if you use one VMID on multiple clusters and then remove the guest.

 

[crispyduck]

Thanks! Is just for the VMs "/etc/pve/lxc" and "/etc/pve/qemu-server" enough?

The host setup is more or less static, so backup is not so frequently needed. But VMs come and go.
For a system I use alone I think I know what and when to backup. At the Lab environment I also give access to several colleges, they can create and delete VMs as needed. If I know just periodically backup the VM configs, It could be that I have a 101.conf file from a VM that was in the meantime destroyed and recreated.
Is there a better backup strategy for the configs? I don't need and want to make full backups, as the underlying shared storage is anyway backed up, redundant,....
On the same Shared storage are also ESXi hosts, here it is easy, as everything needed is stored with the disks in the vm folder on the storage.
Now switching some of the hosts to PVE, most of them standalone as this will be the main setup how we could use PVE also on site on customer locations I need to find a backup strategy that tolerates host failures keeping the VMs accessible.

So as you mentioned one NFS export per host, and a cron that copies or rsyncs the vm config files into a folder on this specific export. Replacing the host with a new one or another would mean simply adding the NFS export and copying back the vm configs. ?

What about the VMIDs, can they be changed by simply adapting the config files? Is there a way to force a node/host to use a specific VMID range? Eg. my PVE1 100-199, PVE2 200-299,...? So it should be possible to ensure there is no overlapping and keeping the VMIDs unique globally.

best regards,
cd

 

[fabian]

well, things like replication and backup jobs, firewall rules, HA settings are stored in other files, so probably saving all of /etc/pve is the safer approach unless you know for sure you don't want any of the other files.

 

[crispyduck]

Ok, seems I have to play and test a little bit what's needed just to have the VMs.

Would like to have alternative to ESXi and so far I like PVE a lot. At least clustering is quiet easy.

Should also fit most Customer installs, small systems with independent A/B node and also bigger setups with two clusters.

I did not test backup/restore so far, due to long HW delivery times these days we often just set up our Systems/VMs in the Lab, export them and later simply import them on site. Think this should not be a Problem to do with PVE, just a bit different handling.

Regarding Enterprise subscription, what happens when it runs out after e.g. two or 3 years? Anything to do on in place systems?

Sorry, is a bit off topic.

 

[fabian]

if your subscription is no longer valid, you need to switch to one of the non pve-enterprise repositories: https://pve.proxmox.com/pve-docs/pve-admin-guide.html#sysadmin_package_repositories , else you won't receive any more updates.

 

[crispyduck]

The systems are all offline with no direct internet connection, so updates can anyway not be be done via the online repo.
It is just important that it keeps working as is, not like on VMWare, where maybe something does not start when the license is expired.

 

[fabian]

there is no change w.r.t. features/usability with or without an active subscription