[SOLVED] Question: Why does proxmox-backup-client skip "etc/pve" in the backup?

[matrix1999]

Hi,

I am interested to leverage proxmox-backup-client to make backups on my hosts. I read several forum posts and tutorials and have successfully created the backup.

Questions:
1/ I noticed that /etc/pve is not included in the backup, nor did I specified it in the .pxarexclude file. Why is that the case? Did I do something incorrectly in the setup?
2/ I assume it is a good idea to have /etc/pve backup, especially the fact that it contains all the information for the clusters and nodes setup. If not, what is the best practice to backup /etc/pve?


Thanks you!

 

[leesteken]

It skips it because /etc/pve is a different filesystem from /etc. Use the option --include-dev /etc/pve to include it. Or backup the underlying configuration database.

 

[Dunuin]

By default, the backup client won't traverse mountpoints unless you explicitely tell it to do so by specifying the mountpoints via "--include-dev".

 

[esi_y]

It skips it because /etc/pve is a different filesystem from /etc. Use the option --include-dev /etc/pve to include it. Or backup the underlying configuration database.

Isn't that (local) config.db going to be overwritten the moment one brings it in (on that one particular node) instantly from the more recent versions of it maintained by the (rest of the) cluster anyhow?

There's a moment when (e.g. joining a cluster) a backup is made of the database, copying it back (as described in the document) then would restore the pre-cluster state of the node, but it's basically just the database that then presents itself via that virtual FUSE mounted (equally) to all nodes in /etc/pve.

Backing it up on an active node (with pve-cluster running) without some sort of snaphost prior to running the backing is not going to result in something dependable anyhow.

 

[matrix1999]

It skips it because /etc/pve is a different filesystem from /etc. Use the option --include-dev /etc/pve to include it. Or backup the underlying configuration database.

Thanks for your prompt reply.

Additional questions:
1/ According to what you mentioned above, since I didn't exclude /var/lib/pre-cluster, so is it save to assume that I have already backed up /var/lib/pve-cluster/config.db? Is there a way I can check if that has been backed up?
2/ Assume /var/lib/pve-cluster/config.db is backed up, does it mean I no longer need to include /etc/pve/*?

 

[matrix1999]

By default, the backup client won't traverse mountpoints unless you explicitely tell it to do so by specifying the mountpoints via "--include-dev".

Thanks for your prompt reply. What other mountpoint is a general good best practice to include using --include-dev other than /etc/pve?

 

[Dunuin]

PBS got single file restore. You could browse the contents of your backup snapshots from your PVEs webUI.

 

[Dunuin]

What other mountpoints is a general good best practice to include using --include-dev other than /etc/pve?

Depends on what you want to backup...
For the stuff I care about it is "/etc/pve" only.

 

[matrix1999]

Isn't that (local) config.db going to be overwritten the moment one brings it in (on that one particular node) instantly from the more recent versions of it maintained by the (rest of the) cluster anyhow?

There's a moment when (e.g. joining a cluster) a backup is made of the database, copying it back (as described in the document) then would restore the pre-cluster state of the node, but it's basically just the database that then presents itself via that virtual FUSE mounted (equally) to all nodes in /etc/pve.

Backing it up on an active node (with pve-cluster running) without some sort of snaphost prior to running the backing is not going to result in something dependable anyhow.

According to "Section 6.5 Recovery", if you have backed up the config.db, then you will need to stop the pve-cluster service, restore config.db (required permission 0600), then reboot.

 

[esi_y]

According to "Section 6.5 Recovery", if you have backed up the config.db, then you will need to stop the pve-cluster service, restore config.db (required permission 0600), then reboot.

I know it's official docs, I just prefer to understand what and why (is going on) and when.

If you have a look at:
https://github.com/proxmox/pve-clus...f24c05a11b0f864f5b9dc/src/PVE/Cluster.pm#L884

You see there is even a "backup" feature made but it clearly is used just (to my knowledge) to make a failed cluster join recoverable because it's then deleted on finishing a join:
https://github.com/proxmox/pve-clus...a11b0f864f5b9dc/src/PVE/Cluster/Setup.pm#L772

The whole "filesystem" mounted into /etc/pve can be nicely seen (with the rest in the same directory):
https://github.com/proxmox/pve-clus...4ff24c05a11b0f864f5b9dc/src/pmxcfs/database.c

Basically what I wanted to say is ... I really wonder how the database is stored in that config.db as in - it might be basically fine to just copy any of the (other healthy cluster node's) file into place (of the node to be revived), it's a "backend" to the same cluster database exposed as a little FUSE filesystem.

Stopping the pve-cluster will make your node go offline in the cluster, you certainly do not want to do it before a long backup job, if you were to do it quickly just for the sake of backing up that one config file I wonder if it's any useful (in case taking any of the remaining node's database copy might be just as good).

I will test this later (it's a lazy way of finding out without reading through the whole code), but even by the docs - it asks you to properly manually populate hostname and hosts file for a reason - I suspect the database is the same everywhere.

Btw the other way is true for sure - if you leave empty database on a node but put in the appropriate things in the /etc/pve, it will update the config. And it's sort of safer to backup filesystem path since the file changes in it are atomic by the very nature of the filesystem.

 

[esi_y]

I know it's official docs, I just prefer to understand what and why (is going on) and when.

If you have a look at:
https://github.com/proxmox/pve-clus...f24c05a11b0f864f5b9dc/src/PVE/Cluster.pm#L884

You see there is even a "backup" feature made but it clearly is used just (to my knowledge) to make a failed cluster join recoverable because it's then deleted on finishing a join:
https://github.com/proxmox/pve-clus...a11b0f864f5b9dc/src/PVE/Cluster/Setup.pm#L772

The whole "filesystem" mounted into /etc/pve can be nicely seen (with the rest in the same directory):
https://github.com/proxmox/pve-clus...4ff24c05a11b0f864f5b9dc/src/pmxcfs/database.c

Basically what I wanted to say is ... I really wonder how the database is stored in that config.db as in - it might be basically fine to just copy any of the (other healthy cluster node's) file into place (of the node to be revived), it's a "backend" to the same cluster database exposed as a little FUSE filesystem.

Stopping the pve-cluster will make your node go offline in the cluster, you certainly do not want to do it before a long backup job, if you were to do it quickly just for the sake of backing up that one config file I wonder if it's any useful (in case taking any of the remaining node's database copy might be just as good).

I will test this later (it's a lazy way of finding out without reading through the whole code), but even by the docs - it asks you to properly manually populate hostname and hosts file for a reason - I suspect the database is the same everywhere.

Btw the other way is true for sure - if you leave empty database on a node but put in the appropriate things in the /etc/pve, it will update the config. And it's sort of safer to backup filesystem path since the file changes in it are atomic by the very nature of the filesystem.

The whole schema of that DB is just the filesystem really.

CREATE TABLE tree ( inode INTEGER PRIMARY KEY NOT NULL, parent INTEGER NOT NULL CHECK(typeof(parent)=='integer'), version INTEGER NOT NULL CHECK(typeof(version)=='integer'), writer INTEGER NOT NULL CHECK(typeof(writer)=='integer'), mtime INTEGER NOT NULL CHECK(typeof(mtime)=='integer'), type INTEGER NOT NULL CHECK(typeof(type)=='integer'), name TEXT NOT NULL, data BLOB)

And when you look at the filesystem, whatever is specific to local node is symlinks (i.e. local, lxc, openvz, qemu-server into respective nodes/$nodename) so it seems that you only need to back this up once from any node of the cluster. But for the case mentioned in the docs (where other nodes are alive and well), it should be fine just to copy it out from any of the existing nodes.

It's all very strange why is this not already a built-in feature (i.e. regularly take atomic snapshot of the config.db and put it into local filesystem of the node with a timestamp).

 

[esi_y]

The whole schema of that DB is just the filesystem really.

CREATE TABLE tree ( inode INTEGER PRIMARY KEY NOT NULL, parent INTEGER NOT NULL CHECK(typeof(parent)=='integer'), version INTEGER NOT NULL CHECK(typeof(version)=='integer'), writer INTEGER NOT NULL CHECK(typeof(writer)=='integer'), mtime INTEGER NOT NULL CHECK(typeof(mtime)=='integer'), type INTEGER NOT NULL CHECK(typeof(type)=='integer'), name TEXT NOT NULL, data BLOB)

And when you look at the filesystem, whatever is specific to local node is symlinks (i.e. local, lxc, openvz, qemu-server into respective nodes/$nodename) so it seems that you only need to back this up once from any node of the cluster. But for the case mentioned in the docs (where other nodes are alive and well), it should be fine just to copy it out from any of the existing nodes.

It's all very strange why is this not already a built-in feature (i.e. regularly take atomic snapshot of the config.db and put it into local filesystem of the node with a timestamp).

@fabian Sorry to spam, but can i quickly ask you is it safe to just grab any cluster's node config.db as a backup (of any particular = all node(s) at once at any time)? say it's on ZFS, it should be safe to grab it off a snapshot too, correct? And then if any node need to kick-start just implant it and start pve-cluster?

 

[fabian]

the safe variant is to re-sync via corosync, anything else is not supported (and what works today, might not in the future!). you can keep backups of the DB (e.g., to have historical versions of files in /etc/pve in case of accidental changes) if you want, but any modification / .. can easily lead to problems and should be avoided. basically that /etc/pve is backed by this sqlite DB is an implementation detail.

 

[esi_y]

the safe variant is to re-sync via corosync, anything else is not supported (and what works today, might not in the future!). you can keep backups of the DB (e.g., to have historical versions of files in /etc/pve in case of accidental changes) if you want, but any modification / .. can easily lead to problems and should be avoided. basically that /etc/pve is backed by this sqlite DB is an implementation detail.

Right, I was mostly asking in the context of the OP's post. The implementation detail is significant (or so I thought) in that if looking for:

i) best way to backup cluster node (OP mentioned he expected /etc/pve be included); and

ii) have a way to handle total cluster loss (e.g. datacentre fire)

For losing a node, no question, it's just most sensible to add new node, re-sync. But in case there's nothing to resync with as in (ii), but one would like to pull out backups to reconstruct the cluster, and only for this edge case:

1) Is it sensible to have a backup of the config.db file and implant it into a node then start the pve-cluster?

2) Asking because the config.db, obviously, was part of all backups (from some point in time) already, as it is in the local filesystem.

3) The config.db for this edge case is better than including the /etc/pve in a backup, isn't it? Because as it is sqlite it will be in some (albeit outdated) consistent state at any given point. A backup tool might have caught different parts of /etc/pve at different times and while any individual file will not be caught mid-write, they might not be capturing a consistent state across files (as the backup took them at different times, there's no way to get a lock on this location during backup).

4) The config.db is identically stored on each node, the local /etc/pve/local, etc. symlinks are artificial FUSE construct, so capturing config.db is more universal than /etc/pve from a viewpoint of a particular node, correct?

If I am completely off, apologies for veering off all the way. I am mostly interested because it seems the most elegant way to have a backup without taking having to take a node offline even.

NB I understand all of the above will be unofficial way to do it anyhow, but is it sensible in the edge case (ii)?

 

[fabian]

3) is also true for the rest of the system though, a host backup is only consistent if you use some sort of file system snapshot as a base
4) is correct now, but might not be correct anymore in the future

I'd still advise to recover individual config files from the backup, and re-doing the cluster setup (order can be mixed of course, e.g., first restore network config/.., then create/join cluster, then restore configs going into /etc/pve)

 

[esi_y]

3) is also true for the rest of the system though, a host backup is only consistent if you use some sort of file system snapshot as a base

Yes, but I suppose one can always get a snapshot on the LVM*, ZFS or BTRFS for such backup, but there's no way to take a snapshot on the mounted pmxcfs (other than take pve-cluster off momentarily, complicating backup chores, generating logfile noise, etc.).

4) is correct now, but might not be correct anymore in the future

Yes I will have to keep an eye on the schema if using this, but do you have anything particular in mind or is this more like being on the safe side with public statements?

I'd still advise to recover individual config files from the backup, and re-doing the cluster setup (order can be mixed of course, e.g., first restore network config/.., then create/join cluster, then restore configs going into /etc/pve)

I know, but it's more manual work. Harder to automate. I actually wondered why there isn't e.g. userland mount tool for the config.db (is in a copied away one, not one being currently in /etc/pve) to make it all easier (to extract individual files) because the config.db is not only inadvertently backed up with each node's local filesystem, but even when making a standalone node a cluster node you do it yourself (it must have been a failsafe). Though I never found it being used, I mean that backup before taking the cluster plunge, so I suppose that's for manual recovery (to advise someone tearing out their hair) by copying it simply back (pretty safe in single node setup recovery I suppose).

* You have a table in the docs mentioning LVM (thick) does not support snapshots:
https://pve.proxmox.com/wiki/Storage#_storage_types

This is obviously not correct within the context of LVM, but during regular ISO install there indeed is no space left for in the VG for using snapshots, which is IMHO a pity.

 

[fabian]

Yes I will have to keep an eye on the schema if using this, but do you have anything particular in mind or is this more like being on the safe side with public statements?

the latter.

I know, but it's more manual work. Harder to automate. I actually wondered why there isn't e.g. userland mount tool for the config.db (is in a copied away one, not one being currently in /etc/pve) to make it all easier (to extract individual files) because the config.db is not only inadvertently backed up with each node's local filesystem, but even when making a standalone node a cluster node you do it yourself (it must have been a failsafe). Though I never found it being used, I mean that backup before taking the cluster plunge, so I suppose that's for manual recovery (to advise someone tearing out their hair) by copying it simply back (pretty safe in single node setup recovery I suppose).

the problem is that for the backup to be consistent, you must stop pmxcfs. and obviously, since everything relies on that running, you don't want that during regular operations. I guess we could add it to pmxcfs itself as well, to allow triggering a consistent dump with it running. or do it as part of rebooting a node, but then the schedule is much more variable.

* You have a table in the docs mentioning LVM (thick) does not support snapshots:
https://pve.proxmox.com/wiki/Storage#_storage_types

This is obviously not correct within the context of LVM, but during regular ISO install there indeed is no space left for in the VG for using snapshots, which is IMHO a pity.

yeah, that table is meant to state whether a storage supports snapshots in the context of PVE, where snapshots are potentially kept indefinitely. LVM snapshots don't work there, they are meant as temporary consistency measure while pulling a backup (and have a lot of constraints and performance penalties), and basically need to be deleted ASAP once you don't need it anymore.

 

[esi_y]

the problem is that for the backup to be consistent, you must stop pmxcfs. and obviously, since everything relies on that running, you don't want that during regular operations. I guess we could add it to pmxcfs itself as well, to allow triggering a consistent dump with it running. or do it as part of rebooting a node, but then the schedule is much more variable.

Reading my mind. I could add this to Bugzilla as enhancement request, but that would be my first (enhacement one). Idk this is your call, e.g. if you were planning something full-service as part of your PBS. I only know for now it's actually good to know the config.db is there (in the backup) and it could be dumped. (Just knowing this is valuable, I thought - from how this thread started.) I would definitely +1 myself on having an (official ) dump tool.

yeah, that table is meant to state whether a storage supports snapshots in the context of PVE, where snapshots are potentially kept indefinitely. LVM snapshots don't work there, they are meant as temporary consistency measure while pulling a backup (and have a lot of constraints and performance penalties), and basically need to be deleted ASAP once you don't need it anymore.

I know, but for folks running backup of their hypervisor as the OP, it would have been nice to have even just a little extra space left there (who knows how to do it will use it and ditch it right after backup, but arguable those can also manage custom partitioning install altogether). It doesn't take too long to back up those ~2GB, I know some people do that before update/upgrade and would have been better with LVM snapshot for the moment.

Thanks for the answers! Appreciate it (maybe more than the OP, but he set it as solved, maybe someone nosy finds this helpful later too.)

 

[fabian]

I know, but for folks running backup of their hypervisor as the OP, it would have been nice to have even just a little extra space left there (who knows how to do it will use it and ditch it right after backup, but arguable those can also manage custom partitioning install altogether). It doesn't take too long to back up those ~2GB, I know some people do that before update/upgrade and would have been better with LVM snapshot for the moment.

you can just set minfree in the installer if you want that

https://pve.proxmox.com/pve-docs/chapter-pve-installation.html#advanced_lvm_options

 

[esi_y]

you can just set minfree in the installer if you want that

https://pve.proxmox.com/pve-docs/chapter-pve-installation.html#advanced_lvm_options

My bad , I am the kind of person who reads the code before the docs. Thanks I didn't know it was already thought of!