PBS backing up to NFS--backing up the NFS VM

[crashtest]

So I have configured my PBS datastore to use an NFS mount. It took a while but, after disabling ALL NFS caching functions from NFS client on the PBS to get around ESTSALE errors, I was finally able to get back ups working. I have successful VM backups on both of my nodes.

However, my dilemma is that the NFS server is itself a proxmox VM that I would like to backup. Everytime it attempts the backup of this VM though, I receive

INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 205 qmp command 'backup' failed - backup connect failed: command error: http upgrade request timed out
INFO: aborting backup job
NFO: resuming VM again

My guess is that PBS is trying to access the NFS share while the server is paused for snapshot? I'm not really clear on where PBS would be asking for an HTTP upgrade other than the PVE node though and other VMs on that node backup without error. In any event, is there a way around this?

ETA: The NFS server VM is a PCI passthrough of the drive controllers so it's stuck on the node it's on.

 

[ggoller]

Why would you want to backup the PBS datastore? This won't add any additional protection. I recommend adding another PBS instance and use a sync job (https://pbs.proxmox.com/docs/managing-remotes.html#sync-jobs).

Some context: when starting the backup the client will make a HTTP upgrade, then send the backup chunks on the upgraded connection.

 

[crashtest]

It's not the PBS datastore I'm backing up. It's the VM that runs the NFS server that the datastore connects to that I'm trying to backup. However, I am kind of back to square one as the ESTALE errors when trying to open the finish blob came back.

 

[ggoller]

Hi!
well yes, that is the same thing. I want to emphasize this again: this is kind of pointless, you don't add any additional protection to the data and the deduplication won't even work because the chunks look different. This means you have a recursive datastore which is just a waste of storage space.
If you have any other important data not related to PBS on the NFS server I strongly recommend to move them another instance.

Technically this should work as we do a vzdump backup, then we chunk and upload it (never tested it though).

 

[Dunuin]

And I don't see the point in general unless you also sync your backup snapshots to some proper second PBS.
Otherwise you are basically screwed once you are hit by ransomware, human error, failing hardware, lightning, fire, theft, ... as you will then lose all your guests as well as your backups.
A failing PSU for example could fry all your disks of that server at the sane time.
There are reasons why ita recommended to run PBS on a dedicated server or even better multiple with one offsite.

 

[crashtest]

Hi!
well yes, that is the same thing. I want to emphasize this again: this is kind of pointless, you don't add any additional protection to the data and the deduplication won't even work because the chunks look different. This means you have a recursive datastore which is just a waste of storage space.
If you have any other important data not related to PBS on the NFS server I strongly recommend to move them another instance.

Technically this should work as we do a vzdump backup, then we chunk and upload it (never tested it though).

Are you saying PBS will try to backup every mount in the VM and not JUST the VM snapshot? Because, that's an entire different problem if that's the case. All I want is the VM and the virtual disks with that VM backed up, all physical non-virtual mounts in the VM are backed up separately from proxmox.

 

[crashtest]

And I don't see the point in general unless you also sync your backup snapshots to some proper second PBS.
Otherwise you are basically screwed once you are hit by ransomware, human error, failing hardware, lightning, fire, theft, ... as you will then lose all your guests as well as your backups.
A failing PSU for example could fry all your disks of that server at the sane time.
There are reasons why ita recommended to run PBS on a dedicated server or even better multiple with one offsite.

At the moment I'm just trying to get a second copy of my VMs. One that is on separate media from the SSD ZFS pool that they run off of. I am NOT trying to backup the data that the NFS serves.

As far as backups go, I have to work with what I have, If I could afford a full backup implementation, I would already have one. As it is, I'm stuck triaging the data that I have.

 

[crashtest]

Why would you want to backup the PBS datastore? This won't add any additional protection. I recommend adding another PBS instance and use a sync job (https://pbs.proxmox.com/docs/managing-remotes.html#sync-jobs).

Some context: when starting the backup the client will make a HTTP upgrade, then send the backup chunks on the upgraded connection.

So does that mean that the timeout is happening on waiting for a response from PBS and not something PBS is waiting on?

 

[Dunuin]

As far as backups go, I have to work with what I have, If I could afford a full backup implementation, I would already have one. As it is, I'm stuck triaging the data that I have.

I ran PBS even on low hardware like 2 cores and 3GB of RAM. Any cheap 10+ years old office PC you get for free at the dump would do the job if you don't care about performance. Most demanding thing is the storage but sounds like you already got the disks. And if you care about power consumption, put that dedicated PBS on a cheap timer-clock-power-plug. Or even cheaper and totally free, use a VZDump hook script to wake up that PBS via wake-on-lan when the backup job starts and send it to sleep via API when finished.

 

[crashtest]

I ran PBS even on low hardware like 2 cores and 3GB of RAM. Any cheap 10+ years old office PC you get for free at the dump would do the job if you don't care about performance. Most demanding thing is the storage but sounds like you already got the disks. And if you care about power consumption, put that dedicated PBS on a cheap timer-clock-power-plug. Or even cheaper and totally free, use a VZDump hook script to wake up that PBS via wake-on-lan when the backup job starts and send it to sleep via API when finished.

It's more a matter of storage than processing. I use rasp pis for all kinds of things, even my quorum vote is a pi, my one PBS(so far) runs in a tiny LXC container. All my HDD bays are full and in use. What I have is a large array of rust and a bunch of CPU cores running VMs and Containers off SSDs. All I'm trying to accomplish here is to get backups of the things on the SSDs onto the rust. This is complete homelab stuff. If PBS isn't the best solution for that then I am completely open to suggestions, I just thought the integration of PVE and PBS made sense.

PS I have also tried skipping PBS and just setting PVE to backup to the NAS by adding the NFS as a storage in the PVE GUI... I haven't been able to solve the ESTALE issue with that approach though.

 

[Dunuin]

PBS(so far) runs in a tiny LXC container

Again, not a thing you should do. Running PBS in a LXC is not a supported configuration and the devs told us to better not do this. While it still works right now (by accident ), no one cares while developing it to not break things and this also isn't covered by internal testing so any PBS upgrade could break your disaster recovery capabilities.

 

[crashtest]

Again, not a thing you should do. Running PBS in a LXC is not a supported configuration and the devs told us to better not do this. While it still works right now (by accident ), no one cares while developing it to not break things and this also isn't covered by internal testing so any PBS upgrade could break your disaster recovery capabilities.

So the allure in doing this is I am limited to one array with the storage capacity that has bit rot and parity protection. Even if I were to offload PBS to it's own device, I would still be using that array as a datastore to be "first line of defense". Breaking changes don't really bother me because uptime isn't that critical of a requirement, I have the ability to sit on a version for a bit to wait out workarounds, or dedicate resource collection to completely reconfigure. As it is, it really doesn't take long for me to recreate these VMs from scratch, nearly all of my data exists outside the VM system. However, doing so is quite the headache, so I like to back up my "infrastructure" as much as I am able. In the end, all of this is just tinkering on a home lab. Would I like to do everything by best practices, absolutely. I learn a lot about best practices trying to implement as much of them as I can, but I don't have the resources(money) to do so fully thus "doing the best I can with what I got".

Given the issues I am having, in the future, I'd probably dedicate one of my Pis to PBS and kluge a PBS install on it. If I can't get the array to work it will end up being a single cheap USB drive that holds the backups of all my VMs.