Help Setting Up Push Sync From PBS to Remote PBS: Confusing Documentation re: Token API Permissions, Not Sure What to Do

[SInisterPisces]

I'm trying to set up a push sync from my in-LAN PBS to a remote Tuxis PBS.

I've gotten stuck on the Permissions setup for my API keys, here: https://pbs.proxmox.com/docs/managing-remotes.html#sync-direction-push

The following permissions are required for a sync job in push direction:

1. Remote.Audit on /remote/{remote} and Remote.DatastoreBackup on /remote/{remote}/{remote-store}/{remote-ns} path or subnamespace.
2. At least Datastore.Read and Datastore.Audit on the local source datastore namespace (/datastore/{store}/{ns}) or Datastore.Backup if owner of the sync job.
3. Remote.DatastorePrune on /remote/{remote}/{remote-store}/{remote-ns} path to remove vanished snapshots and groups. Make sure to use a dedicated remote for each sync job in push direction as noted above.
4. Remote.DatastoreModify on /remote/{remote}/{remote-store}/{remote-ns} path to remove vanished namespaces.

I've created a user on my local PBS with an API token to use for this. But I'm having a bit of trouble figuring out what Role to give that user. The documentation is not clear about this.

• Remote.Audit and Remote.DatastoreBackup are the only permissions in the RemoteSyncPushOperator role.
• Remote.DatastorePrune: If I want to add this to my local user that runs the sync job, RemoteSyncPushOperator doesn't have it. I'd need to escalate to RemoteDatastorePowerUser, right? Then what's the point of RemoteSyncPushOperator? A user that can just send backups but not clean them up?
• If I need all of these things ("permissions are requried"), then why not just tell me to set RemoteDatastorePowerUser?
• "Remote.DatastoreModify on /remote/{remote}/{remote-store}/{remote-ns} path to remove vanished namespaces." What is a "vanished namespace?" When do I need to be able to do this? It requires the RemoteDatastoreAdmin privilege. At that point, I don't feel like I'm actually restricting any permissions at all.
• For the local side, Datastore.Reader grants Datastore.Read and Datastore.Audit as per item (2); but
• For the local side, "Datastore.Backup if owner of the sync job." Who is the owner of the sync job? I have User A on PBS that PVE logs into to do the backup, and then User B that's going to handle the remote sync, since their permissions are different. It's my server, that I am the only admin of, so I'm the "owner," but I'm not sure that's what it means.

A remote user with limited access should be used on the remote backup server instance. Consider the implications…

Remote Config:
The manual doesn't actual explain how to configure the remote PBS's user permissions…

I'm using @tuxis 's paid 1 TB plan, so I only have one user, and they've got the DatastoreAdmin role already.

Assumed Correct Answer: I only have one namespace on the local PBS (Root/Encrypted), and one namespace on the remote (Root). I don't contemplate ever having a vanished namespace, or needing to remove one.

Is this the correct config?

• Locally, I need to set RemoteDatastorePowerUser, because I assume I need to be able to prune the Remote but never plan to delete any namespaces.
• On the Remote, I need to set DatastorePowerUser, because the user that's doing all the work needs those permissions (this is what I'd set if I were just backing up directly to the Tuxis remote instance).

Help? This has all become more complicated than I think it should be.

 

[fabian]

I'm trying to set up a push sync from my in-LAN PBS to a remote Tuxis PBS.

I've gotten stuck on the Permissions setup for my API keys, here: https://pbs.proxmox.com/docs/managing-remotes.html#sync-direction-push



I've created a user on my local PBS with an API token to use for this. But I'm having a bit of trouble figuring out what Role to give that user. The documentation is not clear about this.

• Remote.Audit and Remote.DatastoreBackup are the only permissions in the RemoteSyncPushOperator role.
• Remote.DatastorePrune: If I want to add this to my local user that runs the sync job, RemoteSyncPushOperator doesn't have it. I'd need to escalate to RemoteDatastorePowerUser, right? Then what's the point of RemoteSyncPushOperator? A user that can just send backups but not clean them up?

yes, this is a common use case (append-only syncing)

• If I need all of these things ("permissions are requried"), then why not just tell me to set RemoteDatastorePowerUser?

because you might want to define your own custom roles, and/or you might not want to give that level of privilege (see above)

• "Remote.DatastoreModify on /remote/{remote}/{remote-store}/{remote-ns} path to remove vanished namespaces." What is a "vanished namespace?" When do I need to be able to do this? It requires the RemoteDatastoreAdmin privilege. At that point, I don't feel like I'm actually restricting any permissions at all.

if you do a recursive sync, you can either pre-create matching namespaces on both sides, and don't give the sync users/tokens enough privileges to manage them, or you can let the sync handle it, but then the involved users/tokens need enough privileges.

a vanished namespace is one that exists on the target, but doesn't exist on the source (like a vanished group or snapshot).

• For the local side, Datastore.Reader grants Datastore.Read and Datastore.Audit as per item (2); but
• For the local side, "Datastore.Backup if owner of the sync job." Who is the owner of the sync job? I have User A on PBS that PVE logs into to do the backup, and then User B that's going to handle the remote sync, since their permissions are different. It's my server, that I am the only admin of, so I'm the "owner," but I'm not sure that's what it means.

to read a backup, you either need Datastore.Read and Datastore.Audit (which allows reading the contents of and listing all groups), or you need Datastore.Backup and be the owner of the backup group. that is indeed phrased weirdly, what it means is that if you are the "owner" of the sync job (that is an option inside the sync.cfg), then you only need Datastore.Backup (and the sync job is limited to backups owned by your user in that case, unless you have higher privileges).

Remote Config:
The manual doesn't actual explain how to configure the remote PBS's user permissions…

I'm using @tuxis 's paid 1 TB plan, so I only have one user, and they've got the DatastoreAdmin role already.

Assumed Correct Answer: I only have one namespace on the local PBS (Root/Encrypted), and one namespace on the remote (Root). I don't contemplate ever having a vanished namespace, or needing to remove one.

Is this the correct config?

• Locally, I need to set RemoteDatastorePowerUser, because I assume I need to be able to prune the Remote but never plan to delete any namespaces.

yes, if you want the sync job to handle pruning via remove-vanished. if you want the remote PBS to handle pruning independent from the source, then you don't need PowerUser, as described above.

• On the Remote, I need to set DatastorePowerUser, because the user that's doing all the work needs those permissions (this is what I'd set if I were just backing up directly to the Tuxis remote instance).

yes, if you want the sync job to handle pruning. if not, then the user/token on the tuxis side just needs to be able to do backups.

Help? This has all become more complicated than I think it should be.

it is. maybe a few examples for common scenarios would help?

 

[SInisterPisces]

it is. maybe a few examples for common scenarios would help?

Thanks, @fabian . I hope I didn't sound too aggravated in my original post; I'd been trying to figure out how to make it work for a while before I posted, and felt like it was something I should have been able to figure out.

I'm going to need to read over that a few times to make sure I understand. I really appreciate you typing it out.

Examples for common scenarios would be awesome. That would be really helpful.

Related to that, maybe a bit more of an explanation of the benefits of using Remotes/Push Sync from Local PBS to Remote PBS, versus having two separate backup jobs (one to Local PBS, one to Remote PBS) on the PVE cluster, instead. My understanding is that it frees up the PVE cluster to spend less time doing backups (and thus, less time suspending/snapshotting running VMs).

 

[fabian]

I really understand that this is complicated and sympathize with the pain that entails, don't worry!

the advantage of sync vs. two backup targets are many:
- less exposure for the offsite PBS (it only needs to talk to the primary PBS)
- less load on the PVE side
- faster backups for VMs because bitmaps don't get invalidated (although that will be fixed by multi bitmap support at some point)
- better recovery (chunk gets corrupt on primary? sync it back from offsite)

 

[SInisterPisces]

I really understand that this is complicated and sympathize with the pain that entails, don't worry!

the advantage of sync vs. two backup targets are many:
- less exposure for the offsite PBS (it only needs to talk to the primary PBS)
- less load on the PVE side
- faster backups for VMs because bitmaps don't get invalidated (although that will be fixed by multi bitmap support at some point)
- better recovery (chunk gets corrupt on primary? sync it back from offsite)

Thanks for the kind words, @fabian . I was feeling a bit overwhelmed with all the half-functioning tech bits my life right now.

As an indicator of how behind the curve I am, I didn't realize you could assign individual permissions without assigning roles.
I prefer using the roles, honestly. It helps keep me aware of why and why certain privileges go together. I'm new to ACLs and privileges for users/groups, so it's been a useful way to learn and internalize some concepts that are useful elsewhere, as well.

That said, I should figure out how to work with individual permissions.

Broadly, this is what I was hoping to do:

1. Set up an encrypted backup to my local PBS, with verification, prune, and GC jobs.
2. Use a remote sync to send those backups, to my cloud PBS instance, still encrypted.
3. have the cloud PBS handle verification, pruning, and GC of hte data there itself, so I have the flexibility to have the local and remote schedules for those things diverge.
4. Have some way of monitoring the status of syncs to the remote PBS without having to go look at the remote PBS. (I assume the local PVE would warn me if the sync job failed.)
   1. Yes, the remote PBS can send me notifications; but
   2. I really like having the big red warning in my log at the bottom of the screen when a backup goes wrong.

If I could get this set up, I'd still probably end up adding the cloud PBS as a storage in the PVE cluster. Not to interact directly with it, but so I could click on it and check its summary page to see free space, whether it's online, etc. This part is easy, as I think it's completely separate from the sync.

 

[fabian]

yes, mostly correct. ACLs work with roles, but roles are made up of privileges - in PVE you can define your own, we haven't added that to PBS yet because the ACLs are a lot simpler there (still).

a failing sync job should trigger a notification on the the PBS driving it (source for push, target for pull). you will likely also see failed reader/writer tasks in such a case. adding the remote PBS as a storage on PVE can make sense, in particular if you have different pruning schedules - you might want to restore directly from offsite, for example. you might want to make this access token read only though