[SOLVED] error listing snapshots - 400 Bad Request (500)

[DC-CA1]

Hi we Added a Namespace under a existing Datastore and cannot Reach it trough any of our Nodes.

the Namespace is working well and PBS is retrieving a Daily Sync from another .
i see all the Content VMs in PBS viewer tab

We first disconnected ( removed from the cluster the storage that was pointing to the root and reconnected with the corresponding namespace.
no luck
we then reconnected using a different ID
no luck
we upgraded a 7.211 node to 7.3.4 and rebooted it
no luck

Proxmox cluster 7.2.11 to 7.3.4 sames issues

Proxmox PBS 2.3.1 .

we dont have any issues adding the Datastore trough the Cluster storage and specifing the Datastore and Namespace.
we can than retrieve the Data usage from any Nodes but nothing else.

if we click on Backups we get a 400 Bad Request ( 500 )

 

[fweber]

Hi, could you verify that the namespace is spelled correctly in /etc/pve/storage.cfg on PVE? Can you see the snapshots using proxmox-backup-client, e.g. with proxmox-backup-client snapshot list --ns yournamespace?

 

[DC-CA1]

yest its correctly spelled in the file. i have another target on another PBS with the same name but a different user , and i can reach it .

maybe i cannot have 2 namespace with the same name but i dont think its that as i have this issue only with this namespace

 

[fweber]

Thanks for checking. Could you try running the proxmox-backup-client command above? This will probably also give an error, but it will print a more detailed error message, so we might see what's going on.
You'll need to do something like

export PBS_REPOSITORY=...
proxmox-backup-client login
proxmox-backup-client snapshot list --ns yournamespace

 

[fabian]

please verify that the permissions of the user/token you have configured in PVE are correct - the namespace is a separate entity in the ACL config, if your user/token only has access for the datastore itself without the propagate flag set, you won't be able to use the namespace!

you can verify this by running proxmox-backup-manager user permissions <username-from-pve-storage.cfg> on the PBS host - please post the result here.

the output of pvesm list <STORAGENAME> on the PVE host would also be interesting.

 

[DC-CA1]

Thanks for checking. Could you try running the proxmox-backup-client command above? This will probably also give an error, but it will print a more detailed error message, so we might see what's going on.
You'll need to do something like

export PBS_REPOSITORY=...
proxmox-backup-client login
proxmox-backup-client snapshot list --ns yournamespace

root@CL1-MTL1-BL2:~# proxmox-backup-client snapshot list --ns BACKUPS-1
Error: unable to get (default) repository

 

[DC-CA1]

please verify that the permissions of the user/token you have configured in PVE are correct - the namespace is a separate entity in the ACL config, if your user/token only has access for the datastore itself without the propagate flag set, you won't be able to use the namespace!

you can verify this by running proxmox-backup-manager user permissions <username-from-pve-storage.cfg> on the PBS host - please post the result here.

the output of pvesm list <STORAGENAME> on the PVE host would also be interesting.

root@CL1-MTL1-BL2:~# pvesm list CL1-MTL1-PBS-RP1
error listing snapshots - 400 Bad Request

 

[DC-CA1]

please verify that the permissions of the user/token you have configured in PVE are correct - the namespace is a separate entity in the ACL config, if your user/token only has access for the datastore itself without the propagate flag set, you won't be able to use the namespace!

you can verify this by running proxmox-backup-manager user permissions <username-from-pve-storage.cfg> on the PBS host - please post the result here.

the output of pvesm list <STORAGENAME> on the PVE host would also be interesting.

propagate is set to yes.
and another 2.3.1 PBS is sucessfuly syncing a DATASTORE from another PBS to this problematic one with the same user configured.
the only thing i see is that the bakcup files written on the original datastore are not using the same USER than the user we use on the SYNC and THE PVE Hosts.

the goal is to access our Replication from the secondary PBS in case of a hardware failure appen on the main PBS. this was working correctly prior that i start using NAMESPACE.
if i reconnect with the root Namespace i will have access to the existing DATA..
also i see al the daily VM adding up to our NAMESPACE from PBS , so i dont think its user related

 

[fabian]

then please check the logs (journal and proxmox-backup-proxy logs in /var/log/proxmox-backup/api) on the PBS side for any warnings or errors

 

[DC-CA1]

then please check the logs (journal and proxmox-backup-proxy logs in /var/log/proxmox-backup/api) on the PBS side for any warnings or errors

hi Fabian the error is mine. We had a NAMESPACE with the identical same name as the DATASTORE NAME from PBS server and had another NAMESPACE ( the one i use ) inside of it.

good to know i didnt tough we can have multiple NAMESPACE after the root

 

[fabian]

yes, nested namespaces are supported:

https://pbs.proxmox.com/docs/storage.html#backup-namespaces

the idea is to support most reasonable setups that want logical separation (for preventing name collisions, and easily separating ACLs) while sharing the underlying chunk store for deduplication. e.g., you could have have a top-level namespace for each location, with sub-namespaces for different departments/PVE clusters/environments/.., with the option of further sub-dividing that (e.g. per team, or ..)

 

[ViperD21]

Hello,

I am unsure if I should start a new thread or add to this one. I have tried the above items, and PVE has access to the PBS showing the space used and the space remaining. However, when I click on the backup tab, it shows "error listing snapshots - 400 Bad Request (500)". This all started when I was trying to set up user API tokens to prevent pruning in case of ransomware or viruses. I wish I didn't because this made a whole mess of things, and the backups are not working. I tried deleting all the API Tokens and permissions to go back to the way it was working initially. Now, I can no longer see the snapshots.

What should I do next to get this working?

I appreciate your help!

I am using root@pam for the user for both servers with the same password to see if that stops any permission errors, but still no luck. Also:
on PVE:
root@pve:~# pvesm list T320backup
storage 'T320backup' does not exist

on PBS:
root@pbs:~# proxmox-backup-client change-owner vm/101 root@pam
Error: unable to get (default) repository

I had gotten the Error: unable to get (default) repository when I did another command previously; I just can't remember what command. I was looking and trying to follow so many Forum posts to try and get this working again.

 

[fabian]

please post your storage.cfg entry for the "pbs" storage..

 

[ViperD21]

Hello Fabian,

Please let me know what else you require.

WORKING (I created a new namespace under Root):
pbs: pbs-T320
datastore T320backup
server 192.168.1.251
content backup
fingerprint 7e:52:53:0d:44:76:cf:b8:01:67:55:f0:b3:e0:1c:d1:c5:b1:a9:89:bd:c2:50:f1:1c:95:de:38:b9:f5:f8:91
namespace T320
nodes pve
prune-backups keep-all=1
username API@pbs!mYv2EYF8GM9Hcn7CgC6JMwkC

Still Not Working:
pbs: pbs
datastore T320backup
server 192.168.1.251
content backup
fingerprint 7e:52:53:0d:44:76:cf:b8:01:67:55:f0:b3:e0:1c:d1:c5:b1:a9:89:bd:c2:50:f1:1c:95:de:38:b9:f5:f8:91
namespace Root
prune-backups keep-all=1
username API@pbs!mYv2EYF8GM9Hcn7CgC6JMwkC

I figured it's some sort of permission issue. I am new when it comes to that. I was trying everything to get it working and probably did it incorrectly. See the attached permissions. I want to set it up so the PVE can back up to the PBS but can not delete or change anything on the PBS in case there is ransomware or any other type of compromise to the PVE. I added Admin permissions to API user because I was trying to get it working somehow and that was the only way I could get it showing under the new T320 Namespace.

What do you recommend for permissions for the scenario I am describing? Is that even possible?

Thank you!

 

[fabian]

remove the "namespace" line from "pbs" - the root namespace is always implicitly used if no explicit namespace is set.

 

[ViperD21]

Fabian,

I appreciate your help! That fixed the issue.

Any suggestions on my permissions? Is it correctly set up so the PVE can't remove backups from PBS?

Thanks again!

 

[fabian]

usually in such a case you'd only hand out "DatastoreBackup" which allows creating new snapbshots and restoring from them, if the backup group belongs to the accessing token. "DatastoreReader" gives you read access to *all backup groups*, even ones owned by other users. RemoteSyncOperator is for handling access to configured remotes, and makes no sense on a datastore or namespace.

 

[ViperD21]

Great, thanks again for all your help!