backup job process only for 3 VM did not work issuing guest-agent 'fs-freeze and issuing guest-agent 'fs-thaw'

[sysbitnet]

I tested on a local lab environment and got a little problem, which is a little weird.

For testing, i have one Proxmox Server DC1 and one Proxmox Backup Server pbs-00

Inside the Proxmox Backup Server, i create a new user dc1@pbs who communicates with Proxmox and Proxmox backup Server in Proxmox Datacenter => Storage add like new type Proxmox Backup Server and add all information needed from Proxmox Backup Server who can link. And that part was completed successfully.

When i want to manually trigger any other VM how can create and store the backup on pbs-00 that works perfectly, but Inside Proxmox i manually trigger to create a backup that has been stored on pbs-00 but i get an error only on this tree VM machine.

INFO: starting new backup job: vzdump 200 --storage pbs-00 --notification-mode auto --mode snapshot --notes-template '{{guestname}}' --node dc1 --remove 0
INFO: Starting Backup of VM 200 (qemu)
INFO: Backup started at 2024-07-29 23:10:18
INFO: status = running
INFO: VM Name: Ubuntu
INFO: include disk 'sata0' 'local:200/vm-200-disk-0.raw' 10G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/200/2024-07-29T21:10:18Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 200 qmp command 'backup' failed - backup connect failed: command error: backup owner check failed (dc1@pbs != root@pam)
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 200 failed - VM 200 qmp command 'backup' failed - backup connect failed: command error: backup owner check failed (dc1@pbs != root@pam)
INFO: Failed at 2024-07-29 23:10:19
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

INFO: starting new backup job: vzdump 300 --notes-template '{{guestname}}' --node dc1 --remove 0 --storage pbs-00 --mode snapshot --notification-mode auto
INFO: Starting Backup of VM 300 (qemu)
INFO: Backup started at 2024-07-29 23:10:02
INFO: status = running
INFO: VM Name: Debian
INFO: include disk 'scsi0' 'local:300/vm-300-disk-0.raw' 10G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/300/2024-07-29T21:10:02Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 300 qmp command 'backup' failed - backup connect failed: command error: backup owner check failed (dc1@pbs != root@pam)
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 300 failed - VM 300 qmp command 'backup' failed - backup connect failed: command error: backup owner check failed (dc1@pbs != root@pam)
INFO: Failed at 2024-07-29 23:10:02
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

INFO: starting new backup job: vzdump 400 --mode snapshot --notification-mode auto --storage pbs-00 --remove 0 --notes-template '{{guestname}}' --node dc1
INFO: Starting Backup of VM 400 (qemu)
INFO: Backup started at 2024-07-29 23:09:44
INFO: status = running
INFO: VM Name: Fedora
INFO: include disk 'scsi0' 'local:400/vm-400-disk-0.raw' 10G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/400/2024-07-29T21:09:44Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 400 qmp command 'backup' failed - backup connect failed: command error: backup owner check failed (dc1@pbs != root@pam)
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 400 failed - VM 400 qmp command 'backup' failed - backup connect failed: command error: backup owner check failed (dc1@pbs != root@pam)
INFO: Failed at 2024-07-29 23:09:45
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

When i go inside the Proxmox Backup Server and on Server Administration in Syslog see this

Jul 29 23:10:19 pbs-00 proxmox-backup-proxy[22514]: GET //api2/json/backup?backup-id=200&backup-time=1722287418&backup-type=vm&benchmark=false&debug=false&store=Storage: 400 Bad Request: [client [::ffff:91.192.103.5]:41912] backup owner check failed (dc1@pbs != root@pam)

Jul 29 23:10:02 pbs-00 proxmox-backup-proxy[22514]: GET //api2/json/backup?backup-id=300&backup-time=1722287402&backup-type=vm&benchmark=false&debug=false&store=Storage: 400 Bad Request: [client [::ffff:91.192.103.5]:40706] backup owner check failed (dc1@pbs != root@pam)

Jul 29 23:09:45 pbs-00 proxmox-backup-proxy[22514]: GET //api2/json/backup?backup-id=400&backup-time=1722287384&backup-type=vm&benchmark=false&debug=false&store=Storage: 400 Bad Request: [client [::ffff:91.192.103.5]:52694] backup owner check failed (dc1@pbs != root@pam)

But when inside Proxmox on Datacenter => Storage add type Proxmox Backup Server when add user root@pam with fallow information and trigger to backup this VM all works perfectly.

I need to say the user dc1@pbs has Permissions Patch / like root and Rule Admin

If someone can help that would be nice, how can be resolve this strange case problem?

Thank you,

 

[sysbitnet]

In this moment i solved this problem as I changed it configuration.


On Proxmox Backup Server i create a Namespace name dc1 and on Proxmox remove the current and create a new config that looks like.



When i trigger again back up again, it's successfully working.

Then in the console copy the current backup content to the namespace you can see it inside Proxmox Backup Server and Proxmox

root@pbs-00:/mnt/datastore/Storage# cp -r vm /mnt/datastore/Storage/ns/dc1/
root@pbs-00:/mnt/datastore/Storage# cp -r ct /mnt/datastore/Storage/ns/dc1/

Then i be sure it working fine, I remove a copy of the backup that has been storage on /mnt/datastore/Storage/vm and /mnt/datastore/Storage/ct

 

[sysbitnet]

But the weird problem with the backup machine VM Name: Ubuntu, VM Name: Debian, VM Name: Fedora i have not figured out what is the problem, because when i create a new VM from the clone, and triggers to backup work perfectly. Or case when the clone moves the machine to the second Proxmox and from that trigger backup again works normally.

 

[fiona]

Hi,
a backup group in Proxmox Backup Server has an owner. It gets automatically set when the first backup to the group is added. If another user tries to add a backup to that group later, it will fail. You can change the owner e.g. via the Proxmox Backup Server Web UI under Actions with the "person silhouette" icon.

 

[sysbitnet]

Hi Fiona,

I think something like this

a backup group in Proxmox Backup Server has an owner. It gets automatically set when the first backup to the group is added. If another user tries to add a backup to that group later, it will fail.

I try to find where is this part

You can change the owner e.g. via the Proxmox Backup Server Web UI under Actions with the "person silhouette" icon.

But can not find it, so can I politely you to take a screenshot and share it with the community?

 

[fiona]

But can not find it, so can I politely you to take a screenshot and share it with the community?

When you go to Datastore > [datastore] > Contents, there is an Actions column.

 

[sysbitnet]

Hi fiona,

Now i understand what you mean, and let me share the screenshot for future who read this.

When you go to Datastore > [datastore] > Contents, there is an Actions column.

When you click on the user icon which is marked green, you get the little window to Change Owner.

This can fix the problem of what is at the very beginning of this post.


But what in the case when did not have a backup from the VM machine VM Name: Ubuntu, VM Name: Debian, VM Name: Fedora like from the beginning of this post?

Thank you

 

[sysbitnet]

Now i get one more error case

Inside Proxmox

INFO: starting new backup job: vzdump 100 --mode snapshot --notification-mode auto --storage pbs-00 --remove 0 --notes-template '{{guestname}}' --node dc1
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2024-07-31 01:09:25
INFO: status = running
INFO: VM Name: AlmaLinux
INFO: include disk 'scsi0' 'local:100/vm-100-disk-0.raw' 150G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/100/2024-07-30T23:09:25Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
ERROR: VM 100 qmp command 'backup' failed - backup connect failed: command error: Permission denied (os error 13)
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 100 failed - VM 100 qmp command 'backup' failed - backup connect failed: command error: Permission denied (os error 13)
INFO: Failed at 2024-07-31 01:09:26
INFO: Backup job finished with errors
INFO: notified via target `mail-to-root`
TASK ERROR: job errors

Inside Proxmox Backup Server

Jul 31 01:00:06 pbs-00 proxmox-backup-proxy[22514]: GET //api2/json/backup?backup-id=100&backup-time=1722380405&backup-type=vm&benchmark=false&debug=false&ns=dc1&store=Storage: 400 Bad Request: [client [::ffff:192.168.1.10]:45294] Permission denied (os error 13)

What is working early now is not working,

Now from ProxMox to Proxmox Backup Server, the backup process is working only for VM Name: Ubuntu, VM Name: Debian, and VM Name: Fedora, but for all others did not work.

 

[fiona]

But what in the case when did not have a backup from the VM machine VM Name: Ubuntu, VM Name: Debian, VM Name: Fedora like from the beginning of this post?

For a new backup group, the owner will be the user creating the first backup in the group.

Inside Proxmox Backup Server

Jul 31 01:00:06 pbs-00 proxmox-backup-proxy[22514]: GET //api2/json/backup?backup-id=100&backup-time=1722380405&backup-type=vm&benchmark=false&debug=false&ns=dc1&store=Storage: 400 Bad Request: [client [::ffff:192.168.1.10]:45294] Permission denied (os error 13)

What permissions are configured for your user on the Proxmox Backup Server side?

What is the output of the following?

ls -l /mnt/datastore/Storage/ns/
ls -l /mnt/datastore/Storage/ns/dc1
ls -l /mnt/datastore/Storage/ns/dc1/vm

 

[sysbitnet]

What is the output of the following?

ls -l /mnt/datastore/Storage/ns/
ls -l /mnt/datastore/Storage/ns/dc1
ls -l /mnt/datastore/Storage/ns/dc1/vm

My result looks like

root@pbs-00:~# ls -l /mnt/datastore/Storage/ns/
total 4
drwxr-xr-x 4 backup backup 4096 Jul 30 10:55 dc1

root@pbs-00:~# ls -l /mnt/datastore/Storage/ns/dc1
total 8
drwxr-xr-x  3 root   root   4096 Jul 30 10:55 ct
drwxr-xr-x 13 backup backup 4096 Jul 30 17:36 vm

root@pbs-00:~# ls -l /mnt/datastore/Storage/ns/dc1/vm
total 44
drwxr-xr-x 5 root   root   4096 Jul 30 10:54 100
drwxr-xr-x 5 root   root   4096 Jul 30 10:54 120
drwxr-xr-x 4 backup backup 4096 Jul 31 01:30 200
drwxr-xr-x 5 root   root   4096 Jul 30 10:54 201
drwxr-xr-x 4 backup backup 4096 Jul 31 01:30 300
drwxr-xr-x 5 root   root   4096 Jul 30 10:54 301
drwxr-xr-x 5 root   root   4096 Jul 30 10:54 302
drwxr-xr-x 4 backup backup 4096 Jul 31 01:31 400
drwxr-xr-x 5 root   root   4096 Jul 30 10:54 401
drwxr-xr-x 2 root   root   4096 Jul 30 10:54 402
drwxr-xr-x 5 root   root   4096 Jul 30 10:54 500

With command chown -R backup:backup /mnt/datastore/Storage/ns/dc1/* i changed it right and now the test lab working fine.

Thank you

 

[sysbitnet]

Question out of curiosity, when i check right on dir like

root@pbs-00:/mnt/datastore/Storage/ns/dc1/ct/101# ls -l
total 20
drwxr-xr-x 2 backup backup 4096 Jul 30 10:55 2024-07-27T23:04:46Z
drwxr-xr-x 2 backup backup 4096 Jul 30 10:55 2024-07-28T23:00:51Z
drwxr-xr-x 2 backup backup 4096 Jul 30 10:55 2024-07-29T23:00:55Z
drwxr-xr-x 2 backup backup 4096 Jul 31 18:09 2024-07-31T16:09:27Z
-rw-r--r-- 1 backup backup   10 Jul 30 10:55 owner

root@pbs-00:/mnt/datastore/Storage/ns/dc1/vm/200# ls -l
total 12
drwxr-xr-x 2 backup backup 4096 Jul 30 17:47 2024-07-30T15:34:27Z
drwxr-xr-x 2 backup backup 4096 Jul 31 03:00 2024-07-30T23:30:05Z
-rw-r--r-- 1 backup backup   10 Jul 30 17:34 owner

On any type and in any dir have file owner, and when i open this i see user dc1@pbs, can you explain more what is the point of this file?

Thank you one more time.

 

[fiona]

With command chown -R backup:backup /mnt/datastore/Storage/ns/dc1/* i changed it right and now the test lab working fine.

Yes, they need to belong to the backup system user. Glad that it worked

root@pbs-00:/mnt/datastore/Storage/ns/dc1/vm/200# ls -l
total 12
drwxr-xr-x 2 backup backup 4096 Jul 30 17:47 2024-07-30T15:34:27Z
drwxr-xr-x 2 backup backup 4096 Jul 31 03:00 2024-07-30T23:30:05Z
-rw-r--r-- 1 backup backup   10 Jul 30 17:34 owner

On any type and in any dir have file owner, and when i open this i see user dc1@pbs, can you explain more what is the point of this file?

The owner file contains the name of the user the backup group belongs to. From Proxmox Backup Server's perspective there are name@pbs users, name@pam users, etc. and the backup server has its own permission management and needs to remember which backup groups belong to who. Creating/modifying the files is done as the backup system user on the file system no matter which user triggered the operation.