Some backups are not saved in the namespace

[ednt]

Hi,

I just tried to store the backups in a namespace.
Most VMs and CTs are now stored inside the namespace, but some machines are stored in root.
They are all in the same backup job.


PVE Storage:





PVE Backup job:




PBS datastore:




All other machines are inside of the namespace.

All of the wrong stored backups are located on 1 PVE server of the cluster.

How is this possible?

Or better: where is the bug und how to solve it?


PBS: 2.2-3 latest stuff from no-subscription
PVE: 7.2-7 latest stuff from no-subscription

Best regards.

 

[fabian]

is your PVE-side a cluster? maybe not all nodes have been updated correctly and some are ignoring/don't yet understand the namespace setting in storage.cfg?

edit: another potential source would be a forgotten sync job?

 

[ednt]

I already edited the first post.
It is a cluster and all wrong stored machines are running on one PVE.
But this PVE is also on the latest software release.
'Upgrade' shows no available packages and when I connect to the GUI on this PVE I get also 7.2-7 as version.

What do you mean with forgotten sync job?
The job was started at the correct time the last 4 days. (I switched to namespace 5 days ago)

The /etc/pve/storage.conf is up to date and contains the pbs with namespace.

 

[fabian]

I mean that maybe there is a sync job between two PBS that could cause groups to appear in the root namespace - but if you don't have such a job set up, then it can't be the cause

can you post pveversion -v from the "faulty" node?

 

[ednt]

proxmox-ve: 7.2-1 (running kernel: 5.11.22-5-pve)
pve-manager: 7.2-7 (running version: 7.2-7/d0dd0e85)
pve-kernel-5.15: 7.2-6
pve-kernel-helper: 7.2-6
pve-kernel-5.13: 7.1-9
pve-kernel-5.15.39-1-pve: 5.15.39-1
pve-kernel-5.15.35-3-pve: 5.15.35-6
pve-kernel-5.15.27-1-pve: 5.15.27-1
pve-kernel-5.15.19-2-pve: 5.15.19-3
pve-kernel-5.13.19-6-pve: 5.13.19-15
pve-kernel-5.11.22-5-pve: 5.11.22-10
ceph: 16.2.9-pve1
ceph-fuse: 16.2.9-pve1
corosync: 3.1.5-pve2
criu: 3.15-1+pve-1
glusterfs-client: 9.2-1
ifupdown: residual config
ifupdown2: 3.1.0-1+pmx3
ksm-control-daemon: 1.4-1
libjs-extjs: 7.0.0-1
libknet1: 1.24-pve1
libproxmox-acme-perl: 1.4.2
libproxmox-backup-qemu0: 1.3.1-1
libpve-access-control: 7.2-3
libpve-apiclient-perl: 3.2-1
libpve-common-perl: 7.2-2
libpve-guest-common-perl: 4.1-2
libpve-http-server-perl: 4.1-3
libpve-storage-perl: 7.2-5
libqb0: 1.0.5-1
libspice-server1: 0.14.3-2.1
lvm2: 2.03.11-2.1
lxc-pve: 5.0.0-3
lxcfs: 4.0.12-pve1
novnc-pve: 1.3.0-3
openvswitch-switch: 2.15.0+ds1-2+deb11u1
proxmox-backup-client: 2.2.3-1
proxmox-backup-file-restore: 2.2.3-1
proxmox-mini-journalreader: 1.3-1
proxmox-widget-toolkit: 3.5.1
pve-cluster: 7.2-1
pve-container: 4.2-1
pve-docs: 7.2-2
pve-edk2-firmware: 3.20210831-2
pve-firewall: 4.2-5
pve-firmware: 3.4-2
pve-ha-manager: 3.3-4
pve-i18n: 2.7-2
pve-qemu-kvm: 6.2.0-11
pve-xtermjs: 4.16.0-1
qemu-server: 7.2-3
smartmontools: 7.2-pve3
spiceterm: 3.2-2
swtpm: 0.7.1~bpo11+1
vncterm: 1.7-1
zfsutils-linux: 2.1.4-pve1

Was not able to reboot it up to now, so the kernel is not the latest.
But that's also on the other PVE servers in the cluster.

Only the ceph servers are on the latest kernel, since they are dedicated to ceph and 'rebootable' thanks to ceph.

 

[fabian]

could you post the 'proxmox-support' part of 'qm status XXX --verbose' for one of the affected VMs on the faulty node?

if you do a new backup, does the snapshot get added to the root namespace or the correct one (on the faulty node)? could you post the backup task log?

 

[ednt]

ct/225 is a container.

From vm/236:

proxmox-support:
        pbs-dirty-bitmap: 1
        pbs-dirty-bitmap-migration: 1
        pbs-dirty-bitmap-savevm: 1
        pbs-library-version: 1.2.0 (6e555bc73a7dcfb4d0b47355b958afd101ad27b5)
        pbs-masterkey: 1
        query-bitmap-info: 1
qmpstatus: running

If I went to the backups of the VM 236 I see no backup., but iot is available in root datastore.
If I try to start a backup it results in:

INFO: starting new backup job: vzdump 236 --storage pm-pbs01-ednt --node pm-cpu-175 --remove 0 --notes-template '{{guestname}}' --mode snapshot
INFO: Starting Backup of VM 236 (qemu)
INFO: Backup started at 2022-07-18 14:09:35
INFO: status = running
INFO: VM Name: zldap-01
INFO: include disk 'sata0' 'ssd:vm-236-disk-0' 20G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/236/2022-07-18T12:09:35Z'
ERROR: VM 236 qmp command 'backup' failed - Parameter 'backup-ns' is unexpected
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 236 failed - VM 236 qmp command 'backup' failed - Parameter 'backup-ns' is unexpected
INFO: Failed at 2022-07-18 14:09:36
INFO: Backup job finished with errors
TASK ERROR: job errors

which is strange, since I have backups of the last 4 days in root.

 

[fabian]

yeah, so that VM (and possibly others) are running a pre-NS-support PBS library - as indicated in the release notes, you need to stop/start or live-migrate these VMs to pick up the updated library with namespace support..

 

[ednt]

I did now an online migration with the VMs.
They report now 1.31 as pbs-library.

But what's with the CTs ?
Do they also need a 'restart' ?

 

[fabian]

no, they use the client directly. but the fact that the backups of the VMs only now started to fail (and worked but with the root namespace in the days before) is a pretty clear indication that you were missing an upgrade on that node until very recently..

 

[ednt]

Hi,

partly success.
But ...

still some VMs and all CTs are not stored in the namespace.



All with count 5 are still wrong.

Here is the status of vm/114

proxmox-support:
        pbs-dirty-bitmap: 1
        pbs-dirty-bitmap-migration: 1
        pbs-dirty-bitmap-savevm: 1
        pbs-library-version: 1.3.1 (4d450bb294cac5316d2f23bf087c4b02c0543d79)
        pbs-masterkey: 1
        query-bitmap-info: 1
qmpstatus: running
running-machine: pc-i440fx-6.1+pve0
running-qemu: 6.2.0
shares: 1000
status: running
uptime: 61414
vmid: 114

It uses now pbs-library 1.3.1

And the PVE node has definately all updates.

Any ideas?

Ups... all the VMs which worked now are not on these node.
So it is more surprising, because I only migrated VMs on this node. I don't know why these VMs are now stored in the namespace.

I just installed the new updates with:

libpve-storage-perl 7.2-7
proxmox-backup-client 2.2.4-1

maybe ...

 

[fabian]

something is missing from your description.. what exactly did you do with the VMs that previously were wrong and are now right? migrate them from the faulty node to a working node? or migrate them away and back?

you can try the following as well on the faulty node: systemctl reload-or-restart pveproxy pvedaemon pvescheduler

 

[ednt]

More ups ...

Some VMs of this node were backuped twice this night.

For example vm/114:

stored in root at 2022-07-18 23:00:12 (count 5)
stored in ednt at 2022-07-19 00:09:13 (count 1)

But as you can see in the first post, there is only one backup job where this VM is included.

 

[ednt]

something is missing from your description.. what exactly did you do with the VMs that previously were wrong and are now right? migrate them from the faulty node to a working node? or migrate them away and back?

That is the great mystery, I only migrated the VMs from the one node away and back.
I did nothing with VMs on other nodes.

 

[ednt]

In the backup log of this node I see only the 'right' backup from 2022-07-19 00:09:13:

INFO: trying to get global lock - waiting...
INFO: got global lock
INFO: starting new backup job: vzdump 118 246 140 141 199 128 145 181 1241 185 186 100 107 225 227 231 228 233 238 234 236 237 201 240 254 275 123 170 130 180 148 232 230 229 142 143 1113 149 152 162 164 171 129 139 158 244 252 108 138 174 183 182 188 190 189 222 241 255 161 196 178 251 114 113 --quiet 1 --mailnotification always --mailto edv@ednt.de --mode snapshot --storage pm-pbs01-ednt
INFO: skip external VMs: 100, 107, 108, 113, 118, 123, 128, 129, 130, 138, 139, 140, 141, 142, 143, 145, 148, 149, 152, 158, 161, 162, 164, 170, 171, 174, 178, 180, 181, 182, 183, 185, 186, 188, 189, 190, 196, 199, 201, 222, 229, 230, 231, 232, 241, 244, 246, 251, 252, 254, 255, 1113, 1241
INFO: Starting Backup of VM 114 (qemu)
INFO: Backup started at 2022-07-19 00:09:13
INFO: status = running
INFO: VM Name: EDNT-SRV-TS10
INFO: include disk 'virtio0' 'raid10:114/vm-114-disk-0.qcow2' 250G
INFO: include disk 'virtio1' 'raid10:114/vm-114-disk-1.qcow2' 150G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/114/2022-07-18T22:09:13Z'
INFO: started backup task 'ae74b312-0d56-4fc1-91c7-b1b4fc11b211'
INFO: resuming VM again

 

[fabian]

is there another cluster accessing the same PBS without namespace configured (but with overlapping guest IDs)? please double check that there is no sync job configured on the PBS side either..

 

[ednt]

No, on this PBS are only stored VMs and CTs from one cluster.
But I want 2 namespaces since the VMs and CTs are owned by 2 different customers.
So there is no chance to have double IDs.
This PBS is synced to a second PBS, but as slave. The other PBS syncs from this one.

 

[ednt]

Hi,

still the same problem, but ...
I saw a new error message in the backup log:

file /etc/pve/storage.cfg line 158 (section 'pm-pbs01-ednt') - unable to parse value of 'namespace': unexpected property 'namespace'

But I swear, the node has no missing updates.
So I try now the restart of the services:

systemctl reload-or-restart pveproxy pvedaemon pvescheduler

From which service or program is the message from above?

 

[fabian]

that depends on whether the backup was run manually on the CLI (in which case, code is directly read from the on disk perl module) or manually via the API/GUI (in which case it's likely pvedaemon) or scheduled (pvescheduler).

 

[ednt]

The backups are running via the backup-job, because I'm not in the company at 11pm.

I just checked PBSPlugin.pm on this node.
in sub options {
is definately

namespace => { optional => 1 },

inside.