[SOLVED] Error: not a vaild user id - cannot backup CTs

[Proxomx]

Dear Community,

Another day and I created another problem to myself

I have Proxmox Backup Server which is running as a VM on my node. I want to backup three of my current LXC containers to PBS (there is mounted SSD which is used only by server). Sadly only 1 of 3 containers is able to finish backing up.
The problem with backup arises only when I choose to backup to my PBS - if I choose local storage on PVE as a folder for backups everything is backing up without issues and later I am able to restore it (I checked already).
PVE is up to date (7.1.8) as is PBS (2.1).
I operated as root@pam on PVE and PBS - not working. I changed to be user patryk on PBS and PVE (with admin permissions) - error still occurs.
I am also mapped from PVE host to LXC containers as user patryk 1000:1000 - container 102 (unprivileged) backs up without issues, then container 103 (also unprivileged) won't back up.
The most disturbing to me is that one container backs up without problem - if it was all of them I think would be more logical and easy to troubleshoot

Below LXC 102 which backs up correctly ->

INFO: starting new backup job: vzdump 102 --remove 0 --node alexpiesel --storage pbs --mode snapshot
INFO: Starting Backup of VM 102 (lxc)
INFO: Backup started at 2022-01-15 20:00:58
INFO: status = running
INFO: CT Name: samba
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/mnt/data/share') from backup (not a volume)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/102/2022-01-15T19:00:58Z'
INFO: run: lxc-usernsexec -m u:0:100000:1000 -m u:1000:1000:1 -m u:1001:101001:64535 -m g:0:100000:1000 -m g:1000:1000:1 -m g:1001:101001:64535 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp24624_102/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 102 --backup-time 1642273258 --repository root@pam@192.168.1.164:backup1
INFO: Starting backup: ct/102/2022-01-15T19:00:58Z
INFO: Client name: alexpiesel
INFO: Starting backup protocol: Sat Jan 15 20:00:58 2022
INFO: Downloading previous manifest (Sat Jan 15 19:43:49 2022)
INFO: Upload config file '/var/tmp/vzdumptmp24624_102/etc/vzdump/pct.conf' to 'root@pam@192.168.1.164:8007:backup1' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@192.168.1.164:8007:backup1' as root.pxar.didx
INFO: root.pxar: had to backup 42.852 MiB of 1.046 GiB (compressed 3.727 MiB) in 2.43s
INFO: root.pxar: average backup speed: 17.623 MiB/s
INFO: root.pxar: backup was done incrementally, reused 1.004 GiB (96.0%)
INFO: Uploaded backup catalog (608.883 KiB)
INFO: Duration: 2.49s
INFO: End Time: Sat Jan 15 20:01:00 2022
INFO: cleanup temporary 'vzdump' snapshot
INFO: Finished Backup of VM 102 (00:00:02)
INFO: Backup finished at 2022-01-15 20:01:00
INFO: Backup job finished successfully
TASK OK

LXC 102 config below ->

arch: amd64
cores: 1
features: nesting=1
hostname: samba
memory: 1024
mp0: /mnt/data,mp=/mnt/data/share
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=0E:0E:6C:26:FA:F8,ip=192.168.1.163/24,ip6=dhcp,type=veth
onboot: 1
ostype: ubuntu
rootfs: encrypted_zfs:subvol-102-disk-1,size=8G
swap: 1024
unprivileged: 1
lxc.idmap: u 0 100000 1000
lxc.idmap: u 1000 1000 1
lxc.idmap: u 1001 101001 64535
lxc.idmap: g 0 100000 1000
lxc.idmap: g 1000 1000 1
lxc.idmap: g 1001 101001 64535

Then I have LXC 103 which sadly cannot back up ->

INFO: starting new backup job: vzdump 103 --storage pbs --mode snapshot --node alexpiesel --remove 0
INFO: Starting Backup of VM 103 (lxc)
INFO: Backup started at 2022-01-15 20:00:11

INFO: status = running
INFO: CT Name: wireguard
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/103/2022-01-15T19:00:11Z'
INFO: run: lxc-usernsexec -m u:0:100000:1000 -m u:1000:1000:1 -m u:1001:101001:64535 -m g:0:100000:1000 -m g:1000:1000:1 -m g:1001:101001:64535 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp23722_103/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 103 --backup-time 1642273211 --repository root@pam@192.168.1.164:backup1
INFO: Starting backup: ct/103/2022-01-15T19:00:11Z
INFO: Client name: alexpiesel
INFO: Starting backup protocol: Sat Jan 15 20:00:13 2022
INFO: Error: not a valid user id
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 103 failed - command 'lxc-usernsexec -m u:0:100000:1000 -m u:1000:1000:1 -m u:1001:101001:64535 -m g:0:100000:1000 -m g:1000:1000:1 -m g:1001:101001:64535 -- /usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp23722_103/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 103 --backup-time 1642273211 --repository root@pam@192.168.1.164:backup1' failed: exit code 255
INFO: Failed at 2022-01-15 20:00:13
INFO: Backup job finished with errors
TASK ERROR: job errors

Below contents of 103.conf

arch: amd64
cores: 1
features: nesting=1
hostname: wireguard
memory: 2048
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=9A:D2:10:F4:DA:89,ip=192.168.1.160/24,ip6=dhcp,type=veth
onboot: 1
ostype: ubuntu
rootfs: encrypted_zfs:subvol-103-disk-0,size=16G
swap: 2048
unprivileged: 1
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir
lxc.idmap: u 0 100000 1000
lxc.idmap: u 1000 1000 1
lxc.idmap: u 1001 101001 64535
lxc.idmap: g 0 100000 1000
lxc.idmap: g 1000 1000 1
lxc.idmap: g 1001 101001 64535

And last LXC 104 which also I am not able to back up (this is privileged LXC, plex with gpu passthrough) ->

INFO: starting new backup job: vzdump 104 --remove 0 --node alexpiesel --mode snapshot --storage pbs
INFO: Starting Backup of VM 104 (lxc)
INFO: Backup started at 2022-01-15 20:09:17
INFO: status = running
INFO: CT Name: plex
INFO: including mount point rootfs ('/') in backup
INFO: excluding bind mount point mp0 ('/mnt/data/share') from backup (not a volume)
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
INFO: creating Proxmox Backup Server archive 'ct/104/2022-01-15T19:09:17Z'
INFO: run: /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp30863_104/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 104 --backup-time 1642273757 --repository root@pam@192.168.1.164:backup1
INFO: Starting backup: ct/104/2022-01-15T19:09:17Z
INFO: Client name: alexpiesel
INFO: Starting backup protocol: Sat Jan 15 20:09:18 2022
INFO: Error: not a valid user id
INFO: cleanup temporary 'vzdump' snapshot
ERROR: Backup of VM 104 failed - command '/usr/bin/proxmox-backup-client backup '--crypt-mode=none' pct.conf:/var/tmp/vzdumptmp30863_104/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found '--exclude=/tmp/?*' '--exclude=/var/tmp/?*' '--exclude=/var/run/?*.pid' --backup-type ct --backup-id 104 --backup-time 1642273757 --repository root@pam@192.168.1.164:backup1' failed: exit code 255
INFO: Failed at 2022-01-15 20:09:18
INFO: Backup job finished with errors
TASK ERROR: job errors

Config 104 below ->

arch: amd64
cores: 3
features: mount=cifs
hostname: plex
memory: 4096
mp0: /mnt/data,mp=/mnt/data/share,ro=1
net0: name=eth0,bridge=vmbr0,firewall=1,gw=192.168.1.1,hwaddr=BA:D8:25:4C:20:69,ip=192.168.1.161/24,ip6=dhcp,type=veth
onboot: 1
ostype: ubuntu
rootfs: encrypted_zfs:subvol-104-disk-0,size=16G
swap: 4096
lxc.cgroup2.devices.allow: c 226:0 rwm
lxc.cgroup2.devices.allow: c 226:128 rwm
lxc.cgroup2.devices.allow: c 4:7 rwm
lxc.cgroup2.devices.allow: c 29:0 rwm
lxc.autodev: 1
lxc.hook.autodev: /var/lib/lxc/104/mount_hook.sh

I tried to stop LXC and then back them up - still got the error. I restarted server - still got the error.
The PBS storage (called also pbs) was added in PVE GUI, is visible in PVE (otherwise I wouldnt be able to back up this one LXC at all). I think it got to something with user permissions but I am out of ideas here - perhaps someone would be so kind (looking at you @Dunuin ) and throw some ideas how to resolve this problem?

Best wishes!
Patryk

 

[Dunuin]

Can't see anything obvious. Are there any errors in the syslog at the time of the backup?

 

[Proxomx]

@Dunuin I checked and nothing more there, only the same error which is created during backup I installed proxmox-backup-client in one of LXC which couldnt be backed up and I was able to send backup to PBS. Currently only automatic option for backup from PVE GUI is not possible to process... I will look more into this today, hopefully I will stumble upon solution somehwere

EDIT: I deleted VM with PBS, wiped SSD, created once again new VM with PBS, added again wiped storage (basically everything I did at first place) and now everything is working ok... I mark this one as solved but it is really strange

@Dunuin Thank you for trying to help me again!! I think we will "see" each other soon again haha