[SOLVED] backups to pbs works, but no Contents on pbs visible

[uldise]

Hi,
i'm trying to set up my first pbs, and have a very strange issue - when i issue manual backup from pve, it's reports task ok, but i can't find any backups on pbs side and on pve as well.. pbs Datastore Content section is just empty.
my pbs is on version 2.2-3, pve is on version 6.3-6. maybe my pve is too old to work with this new pbs?
if i look in datastore filelevel - i see backup contents..

see bellow task log example:

INFO: starting new backup job: vzdump 407 --mode snapshot --remove 0 --node proxmoxServer --storage pbs
INFO: Starting Backup of VM 407 (lxc)
INFO: Backup started at 2022-06-26 16:37:00
INFO: status = running
INFO: CT Name: <reducted>
INFO: including mount point rootfs ('/') in backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: create storage snapshot 'vzdump'
  Logical volume "snap_vm-407-disk-1_vzdump" created.
INFO: creating Proxmox Backup Server archive 'ct/407/2022-06-26T13:37:00Z'
INFO: run: lxc-usernsexec -m u:0:100000:65536 -m g:0:100000:65536 -- /usr/bin/proxmox-backup-client backup --crypt-mode=none pct.conf:/var/tmp/vzdumptmp27371_407/etc/vzdump/pct.conf root.pxar:/mnt/vzsnap0 --include-dev /mnt/vzsnap0/./ --skip-lost-and-found --exclude=/tmp/?* --exclude=/var/tmp/?* --exclude=/var/run/?*.pid --backup-type ct --backup-id 407 --backup-time 1656250620 --repository root@pam@192.168.50.242:pbs
INFO: Starting backup: ct/407/2022-06-26T13:37:00Z
INFO: Client name: proxmoxServer
INFO: Starting backup protocol: Sun Jun 26 16:37:00 2022
INFO: No previous manifest available.
INFO: Upload config file '/var/tmp/vzdumptmp27371_407/etc/vzdump/pct.conf' to 'root@pam@192.168.50.242:8007:pbs' as pct.conf.blob
INFO: Upload directory '/mnt/vzsnap0' to 'root@pam@192.168.50.242:8007:pbs' as root.pxar.didx
INFO: root.pxar: had to upload 3.32 GiB of 3.57 GiB in 29.19s, average speed 116.57 MiB/s).
INFO: root.pxar: backup was done incrementally, reused 254.51 MiB (7.0%)
INFO: Uploaded backup catalog (2.20 MiB)
INFO: Duration: 29.30s
INFO: End Time: Sun Jun 26 16:37:29 2022
INFO: cleanup temporary 'vzdump' snapshot
  Logical volume "snap_vm-407-disk-1_vzdump" successfully removed
INFO: Finished Backup of VM 407 (00:00:30)
INFO: Backup finished at 2022-06-26 16:37:30
Result: {
  "data": null
}
INFO: Backup job finished successfully
TASK OK

 

[dcsapak]

can you post the output of

proxmox-backup-client snapshot list

(include the correct --repository etc. parameters ofc)

?

is there any journal entry on the pbs side while you're trying to list the datastore?

 

[uldise]

proxmox-backup-client snapshot list

i tried this on both - pbs or pve. and it returns nothing.

on pve i executed this command:

proxmox-backup-client snapshot list --repository 192.168.50.242:pbs

after entering password and accepting fingerprint twice, it returns nothing.

and no new lines on syslog on pbs side.

 

[dcsapak]

mhmm ok thats weird. can you post the complete journal from the pbs since the last boot? (journalctl -b)

 

[uldise]

since the last boot

i just did a fresh reboot, issued snapshot list commands too.
see attached.

 

[dcsapak]

ok can you post your datastore config ? (/etc/proxmox-backup/datastore.cfg) and a directory listing from in there:

ls -lha /path/to/datastore

?

 

[uldise]

datastore config

/etc/proxmox-backup/datastore
datastore: pbs
    comment.
    gc-schedule daily
    path /mnt/pbs
    prune-schedule daily

root@pbs:~# ls -lha /mnt/pbs
total 4.0K
drwxrwxrwx 1 backup backup   28 Jun 26 16:37 .
drwxr-xr-x 3 root   root   4.0K Jun 26 12:42 ..
drwxr-x--- 1 backup backup 512K Jun 26 16:34 .chunks
-rw-r--r-- 1 backup backup    0 Jun 26 16:33 .lock
drwxr-xr-x 1     99 users     6 Jun 26 16:37 ct
root@pbs:~#

 

[dcsapak]

drwxr-xr-x 1 99 users 6 Jun 26 16:37 ct

it does not belong to the correct user,
the owner and group must be 'backup'

 

[uldise]

group must be 'backup'

ok, but pbs is creating these folders..
it's is NFS share, and on NFS server all belogs to 'users'..
so, a simple workaround would be change owners? of the /mnt/pbs folder?

 

[dcsapak]

i guess you must configure your nfs mount in such a way that the 'backup' user really acts as the 'backup' user...

 

[uldise]

you must configure your nfs mount

i just did that, created datastore from scratch in empty folder, issued manual backup, and still no go..

root@pbs:~# ls -lha /mnt/pbs
total 4.0K
drwxrwxrwx 1 backup backup   28 Jun 27 15:31 .
drwxr-xr-x 3 root   root   4.0K Jun 26 12:42 ..
drwxr-x--- 1 backup backup 512K Jun 27 15:31 .chunks
-rw-r--r-- 1 backup backup    0 Jun 27 15:30 .lock
drwxr-xr-x 1 backup backup    6 Jun 27 15:31 ct

 

[dcsapak]

can you also show the file listings below ct?

e.g. with

ls -lhaR /mnt/pbs/ct

 

[uldise]

below ct

here you go:

root@pbs:~# ls -lhaR /mnt/pbs/ct
/mnt/pbs/ct:
total 0
drwxr-xr-x 1 backup backup  6 Jun 27 15:31 .
drwxrwxrwx 1 backup backup 28 Jun 27 15:31 ..
drwxr-xr-x 1 backup backup 50 Jun 27 15:31 407

/mnt/pbs/ct/407:
total 4.0K
drwxr-xr-x 1 backup backup  50 Jun 27 15:31 .
drwxr-xr-x 1 backup backup   6 Jun 27 15:31 ..
drwxr-xr-x 1 backup backup 150 Jun 27 15:34 2022-06-27T12:31:47Z
-rw-r--r-- 1 backup backup   9 Jun 27 15:31 owner

'/mnt/pbs/ct/407/2022-06-27T12:31:47Z':
total 60K
drwxr-xr-x 1 backup backup  150 Jun 27 15:34 .
drwxr-xr-x 1 backup backup   50 Jun 27 15:31 ..
-rw-r--r-- 1 backup backup 4.2K Jun 27 15:34 catalog.pcat1.didx
-rw-r--r-- 1 backup backup  843 Jun 27 15:34 client.log.blob
-rw-r--r-- 1 backup backup  448 Jun 27 15:34 index.json.blob
-rw-r--r-- 1 backup backup  242 Jun 27 15:31 pct.conf.blob
-rw-r--r-- 1 backup backup  38K Jun 27 15:34 root.pxar.didx
root@pbs:~#

 

[dcsapak]

ok, looks normal.

the behaviour is definitely not normal and something strange is going on..
you could install strace and try to trace the proxmox-backup-proxy during listing e.g. with

strace -fyy -p <pid-of-proxmox-backup-proxy>

this will output a lot of stuff, so you maybe want to redirect it to a file

strace -fyy -p <pid-of-proxmox-backup-proxy> 2>&1 > /tmp/strace.log

and stop the strace with a ctrl+c after trying to list the backups

 

[uldise]

i strated strace, and hit Reload button on Contents tab. see output attached.

 

[dcsapak]

thanks,

the strace output does not really make sense though...

828   openat(AT_FDCWD, "/mnt/pbs/", O_RDONLY) = 22</mnt/pbs>                 
828   fstat(22</mnt/pbs>, {st_mode=S_IFDIR|0777, st_size=28, ...}) = 0       
828   fcntl(22</mnt/pbs>, F_GETFL)      = 0x8000 (flags O_RDONLY|O_LARGEFILE)
828   fcntl(22</mnt/pbs>, F_SETFD, FD_CLOEXEC) = 0                           
828   getdents64(22</mnt/pbs>, 0x7facd40476c0 /* 5 entries */, 32768) = 136  
828   getdents64(22</mnt/pbs>, 0x7facd40476c0 /* 0 entries */, 32768) = 0

here the proxy tries to list the dirs in /mnt/pbs abd should try to open the subdirs

e.g. here is an excerpt from a working listing

fcntl(18</temporary>, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)   
fcntl(18</temporary>, F_SETFD, FD_CLOEXEC) = 0                         
getdents64(18</temporary>, 0x7f2fc8033d40 /* 8 entries */, 32768) = 216
openat(18</temporary>, "ct", O_RDONLY) = 27</temporary/ct>             
fstat(27</temporary/ct>, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

you can see that it tries to open /temporary/ct directly after getdents

the only thing that comes to mind is that the nfs server does something weird?

can you post the output of the 'mount' command on pbs ?

 

[uldise]

output of the 'mount' command on pbs

from fstab file:

192.168.50.247:/mnt/user/pbs /mnt/pbs nfs local_lock=all,soft,intr 0 0

 

[dcsapak]

no i meant the real output from the command

mount

 

[uldise]

sorry, here you go:

root@pbs:~# mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=1976288k,nr_inodes=494072,mode=755,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=402000k,mode=755,inode64)
/dev/mapper/pbs-root on / type ext4 (rw,relatime,errors=remount-ro)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14820)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
192.168.50.247:/mnt/user/pbs on /mnt/pbs type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.50.247,mountvers=3,mountport=53796,mountproto=udp,local_lock=all,addr=192.168.50.247)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=401996k,nr_inodes=100499,mode=700,inode64)

 

[dcsapak]

output looks normal so far...

i currently don't have any idea what could trigger that...
could you maybe make another datastore on a local filesystem and test a backup there? (just to rule out the nfs mount)