Remote sync not starting

[Andrei_W4TL]

Hello everyone.

I'm trying to sync my local datastore to a "remote" NAS, which is also added as a datastore BUT as a nfs-share. (don't pull out the pitchforks please)

I have already a sync up and running to another NAS which works alright.

/mnt/bk-nas-rotate from 192.168.207.199:/mnt/array1/BACKUP
 Flags: rw,relatime,vers=3,rsize=524288,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.207.199,mountvers=3,mountport=2049,mountproto=udp,local_lock=none,addr=192.168.207.199


/mnt/bk-nas-01 from 192.168.200.185:/BACKUP
 Flags: rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.200.143,local_lock=none,addr=192.168.200.185

The /mnt/bk-nas-rotate is not really doing anything when the sync job starts. It is not even possible to stop the sync job after starting it.

This is the sync jobs output:
2024-10-15T22:40:00+02:00: Starting datastore sync job '-:pbs-local:BK-NAS-ROTATE:PMX-HOST-01:s-a0e12d1d-9ed4'
2024-10-15T22:40:00+02:00: task triggered by schedule 'hourly'
2024-10-15T22:40:00+02:00: sync datastore 'BK-NAS-ROTATE' from 'pbs-local'
2024-10-15T22:47:52+02:00: received abort request ...

Writing to the mount is possible and speedwise okay-ish:

root@PMX-BACKUP-01:~# dd if=/dev/zero of=/mnt/bk-nas-rotate/test bs=1M count=100 status=progress
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 2.8445 s, 36.9 MB/s

to compare it with the other NFS share (which is nfs 4.1):

root@PMX-BACKUP-01:~# dd if=/dev/zero of=/mnt/bk-nas-01/test bs=1M count=100 status=progress
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 1.60522 s, 65.3 MB/s

Here is my /etc/fstab:

root@PMX-BACKUP-01:~# cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/pbs/root / ext4 errors=remount-ro 0 1
/dev/pbs/swap none swap sw 0 0
proc /proc proc defaults 0 0


# NAS 1
192.168.200.185:/BACKUP /mnt/bk-nas-01 nfs nfsvers=4,minorversion=1,rw 0 0


# NAS 2 - COLD STORAGE/WEEKLY
192.168.207.199:/mnt/array1/BACKUP  /mnt/bk-nas-rotate nfs rw,vers=3  0 0

Any ideas where to look? When doing a df -h even after hours nothing has been written to the NAS so I guess the sync does not even start?

First initializing the NAS (writing of .chunks) took almost 24hrs..

Thanks in advance.

Greetings

 

[Chris]

Hi,
while your setup is not really up for the task, the sync job should nevertheless start to pull contents from the source to the target, so something seems off. Please post the job and datastore config cat /etc/proxmox-backup/{datastore,sync}.cfg and check the systemd journal for output related to the sync job.

 

[Andrei_W4TL]

Hi,
while your setup is not really up for the task, the sync job should nevertheless start to pull contents from the source to the target, so something seems off. Please post the job and datastore config cat /etc/proxmox-backup/{datastore,sync}.cfg and check the systemd journal for output related to the sync job.

Hi Chris,

root@PMX-BACKUP-01:/var/log/proxmox-backup/tasks# cat /etc/proxmox-backup/{datastore,sync}.cfg
datastore: pbs-local
        gc-schedule sat 18:15
        path /mnt/datastore/pbs-local


datastore: BK-NAS-01
        comment
        gc-schedule daily
        notification-mode notification-system
        path /mnt/bk-nas-01/


datastore: BK-NAS-ROTATE
        comment
        gc-schedule daily
        notification-mode notification-system
        path /mnt/bk-nas-rotate/
sync: s-ab906c7d-1e76
        ns
        owner root@pam
        remote-ns
        remote-store pbs-local
        remove-vanished false
        schedule 21:00
        store BK-NAS-01


sync: s-a0e12d1d-9ed4
        ns PMX-HOST-01
        owner root@pam
        remote-ns PMX-HOST-01
        remote-store pbs-local
        remove-vanished false
        store BK-NAS-ROTATE

Also from journalctl I couldn't get any useful output?
Here is the job itself:

root@PMX-BACKUP-01:/var/log/proxmox-backup/tasks# proxmox-backup-manager task log "UPID:PMX-BACKUP-01:00000408:000003B5:00000000:670ED320:syncjob:\x2d\x3apbs\x2dlocal\x3aBK\x2dNAS\x2dROTATE\x3aPMX\x2dHOST\x2d01\x3as\x2da0e12d1d\x2d9ed4:root@pam:"
Starting datastore sync job '-:pbs-local:BK-NAS-ROTATE:PMX-HOST-01:s-a0e12d1d-9ed4'
task triggered by schedule 'hourly'
sync datastore 'BK-NAS-ROTATE' from 'pbs-local'
received abort request ...

 

[Chris]

Please do post an excerpt of the systemd journal covering the time range of the sync job, this could give at least some information on what steps get executed. You can generate it via journalctl --since <DATETIME> --until <DATETIME> > journal.txt and upload it here as attachment.

 

[Andrei_W4TL]

Please do post an excerpt of the systemd journal covering the time range of the sync job, this could give at least some information on what steps get executed. You can generate it via journalctl --since <DATETIME> --until <DATETIME> > journal.txt and upload it here as attachment.

Thanks for taking your time Chris, you'll have a new customer after this I'm pretty sure. Now I see why support might be needed.

Attached is the log, I just restarted the server and manually started the sync job for this. I don't see anything related the job though.

 

[Chris]

Thanks for the journal, the first line might already indicate a problem:
kernel: lockd: server 192.168.207.199 not responding, still trying

So it seems the kernel cannot get a lock on the NFS share. So you might want to start debug there, e.g. by following https://access.redhat.com/solutions/28211