backup to NFS failing input/output error

[veehexx]

i've recently changed my NFS export from inside a VM with passthru disks, to the proxmox host.
Figurging it was a simple case of remount disks, copy /etc/exports over, and re-point the Datacenter Storages.
in the specific example below, the 'Backups-NFS' is essentially localhost (pve1; .201 IP), however the same occurs from another cluster member (pve4; .205 IP)
pve1 hosts the NFS export.

# cat /etc/fstab
...
UUID=4126724a-18dd-426e-a6a1-e5fe6a574730 /mnt/btrfs-ssd1 auto noauto,degraded,rw,ssd_spread,noatime,compress,discard=async,space_cache=v2 0 0

# cat /etc/exports
/mnt/btrfs-ssd1/data/backup/pve 192.168.1.201(rw,sync,no_subtree_check,no_root_squash) 192.168.1.205(rw,sync,no_subtree_check,no_root_squash)

# ls -lah /mnt/btrfs-ssd1/data/backup/pve
total 4.0K
drwxrwxrwx 1 root root   24 Jul 22 15:24 .
drwxrwxr-- 1 REMOVED  users 138 Feb 25 08:48 ..
drwxrwxrwx 1 root root  59K Jul 23 10:50 dump
drwxrwxrwx 1 root root    6 Jul 22 15:24 template

# vzdump 118 --storage Backups-NFS
INFO: starting new backup job: vzdump 118 --storage Backups-NFS
INFO: Starting Backup of VM 118 (qemu)
INFO: Backup started at 2023-07-23 10:49:20
INFO: status = running
INFO: VM Name: Print
INFO: include disk 'scsi0' 'local-storage:118/vm-118-disk-0.raw' 64G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: snapshots found (not included into backup)
INFO: creating vzdump archive '/mnt/pve/Backups-NFS/dump/vzdump-qemu-118-2023_07_23-10_49_20.vma'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'df07eb22-96e7-4709-bde9-75eb4e3984e8'
INFO: resuming VM again
INFO:   2% (1.7 GiB of 64.0 GiB) in 3s, read: 569.8 MiB/s, write: 294.9 MiB/s
INFO:   4% (2.7 GiB of 64.0 GiB) in 6s, read: 342.1 MiB/s, write: 263.2 MiB/s
INFO:   5% (3.6 GiB of 64.0 GiB) in 9s, read: 323.0 MiB/s, write: 245.2 MiB/s
INFO:   6% (3.9 GiB of 64.0 GiB) in 13s, read: 59.8 MiB/s, write: 34.9 MiB/s
INFO:   7% (4.9 GiB of 64.0 GiB) in 16s, read: 342.0 MiB/s, write: 34.3 MiB/s
INFO:   8% (5.2 GiB of 64.0 GiB) in 22s, read: 53.2 MiB/s, write: 42.8 MiB/s
INFO:  11% (7.2 GiB of 64.0 GiB) in 34s, read: 177.7 MiB/s, write: 49.3 MiB/s
INFO:  14% (9.2 GiB of 64.0 GiB) in 37s, read: 657.1 MiB/s, write: 223.0 MiB/s
INFO:  15% (10.0 GiB of 64.0 GiB) in 40s, read: 293.6 MiB/s, write: 276.4 MiB/s
INFO:  16% (10.5 GiB of 64.0 GiB) in 44s, read: 130.5 MiB/s, write: 130.5 MiB/s
INFO:  22% (14.4 GiB of 64.0 GiB) in 47s, read: 1.3 GiB/s, write: 265.8 MiB/s
INFO:  39% (25.0 GiB of 64.0 GiB) in 50s, read: 3.5 GiB/s, write: 0 B/s
INFO:  53% (33.9 GiB of 64.0 GiB) in 53s, read: 3.0 GiB/s, write: 0 B/s
INFO:  68% (43.5 GiB of 64.0 GiB) in 56s, read: 3.2 GiB/s, write: 0 B/s
INFO:  81% (52.5 GiB of 64.0 GiB) in 59s, read: 3.0 GiB/s, write: 0 B/s
INFO:  92% (59.2 GiB of 64.0 GiB) in 1m 2s, read: 2.2 GiB/s, write: 0 B/s
INFO:  99% (63.6 GiB of 64.0 GiB) in 1m 5s, read: 1.4 GiB/s, write: 0 B/s
INFO: 100% (64.0 GiB of 64.0 GiB) in 1m 8s, read: 152.1 MiB/s, write: 2.7 KiB/s
Warning: unable to close filehandle GEN82 properly: Input/output error at /usr/share/perl5/PVE/VZDump/QemuServer.pm line 823.
ERROR: vma_writer_close: close failed - Input/output error
INFO: aborting backup job
INFO: resuming VM again
ERROR: Backup of VM 118 failed - vma_writer_close: close failed - Input/output error
INFO: Failed at 2023-07-23 10:50:28
INFO: Backup job finished with errors
job errors

# mount
...
pve1:/mnt/btrfs-ssd1/data/backup/pve on /mnt/pve/Backups-NFS type nfs (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.1.201,mountvers=3,mountport=44000,mountproto=udp,local_lock=none,addr=192.168.1.201)

apart from the disks from VM passthru to local and the nfs export move, i cant think of anything else i've changed that could cause this. Last weeks backups ran fine.

 

[veehexx]

i've found something that looks like it might be the problem.
the btrfs mount isnt mounting at boot, but fine if i manually mount after login. that then messes up the NFS side so appears to be something along the lines of NFS stuck with the boot OS mount, when the manual btrfs mount has mounted ontop of that path.

i've manually fixed the mount/service order and i've sucessfully ran a backup.