Scheduled backups are failing suddenly (but work when I do them manually)

[eleanor]

My automated backups fail due to a free space error, but the disks are far from full. When I invoke a manual backup to the same storage, it works fine. Here are the emailed logs from both situations. The backups are heading to a PBS instance running as an unRAID container which has an external USB 3.0 enclosure connected to it. If you have any ideas, I'd be thankful for some input! Thanks!

Excerpt from failing automatic backup log:

vzdump 104 107 100 101 112 113 201 106 115 116 120 103 102 108 118 --notes-template '{{guestname}}' --prune-backups 'keep-daily=4,keep-last=4,keep-monthly=2,keep-weekly=4,keep-yearly=2' --mailto sshank@gmail.com --compress gzip --fleecing 0 --mailnotification failure --quiet 1 --storage unraid --mode snapshot


100: 2025-01-25 03:00:05 INFO: Starting Backup of VM 100 (qemu)
100: 2025-01-25 03:00:05 INFO: status = stopped
100: 2025-01-25 03:00:05 INFO: backup mode: stop
100: 2025-01-25 03:00:05 INFO: ionice priority: 7
100: 2025-01-25 03:00:05 INFO: VM Name: ubuntu-server
100: 2025-01-25 03:00:05 INFO: include disk 'scsi0' 'local-btrfs:100/vm-100-disk-0.raw' 122G
100: 2025-01-25 03:00:05 INFO: creating Proxmox Backup Server archive 'vm/100/2025-01-25T10:00:05Z'
100: 2025-01-25 03:00:05 INFO: starting kvm to execute backup task
100: 2025-01-25 03:00:07 INFO: started backup task 'cd93ac72-ea99-498c-acdd-1e2b87f196c7'
100: 2025-01-25 03:00:07 INFO: scsi0: dirty-bitmap status: created new
100: 2025-01-25 03:00:10 INFO: 1% (2.2 GiB of 122.0 GiB) in 3s, read: 757.3 MiB/s, write: 0 B/s
100: 2025-01-25 03:00:13 INFO: 2% (2.9 GiB of 122.0 GiB) in 6s, read: 234.7 MiB/s, write: 0 B/s
100: 2025-01-25 03:00:18 INFO: 3% (3.7 GiB of 122.0 GiB) in 11s, read: 168.8 MiB/s, write: 0 B/s
100: 2025-01-25 03:00:24 INFO: 4% (5.1 GiB of 122.0 GiB) in 17s, read: 232.0 MiB/s, write: 0 B/s
100: 2025-01-25 03:00:27 INFO: 5% (6.2 GiB of 122.0 GiB) in 20s, read: 373.3 MiB/s, write: 0 B/s
100: 2025-01-25 03:00:31 INFO: 6% (7.7 GiB of 122.0 GiB) in 24s, read: 381.0 MiB/s, write: 0 B/s
[ snip ]
100: 2025-01-25 03:04:45 INFO: 81% (99.2 GiB of 122.0 GiB) in 4m 38s, read: 266.4 MiB/s, write: 0 B/s
100: 2025-01-25 03:04:48 INFO: 86% (105.1 GiB of 122.0 GiB) in 4m 41s, read: 2.0 GiB/s, write: 0 B/s
100: 2025-01-25 03:04:51 INFO: 91% (112.2 GiB of 122.0 GiB) in 4m 44s, read: 2.4 GiB/s, write: 0 B/s
100: 2025-01-25 03:04:54 INFO: 93% (114.7 GiB of 122.0 GiB) in 4m 47s, read: 834.7 MiB/s, write: 0 B/s
100: 2025-01-25 03:04:57 INFO: 96% (117.2 GiB of 122.0 GiB) in 4m 50s, read: 861.3 MiB/s, write: 0 B/s
100: 2025-01-25 03:05:00 INFO: 100% (122.0 GiB of 122.0 GiB) in 4m 53s, read: 1.6 GiB/s, write: 0 B/s
100: 2025-01-25 03:05:00 INFO: Waiting for server to finish backup validation...
100: 2025-01-25 03:19:17 INFO: backup is sparse: 35.38 GiB (28%) total zero data
100: 2025-01-25 03:19:17 INFO: backup was done incrementally, reused 122.00 GiB (100%)
100: 2025-01-25 03:19:17 INFO: transferred 122.00 GiB in 1150 seconds (108.6 MiB/s)
100: 2025-01-25 03:19:17 INFO: stopping kvm after backup task
100: 2025-01-25 03:19:18 INFO: adding notes to backup
100: 2025-01-25 03:19:18 INFO: prune older backups with retention: keep-daily=4, keep-last=4, keep-monthly=2, keep-weekly=4, keep-yearly=2
100: 2025-01-25 03:19:18 INFO: running 'proxmox-backup-client prune' for 'vm/100'
100: 2025-01-25 03:19:18 ERROR: prune 'vm/100': proxmox-backup-client failed: Error: mkstemp "/var/log/proxmox-backup/tasks/45/UPID:Tower:00000027:0256E545:00000001:6794BAA6rune:Unraid\\x3a\\x3avm-100:admin@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
100: 2025-01-25 03:19:18 ERROR: Backup of VM 100 failed - error pruning backups - check log

101: 2025-01-25 03:19:18 INFO: Starting Backup of VM 101 (qemu)
101: 2025-01-25 03:19:18 INFO: status = running
101: 2025-01-25 03:19:18 INFO: VM Name: DietPi
101: 2025-01-25 03:19:18 INFO: include disk 'scsi0' 'local-btrfs:101/vm-101-disk-0.raw' 8G
101: 2025-01-25 03:19:18 INFO: backup mode: snapshot
101: 2025-01-25 03:19:18 INFO: ionice priority: 7
101: 2025-01-25 03:19:18 INFO: creating Proxmox Backup Server archive 'vm/101/2025-01-25T10:19:18Z'
101: 2025-01-25 03:19:18 INFO: issuing guest-agent 'fs-freeze' command
101: 2025-01-25 03:19:19 INFO: issuing guest-agent 'fs-thaw' command
101: 2025-01-25 03:19:19 ERROR: VM 101 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/45/UPID:Tower:00000027:0256E545:00000002:6794BAA7:backup:Unraid\\x3avm-101:admin@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
101: 2025-01-25 03:19:19 INFO: aborting backup job
101: 2025-01-25 03:19:19 INFO: resuming VM again
101: 2025-01-25 03:19:19 ERROR: Backup of VM 101 failed - VM 101 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/45/UPID:Tower:00000027:0256E545:00000002:6794BAA7:backup:Unraid\\x3avm-101:admin@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device

102: 2025-01-25 03:19:19 INFO: Starting Backup of VM 102 (qemu)
102: 2025-01-25 03:19:19 INFO: status = stopped
102: 2025-01-25 03:19:19 INFO: backup mode: stop
102: 2025-01-25 03:19:19 INFO: ionice priority: 7
102: 2025-01-25 03:19:19 INFO: VM Name: Kali-Linux
102: 2025-01-25 03:19:19 INFO: include disk 'virtio0' 'local-btrfs:102/vm-102-disk-0.raw' 30G
102: 2025-01-25 03:19:19 INFO: creating Proxmox Backup Server archive 'vm/102/2025-01-25T10:19:19Z'
102: 2025-01-25 03:19:19 INFO: starting kvm to execute backup task
102: 2025-01-25 03:19:20 ERROR: VM 102 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/45/UPID:Tower:00000027:0256E545:00000003:6794BAA8:backup:Unraid\\x3avm-102:admin@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
102: 2025-01-25 03:19:20 INFO: aborting backup job
102: 2025-01-25 03:19:20 INFO: stopping kvm after backup task
102: 2025-01-25 03:19:20 ERROR: Backup of VM 102 failed - VM 102 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/45/UPID:Tower:00000027:0256E545:00000003:6794BAA8:backup:Unraid\\x3avm-102:admin@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device

103: 2025-01-25 03:19:20 INFO: Starting Backup of VM 103 (qemu)
103: 2025-01-25 03:19:20 INFO: status = stopped
103: 2025-01-25 03:19:20 INFO: backup mode: stop
103: 2025-01-25 03:19:20 INFO: ionice priority: 7
103: 2025-01-25 03:19:20 INFO: VM Name: ubuntu-20-server
103: 2025-01-25 03:19:20 INFO: include disk 'scsi0' 'local-btrfs:103/vm-103-disk-0.raw' 20G
103: 2025-01-25 03:19:20 INFO: creating Proxmox Backup Server archive 'vm/103/2025-01-25T10:19:20Z'
103: 2025-01-25 03:19:20 INFO: starting kvm to execute backup task
103: 2025-01-25 03:19:21 ERROR: VM 103 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/45/UPID:Tower:00000027:0256E545:00000004:6794BAA9:backup:Unraid\\x3avm-103:admin@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
103: 2025-01-25 03:19:21 INFO: aborting backup job
103: 2025-01-25 03:19:21 INFO: stopping kvm after backup task
103: 2025-01-25 03:19:22 ERROR: Backup of VM 103 failed - VM 103 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/45/UPID:Tower:00000027:0256E545:00000004:6794BAA9:backup:Unraid\\x3avm-103:admin@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device

104: 2025-01-25 03:19:22 INFO: Starting Backup of VM 104 (qemu)
104: 2025-01-25 03:19:22 INFO: status = running
104: 2025-01-25 03:19:22 INFO: VM Name: windows-qb-dan-2
104: 2025-01-25 03:19:22 INFO: include disk 'sata1' 'local-btrfs:104/vm-104-disk-0.raw' 50G
104: 2025-01-25 03:19:22 INFO: backup mode: snapshot
104: 2025-01-25 03:19:22 INFO: ionice priority: 7
104: 2025-01-25 03:19:22 INFO: creating Proxmox Backup Server archive 'vm/104/2025-01-25T10:19:22Z'
104: 2025-01-25 03:19:22 INFO: issuing guest-agent 'fs-freeze' command
104: 2025-01-25 03:19:24 INFO: issuing guest-agent 'fs-thaw' command
104: 2025-01-25 03:19:24 ERROR: VM 104 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/45/UPID:Tower:00000027:0256E545:00000005:6794BAAC:backup:Unraid\\x3avm-104:admin@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
104: 2025-01-25 03:19:24 INFO: aborting backup job
104: 2025-01-25 03:19:24 INFO: resuming VM again
104: 2025-01-25 03:19:24 ERROR: Backup of VM 104 failed - VM 104 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/45/UPID:Tower:00000027:0256E545:00000005:6794BAAC:backup:Unraid\\x3avm-104:admin@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device

106: 2025-01-25 03:19:24 INFO: Starting Backup of VM 106 (qemu)
106: 2025-01-25 03:19:24 INFO: status = running
106: 2025-01-25 03:19:24 INFO: VM Name: LibreNMS
106: 2025-01-25 03:19:24 INFO: include disk 'scsi0' 'local-btrfs:106/vm-106-disk-1.raw' 19532M
106: 2025-01-25 03:19:24 INFO: backup mode: snapshot
106: 2025-01-25 03:19:24 INFO: ionice priority: 7
106: 2025-01-25 03:19:24 INFO: skip unused drive 'local-btrfs:106/vm-106-disk-0.raw' (not included into backup)
106: 2025-01-25 03:19:24 INFO: creating Proxmox Backup Server archive 'vm/106/2025-01-25T10:19:24Z'
106: 2025-01-25 03:19:24 INFO: issuing guest-agent 'fs-freeze' command
106: 2025-01-25 03:19:25 INFO: issuing guest-agent 'fs-thaw' command
106: 2025-01-25 03:19:25 ERROR: VM 106 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/45/UPID:Tower:00000027:0256E545:00000006:6794BAAD:backup:Unraid\\x3avm-106:admin@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device
106: 2025-01-25 03:19:25 INFO: aborting backup job
106: 2025-01-25 03:19:25 INFO: resuming VM again
106: 2025-01-25 03:19:25 ERROR: Backup of VM 106 failed - VM 106 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/45/UPID:Tower:00000027:0256E545:00000006:6794BAAD:backup:Unraid\\x3avm-106:admin@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device



Excerpt from successful manual backup log:
vzdump 201 --remove 0 --compress zstd --notes-template '{{guestname}} manual' --storage local --node pve2 --notification-mode auto --mode snapshot


201: 2025-01-25 20:42:47 INFO: Starting Backup of VM 201 (lxc)
201: 2025-01-25 20:42:47 INFO: status = running
201: 2025-01-25 20:42:47 INFO: CT Name: docker
201: 2025-01-25 20:42:47 INFO: including mount point rootfs ('/') in backup
201: 2025-01-25 20:42:47 INFO: backup mode: snapshot
201: 2025-01-25 20:42:47 INFO: ionice priority: 7
201: 2025-01-25 20:42:47 INFO: create storage snapshot 'vzdump'
201: 2025-01-25 20:42:47 INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-lxc-201-2025_01_25-20_42_47.tar.zst'
201: 2025-01-25 20:46:30 INFO: Total bytes written: 28455639040 (27GiB, 122MiB/s)
201: 2025-01-25 20:46:31 INFO: archive file size: 8.83GB
201: 2025-01-25 20:46:31 INFO: adding notes to backup
201: 2025-01-25 20:46:31 INFO: cleanup temporary 'vzdump' snapshot
201: 2025-01-25 20:46:32 INFO: Finished Backup of VM 201 (00:03:45)

vzdump 106 --notes-template '{{guestname}} manual' --compress zstd --remove 0 --storage local --node pve2 --notification-mode auto --mode snapshot


106: 2025-01-25 20:54:45 INFO: Starting Backup of VM 106 (qemu)
106: 2025-01-25 20:54:45 INFO: status = running
106: 2025-01-25 20:54:45 INFO: VM Name: LibreNMS
106: 2025-01-25 20:54:45 INFO: include disk 'scsi0' 'local-btrfs:106/vm-106-disk-1.raw' 19532M
106: 2025-01-25 20:54:45 INFO: backup mode: snapshot
106: 2025-01-25 20:54:45 INFO: ionice priority: 7
106: 2025-01-25 20:54:45 INFO: skip unused drive 'local-btrfs:106/vm-106-disk-0.raw' (not included into backup)
106: 2025-01-25 20:54:45 INFO: creating vzdump archive '/var/lib/vz/dump/vzdump-qemu-106-2025_01_25-20_54_45.vma.zst'
106: 2025-01-25 20:54:45 INFO: issuing guest-agent 'fs-freeze' command
106: 2025-01-25 20:54:45 INFO: issuing guest-agent 'fs-thaw' command
106: 2025-01-25 20:54:45 INFO: started backup task 'fa9b5892-7524-436b-a219-067940499f97'
106: 2025-01-25 20:54:45 INFO: resuming VM again
106: 2025-01-25 20:54:48 INFO: 6% (1.2 GiB of 19.1 GiB) in 3s, read: 404.8 MiB/s, write: 240.4 MiB/s
106: 2025-01-25 20:54:51 INFO: 10% (2.1 GiB of 19.1 GiB) in 6s, read: 306.9 MiB/s, write: 255.1 MiB/s
106: 2025-01-25 20:54:54 INFO: 15% (3.0 GiB of 19.1 GiB) in 9s, read: 328.7 MiB/s, write: 281.0 MiB/s
106: 2025-01-25 20:54:57 INFO: 17% (3.3 GiB of 19.1 GiB) in 12s, read: 85.8 MiB/s, write: 72.1 MiB/s
106: 2025-01-25 20:55:00 INFO: 20% (3.9 GiB of 19.1 GiB) in 15s, read: 195.8 MiB/s, write: 173.6 MiB/s
[snip]
106: 2025-01-25 20:55:51 INFO: 83% (16.0 GiB of 19.1 GiB) in 1m 6s, read: 257.4 MiB/s, write: 235.7 MiB/s
106: 2025-01-25 20:55:54 INFO: 88% (16.8 GiB of 19.1 GiB) in 1m 9s, read: 280.3 MiB/s, write: 239.4 MiB/s
106: 2025-01-25 20:55:57 INFO: 91% (17.4 GiB of 19.1 GiB) in 1m 12s, read: 200.6 MiB/s, write: 192.7 MiB/s
106: 2025-01-25 20:56:00 INFO: 94% (18.0 GiB of 19.1 GiB) in 1m 15s, read: 203.0 MiB/s, write: 198.4 MiB/s
106: 2025-01-25 20:56:03 INFO: 95% (18.2 GiB of 19.1 GiB) in 1m 18s, read: 89.4 MiB/s, write: 88.0 MiB/s
106: 2025-01-25 20:56:06 INFO: 98% (18.8 GiB of 19.1 GiB) in 1m 21s, read: 189.7 MiB/s, write: 154.1 MiB/s
106: 2025-01-25 20:56:08 INFO: 100% (19.1 GiB of 19.1 GiB) in 1m 23s, read: 141.6 MiB/s, write: 128.1 MiB/s
106: 2025-01-25 20:56:08 INFO: backup is sparse: 1.96 GiB (10%) total zero data
106: 2025-01-25 20:56:08 INFO: transferred 19.07 GiB in 83 seconds (235.3 MiB/s)
106: 2025-01-25 20:56:08 INFO: archive file size: 5.56GB
106: 2025-01-25 20:56:08 INFO: adding notes to backup
106: 2025-01-25 20:56:08 INFO: Finished Backup of VM 106 (00:01:23)

 

[Chris]

ERROR: Backup of VM 103 failed - VM 103 qmp command 'backup' failed - backup connect failed: command error: mkstemp "/var/log/proxmox-backup/tasks/45/UPID:Tower:00000027:0256E545:00000004:6794BAA9:backup:Unraid\\x3avm-103:admin@pbs:.tmp_XXXXXX" failed: ENOSPC: No space left on device

As stated in the error message, creating the files to store the task logs fails. You have no more storage space available on the filesystem backing the /var/log/proxmox-backup path of your PBS instance. Please post the output of df -h and mount, that should tell more.

Edit: Also see https://forum.proxmox.com/threads/b...ace-left-on-device-but-there-is-space.160905/

 

[eleanor]

Thanks, here is the output.

root@Tower:/mnt/user/appdata/pbs/logs# du -sh *
123M    api
1.8M    tasks
root@Tower:/mnt/user/appdata/pbs/logs# df -h .
Filesystem      Size  Used Avail Use% Mounted on
shfs            477G  350G  128G  74% /mnt/user
root@Tower:/mnt/user/appdata/pbs/logs# mount
rootfs on / type rootfs (rw,size=16280356k,nr_inodes=4070089,inode64)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=131072k,mode=755,inode64)
/dev/sda1 on /boot type vfat (rw,noatime,nodiratime,fmask=0177,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,flush,errors=remount-ro)
/boot/bzmodules on /lib type squashfs (ro,relatime,errors=continue)
overlay on /lib type overlay (rw,relatime,lowerdir=/lib,upperdir=/var/local/overlay/lib,workdir=/var/local/overlay-work/lib)
/boot/bzfirmware on /usr type squashfs (ro,relatime,errors=continue)
overlay on /usr type overlay (rw,relatime,lowerdir=/usr,upperdir=/var/local/overlay/usr,workdir=/var/local/overlay-work/usr)
devtmpfs on /dev type devtmpfs (rw,relatime,size=8192k,nr_inodes=4070089,mode=755,inode64)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /dev/shm type tmpfs (rw,relatime,inode64)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
hugetlbfs on /hugetlbfs type hugetlbfs (rw,relatime,pagesize=2M)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
tmpfs on /var/log type tmpfs (rw,relatime,size=131072k,mode=755,inode64)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
rootfs on /mnt type rootfs (rw,size=16280356k,nr_inodes=4070089,inode64)
nfsd on /proc/fs/nfs type nfsd (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
disk1 on /mnt/disk1 type zfs (rw,noatime,xattr,posixacl)
disk2 on /mnt/disk2 type zfs (rw,noatime,xattr,posixacl)
/dev/nvme0n1p1 on /mnt/cache type btrfs (rw,noatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)
disk1/system on /mnt/disk1/system type zfs (rw,noatime,xattr,posixacl)
disk1/Media on /mnt/disk1/Media type zfs (rw,noatime,xattr,posixacl)
disk1/pbs_backup on /mnt/disk1/pbs_backup type zfs (rw,noatime,xattr,posixacl)
...
...
disk2/pbs_backup on /mnt/disk2/pbs_backup type zfs (rw,noatime,xattr,posixacl)
shfs on /mnt/user0 type fuse.shfs (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
shfs on /mnt/user type fuse.shfs (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)
/dev/nvme0n1p1 on /var/lib/docker type btrfs (rw,noatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)
/dev/nvme0n1p1 on /var/lib/docker/btrfs type btrfs (rw,noatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)
/dev/nvme0n1p1 on /mnt/cache/docker/btrfs type btrfs (rw,noatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/)
nsfs on /run/docker/netns/c3da7444d6de type nsfs (rw)
/mnt/disk1/system/libvirt/libvirt.img on /etc/libvirt type btrfs (rw,noatime,ssd,space_cache=v2,subvolid=5,subvol=/)
nsfs on /run/docker/netns/4337fbab8ff3 type nsfs (rw)
nsfs on /run/docker/netns/0f71cc7bc1c3 type nsfs (rw)
nsfs on /run/docker/netns/c3fe778e5b0c type nsfs (rw)
nsfs on /run/docker/netns/default type nsfs (rw)
nsfs on /run/docker/netns/2d5a78bd74b7 type nsfs (rw)
nsfs on /run/docker/netns/dbf2d4d64d7c type nsfs (rw)
nsfs on /run/docker/netns/94bdb0c2f813 type nsfs (rw)
nsfs on /run/docker/netns/ef02b11ed5ac type nsfs (rw)
nsfs on /run/docker/netns/ac45a341c0c1 type nsfs (rw)
nsfs on /run/docker/netns/a9df71ac5faf type nsfs (rw)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3259508k,nr_inodes=814877,mode=700,inode64)

 

[Chris]

shfs on /mnt/user type fuse.shfs (rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other)

root@Tower:/mnt/user/appdata/pbs/logs# du -sh * 123M api 1.8M tasks root@Tower:/mnt/user/appdata/pbs/logs# df -h . Filesystem Size Used Avail Use% Mounted on shfs 477G 350G 128G 74% /mnt/user

So given your output from above, I assume the shown mountpoint using shfs is backing the filesystem the PBS tasks should be written to. Please try to exclude that there is an issue with this shfs based mountpoint in combination with the mkostemp call used to create the temporary file by moving this to a regular, non network based filesystem.

Please also share how above mountpoints are related to the PBS container.