virtualized PBS with shared mount

[maccmiles]

Hi folks, Relative noob here trying to setup a proxmox backup server, but running into some issues with backups completing.

My PBS is running as a VM under Unraid (offsite from cluster), with a shared folder being passed to the VM. I'm passing my backup share to the VM as follows:

  <filesystem type='mount' accessmode='passthrough'>
      <source dir='/mnt/user/Backups/'/>
      <target dir='pbsdir'/>
      <alias name='fs0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </filesystem>

With that, I'm mounting in PBS as /Backup in /etc/fstab

pbsdir /Backup 9p trans=virtio,version=9p2000.L,_netdev,rw 0 0

I can read/write to /Backup in commandline, and when I run a backup job from my cluster, the backup filestructure gets generated on the drive (ex: /vm/121/owner) But the backups fail with "EINVAL: Invalid argument"

121: 2021-11-26 15:51:01 INFO: Starting Backup of VM 121 (qemu)
121: 2021-11-26 15:51:01 INFO: status = running
121: 2021-11-26 15:51:01 INFO: VM Name: REDACTED
121: 2021-11-26 15:51:01 INFO: include disk 'scsi0' 'data:vm-121-disk-0' 70G
121: 2021-11-26 15:51:01 INFO: backup mode: snapshot
121: 2021-11-26 15:51:01 INFO: ionice priority: 7
121: 2021-11-26 15:51:01 INFO: snapshots found (not included into backup)
121: 2021-11-26 15:51:01 INFO: creating Proxmox Backup Server archive 'vm/121/2021-11-26T20:51:01Z'
121: 2021-11-26 15:51:01 INFO: enabling encryption
121: 2021-11-26 15:51:02 ERROR: VM 121 qmp command 'backup' failed - backup register image failed: command error: EINVAL: Invalid argument
121: 2021-11-26 15:51:02 INFO: aborting backup job
121: 2021-11-26 15:51:02 ERROR: Backup of VM 121 failed - VM 121 qmp command 'backup' failed - backup register image failed: command error: EINVAL: Invalid argument

Going to Datastore>Content on pbs web returns "Bad Request (400) unable to detect file type"

My research into this already suggests maybe some filetype issue or permissions issue, but I'm unsure where to check/diagnose this, and I'm confused as to why I am able to see contents and it's writing folders if that would be the case.
Kinda stumped on this one, looking for any guidance / suggestions on what to try / where to look.

Cheers!

 

[dcsapak]

the 9p filesystem probably does not support some features we need... can you check the task log on the pbs side for the backup?

 

[maccmiles]

Hi Dominik, appreciate the assistance.

my PBS task logs weren't very helpful, to me atleast

()
2021-11-26T15:57:29-05:00: starting new backup on datastore 'Backup': "vm/111/2021-11-26T20:57:28Z"
2021-11-26T15:57:29-05:00: GET /previous: 400 Bad Request: no valid previous backup
2021-11-26T15:57:30-05:00: POST /fixed_index: 400 Bad Request: EINVAL: Invalid argument
2021-11-26T15:57:30-05:00: backup ended and finish failed: backup ended but finished flag is not set.
2021-11-26T15:57:30-05:00: removing unfinished backup
2021-11-26T15:57:30-05:00: TASK ERROR: backup ended but finished flag is not set.

Just seems to throw a 400 and complain about EINVAL

I was doing some readings about users getting it to work with 9p, I thought it would be faster than trying to export a fileshare over NFS and mounting it that way.

Notably, the backup prune function seems to complain about filetype.

2021-12-01T00:00:00-05:00: task triggered by schedule 'daily'
2021-12-01T00:00:00-05:00: Starting datastore prune on store "Backup"
2021-12-01T00:00:00-05:00: retention options: --keep-last 3 --keep-daily 6 --keep-weekly 8 --keep-monthly 5 --keep-yearly 1
2021-12-01T00:00:00-05:00: TASK ERROR: unable to detect file type

I'm just confused as to why it would make directories and files on the share, then give up. It seems to be succesfully removing the failed backups, and the vm dirs / manifests remain on-disk.

Thanks,
~Miles

 

[dcsapak]

can you check the syslog of the pbs ? it seems that some file operation still does not work like expected
the error indicates that there is some file where the file type cannot be detected (so its no directory/normal file/symlink/etc..)

 

[maccmiles]

Just got around to having another run.

root@pbs:~# tail -f /var/log/syslog
Dec  9 15:36:08 pbs systemd[12281]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Dec  9 15:36:08 pbs systemd[12281]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Dec  9 15:36:08 pbs systemd[12281]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Dec  9 15:36:08 pbs systemd[12281]: Listening on GnuPG cryptographic agent and passphrase cache.
Dec  9 15:36:08 pbs systemd[12281]: Reached target Sockets.
Dec  9 15:36:08 pbs systemd[12281]: Reached target Basic System.
Dec  9 15:36:08 pbs systemd[12281]: Reached target Main User Target.
Dec  9 15:36:08 pbs systemd[12281]: Startup finished in 186ms.
Dec  9 15:36:08 pbs systemd[1]: Started User Manager for UID 0.
Dec  9 15:36:08 pbs systemd[1]: Started Session 355 of user root.
Dec  9 15:37:47 pbs proxmox-backup-proxy[620]: starting new backup on datastore 'Backup': "vm/121/2021-12-09T20:37:46Z"
Dec  9 15:37:47 pbs proxmox-backup-proxy[620]: GET /previous: 400 Bad Request: no valid previous backup
Dec  9 15:37:47 pbs proxmox-backup-proxy[620]: POST /fixed_index: 400 Bad Request: EINVAL: Invalid argument
Dec  9 15:37:47 pbs proxmox-backup-proxy[620]: backup ended and finish failed: backup ended but finished flag is not set.
Dec  9 15:37:47 pbs proxmox-backup-proxy[620]: removing unfinished backup
Dec  9 15:37:47 pbs proxmox-backup-proxy[620]: removing backup snapshot "/Backup/vm/121/2021-12-09T20:37:46Z"
Dec  9 15:37:47 pbs proxmox-backup-proxy[620]: TASK ERROR: backup ended but finished flag is not set.

Seems to be similar if not same errors. Are there other methods of mounting you'd recommend I try?

 

[dcsapak]

as i said in the beginning, i believe that the filesystem does not support some feature we need, which though i cannot say from the output...

you can check the files in the datastore and see if something looks weird... e.g. with 'ls -lR' (caution, that will recursively print a list of all files from the current directory (and subdirs))

 

[Matchlighter]

@maccmiles If this is still an issue for you, I figured out a workaround while I was trying to do the same thing: instead of sharing the `/mnt/user/pbs` directory from Unraid into the PBS VM, share `/mnt/disk0/pbs` (of course updating `pbs` and `disk0` to your situation). You may also want to update the Unraid Share to use _only_ that disk and not use caching - though this is optional and won't affect PBS directly - only if another system writes to the share, in which case there's a chance PBS won't see changes that that system makes.

Offering some technical reasoning, I'm guessing that @dcsapak was right about an FS not implementing a required feature, except it wasn't 9p that was the problem, but instead the SHFS that Unraid uses to merge files from the various storage "backends" (eg disk0 to diskN (XFS), and any cache disks (BTRFS)) into a single file tree.

 

[kevinkok]

Hey @maccmiles, I had the same issue as you when using 9p, in my case, CT backup works, but VM backup would fail. Now, I'm using virtiofs instead and it is working for me. You can check out my post for how I set mine up: https://forum.proxmox.com/threads/h...d-uses-virtiofs-to-passthrough-shares.120271/

Hope it helps!