[SOLVED] Cannot read remotely mounted backup datastore

[lifeboy]

I have installed PVS 2.0 on a LXC to test an play around with. All is good, except the following:

To have some decent storage to write backup to I have mounted a location with sftp as follows:

sshfs backup@192.168.121.34:/mnt/pve/cephfs /mnt/cephfs

(I have 'hijacked' the backup account, given it bash login and a password for this purpose)

Here it is mounted:

# ls -la
total 585105520
drwxr-xr-x 1 backup backup           13 Nov  3 18:46  .
drwxr-xr-x 1 backup backup           13 Nov  3 18:46  ..
drwxr-xr-x 1 backup backup        65536 Nov  3 18:59  .chunks
-rwxr-xr-x 1 backup backup            0 Nov  3 18:46  .lock

However, when I add the location /mnt/cephfs to PBS, I get the error:

Bad Request (400) unable to open chunk store 's2-cephfs' at "/mnt/cephfs/.chunks" - Permission denied (os error 13)

I'm at a loss. What am I missing? I use sftp very often and it never gives me any trouble. Also, when I first added the datastore, the system added the .chunks and .lock directories, so it was able to write there!

 

[dietmar]

We do not support sshfs for backup datastores, because it does not provide required features (e.g. missing O_TMPFILE).

Besides, it is highly recommended to use a local file system. Else you double the network traffic!

 

[lifeboy]

We do not support sshfs for backup datastores, because it does not provide required features (e.g. missing O_TMPFILE).

Oh, I see. Can we put a note in the documentation somewhere to that effect please?

Besides, it is highly recommended to use a local file system. Else you double the network traffic!

It is essentially on the same metal machine on which the lxc with pbs is running. It's still a test at this stage, though I hear you in terms of traffic.

Thanks for the response.

 

[lifeboy]

On this point, let me ask this related question then.

I have a proxmox cluster of older machines with about 20 spinning disks on 7 nodes which we use for development work & testing. Nothing to write home about, but stable and due the use of ceph as virtualised storage a dependable and stable cluster.

I have recently upgraded the cluster to PVE 7.

I'd like to use Proxmox Backup Server 2 to store some backups from other clusters here. So I have so far:

• Installed PBS in an LXC using Debian 11 (bullseye) as guest OS
• PBS is running fine for now

However, it doesn't seem to be easy to add ceph based storage to it as a datastore. I tried sftp to mount cephfs as explained above. I have tried to add /mnt/pve/cephfs as a mountpoint to the lxc as below, but it prevents the container from starting.

pct set 100 -mp0 /mnt/cephfs,mp=/mnt/pve/cephfs

It gives me an error:

# lxc-start -F -n 100
lxc-start: 100: conf.c: run_buffer: 316 Script exited with status 2
lxc-start: 100: start.c: lxc_init: 816 Failed to run lxc.hook.pre-start for container "100"
lxc-start: 100: start.c: __lxc_start: 2007 Failed to initialize container "100"
lxc-start: 100: tools/lxc_start.c: main: 308 The container failed to start
lxc-start: 100: tools/lxc_start.c: main: 313 Additional information can be obtained by setting the --logfile and --logpriority options

I guess that it is because /mnt/pve/cephfs is actually also a mountpoint on the host machine?

Any ideas on how I could get PBS to write to ceph based storage? I would set up a new replicated pool with only 2 copies to save storage space and because loosing a backup is not critical in this test phase. It would however be better still than using a disk directly, since a disk failure would then result in loosing the backups anyway.

thanks in advance

 

[lifeboy]

Bump. Anyone?

 

[lifeboy]

I have changed my approach somewhat now by mounting an actual directory of the host OS, which works fine, except that I now have to map the users properly, which I'll do next.

 

[aaron]

Bump. Anyone?

It has been the weekend in between

And this question would be better of in the PVE subforum. But nonetheless; you could also try to mount the CephFS directly in the container / VM. You'd need to install the necessary packages and provide a valid keyring file that has permissions in the cluster to access the cephfs.

 

[lynx]

We do not support sshfs for backup datastores, because it does not provide required features (e.g. missing O_TMPFILE).

Putting aside all performance considerations, things seem to work for us and sshfs. Need to note that this storage is only used for a sync job.

Is "missing O_TMPFILE" something that can break the backups in any way?

 

[lifeboy]

I have resolved the problem in a really simply way. Put another way: I was being really stupid.

The lxc is already using ceph as storage, so if I simply increase the size of the lxc disk resource and use a directory as a datastore, I can provision enough storage for all my backups.

What I now have to do is configure a ceph pool to make most efficient use of the storage, which probably means I'll create a erasure coded pool and set up the lxc manually from the command line.

I sincerely hope that the Proxmox devs acknowledge and implement this important feature of ceph soon!

 

[aaron]

The lxc is already using ceph as storage, so if I simply increase the size of the lxc disk resource and use a directory as a datastore, I can provision enough storage for all my backups.

One general question as I may have missed that information from going over this thread: Is the PBS LXC running in the PVE cluster that it is storing backups for?

If so, and if you want to be able to restore from these backups should your PVE cluster fail completely for some reason, you should rethink your architecture. Either have the PBS datastores synced to another "external" PBS server or move that PBS outside your PVE cluster. Otherwise, you do have a circular dependency and in a disaster situation it will be hard to impossible to access the backups

 

[lifeboy]

One general question as I may have missed that information from going over this thread: Is the PBS LXC running in the PVE cluster that it is storing backups for?

If so, and if you want to be able to restore from these backups should your PVE cluster fail completely for some reason, you should rethink your architecture. Either have the PBS datastores synced to another "external" PBS server or move that PBS outside your PVE cluster. Otherwise, you do have a circular dependency and in a disaster situation it will be hard to impossible to access the backups

No. Like I said higher up: "I'd like to use Proxmox Backup Server 2 to store some backups from other clusters here." It's all a test at this stage. We use clusters in different geographical locations and this "test" cluster is in yet another location.

But as you can see from my previous post, the problem is resolved. Thanks for your input anyway!

 

[aaron]

Ah okay then