S3 Local datastore recommendations?

C

cfgmgr

Member
Jul 25, 2023
76
17
13
Greetings!

I was looking at the document that talks about configuring a backend for S3. I was curious about the local datastore. This part here discusses it.
A new datastore with S3 backend can be created using one of the configured S3 endpoints. Although storing all contents on the S3 object store, the datastore requires nevertheless a local persistent cache, used to increase performance and reduce the number of requests to the backend. For this, a local filesystem path has to be provided during datastore creation, just like for regular datastore setup. However, unlike for regular datastores the size of the local cache can be limited, 64 GiB to 128 GiB are recommended given that cached datastore contents include also data chunks. Best is to use a dedicated disk, partition or ZFS dataset with quota as local cache. Note however, it is not possible to use a pre-existing regular datastore for this. Further, the use of volatile memory only for the cache is currently not possible.
Click to expand...

Does this device need to have any sort of HA/redundant capabilities? I think I have a single open NVMe slot and was wondering if that would suffice. If the device fails - is there anything that need to be backed up etc, or is it all just purely pass through?

I have a local 5PiB ceph cluster that I would back up to.

Thanks!
~J
 
IsThisThingOn said:
No, why should it? It is cache.


I hope this is a typo, otherwise good luck with your S3 bill ;)
Click to expand...

I wouldn't think the device would need to be redundant but I didn't want to make any assumptions.

It's a standalone ceph cluster thats local/adjacent to PBS. no AWS.

Thanks!
 
You must log in or register to reply here.
Top Bottom
Back