Tuxis launches free Proxmox Backup Server BETA service

@tuxis Hello, there! Quick question about the API Token tutorial at https://support.tuxis.nl/en/knowledgebase/article/how-to-add-a-api-token-used-for-backups . I decided to post here in case the answer helps someone else learn, as well.

The tutorial API Token has us assign the token the DatastorePowerUser permission, which includes pruning in addition to backing up. My understanding was that pruning is scheduled on the server side. Is there a recommended workflow that involves client-side pruning?

Edit: Never mind, I'm an idiot. I forgot that you could manually delete backups from the PVE node interface. :P
 
Last edited:
Hi everyone,

Just wanted to share an update regarding the free accounts, of which we disabled the possibility to create them as of yesterday. As this thread shows, we've been offering these free accounts since september 2020, over more than four years. We've done this for three reasons:
1: Test PBS
2: Give something back to the community
3: Marketing, obviously. We're still a professional company trying to make some money.

Since september 2020, our fleet of PBS'es have grown from a single server to eight. And the number of non-paying users is far beyond 2000. We've been able to manage the (manual) workload by incorporating creation of accounts into our new portal, so that's doable (although about 3 times per day, users fill their datastores completely, requiring our intervention).

The amount of storage that these users use, is not really an issue. If all the users would fill up their datastore completely, it would be about 300Tb, that doesn't scare us. What IS an issue though, is performance for active users.

We see two issues there:
1: Verify-jobs. Everyone wants to see a green checked-icon next to their backup. Even though we use ZFS and verifies are pretty much useless, the amount of IO this causes is enourmous. For non-paying users, we circumvented this by implementing https://support.tuxis.nl/en/knowled...garbage-collection-jobs-for-free-pbs-accounts
2: PBS was not built for large amounts of datastores and accounts. The amount of requests we handle, without any running tasks, exceeds 250 requests per second on a single server. These are authentication and datastore-status-requests. About 800% of CPU-power is required to handle just those requests.
We've created https://bugzilla.proxmox.com/show_bug.cgi?id=6049 for that.

We've decided to disable creation of free accounts, until we see bug 6049 fixed and the load on our PBS'es lowered, to prevent our paying customers to experience performance issues.

So this is the time to let Open Source shine. If you are able to code in Rust (we're not), dive into bug 6049 and help the Proxmox Team in their efforts to create these cool products.

Please note, that we do *not* feel that this is a problem in PBS. AFAIC the way we are deploying PBS is (far) from the normal situation that PBS runs in.
 
  • Like
Reactions: MaroonGorilla, SInisterPisces, coolhuman and 2 others
  • Like
Reactions: Johannes S, SInisterPisces and coolhuman
Jeeves_ said:
2: PBS was not built for large amounts of datastores and accounts. The amount of requests we handle, without any running tasks, exceeds 250 requests per second on a single server. These are authentication and datastore-status-requests. About 800% of CPU-power is required to handle just those requests.
We've created https://bugzilla.proxmox.com/show_bug.cgi?id=6049 for that.
Click to expand...
I'm updating PBS to 3.3.3-1 today (I run a local instance as well as the Tuxis one). I saw this in the release notes.
Code: 
* api: datastore list: move checking if a datastore is mounted after we
    ensured that the user may actually access it. While this had no effect
    security wise, it could significantly increase the cost of this API
    endpoint in big setups with many datastores and many tenants that each
    have only access to one, or a small set, of datastores.

Not the same thing as 6049, but it sounds like it might help somehow for Tuxis' specific setup, which is definitely "many datastores and many tenants that each have only access to one, or a small set, of datastores." :)
 
  • Like
Reactions: Johannes S
SInisterPisces said:
I'm updating PBS to 3.3.3-1 today (I run a local instance as well as the Tuxis one). I saw this in the release notes.
Code: 
* api: datastore list: move checking if a datastore is mounted after we
    ensured that the user may actually access it. While this had no effect
    security wise, it could significantly increase the cost of this API
    endpoint in big setups with many datastores and many tenants that each
    have only access to one, or a small set, of datastores.

Not the same thing as 6049, but it sounds like it might help somehow for Tuxis' specific setup, which is definitely "many datastores and many tenants that each have only access to one, or a small set, of datastores." :)
Click to expand...
That fix is the cousin of 6049, was fixed via the mailinglist. I see that that also includes the revert of the stat-issue that was introduced earlier. We will keep track for when that is available in the enterprise repository. Thanks!
 
  • Like
Reactions: Johannes S
I tested that version by creating 500 (useless) datastores and ab the hell out of it. 3.3.3 has a slightly higher performance, but the difference is minimal.

Code: 
for i in `seq 1 10`; do ab -n 1000 -c 200 -H 'Authorization: PBSAPIToken=XXXX@pbs!XXX' https://XXX.tuxis.net:8007/api2/json/admin/datastore/backup/status > run-$i.txt; done

Code: 
root@pbs:~/pre-upgrade# grep -h 'across all concurrent requests' *.txt
Time per request:       3.699 [ms] (mean, across all concurrent requests)
Time per request:       3.622 [ms] (mean, across all concurrent requests)
Time per request:       3.658 [ms] (mean, across all concurrent requests)
Time per request:       3.637 [ms] (mean, across all concurrent requests)
Time per request:       3.700 [ms] (mean, across all concurrent requests)
Time per request:       3.657 [ms] (mean, across all concurrent requests)
Time per request:       3.572 [ms] (mean, across all concurrent requests)
Time per request:       3.781 [ms] (mean, across all concurrent requests)
Time per request:       3.661 [ms] (mean, across all concurrent requests)
Time per request:       3.645 [ms] (mean, across all concurrent requests)

Code: 
root@pbs:~/post-upgrade# grep -h 'across all concurrent requests' *.txt
Time per request:       3.560 [ms] (mean, across all concurrent requests)
Time per request:       3.691 [ms] (mean, across all concurrent requests)
Time per request:       3.626 [ms] (mean, across all concurrent requests)
Time per request:       3.564 [ms] (mean, across all concurrent requests)
Time per request:       3.609 [ms] (mean, across all concurrent requests)
Time per request:       3.524 [ms] (mean, across all concurrent requests)
Time per request:       3.723 [ms] (mean, across all concurrent requests)
Time per request:       3.539 [ms] (mean, across all concurrent requests)
Time per request:       3.697 [ms] (mean, across all concurrent requests)
Time per request:       3.738 [ms] (mean, across all concurrent requests)

With a single datastore:
Code: 
root@pbs:~/no-datastores# grep -h 'across all concurrent requests' *.txt
Time per request:       3.190 [ms] (mean, across all concurrent requests)
Time per request:       3.166 [ms] (mean, across all concurrent requests)
Time per request:       3.168 [ms] (mean, across all concurrent requests)
Time per request:       3.125 [ms] (mean, across all concurrent requests)
Time per request:       3.219 [ms] (mean, across all concurrent requests)
Time per request:       3.088 [ms] (mean, across all concurrent requests)
Time per request:       3.212 [ms] (mean, across all concurrent requests)
Time per request:       3.129 [ms] (mean, across all concurrent requests)
Time per request:       3.272 [ms] (mean, across all concurrent requests)
Time per request:       3.151 [ms] (mean, across all concurrent requests)
 
  • Like
Reactions: coolhuman and Johannes S
itjamie said:
sorry if this has been answered elsewhere.
how are you doing the quota per datastore?
is the storage local or on a remote san?
Click to expand...
ZFS filesystems per datastore, with a quota.
 
  • Like
Reactions: coolhuman and Johannes S
I am running into limits of the free tier, I have attempted to upgrade the service but I see "Paid services not allowed" when trying to activate a paid PBS service.

Earlier in this thread it was mentioned you can only to B2B, does this apply outside the EU?

I would love to upgrade to a paid account and stick with Tuxis, please let me know if there is a way forward.
 
bonkas said:
I am running into limits of the free tier, I have attempted to upgrade the service but I see "Paid services not allowed" when trying to activate a paid PBS service.

Earlier in this thread it was mentioned you can only to B2B, does this apply outside the EU?

I would love to upgrade to a paid account and stick with Tuxis, please let me know if there is a way forward.
Click to expand...
Oh, you can't buy it as normal consumer? That is sad.
 
You must log in or register to reply here.
Top Bottom