Tuxis launches free Proxmox Backup Server BETA service

[Jarvar]

How can we see the data store location fully?
I used to just put the name, but there's an error adding that method now. I think they want /mnt
Thank you.
Looks like I figured out the solution. I just needed to add the storage name, but I also had to remove the previous fingerprint.
Thank you.

 

[tuxis]

I don't understand what you mean?

 

[Fantomas]

Hi community,

it seems that as of 18th May my free PBS plan has stopped working. vzdump reports the error "backup failed: could not activate storage 'tuxis': tuxis: Cannot find datastore ...". When logging into PBS, I am unable to find my datastore. Is there any activity ongoing regarding the upgrade to PBS 2.2?

Thanks and kind regards,
Jan

 

[tuxis]

Hi community,

it seems that as of 18th May my free PBS plan has stopped working. vzdump reports the error "backup failed: could not activate storage 'tuxis': tuxis: Cannot find datastore ...". When logging into PBS, I am unable to find my datastore. Is there any activity ongoing regarding the upgrade to PBS 2.2?

Thanks and kind regards,
Jan

Could you please create a ticket via https://support.tuxis.nl/en/tickets/create/step1
Please also mention your debtor number.

 

[apollo13]

I am seeing slow backup speed towards tuxis currently. Dumping onto a local disk yields a few hundred mb per second but running the proxmox server benchmark against tuxis yields:

Uploaded 16 chunks in 11 seconds.
Time per request: 744015 microseconds.
TLS speed: 5.64 MB/s
SHA256 speed: 1716.83 MB/s
Compression speed: 716.49 MB/s
Decompress speed: 958.61 MB/s
AES256/GCM speed: 2229.85 MB/s
Verify speed: 609.08 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 5.64 MB/s (0%)     │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 1716.83 MB/s (85%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 716.49 MB/s (95%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 958.61 MB/s (80%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 609.08 MB/s (80%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2229.85 MB/s (61%) │
└───────────────────────────────────┴────────────────────┘

The line speed (which I am able to reach) of my internet connection is 100mbit. No idea why it only manages to utilize half of it. Uploading a backup from Proxmox currently looks like this:

INFO: starting new backup job: vzdump 100 --storage pbs-tuxis --mode stop --remove 0 --notes-template '{{guestname}}' --node ps-pmx3
INFO: Starting Backup of VM 100 (qemu)
INFO: Backup started at 2022-06-01 20:04:07
INFO: status = stopped
INFO: backup mode: stop
INFO: ionice priority: 7
INFO: VM Name: dummy
INFO: include disk 'scsi0' 'vm_nvme:vm-100-disk-0' 400G
INFO: creating Proxmox Backup Server archive 'vm/100/2022-06-01T18:04:07Z'
INFO: starting kvm to execute backup task
INFO: enabling encryption
INFO: started backup task '27f4b6e3-84c7-46f4-bc23-1ee790384f46'
INFO: scsi0: dirty-bitmap status: created new
INFO:   0% (100.0 MiB of 400.0 GiB) in 3s, read: 33.3 MiB/s, write: 33.3 MiB/s
INFO:   1% (4.0 GiB of 400.0 GiB) in 10m 48s, read: 6.2 MiB/s, write: 6.2 MiB/s
INFO:   2% (8.0 GiB of 400.0 GiB) in 23m 9s, read: 5.5 MiB/s, write: 5.5 MiB/s
INFO:   3% (12.0 GiB of 400.0 GiB) in 37m 32s, read: 4.7 MiB/s, write: 4.7 MiB/s
INFO:   4% (16.0 GiB of 400.0 GiB) in 54m 16s, read: 4.1 MiB/s, write: 4.1 MiB/s
INFO:   5% (20.0 GiB of 400.0 GiB) in 1h 11m 27s, read: 4.0 MiB/s, write: 4.0 MiB/s
INFO:   6% (24.0 GiB of 400.0 GiB) in 1h 28m 39s, read: 4.0 MiB/s, write: 4.0 MiB/s
INFO:   7% (28.0 GiB of 400.0 GiB) in 1h 45m 26s, read: 4.1 MiB/s, write: 4.1 MiB/s

Somewhere on the way to tuxis I seem to be loosing a lot of bandwidth :/

 

[fabian]

was it better before? because you will lose some of your theoretical bandwidth to overhead, especially on slower and/or higher latency links.. the (TLS part of the) benchmark does basically nothing except upload in-memory chunks to the server where they get thrown away, so if there isn't some hickup going on over at Tuxis that's just the HTTP2 over TLS performance you are able to reach over that link (which will likely be slower than something like iperf)..

 

[apollo13]

Yeah it was better, I was close to 100 MBit:


granted it somewhat fluctuates, but looking at the yearly maximum transfer rates since we started using tuxis (not that long ago):

I understand that due to the delta uploads etc there might not even be enough data to saturate the link. But I am currently uploading 300 GB which cannot be deduplicated. So at some point something somewhere got worse. Given that the TLS speeds are "just " 5 MB/s I'll try that from another server in the cloud to rule out other local problems.

Cheers,
Florian

EDIT: Seems to be something between my ISP and Tuxis, gotta double check, but this is what I get from the cloud:

Uploaded 352 chunks in 5 seconds.
Time per request: 14502 microseconds.
TLS speed: 289.20 MB/s
SHA256 speed: 332.34 MB/s
Compression speed: 313.74 MB/s
Decompress speed: 463.19 MB/s
AES256/GCM speed: 973.37 MB/s
Verify speed: 192.30 MB/s
┌───────────────────────────────────┬───────────────────┐
│ Name                              │ Value             │
╞═══════════════════════════════════╪═══════════════════╡
│ TLS (maximal backup upload speed) │ 289.20 MB/s (23%) │
├───────────────────────────────────┼───────────────────┤
│ SHA256 checksum computation speed │ 332.34 MB/s (16%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 compression speed    │ 313.74 MB/s (42%) │
├───────────────────────────────────┼───────────────────┤
│ ZStd level 1 decompression speed  │ 463.19 MB/s (39%) │
├───────────────────────────────────┼───────────────────┤
│ Chunk verification speed          │ 192.30 MB/s (25%) │
├───────────────────────────────────┼───────────────────┤
│ AES256 GCM encryption speed       │ 973.37 MB/s (27%) │
└───────────────────────────────────┴───────────────────┘

 

[tuxis]

I am seeing slow backup speed towards tuxis currently. Dumping onto a local disk yields a few hundred mb per second but running the proxmox server benchmark against tuxis yields:

Somewhere on the way to tuxis I seem to be loosing a lot of bandwidth :/

Hi apollo13,

please create a ticket on https://support.tuxis.nl and i will look into it. My guess is that you already did, if so you got a reply in your inbox about 5 minutes ago .

Br,
Richard

 

[apollo13]

Hi Richard,

thanks -- yeah we wrote our posts at the same time I fear I'll update here once I figured out where the slow down occurs.

Thanks,
Florian

 

[apollo13]

So, short update after further debugging. After moving the testclient outside the perimeter (or at least one firewall) results didn't get better. What did help though was tunneling the traffic to tuxis through a VPN. Which leaves me with the following possibilities:
* The VPN also changes the traceroute through tuxis, so maybe something on the original route is slow
* The VPN also hides the fact that we are using TLS/HTTP, so maybe a content inspecting firewall tries something stupid here.
* MTU issues which are gone through the VPN?

Any ideas on how to debug this further?

 

[tuxis]

...Which leaves me with the following possibilities:
* The VPN also changes the traceroute through tuxis, so maybe something on the original route is slow
* The VPN also hides the fact that we are using TLS/HTTP, so maybe a content inspecting firewall tries something stupid here.

My guess is still at one of these above.
So i took a look at your provider to see if we could establish a direct peering session but that is not the case.
We do not have a IX where we are both present. I did however tickle one of our routers in a way that might impact the path your traffic takes.

 

[apollo13]

My guess is still at one of these above.
So i took a look at your provider to see if we could establish a direct peering session but that is not the case.
We do not have a IX where we are both present. I did however tickle one of our routers in a way that might impact the path your traffic takes.

Well you tickled it well enough. I am now getting routed via nextlayer in vienna:

 1  _gateway (10.7.21.1)  0.470 ms  0.428 ms  0.424 ms
 2  *  * * * *
 3  tengig0020-1702.cr1.grazkom.at (85.237.12.1)  4.206 ms  4.476 ms  4.556 ms
 4  xe-0-0-5-r25.inx.vie.nextlayer.net (81.16.152.121)  10.065 ms  10.062 ms  10.085 ms
 5  ae5-0-r03.inx.vie.nextlayer.net (92.60.0.244)  9.506 ms  9.503 ms  9.548 ms
 6  ae1-0-r60.inx.vie.nextlayer.net (92.60.0.233)  3.901 ms  3.579 ms  3.568 ms
 7  ae10-0-r06.nikh.ams.nextlayer.net (92.60.3.31)  21.014 ms  21.207 ms  21.008 ms
 8  amsix.tuxis.net (80.249.209.2)  22.753 ms  22.757 ms  22.766 ms
 9  pbs003.tuxis.net (185.119.31.23)  22.856 ms * *

and speeds are back up to 12 MB/second (an improvement with a factor of 3 more or less). Is that change persistent (or can you keep it persistent) or will the router loose it again once it stops laughing? Is there anything I could do on my end to improve this (talk to my ISP?)

Either way, thank you very much for your help in debugging!

 

[tuxis]

Well you tickled it well enough. I am now getting routed via nextlayer in vienna:

 1  _gateway (10.7.21.1)  0.470 ms  0.428 ms  0.424 ms
 2  *  * * * *
 3  tengig0020-1702.cr1.grazkom.at (85.237.12.1)  4.206 ms  4.476 ms  4.556 ms
 4  xe-0-0-5-r25.inx.vie.nextlayer.net (81.16.152.121)  10.065 ms  10.062 ms  10.085 ms
 5  ae5-0-r03.inx.vie.nextlayer.net (92.60.0.244)  9.506 ms  9.503 ms  9.548 ms
 6  ae1-0-r60.inx.vie.nextlayer.net (92.60.0.233)  3.901 ms  3.579 ms  3.568 ms
 7  ae10-0-r06.nikh.ams.nextlayer.net (92.60.3.31)  21.014 ms  21.207 ms  21.008 ms
 8  amsix.tuxis.net (80.249.209.2)  22.753 ms  22.757 ms  22.766 ms
 9  pbs003.tuxis.net (185.119.31.23)  22.856 ms * *

and speeds are back up to 12 MB/second (an improvement with a factor of 3 more or less). Is that change persistent (or can you keep it persistent) or will the router loose it again once it stops laughing? Is there anything I could do on my end to improve this (talk to my ISP?)

Either way, thank you very much for your help in debugging!

> Well you tickled it well enough. I am now getting routed via nextlayer in vienna.
That's what we hoped .

> Is that change persistent (or can you keep it persistent) or will the router loose it again once it stops laughing?
> Is there anything I could do on my end to improve this (talk to my ISP?)
We are 'relying' on AMS-IX routeservers, as long as Tuxis and NextLayer are connected it works like it is now. But the internet is a wild place, so the route might change at a later moment.
Regarding your 2nd question, there usually isn't much that you can do as a end-customer. It is either your partner (us) or your ISP or a combination of both that needs to do something router based to change the path.

> Either way, thank you very much for your help in debugging!
Happy to help.

 

[KB19]

@tuxis Is the free 150GB offer still valid?

Placed an order yesterday evening but still no confirmation mail or something like that.

 

[tuxis]

@tuxis Is the free 150GB offer still valid?

Placed an order yesterday evening but still no confirmation mail or something like that.

Still valid, i guess i overlooked your e-mail. You should have a e-mail in ~5min.

 

[KB19]

Still valid, i guess i overlooked your e-mail. You should have a e-mail in ~5min.

Thank you very much

 

[Fathi]

Is the beta still open ?

 

[tuxis]

Is the beta still open ?

Yes. It is not a Beta any more. Running production quite a while now. But you can still order the free 150GB version.
Just go to https://www.tuxis.nl/en/proxmox-backup-server/ and click on: Create an account. Choose the free 150GB version.

 

[Dan Medhurst]

Hi @tuxis , I tried to open an account today but got no email confirmation.. I assume it's a manual process on your end? (and it's also a weekend).. either way i look forward to the account being opened!

 

[rason]

Hey, is it possible to backup PVE virtual machines using an API key instead of the actual login and password? When logged in with my username and password, everything works fine, but I'm getting the following error when trying to back up my VM using an API key:

ERROR: VM 100 qmp command 'backup' failed - backup connect failed: command error: backup owner check failed (DB1000@pbs!api-key != DB1000@pbs)

(number in the username (after DB) and api-key names redacted for privacy)