PBS ZFS disk speed thoughts option

SHA256 speed: 479.43 MB/s
Compression speed: 488.70 MB/s
Decompress speed: 804.15 MB/s
AES256/GCM speed: 3460.90 MB/s
Verify speed: 298.84 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ not tested │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 479.43 MB/s (24%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 488.70 MB/s (65%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 804.15 MB/s (67%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 298.84 MB/s (39%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 3460.90 MB/s (95%) │
└───────────────────────────────────┴────────────────────┘

Did the test on my first machine also. About the same
 
AkUsP said:
The speed is the same. My PBS1 och PBS2 are sister machines.
Click to expand...
Then the TLS speed should be your bottleneck, since the PBS storage is too slow to write above 200MB/s. That is pretty bad for 2-4MB chunks IMHO.
But to be expected with a 6 wide RAIDZ2. You could switch to mirrors, that way you should be able to get 500MB/s, at which point other bottlenecks should arise like your CPU.
 
IsThisThingOn said:
Run the benchmark on PVE with PBS as destination.
Then run the benchmark again on PBS.

Post both results
Click to expand...

Ok will try tomorrow. What would the command look like to throw in on the PVE?


IsThisThingOn said:
Then the TLS speed should be your bottleneck, since the PBS storage is too slow to write above 200MB/s. That is pretty bad for 2-4MB chunks IMHO.
But to be expected with a 6 wide RAIDZ2. You could switch to mirrors, that way you should be able to get 500MB/s, at which point other bottlenecks should arise like your CPU.
Click to expand...

So you are suggesting making 1+1 mirror with 2 disks. Then throw all the 5 mirrors in a VDEV that in my world "looks" like a HW RAID10.
I noticed I can make the mirrors in the web gui. But is it correct that the VDEV parring still need to be done i ZFS tool in CLI ?
 
AkUsP said:
Ok will try tomorrow. What would the command look like to throw in on the PVE?
Click to expand...
proxmox-backup-client benchmark --repository mybackupstore

For me, that was something like this:
Code: 
proxmox-backup-client benchmark --repository 'test@pbs@[2001:XXXXXXXXXXXX::f691]:8007:poolBackup'

AkUsP said:
So you are suggesting making 1+1 mirror with 2 disks. Then throw all the 5 mirrors in a VDEV that in my world "looks" like a HW RAID10.
Click to expand...
Correct, I suggest you create multiple mirrors inside the same vdev. So basically a stripe over all the mirrors. Not sure how that works in the proxmox GUI to be honest. Works like this in CLI

Code: 
zpool create backuppool mirror /dev/disk/by-id/ata-DISK1 /dev/disk/by-id/ata-DISK2 mirror /dev/disk/by-id/ata-DISK3 /dev/disk/by-id/ata-DISK4
 
IsThisThingOn said:
proxmox-backup-client benchmark --repository mybackupstore

For me, that was something like this:
Code: 
proxmox-backup-client benchmark --repository 'test@pbs@[2001:XXXXXXXXXXXX::f691]:8007:poolBackup'


Correct, I suggest you create multiple mirrors inside the same vdev. So basically a stripe over all the mirrors. Not sure how that works in the proxmox GUI to be honest. Works like this in CLI

Code: 
zpool create backuppool mirror /dev/disk/by-id/ata-DISK1 /dev/disk/by-id/ata-DISK2 mirror /dev/disk/by-id/ata-DISK3 /dev/disk/by-id/ata-DISK4
Click to expand...

2001:XXXXXXXXXXXX::f691 is that your IPV6 ?

After creating the VDEV pools using CLI. Will Proxmox GUI still show then with all info and stuff?
 
The PVE GUI will be fine for this, and that is how I created my six-drive RAID10 pool.

Assuming your PBS datastore is added to the PVE host, you should be able to just run this command I mentioned previously:

proxmox-backup-client benchmark --server yourpbsserverhere --repository yourdatastorenamehere
 
Cant get it right..

if we say Im in shell on one of me PVE:
My user on my pbs is jonnyknox@pbs
My IP on my PBS 192.168.1.100:8007
My datastore on PBS is called bigvolume

How should it look then?
 
If you've added your PBS server to the PVE host via the GUI, then you can use my example.

If not, use IsThisThingOn's example with the username/etc. since the PVE host doesn't know where to go.

Code: 
--repository <string>
             Repository URL: [[auth-id@]server[:port]:]datastore

So:

Code: 
proxmox-backup-client benchmark --repository jonnyknox@pbs@192.168.1.100:8007:bigvolume
 
Last edited:
J-Rod said:
If you've added your PBS server to the PVE host via the GUI, then you can use my example.

If not, use IsThisThingOn's example with the username/etc. since the PVE host doesn't know where to go.

Code: 
--repository <string>
             Repository URL: [[auth-id@]server[:port]:]datastore

So:

Code: 
proxmox-backup-client benchmark --repository jonnyknox@pbs@192.168.1.100:8007:bigvolume
Click to expand...

The PBS is added as trusted storage on the PVE cluster.

When i run the command with :8007 i says "error" cant find path.
When I remove the :8007, it asks for the password, as if the PBS user was not added on the PVE cluster.
I then enter the password for the PBS user. It then do something for 1-2 seconds. Then nothing.

Strange...
 
proxmox-backup-client benchmark --repository 'test@pbs@[2001:XXXXXXXXXXXX::f691]:8007:poolBackup'
Click to expand...
proxmox-backup-client benchmark --repository jonnyknox@pbs@192.168.1.100:8007:bigvolume
Click to expand...
Not sure, but is it maybe just the missing ' ?

J-Rod said:
Are you sure the user is @PBS and not @pam?
Click to expand...
That could also be the case. I created the user on pbs, so that is why I am using that. I also just gave it full admin rights and deleted the user afterwards.
 
Not sure what to say, as it works fine for me. Even if you create an @PBS user account and don't give it permissions to the datastore, the tool is able to detect that and throws the appropriate error.

Firewall rule somewhere?
 
J-Rod said:
Not sure what to say, as it works fine for me. Even if you create an @PBS user account and don't give it permissions to the datastore, the tool is able to detect that and throws the appropriate error.

Firewall rule somewhere?
Click to expand...

No firewall. Next week I will show the command to a friend who is more proxmox capable than me. Lets see what he can do.

I want to thank everyone for the input here. Been to very very good help. Awsome forum.

After learning more and more about ZFS behavior a.s.o. I have decided to go for 3x raidz1 with 3x HDD in each. Then spann then in a single VDEV. So 9x5TB disks with usable 30 of the 45TB. With the spaning to mitigate most of the I/O issue of the ”Spinning-Rust”.
 
Just nitpicking - to avoid establishing "wrong" terms:
AkUsP said:
Then spann then in a single VDEV.
Click to expand...
Each of those groups (of three) disks result in a vdev. Any (practical) number of vdevs may be put into one pool. All vdevs in one pool are striped.
 
AkUsP said:
So 9x5TB disks with usable 30 of the 45TB.
Click to expand...
Keep in mind that the usable space will be less as ZFS uses some space "for its own".

Then there is the 80%-rule: for good performance a pool should not get filled up more than 80%. Above 90% IO-operation may get really slow. (( And reaching 100% induces pain as you need to actually fight to free up space ;-) Read about "Quota" to avoid this by any means. ))
 
UdoB said:
Keep in mind that the usable space will be less as ZFS uses some space "for its own".

Then there is the 80%-rule: for good performance a pool should not get filled up more than 80%. Above 90% IO-operation may get really slow. (( And reaching 100% induces pain as you need to actually fight to free up space ;-) Read about "Quota" to avoid this by any means. ))
Click to expand...

Yea, well I dont see ZFS worse than any other filesystem in stealing storage for itself.
Today i got 6x5TB (30TB disk) that gives my in a RAIDZ2 19,41TB So thats just 600GB that is on the loose! ;-)

So I guessed that a 3x5TB RAIDZ1 should give me 9,6TB-ish ? So 9,6TB x 3 = 28,8TB To more more precise ? Assuming that the pool itself does not take storage?

And ofc I understand the 80% rule, I got more by the 50-60% rule to be sure, just like how you plan your ATX PSU to be @ its best to be quiet for your gaming pc :-)

I have today 9,11TB used in my PBS and if my nodes fills up its storage, I think im looking at about 18-19TB backup needed. So im in the 66% range.
If my cluster should need to go beyond my today 4 nodes, other funding will be released, so then also the PBS machines can get new disk, no prob

The issue why I am so keen to use my army of spinning-rust is because I moved from VMware, with intent to keep the hardware and just change the hypervisor. In my VMware cluster, I acctualy had decomissioned the 5TB 5400disks to less important stuff. And was running backup on 2,5" SSD.
But, when moving to Proxmox, when I saw what an masterpiece PBS was, I wanted 2 of those securing my cluster, and I dident have any disks to use.
The SSD are still in use in paralell backupsystem that I havent changed, that doesent concern the cluster itself. So I had to invent the wheel, using what I had. So an army of spinning-rust came to the rescue!

Proxmox PBS is the best thing since sliced bread!
 
  • Like
Reactions: UdoB
You must log in or register to reply here.
Top Bottom
Back