Latest version?

[daros]

Hello,

I say some topic with versions 1.0.11, currently we use 1.08 but dont have any updates.
Where can we find the latest version?

 

[oguz]

hi,

I say some topic with versions 1.0.11, currently we use 1.08 but dont have any updates.
Where can we find the latest version?

are you enterprise subscriber? the current latest on enterprise repositories is 1.08 and the latest in pbs-no-subscription is 1.0.11

after the package has been deemed stable it should also be available in the enterprise repository

 

[daros]

hi,


are you enterprise subscriber? the current latest on enterprise repositories is 1.08 and the latest in pbs-no-subscription is 1.0.11

after the package has been deemed stable it should also be available in the enterprise repository

Hi,

We use the enterprise repository.
Was hoping for an new version because the verify is so slow in this version i was hoping for an bug fix.

 

[oguz]

i'm afraid the verify didn't get much faster in the latest versions either.

it would be interesting to know what kind of hardware you have, and more information about your storage setup. how long does it take on average for the verify to complete?

you should also check if you have "Skip verified snapshots" checked for your verify jobs

 

[daros]

HI.

In the beginning it was 15 hours, currently its more then 24 hours.

zpool status:
config:

NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sde ONLINE 0 0 0
sdf ONLINE 0 0 0
raidz1-1 ONLINE 0 0 0
sdh ONLINE 0 0 0
sdj ONLINE 0 0 0
sdi ONLINE 0 0 0
sdg ONLINE 0 0 0
special
mirror-2 ONLINE 0 0 0
nvme1n1p1 ONLINE 0 0 0
nvme0n1p1 ONLINE 0 0 0

Disks:
2x intel (boot) D3-S4510
2x PM983 nvme (500gb partition) special device
8x Toshiba 12tb sata

Harware:
Supermicro
Xenon 4210
128 ddr

root@prox-pbs:~# proxmox-backup-client benchmark
SHA256 speed: 374.59 MB/s
Compression speed: 429.84 MB/s
Decompress speed: 894.42 MB/s
AES256/GCM speed: 2198.32 MB/s
Verify speed: 259.34 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name │ Value │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ not tested │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 374.59 MB/s (19%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed │ 429.84 MB/s (57%) │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed │ 894.42 MB/s (75%) │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed │ 259.34 MB/s (34%) │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed │ 2198.32 MB/s (60%) │
└───────────────────────────────────┴────────────────────┘

 

[oguz]

In the beginning it was 15 hours, currently its more then 24 hours.

have you checked if you have that option enabled for your verify jobs? Datastore -> Verify Jobs -> double click and check the option for skipping recently verified is enabled.

SHA256 checksum computation speed │ 374.59 MB/s (19%)

is very low in this case. it might have to do with your xeon cpu.

raidz1-0 ONLINE 0 0 0

also worth noting that raidz is also typically slower than raid10

In the beginning it was 15 hours, currently its more then 24 hours.

this is expected since over time you will have more data to verify. how much of your disks are being used currently? can you post df -h?

 

[daros]

have you checked if you have that option enabled for your verify jobs? Datastore -> Verify Jobs -> double click and check the option for skipping recently verified is enabled.

Yes, I see the skipped backups.

you can apt install sysbench and run sysbench cpu run

E: Unable to locate package sysbench

this is expected since over time you will have more data to verify. how much of your disks are being used currently? can you post df -h?

storage/pbs 48T 20T 29T 41% /mnt/datastore/storage/pbs

 

[t.lamprecht]

In the beginning it was 15 hours, currently its more then 24 hours.

I mean, did the usage grow with a similar factor?

storage/pbs 48T 20T 29T 41% /mnt/datastore/storage/pbs

So 20 TiB, and a maximal verify speed (i.e., not accounting for actual IO limitation) of 259.34, lets do a simple calculation (data amount through verification bandwidth per hour)

20*1024*1024 MiB / (259.34 MiB/s * 3600s) = 22.462 h

Figuring your rather big spinner + RAID-Z won't actually deliver that IO speed, at least not permanently and not if there's other things like backup jobs going on during verification, the 24h seem actually not too far off.

I mean, you did not give any stats like amount of snapshots, what type of backup (ct, vm, host), retention period for those backups, how many days you trust the verification result and if that is actually enabled for the verification jobs, so it's hard to tell for sure, but if the verification needs to go through 20T of data then the order of magnitude for the time required seems about right.

 

[daros]

I mean, did the usage grow with a similar factor?


So 20 TiB, and a maximal verify speed (i.e., not accounting for actual IO limitation) of 259.34, lets do a simple calculation (data amount through verification bandwidth per hour)

20*1024*1024 MiB / (259.34 MiB/s * 3600s) = 22.462 h

Figuring your rather big spinner + RAID-Z won't actually deliver that IO speed, at least not permanently and not if there's other things like backup jobs going on during verification, the 24h seem actually not too far off.

I mean, you did not give any stats like amount of snapshots, what type of backup (ct, vm, host), retention period for those backups, how many days you trust the verification result and if that is actually enabled for the verification jobs, so it's hard to tell for sure, but if the verification needs to go through 20T of data then the order of magnitude for the time required seems about right.

Backups runs daily.
All vm's (54) most like 90% are Windows vm's.
Verify every 30days

30 versions
24 daily
3 weekly
3 monthly

Will an extra SSD for log and one for Cache help? Have 2 more 250gb ssd's free in the host.
The 20T is for 18 versions.

I got 8x 12tb disks, will an zfs raid10 be a lot faster?
And do i need 2 mirror with 4 stripes or 4 mirrors with 2 stripes?

EDIT:
And currently there is an sync to an second pbs, thats in its first cycle so probably that will slow don the verify process.