Dedicated Proxmox Backup Server

[TwiX]

Hi !

I plan to install a dedicated PBS for all my PVE clusters.

Which is composed by :
1 Intel Xeon Silver 4216 (2,1GHz, 16C/32T)
64 Go RAM
2 sata disks for OS
10 x 4To sata disk for ZFS
2x 25Gb NICs

I also added 2x SAS SSD drives (400 Go) writes intensive.

What is the best way to configure the zfs pool in order to use SAS SSD cache (L2ARC, ZIL) ?

Thanks !

 

[aaron]

I think you will gain a lot by adding two high endurance SSDs as a mirrored vdev of type `special device` instead of / additionally to an L2ARC or ZIL.

The ZFS metadata will be stored in the special device and you can configure the dataset in which the PBS datastore is located, to store small files (I think 4kB is enough) on the special device. These are accessed much more often and having almost zero access time to them should benefit you much more.

Someone else might be able to give you a bit more details on such a setup.

 

[TwiX]

Thanks !
So you suggest using special device instead of log.

So when building the pool :

zpool create -f -o ashift=12 <pool> raidz2 <sata1> <sata2> <sata3> <sata4>...<sata10> special mirror <ssd1> <ssd2>

and then

zfs set special_small_blocks=4K <pool>

I have 3 clusters (6 nodes each), hope every backups jobs can be launched at the same time...

 

[aaron]

So you suggest using special device instead of log.

If you can have both, go for it. If you can only have one, go with the special device. AFAIK a lot of the operations of the PBS require reading the index files and metadata of the chunk files. Thus having them on SSDs (which practically have zero access time compared to spinning HDDs) should give you a nice performance boost.

Since the special device holds the pool's metadata, you need to set it up in a redundant manner. If the special device vdev fails completely, the pool will be lost.

 

[TwiX]

special mirror <ssd1> <ssd2>

Thanks, that's why I specified special mirror

Thus, is it possible to create a pool like this ?

zpool create -f -o ashift=12 <pool> raidz2 <sata1> <sata2> <sata3> <sata4>...<sata10> cache mirror <ssd1> <ssd2> log mirror <ssd1> <ssd2> special mirror <ssd1> <ssd2>

In order to put everything (zil, l2arc, special) on these 2 SAS SSDs

 

[aaron]

Two thoughts on that.

10 disks in one raidz2 does sound like a lot. I personally would size it smaller and use more raidz vdevs.

Do you plan to use the same SSDs for mirror, log and special device? You will kill the performance as you will have too many different things accessing the same disks. The L2ARC does not need to be mirrored and it can be discussed if the log really needs to be mirrored as well.

You should use dedicated SSDs for each (mirror, log, special device) instead of putting them all on the same SSD which you likely will separate with partitions.

 

[TwiX]

Thanks,

Seems that ZIL used for writes and ARC for reads
cf interesting article here :
https://linuxhint.com/configuring-zfs-cache/

so logs need to be mirrored IMHO, and ZIL is the must important matter in PBS work way.

 

[aaron]

ZIL is used not for all, but only sync writes. Checkout this article if you haven't seen it yet: https://www.servethehome.com/what-is-the-zfs-zil-slog-and-what-makes-a-good-one/

Considering that a verify and garbage collection run will need to read all the small index files, I would rather tend towards the special device giving you overall the better performance when configured that small files are written directly to it.

Though I haven't done any benchmarks and do not know if anyone else has.

 

[shelter]

This was an interesting read. Thanks for the discussion. Not to jack the thread, but I'm looking to build a dedicated PBS using ZFS as well. @aaron, do you have recommendation for high endurance SSDs? Something that's a tiny bit budget friendly? I've already lined up an Optane (the less expensive 32GB m.2 one) for my SLOG. I may kick myself later, but I'd rather not drop gobs of cash on SSDs for the special device.

@TwiX, post performance stats when you're finished with your build? Do you have any recommendations for SSDs?

 

[TwiX]

Hi,

Sure, I will

I'm about to take 2 Samsung PM1635a 400 GB (write intensive) SAS 12Gb and 10 x 4TB SATA 512n.

My pool should be a stripe of 2 raidz2 vdevs or 3 raidz with these 2 SSDs for special device (some kind of raid60 / 50).

That should be convenient.

However, I have to replicate datas to another datacenter, and honestly this issue worries me a bit...

 

[TwiX]

Hi,
I received my Dell PowerEdge R540 with :

2 x Intel(R) Xeon(R) Silver 4208 CPU @ 2.10GHz ( 16 cores / 32 threads )
64 gb RAM
LACP Bond 2 x 25 Gb with DAC cables
ZFS datastore :
RaidZ 5 x 4 TB sata disks + special device 2 x TOSHIBA KPM5XMUG400G 400gb Sas 12gbps (mirrored)
special_small_blocks=4K

I'm a little bit disappointed by the results :

Uploaded 1019 chunks in 5 seconds.
Time per request: 4913 microseconds.
TLS speed: 853.57 MB/s
SHA256 speed: 415.83 MB/s
Compression speed: 607.41 MB/s
Decompress speed: 984.24 MB/s
AES256/GCM speed: 2445.36 MB/s
Verify speed: 293.74 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 853.57 MB/s (69%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 415.83 MB/s (21%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 607.41 MB/s (81%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 984.24 MB/s (82%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 293.74 MB/s (39%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2445.36 MB/s (67%) │
└───────────────────────────────────┴────────────────────┘

I don't know if it's ok or not but the weird thing is that the benchmark seems to not take care of the datastore performance.
Results are exactly the same with a vdev of 2 SAS SSD only.

 

[TwiX]

Here is a PBS backup for a linux VM :
datastore : 1 x fast SAS SSD 400 Gb

I thought it should be better


Again, with a single SATA disk for my datastore, I have the same results.

On both servers, all cores are not at 100 % (some few cores at 7-10 % max during few seconds) and no iowaits.

 

[fabian]

Hi,
I received my Dell PowerEdge R540 with :

2 x Intel(R) Xeon(R) Silver 4208 CPU @ 2.10GHz ( 16 cores / 32 threads )
64 gb RAM
LACP Bond 2 x 25 Gb with DAC cables
ZFS datastore :
RaidZ 5 x 4 TB sata disks + special device 2 x TOSHIBA KPM5XMUG400G 400gb Sas 12gbps (mirrored)
special_small_blocks=4K

I'm a little bit disappointed by the results :

Uploaded 1019 chunks in 5 seconds.
Time per request: 4913 microseconds.
TLS speed: 853.57 MB/s
SHA256 speed: 415.83 MB/s
Compression speed: 607.41 MB/s
Decompress speed: 984.24 MB/s
AES256/GCM speed: 2445.36 MB/s
Verify speed: 293.74 MB/s
┌───────────────────────────────────┬────────────────────┐
│ Name                              │ Value              │
╞═══════════════════════════════════╪════════════════════╡
│ TLS (maximal backup upload speed) │ 853.57 MB/s (69%)  │
├───────────────────────────────────┼────────────────────┤
│ SHA256 checksum computation speed │ 415.83 MB/s (21%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 compression speed    │ 607.41 MB/s (81%)  │
├───────────────────────────────────┼────────────────────┤
│ ZStd level 1 decompression speed  │ 984.24 MB/s (82%)  │
├───────────────────────────────────┼────────────────────┤
│ Chunk verification speed          │ 293.74 MB/s (39%)  │
├───────────────────────────────────┼────────────────────┤
│ AES256 GCM encryption speed       │ 2445.36 MB/s (67%) │
└───────────────────────────────────┴────────────────────┘

I don't know if it's ok or not but the weird thing is that the benchmark seems to not take care of the datastore performance.
Results are exactly the same with a vdev of 2 SAS SSD only.

the benchmark does not actually write anything to disk, so it's not surprising that it's not affected by changing the storage. it's benchmarking how fast your system does the actual backup operations (calculating chunk metadata, compression/encryption/transfer overhead).

 

[fabian]

Here is a PBS backup for a linux VM :
datastore : 1 x fast SAS SSD 400 Gb

I thought it should be better
View attachment 20693

Again, with a single SATA disk for my datastore, I have the same results.

On both servers, all cores are not at 100 % (some few cores at 7-10 % max during few seconds) and no iowaits.

you'd need to post a full log to give more detail. what kind of results did you expect? please keep in mind that the backup cannot be faster than reading from the backed-up storage..

 

[TwiX]

I know

I first tried with lvm-thin backed-up storage (SSD) and then I move the disk image to ceph which is pretty fast (1000 write / 1800 read Mb/s with rados test benchmark)
=> same results.

What kind of logs do you need ?

 

[TwiX]

you'd need to post a full log to give more detail. what kind of results did you expect? please keep in mind that the backup cannot be faster than reading from the backed-up storage..

For the perf I expect, here is a benchmarck for a virtualized PBS on vmware workstation (amd ryzen 5 3600) :
4 G ram and / 2 vcpus :

Uploaded 894 chunks in 5 seconds.
Time per request: 5603 microseconds.
TLS speed: 748.50 MB/s
SHA256 speed: 2047.96 MB/s
Compression speed: 785.10 MB/s
Decompress speed: 1362.86 MB/s
AES256/GCM speed: 3515.72 MB/s
Verify speed: 819.68 MB/s
┌───────────────────────────────────┬─────────────────────┐
│ Name                              │ Value               │
╞═══════════════════════════════════╪═════════════════════╡
│ TLS (maximal backup upload speed) │ 748.50 MB/s (61%)   │
├───────────────────────────────────┼─────────────────────┤
│ SHA256 checksum computation speed │ 2047.96 MB/s (101%) │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 compression speed    │ 785.10 MB/s (104%)  │
├───────────────────────────────────┼─────────────────────┤
│ ZStd level 1 decompression speed  │ 1362.86 MB/s (114%) │
├───────────────────────────────────┼─────────────────────┤
│ Chunk verification speed          │ 819.68 MB/s (108%)  │
├───────────────────────────────────┼─────────────────────┤
│ AES256 GCM encryption speed       │ 3515.72 MB/s (96%)  │
└───────────────────────────────────┴─────────────────────┘

Ouch !!

My VM did it better !

 

[fabian]

yeah, that just means that your Ryzen performs better for those benchmarks (keep in mind it has 1.5-2x the base/boosted clock speed compared to your Intel CPU!). some intel CPUs are also notoriously bad with regards to hashing performance, e.g. my intel i9-10900x only manages ~600MB/s SHA256 computation and 410MB/s chunk verification speed.

the full backup log would give some more numbers, but the read speed there (a bit over 100 MiB/s) looks like the limit for that storage/VM combination and the backup won't magically perform better.

 

[TwiX]

Thanks,

So the good news is I should able to backup few nodes at the same time considering I have lots of cpus...I guess

The full backup log if you're still interested :

Proxmox
Virtual Environment 6.2-12
Search
Node 'th3-prox-02'
Show details
Server View
Logs
()
INFO: starting new backup job: vzdump 101 --mode snapshot --remove 0 --node th3-prox-06 --storage PBS_TH3
INFO: Starting Backup of VM 101 (qemu)
INFO: Backup started at 2020-10-22 10:08:53
INFO: status = running
INFO: VM Name: tco-kube-01
INFO: include disk 'scsi0' 'kvm_pool:vm-101-disk-0' 32G
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating Proxmox Backup Server archive 'vm/101/2020-10-22T08:08:53Z'
INFO: issuing guest-agent 'fs-freeze' command
INFO: issuing guest-agent 'fs-thaw' command
INFO: started backup task 'd39b0b5b-b76f-4b00-bc9b-563cf91f43fc'
INFO: resuming VM again
INFO: scsi0: dirty-bitmap status: existing bitmap was invalid and has been cleared
INFO:   1% (400.0 MiB of 32.0 GiB) in  3s, read: 133.3 MiB/s, write: 129.3 MiB/s
INFO:   2% (812.0 MiB of 32.0 GiB) in  6s, read: 137.3 MiB/s, write: 94.7 MiB/s
INFO:   3% (1.1 GiB of 32.0 GiB) in  9s, read: 112.0 MiB/s, write: 105.3 MiB/s
INFO:   4% (1.4 GiB of 32.0 GiB) in 12s, read: 109.3 MiB/s, write: 109.3 MiB/s
INFO:   5% (1.8 GiB of 32.0 GiB) in 15s, read: 112.0 MiB/s, write: 112.0 MiB/s
INFO:   6% (2.1 GiB of 32.0 GiB) in 18s, read: 102.7 MiB/s, write: 102.7 MiB/s
INFO:   7% (2.4 GiB of 32.0 GiB) in 21s, read: 114.7 MiB/s, write: 114.7 MiB/s
INFO:   8% (2.7 GiB of 32.0 GiB) in 24s, read: 96.0 MiB/s, write: 93.3 MiB/s
INFO:   9% (3.0 GiB of 32.0 GiB) in 27s, read: 105.3 MiB/s, write: 104.0 MiB/s
INFO:  10% (3.3 GiB of 32.0 GiB) in 30s, read: 112.0 MiB/s, write: 102.7 MiB/s
INFO:  11% (3.6 GiB of 32.0 GiB) in 33s, read: 109.3 MiB/s, write: 100.0 MiB/s
INFO:  12% (4.0 GiB of 32.0 GiB) in 36s, read: 114.7 MiB/s, write: 110.7 MiB/s
INFO:  13% (4.3 GiB of 32.0 GiB) in 39s, read: 114.7 MiB/s, write: 89.3 MiB/s
INFO:  14% (4.7 GiB of 32.0 GiB) in 42s, read: 141.3 MiB/s, write: 97.3 MiB/s
INFO:  15% (5.0 GiB of 32.0 GiB) in 45s, read: 102.7 MiB/s, write: 102.7 MiB/s
INFO:  16% (5.4 GiB of 32.0 GiB) in 48s, read: 116.0 MiB/s, write: 105.3 MiB/s
INFO:  17% (5.7 GiB of 32.0 GiB) in 51s, read: 124.0 MiB/s, write: 90.7 MiB/s
INFO:  19% (6.1 GiB of 32.0 GiB) in 54s, read: 124.0 MiB/s, write: 100.0 MiB/s
INFO:  20% (6.5 GiB of 32.0 GiB) in 57s, read: 133.3 MiB/s, write: 94.7 MiB/s
INFO:  21% (7.0 GiB of 32.0 GiB) in  1m  0s, read: 160.0 MiB/s, write: 77.3 MiB/s
INFO:  22% (7.3 GiB of 32.0 GiB) in  1m  3s, read: 124.0 MiB/s, write: 114.7 MiB/s
INFO:  23% (7.6 GiB of 32.0 GiB) in  1m  6s, read: 106.7 MiB/s, write: 106.7 MiB/s
INFO:  24% (8.0 GiB of 32.0 GiB) in  1m  9s, read: 113.3 MiB/s, write: 113.3 MiB/s
INFO:  25% (8.3 GiB of 32.0 GiB) in  1m 12s, read: 117.3 MiB/s, write: 114.7 MiB/s
INFO:  27% (8.6 GiB of 32.0 GiB) in  1m 15s, read: 116.0 MiB/s, write: 114.7 MiB/s
INFO:  28% (9.1 GiB of 32.0 GiB) in  1m 19s, read: 104.0 MiB/s, write: 104.0 MiB/s
INFO:  29% (9.4 GiB of 32.0 GiB) in  1m 22s, read: 110.7 MiB/s, write: 105.3 MiB/s
INFO:  30% (9.7 GiB of 32.0 GiB) in  1m 25s, read: 117.3 MiB/s, write: 117.3 MiB/s
INFO:  31% (10.1 GiB of 32.0 GiB) in  1m 28s, read: 114.7 MiB/s, write: 114.7 MiB/s
INFO:  32% (10.4 GiB of 32.0 GiB) in  1m 31s, read: 108.0 MiB/s, write: 106.7 MiB/s
INFO:  33% (10.7 GiB of 32.0 GiB) in  1m 34s, read: 108.0 MiB/s, write: 108.0 MiB/s
INFO:  34% (11.0 GiB of 32.0 GiB) in  1m 37s, read: 118.7 MiB/s, write: 118.7 MiB/s
INFO:  35% (11.4 GiB of 32.0 GiB) in  1m 40s, read: 133.3 MiB/s, write: 121.3 MiB/s
INFO:  36% (11.7 GiB of 32.0 GiB) in  1m 43s, read: 104.0 MiB/s, write: 104.0 MiB/s
INFO:  37% (12.1 GiB of 32.0 GiB) in  1m 46s, read: 110.7 MiB/s, write: 110.7 MiB/s
INFO:  38% (12.4 GiB of 32.0 GiB) in  1m 49s, read: 105.3 MiB/s, write: 105.3 MiB/s
INFO:  39% (12.7 GiB of 32.0 GiB) in  1m 52s, read: 97.3 MiB/s, write: 97.3 MiB/s
INFO:  40% (13.0 GiB of 32.0 GiB) in  1m 55s, read: 104.0 MiB/s, write: 104.0 MiB/s
INFO:  41% (13.3 GiB of 32.0 GiB) in  1m 58s, read: 113.3 MiB/s, write: 102.7 MiB/s
INFO:  42% (13.6 GiB of 32.0 GiB) in  2m  1s, read: 106.7 MiB/s, write: 106.7 MiB/s
INFO:  43% (13.9 GiB of 32.0 GiB) in  2m  4s, read: 108.0 MiB/s, write: 97.3 MiB/s
INFO:  44% (14.2 GiB of 32.0 GiB) in  2m  7s, read: 108.0 MiB/s, write: 98.7 MiB/s
INFO:  45% (14.6 GiB of 32.0 GiB) in  2m 10s, read: 130.7 MiB/s, write: 69.3 MiB/s
INFO:  46% (15.0 GiB of 32.0 GiB) in  2m 13s, read: 114.7 MiB/s, write: 106.7 MiB/s
INFO:  47% (15.3 GiB of 32.0 GiB) in  2m 16s, read: 112.0 MiB/s, write: 108.0 MiB/s
INFO:  48% (15.6 GiB of 32.0 GiB) in  2m 19s, read: 108.0 MiB/s, write: 108.0 MiB/s
INFO:  49% (15.9 GiB of 32.0 GiB) in  2m 22s, read: 113.3 MiB/s, write: 106.7 MiB/s
INFO:  50% (16.3 GiB of 32.0 GiB) in  2m 25s, read: 116.0 MiB/s, write: 102.7 MiB/s
INFO:  51% (16.6 GiB of 32.0 GiB) in  2m 28s, read: 105.3 MiB/s, write: 102.7 MiB/s
INFO:  52% (16.9 GiB of 32.0 GiB) in  2m 31s, read: 109.3 MiB/s, write: 97.3 MiB/s
INFO:  53% (17.3 GiB of 32.0 GiB) in  2m 34s, read: 121.3 MiB/s, write: 102.7 MiB/s
INFO:  54% (17.6 GiB of 32.0 GiB) in  2m 37s, read: 113.3 MiB/s, write: 96.0 MiB/s
INFO:  55% (17.9 GiB of 32.0 GiB) in  2m 40s, read: 105.3 MiB/s, write: 89.3 MiB/s
INFO:  56% (18.2 GiB of 32.0 GiB) in  2m 43s, read: 113.3 MiB/s, write: 78.7 MiB/s
INFO:  58% (18.6 GiB of 32.0 GiB) in  2m 46s, read: 133.3 MiB/s, write: 104.0 MiB/s
INFO:  59% (19.0 GiB of 32.0 GiB) in  2m 49s, read: 124.0 MiB/s, write: 84.0 MiB/s
INFO:  60% (19.4 GiB of 32.0 GiB) in  2m 52s, read: 130.7 MiB/s, write: 108.0 MiB/s
INFO:  61% (19.7 GiB of 32.0 GiB) in  2m 55s, read: 116.0 MiB/s, write: 84.0 MiB/s
INFO:  62% (20.0 GiB of 32.0 GiB) in  2m 58s, read: 113.3 MiB/s, write: 93.3 MiB/s
INFO:  63% (20.4 GiB of 32.0 GiB) in  3m  1s, read: 126.7 MiB/s, write: 84.0 MiB/s
INFO:  64% (20.7 GiB of 32.0 GiB) in  3m  4s, read: 112.0 MiB/s, write: 101.3 MiB/s
INFO:  66% (21.3 GiB of 32.0 GiB) in  3m  7s, read: 186.7 MiB/s, write: 72.0 MiB/s
INFO:  69% (22.3 GiB of 32.0 GiB) in  3m 10s, read: 354.7 MiB/s, write: 0 B/s
INFO:  71% (22.8 GiB of 32.0 GiB) in  3m 13s, read: 173.3 MiB/s, write: 84.0 MiB/s
INFO:  73% (23.4 GiB of 32.0 GiB) in  3m 16s, read: 209.3 MiB/s, write: 61.3 MiB/s
INFO:  76% (24.4 GiB of 32.0 GiB) in  3m 19s, read: 338.7 MiB/s, write: 0 B/s
INFO:  77% (24.8 GiB of 32.0 GiB) in  3m 22s, read: 136.0 MiB/s, write: 85.3 MiB/s
INFO:  79% (25.3 GiB of 32.0 GiB) in  3m 25s, read: 169.3 MiB/s, write: 82.7 MiB/s
INFO:  82% (26.3 GiB of 32.0 GiB) in  3m 28s, read: 334.7 MiB/s, write: 0 B/s
INFO:  83% (26.8 GiB of 32.0 GiB) in  3m 31s, read: 166.7 MiB/s, write: 73.3 MiB/s
INFO:  84% (27.1 GiB of 32.0 GiB) in  3m 34s, read: 106.7 MiB/s, write: 101.3 MiB/s
INFO:  87% (28.1 GiB of 32.0 GiB) in  3m 37s, read: 338.7 MiB/s, write: 1.3 MiB/s
INFO:  89% (28.7 GiB of 32.0 GiB) in  3m 40s, read: 217.3 MiB/s, write: 53.3 MiB/s
INFO:  91% (29.2 GiB of 32.0 GiB) in  3m 43s, read: 165.3 MiB/s, write: 93.3 MiB/s
INFO:  94% (30.3 GiB of 32.0 GiB) in  3m 46s, read: 356.0 MiB/s, write: 0 B/s
INFO:  96% (30.8 GiB of 32.0 GiB) in  3m 49s, read: 186.7 MiB/s, write: 72.0 MiB/s
INFO:  99% (31.9 GiB of 32.0 GiB) in  3m 52s, read: 365.3 MiB/s, write: 12.0 MiB/s
INFO: 100% (32.0 GiB of 32.0 GiB) in  3m 53s, read: 120.0 MiB/s, write: 4.0 MiB/s
INFO: backup is sparse: 9.47 GiB (29%) total zero data
INFO: backup was done incrementally, reused 11.27 GiB (35%)
INFO: transferred 32.00 GiB in 233 seconds (140.6 MiB/s)
INFO: Finished Backup of VM 101 (00:03:54)
INFO: Backup finished at 2020-10-22 10:12:47
INFO: Backup job finished successfully
TASK OK

 

[TwiX]

Hi,

I finally put this dedicated server on production, and it runs well.

for example, each day at 21:00 I run backup job for all PVE environnements at the same time.
Transfer rate is approaching 500 Mbytes/sec, Cpu utilisation ~50% and bandwidth 2.5-3 Gbps


I'm able to backup ~200 VMs in 30 minutes...And I think it could do more.
Very nice solution !

However, I noticed that I have continuous writes without running tasks (8-9 Mbytes/sec) - Is it normal ?

 

[fabian]

maybe logs and status queries by PVE nodes using it as storage?