Update on our PBS setups

[tuxis]

Hi,

I thought it would be nice to give you guys an update on our PBS setups, so I've written a blogpost about it. If you have any questions, let me know.

https://www.tuxis.nl/proxmox-backup-server-rocks/

 

[Dunuin]

Great that you use 3-way mirrors. I don't see alot people using them because of the costs, even if a raidz2/3 is slower or a 2-way mirror less reliable.

 

[tuxis]

With large disks like these, the rebuild times are too long to depend on a single replica. Thus, you need three-way mirrors. Each machine also has a hot spare.

 

[leesteken]

Thank you for sharing. Speaking of performance (no complaints whatsoever!): can users help spread out the load for the machines by using or avoiding certain days or times for jobs?

 

[tuxis]

No. Backing up goes fine, we hardly see any issues with that. We do reconfigure verificationjobs (which we feel we don't actually need) or pruning jobs if we feel that they cause too much load during peak backup times.

 

[tjk]

Hi,

I thought it would be nice to give you guys an update on our PBS setups, so I've written a blogpost about it. If you have any questions, let me know.

https://www.tuxis.nl/proxmox-backup-server-rocks/

Amazing read, thank you for sharing!

What are your vdev configs? I also notice no zil and no l2arc, just ssd as special device for metadata.

 

[Dunuin]

I also notice no zil and no l2arc, just ssd as special device for metadata.

I guess a SLOG won't make much sense as PBS is primirily using async writes? And a L2ARC only makes sense if you got files that are so big they won't fit in the way faster RAM (where you get less of when using L2ARC) but you need to access them alot. I guess thats not that useful for a big backup server that just stores 4MB chunks that nearly never get read again except by maintaince tasks. L2ARC for caching metadata would be useful, but not when you already store it on a special device.

Did you test if increasing the recordsize or enableing relatime shows a noticable difference?

 

[tuxis]

What are your vdev configs? I also notice no zil and no l2arc, just ssd as special device for metadata.

Like the article says, stacked three-way mirrors. Indeed no ZIL or L2ARC as the dataset is too large for L2ARC and the writes are not synchonous enough for ZIL.

We have enabled relatime, which makes a real difference for GC & Pruning. GC & Pruning mainly hits the Nvme special vdevs, not so much the spinning disks.

 

[conceptual_continuity]

Dear Tuxis crew,
regarding the special device SSD VDEVs - how full do these get? Did they fill up / overflow into the standard spinning rust VDEVs?

Thanks

 

[Dunuin]

If I remember right rule of thumb is about 0.3% of the size of your data. And in case a special device gets over 75% full metadata will spill over onto the HDDs so the pool still stays functonal.

 

[tuxis]

If I remember right rule of thumb is about 0.3% of the size of your data. And in case a special device gets over 75% full metadata will spill over onto the HDDs so the pool still stays functonal.

Yes, about 0.5% of the active data in the pool.