PBS installation layout

[adambenk]

Hello guys,
I have a HP Prolliant server with 4x2TB HDDs with Hardware raid10, Ubuntu 20.04 installed on it, and is a NFS backup for our Proxmox VE servers.
PBS is awesome, an i want to try it!
What is the recomended installation layout in this situation:
1. Installing PBS on these 4 HDDs (raid controller in JBOD mode) in ZFS RAID10 + 2SSDs in "special device" mirror (log and cache devices are not efficients)
After installation, creating a Datatstore on the root fs.

2.Installing PBS on a single SSD, and after that creating(zfs raid10) on the 4 HDDs and adding Datastore from the GUI?
In this case, is also recomended a special device?

Thanks for any help!
Adam

 

[Alwin]

Hmpf... beware that the JBOD mode is not a passthrough or HBA and it might interfere. Besides that, a special device is recommended as verify & GC will definitely benefit.

 

[adambenk]

Thank you Alwin

The server is a Proliant ML350p Gen8, and the raid card unfortunately doesn't now HBA mode, the only way is hardware raid10.
When a stand-alone PVE server pushing an incremental backup to PBS, there is a major speed or performance difference on PBS, when it has zfs raid10 or hardware raid10?
Incremental backup is working even if PBS doesn't have zfs? The perfect situation is when both (PVE and PBS) have zfs?

Adam

 

[guletz]

The server is a Proliant ML350p Gen8, and the raid card unfortunately doesn't now HBA mode, the only way is hardware raid10.

Hi,

In my own opinion it will be better to create a 4 x raid0(each raid0 will have a single hdd). Then use this 4 raid0 deviceas for your zfs pool. In this case raid10 parity is done by zfs and not by the hw controller.

Good luck / Bafta !

 

[guletz]

Incremental backup is working even if PBS doesn't have zfs?

Yes. Could be ext4.

 

[guletz]

1. Installing PBS on these 4 HDDs (raid controller in JBOD mode) in ZFS RAID10 + 2SSDs in "special device" mirror (log and cache devices are not efficients)
After installation, creating a Datatstore on the root fs.

2.Installing PBS on a single SSD, and after that creating(zfs raid10) on the 4 HDDs and adding Datastore from the GUI?

Each cases have pro/con, like;

1.

- pro: better zfs performance
- con: possible future grub problems
- con: if your ssd's do not have the necessary size it could be possible to not have sifficient free space to write more data

2.

pro: less potential problems in future (grub and so on)
con: less speed compared with 1

Note: better will be in case 2. to create a raid1 with 2 ssd and 4 x raid0(for zfs pool) as I already write about.

Good luck / Bafta !

 

[Alwin]

When a stand-alone PVE server pushing an incremental backup to PBS, there is a major speed or performance difference on PBS, when it has zfs raid10 or hardware raid10?

Seen separately, it depends on the hardware used. ZFS uses memory of the host, while a RAID controller uses its own. Besides that it is a general question of dedicating separate hardware resources or using the systems resources.

Incremental backup is working even if PBS doesn't have zfs? The perfect situation is when both (PVE and PBS) have zfs?

PBS runs independently of the underlying filesystem. We recommend ZFS since it protects against bit rot and other forms of data corruption.

In my own opinion it will be better to create a 4 x raid0(each raid0 will have a single hdd).

Never do that! ZFS on RAID controller alone will cause trouble. Exposing the disks as RAID0 might look promising at first, but since the controller hides the true nature of the disks and still keeps its cache active the disks can run into cache starvation or other forms of latency spikes that can degrade a zpool easily.

 

[guletz]

Never do that! ZFS on RAID controller alone will cause trouble

Hi @Alwin ,

In have guess that the @adambenk , have ONLY HW Raid available, and no HBA is posssible! If HBA is an option, then yes, I agree with you!

Good luck / Bafta!

 

[Alwin]

In have guess that the @adambenk , have ONLY HW Raid available, and no HBA is posssible! If HBA is an option, then yes, I agree with you!

Thanks for agreeing. But just because there came a RAID controller with the box doesn't mean that it can't be replaced. And if there is really, really no way of replacing it, then just use the RAID controller and not ZFS.

 

[adambenk]

@Alwin and @guletz
Thank you very much for your help !!!

 

[adambenk]

Can i have a last question.
There is a form to calculate the size of the special device?
For example by zfs memory usage: zfs_arc_max= 50% of the system memory.

Thanks!

 

[Alwin]

There is a form to calculate the size of the special device?

No that I know of. And it greatly depends on how much metadata will be accumulated. But the special device will spillover to the other zvol in the pool if full.

Special Allocation Class
The allocations in the special class are dedicated to specific block types. By default this includes all metadata, the indirect blocks of user data, and any deduplication tables. The class can also be provisioned to accept small file

A pool must always have at least one normal (non-dedup/special) vdev before other devices can be assigned to the special class. If the special class becomes full, then allocations intended for it will spill back into the normal class.

 

[guletz]

There is a form to calculate the size of the special device?

Hi again,

As a aproximation, could be 0.3% pool size. More detailis could be find here:

https://forum.level1techs.com/t/zfs-metadata-special-device-z/159954

Good luck / Bafta !

 

[adambenk]

@guletz
Thank you / Multumesc!

 

[guletz]

Thank you / Multumesc!

With pleasure / Cu placere !