Advice for new installation.

[WhiteTiger]

I am new to PBS, I apologize for the trivial questions. I am reading the documentation, but in the meantime I would like some first advice.

My server is a Dell R510 with 2x480GB SDDs in RAID1; 3x600GB HDDs in RAID5; 9x300GB HDDs in RAID6.
With the Dell PERC H700 controller I am not able to view the disks individually and therefore I have to configure the RAID from the controller.

I wonder if it is worth installing PBS on SSD or on a 16GB USB stick.
If on SSD, how should I partition it?

The disks were already partitioned before, how can I initialize them to use them from scratch?

Thanks for the help and advice.

 

[Dunuin]

I wonder if it is worth installing PBS on SSD or on a 16GB USB stick.

Not sure if USB will work. Atleast PVE shouldn't be installed on a USB stick because it is writing 30GB per day. Would be intersting to know if PBS is also writing that much. But I think it should be less because there shoudn't be HA cluster config files written every minute.

PBS is by the way not designed to run on HDDs. It will work but isn't fast because PBS needs alot of IOPS because everything is split into small chunks. So you for example got 1 million small chucks and each time you run the garbadge collection or verify task it needs to go through each of the chunks.

The disks were already partitioned before, how can I initialize them to use them from scratch?

If it is like with PVE you can use the wipefs command or use a partition tool of your choice to delete all partitions.

 

[WhiteTiger]

PBS is by the way not designed to run on HDDs. It will work but isn't fast because PBS needs alot of IOPS because everything is split into small chunks. So you for example got 1 million small chucks and each time you run the garbadge collection or verify task it needs to go through each of the chunks.

Thanks for your answer.
Sorry, but I didn't understand why "PBS is by the way not designed to run on HDDs." The documentation makes no mention of it.
However on this server there are only HDDs and, as I said initially, I am forced to use the raid managed by the Dell controller because the PERC H700 does not allow direct access to individual disks. I can at most configure them in the controller as RAID0.
They are now 3x600 in RAID 5 and 9x300 in RAID6.
If I keep this configuration, then either I use them in ext4 or in ZFS RAIDZ-0.

I'm re-installing on the two mirrored SSDs.
They have a capacity of 480GB, I was wondering at the time of installation how to divide this space because I don't think everything should be assigned to PSB.
I was thinking of allocating 50GB of space and minfree for everything else.
Again, in ext4 or in ZFS RAIDZ-0.

 

[Dunuin]

Thanks for your answer.
Sorry, but I didn't understand why "PBS is by the way not designed to run on HDDs." The documentation makes no mention of it.

To quote the documentation:

• Backup storage:
  • Use only SSDs, for best results
  • If HDDs are used: Using a metadata cache is highly recommended, for example, add a ZFS special device mirror.

HDDs are only useful at reading/writing big sequential IO. So writing a big 400GB image file would be fine. But PBS will chop everything in small chunks of max 4MB. For backups of LXCs they might even be way smaller like just some KB per chunk. So you are not writing 1x 400GB file, you will write 100.000x 4MB files. And because of deduplication these chunks will be spread across all drives so what you actually get are alot of smaller random reads where HDDs are really terrible at. And stuff like the garbage collection or verify task will need to regularily open every file and check its metadata. So you will get hundreds of thousands or millions of small files that needs to be opened...

However on this server there are only HDDs and, as I said initially, I am forced to use the raid managed by the Dell controller because the PERC H700 does not allow direct access to individual disks. I can at most configure them in the controller as RAID0.
They are now 3x600 in RAID 5 and 9x300 in RAID6.
If I keep this configuration, then either I use them in ext4 or in ZFS RAIDZ-0.

Its not a good idea to use a ZFS raid0 equivalant ontop of a hardware raid. See here in the OpenZFS documentation why.

 

[WhiteTiger]

HDDs are only useful at reading/writing big sequential IO. So writing a big 400GB image file would be fine. But PBS will chop everything in small chunks of max 4MB. For backups of LXCs they might even be way smaller like just some KB per chunk.
...
Its not a good idea to use a ZFS raid0 equivalant ontop of a hardware raid. See here in the OpenZFS documentation why.

I had read the documentation.

As I wrote previously "I am forced to use the raid managed by the Dell controller because the PERC H700 does not allow direct access to individual disks. I can at most configure them in the controller as RAID0"
So I can't use ZFS, but only EXT4.

My problem remains.
The server has 3.5 12xHDDs and 2xSSDs. If I can't use it, is the solution to throw it away?
I can install PVE in it, but it's still 14 HDDs managed by the controller and for use with EXT4.
Then it only has a 6Core CPU with 16GB RAM, that's not much for PVE.

 

[Dunuin]

You could try to create 3 ZFS pools:

A) 2x SSDs as HW raid1
B) 8x 300GB HDDs as HW raid1
C) 3x 600GB HDDs as HW raid5

1.) install PBS on A but tell PBS to only use something like 32GB so the remaining capacity can be manually partitioned.
2.) create a 1.2 TB single disk ZFS on B. Partition unallocated space on A and use this partition as a special device. That way you atleast get the IOPS of 4 disks and GC should be fast because all the metadata is stored on the SSDs.
3.) create a 1.2 TB single disk ZFS on C and use it to replicate all backups from 2 to 3 to get a additional backup of your backup. Or use it as a additional (but 4 times slower) datastore.

But thats all really not optimal...

 

[ness1602]

You could of course use all that. I have PBS running on a 16GB USB stick, and works fine atm.
AS for disk, you can create raid from them,as you usually do ,and present them as individual datastores. That would work okay for standard PBS installation, but the performanse would be okay at best, but for me that is enough.

 

[WhiteTiger]

You could try to create 3 ZFS pools:
...
But thats all really not optimal...

I'm confused. Hasn't it always been said that using ZFS with RAID disks managed by hardware controllers may result in corrupted data?
That the performances are not the best, at the moment I can accept it and then soon I will buy a different server, but I cannot risk that the backups are corrupted.

 

[ness1602]

Yes, ZFS is not recommended on RAID controllers .

 

[Dunuin]

I'm confused. Hasn't it always been said that using ZFS with RAID disks managed by hardware controllers may result in corrupted data?
That the performances are not the best, at the moment I can accept it and then soon I will buy a different server, but I cannot risk that the backups are corrupted.

Yes, that why I said its not optimal...

 

[WhiteTiger]

OK, so with this new disks configuration:
2x300 HDDs for Boot and OS (RAID1)
10x300 HDDs ZFS pool (RAID-Z2 or Z3)
1x250 NVMe, but in USB 2.0, for ZIL
What is your opinion?

 

[Dunuin]

I think a slog (so ZIL on NVMe) wont help much. It is only caching sync writes and no async writes or any reads. I'm not totally sure but would guess that PBS backups are mostly async writes. And with NVMe over USB2.0 it wont be fast anyway so you would limit your complete pool write performance to the 40MB/s of the USB2.0 bandwidth.

 

[ness1602]

For PBS it is recommended to use special devices instead of zil or arc , but for that you need 2 disks.