Recommended hardware

[The Tech Guy]

Hi,

I have decided to give PBS a try, but have a bit of trouble choosing hardware.

As ZFS storage, I have chosen to use 4 disks in Raid-Z, but I am a little in doubt about the rest.

I have the option to connect 2 SSDs and an NVMe to the motherboard, but I have a little doubt if it will be best to use the two SSDs for a raid 1 to OS disk and then put an NVMe to ZFS cache or is it a bad idea?

There is also a free PCIx port if more NVMe disks are needed?

If this is the above solution I need then how do I figure out what the size of the cache disk should be?

 

[fireon]

It depends on what and how much you want to secure. An SSD cache does not help you with backup. In general, I only use Raid10 in productive operation, also in backup. Everything else is way too slow. Now of course it can be that you don't have that much data and one RaidZ might be enough. Proxmox itself is good when it runs on the SSD's. Since I have already noticed that the web interface no longer reacts at all when the server is under load during the backup.

CPU... yes 8 Cores Minium, 32GB Memory Minimum.

Tell me how much data is involved and how often you would like to back it up.

 

[aaron]

Since the PBS does need to touch the metadata of the files (indexes, chunks, ...) a lot you can use the SSDs as special device vdevs. They will store all the metadata and thus certain PBS operations get a LOT faster. You could also set the ZFS dataset which is used for the PBS datastore to store small files up to a few kbyte directly on the special device vdev (SSDs) instead of the slower HDDs.

The only thing to consider is that if the special device vdev fails, the whole pool fails. So make sure to use decent (datacenter/enterprise) SSDs and to at least mirror them!

The ZFS man pages / documentation have more info about them

 

[The Tech Guy]

Hello,

It depends on what and how much you want to secure. An SSD cache does not help you with backup. In general, I only use Raid10 in productive operation, also in backup. Everything else is way too slow. Now of course it can be that you don't have that much data and one RaidZ might be enough. Proxmox itself is good when it runs on the SSD's. Since I have already noticed that the web interface no longer reacts at all when the server is under load during the backup.

CPU... yes 8 Cores Minium, 32GB Memory Minimum.

Tell me how much data is involved and how often you would like to back it up.

The storage pool is 4x12TB (36TB) DC drives in Raid-Z

But do its need a SSD cache?
If its needed its best to use the M2 slots on the motherboard for a NVMe drive, but its not mirrored, another option is to add a PCIe to m2 adapter and use 2 NVMes in raid?

Another solution is to create 2 partions on the SSDs in a software raid mirrror (100GB for OS and 150GB for SSD cache)

What is the best solution ?

 

[fireon]

The storage pool is 4x12TB (36TB) DC drives in Raid-Z

Nice! i would prefer Raid10. RaidZ is really, really slow. But as I already wrote, it depends on how much you want to back up and how often. A couple of facts would be nice.

As Aaron said, the story could end badly without a mirror. Software Raid will probably not work because you need ZFS as a cache. You can install the whole system on your HDD's. Then put 2 SSD's/NVMe's in a ZFS Mirror for the cache.

 

[aaron]

But do its need a SSD cache?

There are quite a few different technical options that could be understood as "cache". What exactly do you mean by that?

As mentioned, the ZFS special device vdev will give you a huge performance boost for some operations. Pruning and garbage collection are the most prominent ones IIRC. But the vdev should be on a mirror to avoid total pool failure if one disk fails. Therefore I don't recommend to place two NVMEs on the same PCI card, but only one. Just to make sure that should the PCI adapter card fail (not likely as most are only a few resistors and capacitors), the other NVME connected directly to the motherboard will still work.

How large are the NVMEs? You could use them, as mentioned, for the system and the special device. Not ideal but it could work okayish enough.

When you use the PBS installer and select to install it to a ZFS mirror, you can define how much of the disks the installer should use. In the disk selection dialog, switch to the "Advanced Options" tab and set the "hdsize" parameter to the size the system should use. The rest of the disks will be left empty and after the installation you can create a new partition at the end of both NVMEs and use those when you define the special device vdev mirror.

But only do that if you feel comfortable doing so

 

[The Tech Guy]

There are quite a few different technical options that could be understood as "cache". What exactly do you mean by that?

As mentioned, the ZFS special device vdev will give you a huge performance boost for some operations. Pruning and garbage collection are the most prominent ones IIRC. But the vdev should be on a mirror to avoid total pool failure if one disk fails. Therefore I don't recommend to place two NVMEs on the same PCI card, but only one. Just to make sure that should the PCI adapter card fail (not likely as most are only a few resistors and capacitors), the other NVME connected directly to the motherboard will still work.

How large are the NVMEs? You could use them, as mentioned, for the system and the special device. Not ideal but it could work okayish enough.

When you use the PBS installer and select to install it to a ZFS mirror, you can define how much of the disks the installer should use. In the disk selection dialog, switch to the "Advanced Options" tab and set the "hdsize" parameter to the size the system should use. The rest of the disks will be left empty and after the installation you can create a new partition at the end of both NVMEs and use those when you define the special device vdev mirror.

But only do that if you feel comfortable doing so

Hi Aaron

I can see its a problem in using NVMes so i have ordered 2 x 480GB DC SSD, and my plan is to use 100GB in a LVM raid and the rest as vdev, its a bad solution?

The backup solution is only for 5-10 servers (3TB of data) so im not sure about

 

[fireon]

When you use the PBS installer and select to install it to a ZFS mirror, you can define how much of the disks the installer should use. In the disk selection dialog, switch to the "Advanced Options" tab and set the "hdsize" parameter to the size the system should use. The rest of the disks will be left empty and after the installation you can create a new partition at the end of both NVMEs and use those when you define the special device vdev mirror.

 

[The Tech Guy]

Hi,

Now the installation is done but the "speciel device" give some problems, i have crated 2 ZFS pools but how do i use the rest of the SSDs to a speciel device for zpool, the first 100GB is used for rpool to PBS?


NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
rpool 99G 1009M 98.0G - - 0% 0% 1.00x ONLINE -
zpool 43.6T 25.5G 43.6T - - 0% 0% 1.00x ONLINE -