Shared SSD for OS & datastore acceleration

[cyruspy]

Hello!,
While deploying Proxmox Backup Server (PBS) for personal backups (NAS files and PVE test VMs), I noticed that SSDs are preferred for the datastore. Since this is a "cost-effective" scenario, I'll have to use leftover mechanical disks. Would it be possible to use the two SSD boot drives to also host the ZFS special device for metadata caching? Are there any drawbacks to this approach?

Hardware:

• HPE Microserver Gen8
• Xeon(R) CPU E3-1265L V2
• 16GB RAM
• Boot disks: 2 x 200GB (Micron P400m-MTF)
• Data: 4 x 4TB WDC WD40EFRX-68N

The data disks will be set up as ZFS mirrors. I was wondering if I could leave 50GB for the boot pool and use the remaining 150GB to create a mirrored partition for the special device. This would hopefully improve performance without needing to dedicate full SSDs for it.

Any hints?

 

[J-Rod]

My hint is run PVE on the hardware and PBS in a VM. More functionality and flexibility, due to all the reasons we run VMs anyway.

Install PVE on the SSDs with a ZFS mirror, but leave 16GB of space free during setup for swap. Configure the HDDs in PVE, not PBS. Allocate 6GB of RAM for PBS to start. Put the PBS OS and special device virtual disks on SSD and data on HDD, but be conservative with the disk allocations so you can use the free space for other purposes. My special device usage is currently at 0.5% of the used datastore space (special_small_blocks=4K). You can hot-expand the virtual disks as needed any time.

With this setup, if your main PVE host/cluster goes down, you have the option to restore some VMs to the HPE Microserver directly while you work on the issue. You can also run some other VMs that may be helpful outside of your main PVE cluster, such as PDM, or a basic VM/container for shared storage, etc. Lastly, if you decide to replace the host, you can easily migrate the VMs off.

 

[UdoB]

Are there any drawbacks to this approach?

The obvious one: when this computer dies badly both the original data and the backup is gone.

Rule zero: backups must be reliably available when the source is gone.

(( The famous 3-2-1 rule applies to a homelab too. And it has been extended "recently" to become 3-2-1-1-0:

• 3 independent copies of data
• 2 types of media
• 1 stored off-site
• 1 immutable/readonly
• 0 errors: ensure restore does actually work

))

Just saying...

 

[J-Rod]

My understanding is the OP's plan was to load PBS bare metal, but since the PBS OS install by default would take over the entire mirrored SSD pool, they were wanting to partition it or otherwise steal some of the SSD space to be allocated as a special device for the PBS HDD datastore pool. So I don't think they were planning to have "the original data and the backup" on the same hardware.

But to extend the discussion, there is a slight increase in risk any time you have a special device added. It's essentially RAID0 at a logical level, where the loss of the special device means a complete loss of data. But I think it's acceptable if the special device is mirrored and the performance boost is in a way a reduction of risk (backups finish more quickly).

If possible, also have a second PBS instance that is a pull-only from the first PBS instance. Use different account passwords to provide some additional isolation.

 

[cyruspy]

The obvious one: when this computer dies badly both the original data and the backup is gone.

Rule zero: backups must be reliably available when the source is gone.

(( The famous 3-2-1 rule applies to a homelab too. And it has been extended "recently" to become 3-2-1-1-0:

• 3 independent copies of data
• 2 types of media
• 1 stored off-site
• 1 immutable/readonly
• 0 errors: ensure restore does actually work

))

Just saying...

I'm not sure I'm following. This machine is only for PBS, live data are in other computers.

 

[cyruspy]

My understanding is the OP's plan was to load PBS bare metal, but since the PBS OS install by default would take over the entire mirrored SSD pool, they were wanting to partition it or otherwise steal some of the SSD space to be allocated as a special device for the PBS HDD datastore pool. So I don't think they were planning to have "the original data and the backup" on the same hardware.

But to extend the discussion, there is a slight increase in risk any time you have a special device added. It's essentially RAID0 at a logical level, where the loss of the special device means a complete loss of data. But I think it's acceptable if the special device is mirrored and the performance boost is in a way a reduction of risk (backups finish more quickly).

If possible, also have a second PBS instance that is a pull-only from the first PBS instance. Use different account passwords to provide some additional isolation.

Yes, you are correct about the scenario.

In a "let's design this and purchase this" scenario I would purchase dedicated flash storage for metadata.

Is sharing the small boot disks what concerns me.

 

[UdoB]

This machine is only for PBS

Okay, that was a misunderstanding on my side...

In this case I would install PBS only on all spindles, multiple mirrors preferred. Nice side effect: at the end all of them are bootable.

When everything is up and running I would add the two SSD as a mirrored "Special Device"-vdev.

 

[cyruspy]

Okay, that was a misunderstanding on my side...

In this case I would install PBS only on all spindles, multiple mirrors preferred. Nice side effect: at the end all of them are bootable.

When everything is up and running I would add the two SSD as a mirrored "Special Device"-vdev.

Hmm, I didn't see that one coming

I've just shrank my original OS rpool to 50GB. Will think about it.

PS: does it mean the datastore will be just a folder somewhere and not a different pool, correct?

 

[UdoB]

Hmm, I didn't see that one coming

I've just shrank my original OS rpool to 50GB. Will think about it.

There are multiple "valid" solutions! Don't feel pushed to follow my ideas. I just say what worked best for me.

PS: does it mean the datastore will be just a folder somewhere and not a different pool, correct?

My approach? Yes. It would have resulted in one single ZFS-pool. You would create a dedicated dataset (not a folder) to hold the Storage for PBS. It "feels" like a folder but is has independent parameters/options.

Note that PBS in itself does not use any ZFS-specific features. It is "file-system agnostic".

 

[VictorSTS]

My option is:

• Install PVE using ~50G with a mirror of SSD. Leave the rest unused.
• Coinstall PBS (add repo, add package) [1].
• From PVE, add a RAIDz/RAID10 (capacity vs performance decision, although on that hardware won't be too relevant). Could use CLI, but avoid the temptation to use /dev/sd* names as those do change more often than not.
• Add partitions for special device on the free SSD space. Leave like 2-3GB for an optional swap.
• Add mirror with those partitions as special device for the HDD pool, special_small_blocks=0 (avoid surprises on disk usage on datastores with many snapshots).
• zfs create for each client/datastore so you can easily set quotas.
• Add a PBS datastore on that ZFS.

Using a privileged LXC is a nice option that I use sometimes, as it allows to backup PBS configuration easily and configuring a mountpoint is trivial.
My proposal above has the drawback that for updates, etc you will have to wait until PVE and PBS use the same base packages/distro and that maybe for the next major version you can't coinstall anymore (although that seems unlikely).

@J-Rod proposal of using a VM may work on small PBS, but has the big drawback that QEMU needs zvol for ZFS, so will have low IOPs and a big space waste due to padding if using RAIDz. On RAID10 this is less an issue, but you will be restricted to the I/O size set on the zvol on creation, which may affect performance.

@UdoB proposal of installing on the HDD has 3 drawbacks:

1. You will have to attach special device to rpool, which will waste some space on it for PVE/PBS file's metadata (or more, if using special_small_blocks!=0).
2. Your datastore I/O will have to contend with PVE/PBS logging, which can get somewhat intensive (i.e. replication logging).
3. If you lose special device, you will alse lose the OS, as rpool uses it.

Lastly, watch carefully the special device usage: if it reaches 100%, metadata will spill to HDD and performance of PBS will sink, specially for GC.



[1] https://pbs.proxmox.com/docs/installation.html#install-proxmox-backup-server-on-debian