storage server

[Kuonel]

Hi this is not directly related to pve or pbs but please let me borrow your zfs experience

I just got an SX135 from Hetzner
with the following drives:
8 x 22 tb hdd
2 x 2 tb nvme
128 gb of ram

this storage server is going to replace s3,
it will store around 40 tb of data (small files, between 1KB and 2MB),
serve them to around 400 clients using samba
and should have decent read (1 second delay max) and write speed (ideally 100MB/s).

I was thinking about using the hdd drives for the os and main storage,
with two drives for parity (to prevent bit-rot)
and the two nvmes as slog.

Does it makes sense ?


Thank you,
Mike

 

[UdoB]

and the two nvmes as slog.

An SLOG accelerates ONLY sync writes.

I recommend to search for the term "Special Device" and learn the intended use.

You can also split the 2 TB NVMe (by classic partitioning) and implement both of it - of course as mirrors. Make sure to understand the construction: as soon something is added as a "Special Device" it becomes integral part of the pool and can not be lost!

 

[Dunuin]

And for good performance, lots of small files and users, reasonable resilvering time with such big and slow drives you probably want a striped mirror for those HDDs.

 

[tcabernoch]

special vdev
https://forum.level1techs.com/t/zfs-metadata-special-device-z/159954

more special vdev. this is gold.
https://klarasystems.com/articles/openzfs-understanding-zfs-vdev-types/

 

[Kuonel]

Lots of suggestions and information here, thank you all !
Let me know if you are perhaps willing to provide professional consulting,
I mean performing the setup and documenting it (so that I can format the server and replicate it, for security reasons).
Contact me here on PM me.

The server is:
8 x 22 tb hdd
2 x 2 tb nvme
and hetzner already told me that no drives can be added (just an usb drive)
so I wonder where to put the OS.

 

[tcabernoch]

I assumed you were building a Proxmox server.
You are on the wrong forum.

You need TrueNAS, and they've got a ZFS forum right here:
https://forums.truenas.com/tag/ZFS

==============================
Below here is all Proxmox info for
8 disk + 2 NVMe special vdev build ...

You build your base ZFS system and then add the special vdev later.
The first decision looks like this. Choose one.

2 vdev setup. You'll get better performance this way.
(vdev1 4 disk raidz1) (vdev2 4 disk raidz1)
raidz1 is equivalent to RAID5 with 1-disk redundancy.

or

1 vdev setup. You can ignore it longer this way. Less maintenance, easier to understand the CLI stuff.
(vdev1 8 disk raidz2)
raidz2 is equivalent to RAID6 with 2-disk redundancy.

----------------


Run the regular proxmox install.
When you get to the disk screen, click the button that says ext4 and change it to one of these ...

2 vdev (performant) setup - Change it to raidz1 and only select 4 disks. Do not select NVME. Proceed with the install.

or

1 vdev (low maint) setup - Change it to raidz2 and leave all the disks selected. Do not select NVME. Proceed with the install.

-------------------

You do the rest from the CLI after install is done.

For the 2 vdev performant setup, you create a second raidz1 vdev with the remaining 4 disks.

---------------

Once you have either single or 2 vdevs added to the zpool and have allocated all your standard disks, add the special vdev.

Do not neglect to adjust your recordsize and special block size, or you don't get any real advantage.

This is just an example ...
zpool add rpool -o -f ashift=12 special mirror scsi-<gptid>-part3 scsi-<gptid>-part3
zfs set recordsize=1M rpool
zfs set special_small_blocks=256K

--------------

You asked about contracting for this build. That second link I posted ... I don't have any relationship with that company, but they contract to do ZFS stuff.