BTRFS

[Maher Khalil]

I have 2 NVMe disks, 4 SSD disks and 4 HDD disks
I want to use all of them in ZFS or BTRFS pool but I am afraid that NVMe and SSD speed will limit to match HDD speed
I need advice on that

 

[LnxBil]

Yeah, using all of those in one pool might NOT be the best choice.

A pool of spinning rust can always be enhanced by using SSD for metadata caching and SLOG (SLOG even on NVMe will be blazing fast, but only needs a few GB). So, you have a lot of other stuff laying around. What are the disk sizes of each?

 

[Maher Khalil]

2 x 250 GB NVMe I want to use for OS
2 x 1.6 TB SSD
4 x 2.4 TB SAS HDD

 

[LnxBil]

2 x 250 GB NVMe I want to use for OS

PVE needs at most 10GB and no speed after it is booted, what about the rest?

One setup could be this setup that uses all block devices and creates two nvme-boosted pools:
Split each NVMes in two 5 GB partitions and in two 120 GB partitions, then build two pools with mirrored slog and mirrored special allocation class for metadata. The data itself is then on the SSDs (pool1) and on the stripped mirror SAS hdds (pool2). If you want to use the SSDs for the PVE install, you can just install on a mirrored pool on the two SSDs, partition the two NVMes and then add the partitions.

Is this clear what the idea is?

 

[Maher Khalil]

PVE needs at most 10GB and no speed after it is booted, what about the rest?

The rest shall be for VM
Please note that the HDD type is SAS 12 Gbps which I think very close to SAS SSD 12Gbps

 

[LnxBil]

Please note that the HDD type is SAS 12 Gbps which I think very close to SAS SSD 12Gbps

In sequential IO it may be in the same ballpark, but still slower. In random IO performance, there is no comparison. A disk has at most 200 IOPS, a SSD over 100k.

 

[Maher Khalil]

So, In this case it may be better to do RAID 0 FOR ssd and use HDD for backups
please note I plan for future expansion of ssd drives

 

[LnxBil]

RAID 0 FOR ssd

RAID0 has no right to exist in a production system.

 

[Maher Khalil]

so, what is your suggestion for better installation

 

[djsami]

RAID0 has no right to exist in a production system.

sata sas you are right
but ssd no ssd risk dying is very difficult
My 5 year old device is 24x1 tb ssd available raid 0 still works
there was not the slightest problem, but of course we use proxmox backup, at the same time we have a total of 1200 vms.

 

[Maher Khalil]

sata sas you are right
but ssd no ssd risk dying is very difficult
My 5 year old device is 24x1 tb ssd available raid 0 still works
there was not the slightest problem, but of course we use proxmox backup, at the same time we have a total of 1200 vms.

Which file system do you use?

 

[djsami]

Which file system do you use?

previous years i was using xfs

now i am using BTRFS

dell r720xd
24x1 TB Samsung pro 860 ssd
Raid 0 + BTRFS

I get 13 14 giga write read

 

[djsami]

2 x 250 GB NVMe I want to use for OS Raid 0 + BTRFS
2 x 1.6 TB SSD ZFS 0
4 x 2.4 TB SAS HDD ZFS 0

Okey

 

[Maher Khalil]

2 x 250 GB NVMe I want to use for OS Raid 0 + BTRFS
2 x 1.6 TB SSD ZFS 0
4 x 2.4 TB SAS HDD ZFS 0

Okey

So, I will use HDD as backups for SSD right?
also for2 x 250 GB NVMe , I will use raid controller then I can get benefits of it, right?

 

[djsami]

So, I will use HDD as backups for SSD right?
also for2 x 250 GB NVMe , I will use raid controller then I can get benefits of it, right?

2x250 nvem BTRFS raid 0 will be already

 

[Maher Khalil]

2x250 nvem BTRFS raid 0 will be already

Great, thank you very much for all

 

[LnxBil]

2 x 250 GB NVMe I want to use for OS Raid 0 + BTRFS
2 x 1.6 TB SSD ZFS 0
4 x 2.4 TB SAS HDD ZFS 0

Okey

Besides the RAID 0 stupidity (seriously!), why use RAID and then BRTFS? BTRFS has built-in raid functionality (mirroring, striping and more).

 

[Maher Khalil]

Besides the RAID 0 stupidity (seriously!), why use RAID and then BRTFS? BTRFS has built-in raid functionality (mirroring, striping and more)

because of different sizes and types of disks, I am afraid of BTRFS lower speed to speed of HDD so what I am thinking is to use raid controller for only operating system with ext4. And then use SSD for BTRFS or zfs and then HDD for BTRFs.
Do you think using BTRFS on different disk types is ok?

 

[LnxBil]

Mixing different disk types for the same kind of data is always not good. You have to mix it with respect to the type of data stored on them - as I described in post #4.

We have not talked about what the goal of your machine is. Normally, goals come first, then you buy the hardware, so why did you buy three different kinds of storage? Nowadays you normally don't buy HDD if you already have NVMe in there.

 

[Maher Khalil]

You are right, but I bought it new assembled with excellent price.
I want to for client to buy c/m and use it as they want
They may use it for storage or databases or stream …etc
Nowadays dell gives option to separate operating system on NVMe with raid they call it boss and other disks with or without raid
In my opinion because btrfs work on block levels it will not matter mixing disk types each block will have its own speed