Advice for a new installation with zfs + a question on the raid/mdadm

[proxtib]

Hello,

I have decide to transfere all my nas HDD on my Promox server and after little issue (see here: Failed to import pool ‘rpool’; ssd ko ? No go to X570D4I-2T BIOS!), it’s done proxmox see they new hdd (and one new ssd )


First one question: Proxmox see my new HDD … and also the raid 5 mdadm/lvm/ext4 that I had done via on nas !

I don’t understand, I always believed that raid mechanism dependent on OS.

That means that no matter the linux distribution, I can tansfer my 5 HDD to another server and they will always be ok ?

___

I have now one proxmox server with :

• mother-carde : ASRock X570D4I-2T
• proc : AMD Ryzen 7 3700X Wraith
• RAM : 36GO(emc)
• storage :
  • 4To ssd (4 x 1To ssd WD red)
  • 20To HDD ( 5 x 4To WD red)

Question : What do you recommend to me as "storage architecture" ?

Before my surprise from above, I wanted something simple like:
2 ZFS pool :

• rpool (aka ssd-pool) raidz with all my ssd
• hdd-pool with all my hdd

But now I have more time, I can always access my old data before reinstalling everything.
It’s my personal-homelab-nas → I will install a smb (or only NFS) server. And I have some important data which are save on an external hard drive. But this is not a reason to restore my backup every month, following a crash of my server !

I am not fluently with ZFS, I question me about the RAM used.
For example, one question on the « ZFS on Linux » wiki parte, they are write something like ZFS use 50 % of the host memory (so for me 18Go) and after « As a general rule of thumb, allocate at least 2 GiB Base + 1 GiB/TiB-Storage ». So for me is 20+4+2= 26Go…

18!=26 … I don’t realy understand … Can you explain to me ?
Have you some tricks for custom my ZFS use ?

For the storage architecture, isn't there something better than my 2 zfs-pool ? Have a pool 'ssd-data' different from rpool ? Have an ARC cache on one of my ssd ?

Thanks for your help.

 

[Dunuin]

Hello,

I have decide to transfere all my nas HDD on my Promox server and after little issue (see here: Failed to import pool ‘rpool’; ssd ko ? No go to X570D4I-2T BIOS!), it’s done proxmox see they new hdd (and one new ssd )


First one question: Proxmox see my new HDD … and also the raid 5 mdadm/lvm/ext4 that I had done via on nas !

I don’t understand, I always believed that raid mechanism dependent on OS.

That means that no matter the linux distribution, I can tansfer my 5 HDD to another server and they will always be ok ?

Only if you install the required software. PVE doesn't officially support mdadm because it is not as save as ZFS. So you can use mdadm with PVE if you install the packages and set it up yourself, but you won't find any support for them in the PVE CLI or GUI.

I have now one proxmox server with :

• mother-carde : ASRock X570D4I-2T
• proc : AMD Ryzen 7 3700X Wraith
• RAM : 36GO(emc)
• storage :
  • 4To ssd (4 x 1To ssd WD red)
  • 20To HDD ( 5 x 4To WD red)

Question : What do you recommend to me as "storage architecture" ?

Before my surprise from above, I wanted something simple like:
2 ZFS pool :

• rpool (aka ssd-pool) raidz with all my ssd
• hdd-pool with all my hdd

But now I have more time, I can always access my old data before reinstalling everything.
It’s my personal-homelab-nas → I will install a smb (or only NFS) server. And I have some important data which are save on an external hard drive. But this is not a reason to restore my backup every month, following a crash of my server !

I am not fluently with ZFS, I question me about the RAM used.
For example, one question on the « ZFS on Linux » wiki parte, they are write something like ZFS use 50 % of the host memory (so for me 18Go) and after « As a general rule of thumb, allocate at least 2 GiB Base + 1 GiB/TiB-Storage ». So for me is 20+4+2= 26Go…

18!=26 … I don’t realy understand … Can you explain to me ?

ZFS will use UP TO 50% of your RAM for its ARC. If other processes need that RAM ZFS will give it back and the ARC will shrink down up to a minimum (I think its 1GB by default). That rule of thumb is if you want good performance. The bigger your ARC is, the better the performance will be. But your pool should also work with 2 GiB Base + 256-512 MiB/TiB-Storage...just slower. And this 50% RAM is just the default. You can set your own minimum and maximum ARC limits. So if you want, you could set your ARC from 16GB to 26GB. See here: https://pve.proxmox.com/wiki/ZFS_on_Linux#sysadmin_zfs_limit_memory_usage
Got a 4x 8TB ZFS pool in my backup NAS running for years with just 16GB of RAM without a problem. But my main NAS also got 32GB RAM for a 4x 8TB ZFS pool.

Have an ARC cache on one of my ssd ?

L2ARC is usually not worth it except you got a special case where oyu know you will need it. In general is recommed to buy more RAM if your ARC is too small and your RAM isn't already maxed out instead of using a way lower SSD as a L2ARC.

Buy the way...ZFS got a hell of overhead and therefore write amplification. Its recommended to use enterprise/datacenter grade SSDs with it and your WD Red SSDs are just consumer SSDs not really great for ZFS because of the low life expectation (TBW/DWPD), bad sync write performance and missing powerloss protection.

 

[proxtib]

Thank you for your answer.

PVE doesn't officially support mdadm because it is not as save as ZFS.

your WD Red SSDs are just consumer SSDs not really great for ZFS

ok.
But what would you recommend for personal use; with these discs ?
Raid lvm with ext4 ?

 

[poing]

Thank you for your answer.




ok.
But what would you recommend for personal use; with these discs ?
Raid lvm with ext4 ?

I also wonder what the best alternative file system for SSD is. Unless I'm mistaking, to make VM snapshots, you need ZFS.

Great info on the ram usage btw, I was initially searching for this kind of info.

 

[Dunuin]

You don't need ZFS for snapshots. LVM-Thin would for example work too or any filesystem based storage when using qcow2. See the storage feature support table to see which storage supports snapshots: https://pve.proxmox.com/wiki/Storage#_storage_types