Proxmox VE SSD-HDD Best practice

[mladenciric]

Hello,
I plan to make three proxmox ve hosts in a cluster. I have at stock SSD and HDD disks.
I am thinking to use it like this:
SSD - for system itself, and some guest machine
HDD - for storage, local backup, guest machine (basically everything else)
NFS - on NAS storage
PBS - for automatic backup of guest machines
I don't have hardware raid cards on my host machines.
All hosts will be equipped with 2xSSD 200GB Kingston + 2xHDD 1TB WD Black
I would like to have some that why I plan to use ZFS.
ZFS array mirror on two SSD and ZFS array mirror on two HDD
First question is this plan good or someone have another suggestion?
Second question what is the right way to make ZFS on HDD? Evidently, ZFS on SDD will be created using installation process but is it better to create ZFS on two HDD also on installation or after installation? After installation before hosts join to cluster and then make zfs for datacenter or create it on every host separately and then join hosts to cluster?
Third question is do I need or must to use ZIL for ZFS?
I will appreciate all suggestions.
Regards/

 

[ph0x]

1. Plan sounds good, you have then to decide which images are to be put on the ssd pool.
Click on options during installation and choose a size for maxroot, then the rest of the disk will be used for image space.
2. The ZFS on the HDDs are to be created after the installation, it's only possible to define the root filesystem during the install.
You should create the pool on each node before you add it as storage.
3. For the SSDs you don't need ZIL, but you could use one partition on the SSDs as ZIL for the HDDs. I would have some gigs spare on the SSDs and add ZIL later, if the performance doesn't fit your needs, since it adds additional load on the SSDs.

 

[mladenciric]

Hello,
I plan to make three proxmox ve hosts in a cluster. I have at stock SSD and HDD disks.
I am thinking to use it like this:
SSD - for system itself, and some guest machine
HDD - for storage, local backup, guest machine (basically everything else)
NFS - on NAS storage
PBS - for automatic backup of guest machines
I don't have hardware raid cards on my host machines.
All hosts will be equipped with 2xSSD 200GB Kingston + 2xHDD 1TB WD Black
I would like to have some that why I plan to use ZFS.
ZFS array mirror on two SSD and ZFS array mirror on two HDD
First question is this plan good or someone have another suggestion?
Second question what is the right way to make ZFS on HDD? Evidently, ZFS on SDD will be created using installation process but is it better to create ZFS on two HDD also on installation or after installation? After installation before hosts join to cluster and then make zfs for datacenter or create it on every host separately and then join hosts to cluster?
Third question is do I need or must to use ZIL for ZFS?
I will appreciate all suggestions.
Regards/

1. Plan sounds good, you have then to decide which images are to be put on the ssd pool.
Click on options during installation and choose a size for maxroot, then the rest of the disk will be used for image space.
2. The ZFS on the HDDs are to be created after the installation, it's only possible to define the root filesystem during the install.
You should create the pool on each node before you add it as storage.
3. For the SSDs you don't need ZIL, but you could use one partition on the SSDs as ZIL for the HDDs. I would have some gigs spare on the SSDs and add ZIL later, if the performance doesn't fit your needs, since it adds additional load on the SSDs.

@ph0x Thank you, i will appreciate if you give me an answers to following questions:
1. do I have solution to choose maxroot option or I must use partition manager to do it. What maxroot size is minimal/recommended/optimal/maximal?
2. it is clear what i have to do. what about names of zfs pools? do i have to name all three pools with the same name, because of joining hosts to cluster? When i create pools do i need to make storage first or to join hosts to cluster and then to make storage for datacenter for all hosts in cluster?I know that both are possible but what is the right way?
3. Yes, of course that for ssd don't need ZIL. My question was for HDD ZFS array? How to partition SSD to reserve some space for ZIL, at installation process using partition manager or after installation? How many gigs to reserve for this purpose? Do you have some useful links which explain how to do that?

 

[ph0x]

1. do I have solution to choose maxroot option or I must use partition manager to do it. What maxroot size is minimal/recommended/optimal/maximal?

There is no partition manager like you may know it from other distributions. But you can klick on options during install and choose maxroot.
Minimal is considered 16 GB, maybe even a bit less. Depends on what you want the hypervisor to do by itself, like run some applications or anything. I would go with 32 GB.

2. it is clear what i have to do. what about names of zfs pools? do i have to name all three pools with the same name, because of joining hosts to cluster? When i create pools do i need to make storage first or to join hosts to cluster and then to make storage for datacenter for all hosts in cluster?I know that both are possible but what is the right way?

Yes, those pools should have the same name. You can create them after cluster creation, but you should create them before defining the storage entry on datacenter level.

3. Yes, of course that for ssd don't need ZIL. My question was for HDD ZFS array? How to partition SSD to reserve some space for ZIL, at installation process using partition manager or after installation? How many gigs to reserve for this purpose? Do you have some useful links which explain how to do that?

Like already mentioned, there is no partitioning during install. After the installation you can use any partition manager you like for that purpose. I usually go with parted, which has to be installed. Everything above maxroot+maxvz will be unpartitioned space that you can layout the way you want. I encourage you to google for ZIL size, since I would do the same. ZIL gets flushed to HDD every five seconds, therefore it has to be bandwidth times 5 seconds large. People oftentimes go with 10 GB and are prepared for 10 GBit scenarios with that.

 

[mladenciric]

There is no partition manager like you may know it from other distributions. But you can klick on options during install and choose maxroot.
Minimal is considered 16 GB, maybe even a bit less. Depends on what you want the hypervisor to do by itself, like run some applications or anything. I would go with 32 GB.

@ph0x option "maxroot" is option if I choose only one disk and ext4 partition type. If I use ZFS array that option don't exist, as attached pics shows.

 

[ph0x]

You're right, I was wrong about that. It's hdsize what I was talking about.