Installing Proxmox on ZFS RAID1 (mirror) Using Remaining Space for L2ARC

[norsemangrey]

I am installing Proxmox on two SSDs in a ZFS mirror (RAID1) configuration (why does the hdsize show 476?).

Proxmox itself should not take up to much space and my VM/LXC space will be on separate drives.

How much space should I make for Proxmox and could I use the rest for an L2ARC stripe across the drives to be used for my HDD pool?

If so how do I accomplish this?

 

[Dunuin]

If you reduce the "hdsize" you should get some unallocated disk space you can partition later and use it for whatever you want.

Keep in mind that it is recommended to use dedicated drives for L2ARC and SLOG so that the caching won't slow down other stuff. And with caching you get alot of writes (primarily the SLOG) so the SSDs might fail earlier and in that case your boot/root drive will fail too.

 

[norsemangrey]

If you reduce the "hdsize" you should get some unallocated disk space you can partition later and use it for whatever you want.

Not sure why I am even allowed to set 476GB when selecting a RAID1 (mirror) configuration. Should the max size not be 223GB?

Keep in mind that it is recommended to use dedicated drives for L2ARC and SLOG so that the caching won't slow down other stuff. And with caching you get alot of writes (primarily the SLOG) so the SSDs might fail earlier and in that case your boot/root drive will fail too.

How much space should I make for Proxmox and could I use the rest for an L2ARC stripe across the drives to be used for my HDD pool?

If so how do I accomplish this?

The disks will not be used for SLOG. I will be using 2 x Intel D3-S4610 256 GB SSDs. I am comfortable with the drives failing earlier. I am looking for the how to do this. How do I allocate space for the L2ARC and how much? Do I create a partition? Can it be done in the GUI or do I use the CLI (if so what commands do I use)? This is not for production, I just want to learn how to do it.

 

[ph0x]

After you set the hdsize you'll have free space on your drives that you can put partitions on.
I don't know a GUI section that would handle that, but I'm the console user anyway, so I use parted for that.
The new partitions can then be added as slog to an existing zpool via zpool add poolname log mirror /dev/disk/by-id/...-partX /dev/disk/by-id/...-partX.

 

[norsemangrey]

Thanks for the reply @ph0x.

After you set the hdsize you'll have free space on your drives that you can put partitions on.

From that i have understood the Proxmox installer will create several partitions on the disks (including root, boot, swap, maybe more?) The hdsize that is defined in the GUI during the install process is that combined space for all of these?

I don't know a GUI section that would handle that, but I'm the console user anyway, so I use parted for that.

I'm comfortable with CLI so that is no issue.

So after installing Proxmox on a portion smaller than the full size of one disk, what I would need to do (and please correct me if I have misunderstood, this is new to me) is to create an equally sized partition of the remaining space on each of the two drives using something like parted.

The new partitions can then be added as slog to an existing zpool via zpool add poolname log mirror /dev/disk/by-id/...-partX /dev/disk/by-id/...-partX.

I am not sure I follow you completely on this part. You are using "log" (I thought the L2ARC was referenced using "cache") and "mirror" (if I have understood correctly there is no need for the L2ARC to be mirrored, so a striped pool is recommended).

I assume the "poolname" is referring to the HDD pool for which I would like to use the L2ARC.

Are there any ZFS properties that should be set specifically for a L2ARC pool (like recordsize, compression, etc.) or are the default value the best to keep?

 

[ph0x]

From that i have understood the Proxmox installer will create several partitions on the disks (including root, boot, swap, maybe more?) The hdsize that is defined in the GUI during the install process is that combined space for all of these?

There will be three partitions afterwards, 1007K BIOS /boot, 512M EFI and hdsize minus the two for the zpool.

So after installing Proxmox on a portion smaller than the full size of one disk, what I would need to do (and please correct me if I have misunderstood, this is new to me) is to create an equally sized partition of the remaining space on each of the two drives using something like parted.

You'll need equally sized partitions if you want to use them as mirror, otherwise you are completely free on how to partition the disks.

I am not sure I follow you completely on this part. You are using "log" (I thought the L2ARC was referenced using "cache") and "mirror" (if I have understood correctly there is no need for the L2ARC to be mirrored, so a striped pool is recommended).

Yeah, you're right on both, I once again had a twist in my mind.

I assume the "poolname" is referring to the HDD pool for which I would like to use the L2ARC.

Exactly.

Are there any ZFS properties that should be set specifically for a L2ARC pool (like recordsize, compression, etc.) or are the default value the best to keep?

L2ARC doesn't become a zpool, the partitions are added as cache and that's it. You can confirm with zpool status poolname afterwards.

 

[norsemangrey]

There will be three partitions afterwards, 1007K BIOS /boot, 512M EFI and hdsize minus the two for the zpool.

Thanks for clarifying. So the boot partition is that where GRUB resides and EFI holds the boot-loader on a UEFI system? (I am trying to learn about the partitioning scheme and boot process).

You'll need equally sized partitions if you want to use them as mirror, otherwise you are completely free on how to partition the disks.

Yes true, but I guess it is best to keep them the same size. Not sure how large I should make the partitions though. I have 32GB of memory (expanding to 64GB later). I have read some places that there is a rule of thumbs stating a 1:5 relation between the amount of RAM and the L2ARC size.

Yeah, you're right on both, I once again had a twist in my mind.

Great, I have not misunderstood completely then

L2ARC doesn't become a zpool, the partitions are added as cache and that's it. You can confirm with zpool status poolname afterwards.

Aha, I see, so zpool add is just fromatting the partition(s) in some fashion that is suitable for use as ZFS pool cache then.

 

[ph0x]

Thanks for clarifying. So the boot partition is that where GRUB resides and EFI holds the boot-loader on a UEFI system? (I am trying to learn about the partitioning scheme and boot process).

I guess so, but I'm no expert in this field.

Yes true, but I guess it is best to keep them the same size. Not sure how large I should make the partitions though. I have 32GB of memory (expanding to 64GB later). I have read some places that there is a rule of thumbs stating a 1:5 relation between the amount of RAM and the L2ARC size.

L2ARC indeed has to be mapped in RAM, so there definitely is a correlation, some say 4:1, some say 5:1 ... Before using L2ARC it is recommended to max out RAM, though, which is a lot faster than any L2ARC.

Great, I have not misunderstood completely then

It was a test.

Aha, I see, so zpool add is just fromatting the partition(s) in some fashion that is suitable for use as ZFS pool cache then.

Yes, so you don't even need to format it, as in creating a filesystem on it. ZFS will take the raw partition and do things with it.

 

[norsemangrey]

I guess so, but I'm no expert in this field.

L2ARC indeed has to be mapped in RAM, so there definitely is a correlation, some say 4:1, some say 5:1 ... Before using L2ARC it is recommended to max out RAM, though, which is a lot faster than any L2ARC.

It was a test.

Yes, so you don't even need to format it, as in creating a filesystem on it. ZFS will take the raw partition and do things with it.

Thanks for the help

One last question though, how much space does the Proxmomx instal require. I could not find a good answer in the documentation. Considering that I will use separate disk for VMs and general storage, what should I set the hdsize value to?

 

[Dunuin]

I'm using 21GiB and 6.32 GiB are used. So if you don't need the root dataset to store ISOs, backups, snippets, templates because you store them somewhere else (NAS shares or directory storage on another pool or just a usb stick with plain ext4 as directory storage) you don't need alot of space.
Especially ISOs and templates you just write once, so they aren'T write heavy and and its no problem to loose them because you can easily download them again. So storing them on a mirrored SSD would be a total waste if a cheap USB-Stick is doing the job too.

 

[norsemangrey]

I'm using 21GiB and 6.32 GiB are used. So if you don't need the root dataset to store ISOs, backups, snippets, templates because you store them somewhere else (NAS shares or directory storage on another pool or just a usb stick with plain ext4 as directory storage) you don't need alot of space.

Super! Thank you.

I will add some space for the ISO though as I do not see the need to store them elsewhere, and I should still have plenty of space for some L2ARC if deciding to use that later.

 

[ph0x]

Once upon a time I played around with the installer in VMs and the lowest I could go without too much hassle were 8GiB, if I remember correctly.
So for a real world deployment I would probably go for 16 GB.