Best practice for ZIL/L2ARC with just 4xSSDs (yet another one :)

[Progratron]

I know this subject was discussed so many times... I have read so many tutorials and so many "best practices" and "suggestions"... Sometimes they advise exactly opposite things... After reading another one I decided to ask community advice before I start

I am building Proxmox VE setup based on a server leased in a DC (so I can't really configure my hardware + budget is pretty limited). What the organisation can afford is a server with 256GB RAM and 4xSSDs where providers say "Enterprise-Grade SSD" without naming the manufacturer exactly.

My initial thinking was to do two separate mirrors out of them. On the first mirror I was planning to make separate partitions for 1) proxmox installation 2) SLOG 3) L2ARC. On the second mirror all the VMs. I do worry about placing SLOG & L2ARC into partitions but seems I can't afford separate drives for that... ZFS Best Practices says "it's certainly acceptable"

The other question is also size of both SLOG and L2ARC. My thoughts were something like 4-8GB SLOG + around 200-300 GB (or whatever is left) for L2ARC. I am considering to limit the ARC to 64 GB instead of standard of 128 GB for my setup.

Thanks in advance for all your thoughts and corrections.

 

[LnxBil]

After reading another one I decided to ask community advice before I start

Maybe you'll get the same opposite answers, but we'll see.

My initial thinking was to do two separate mirrors out of them. On the first mirror I was planning to make separate partitions for 1) proxmox installation 2) SLOG 3) L2ARC. On the second mirror all the VMs. I do worry about placing SLOG & L2ARC into partitions but seems I can't afford separate drives for that... ZFS Best Practices says "it's certainly acceptable"

This advice applies to your actual system with only 4 identical SSDs:

You can do that, but you will get a faster system by using all SSDs in one stripped and mirrored pool - like in LOTR - one to rule them all. SLOG on a ssd as fast as your data disk is also doable, but why would you do such a thing? The only benefit would be that you would write only once to each device group while having slower throughput. The same is true for L2ARC ... your I/O response times are the same if you read from L2ARC or from the disk itself, so why castrate your ARC in favour of L2ARC to have no speed benefit.

I am considering to limit the ARC to 64 GB instead of standard of 128 GB for my setup.

Having L2ARC and then limiting ARC is the worst idea. Your ARC is already reduced by having L2ARC and you lower it further. Again, you can do that, but it is not wise.

 

[Progratron]

Maybe you'll get the same opposite answers, but we'll see.

Thanks for replying me in another thread

You can do that, but you will get a faster system by using all SSDs in one stripped and mirrored pool - like in LOTR - one to rule them all. SLOG on a ssd as fast as your data disk is also doable, but why would you do such a thing? The only benefit would be that you would write only once to each device group while having slower throughput. The same is true for L2ARC ... your I/O response times are the same if you read from L2ARC or from the disk itself, so why castrate your ARC in favour of L2ARC to have no speed benefit.

Yes. This was my other thought.. I assume I'll get 4 identical "enterprise" SSDs... So you think a mirror + stripe and to place everything on a single pool is a better idea? And don't even bother adding separate SLOG/L2ARC caches but just leave the default ARC of 128GB as-is?

Another question - what do you think about

sync=disabled

?

 

[guletz]

Hi,

sync=disable is ok if your server will have ups. If you do not use some DBs, you are ok. If you have a power off, you will lose only the last 5 sec of data write. I prefer to setup zfs to flush the data to disks at 3 seconds insted of 5(the default )

Good luck

 

[alexskysilk]

I'll just leave this here: http://open-zfs.org/wiki/Performance_tuning