zfs/zvol recordsize vs zvolblocksize

adamb

Well-Known Member
Mar 1, 2012
1,065
36
48
We are trying to tune a zfs mirror running on a proxmox host for our database setup.

I am trying to get a better understanding of zfs/zvol recordsize vs zvolblocksize.

If ZFS is set to a record size of 128k, the zvol is default to 8k, which one is actually used? Does the zvolblocksize override the recordsize of the pool?
 

Whatever

Member
Nov 19, 2012
225
5
18
volblocksize is used only for ZVOLs
recordsize is used for datasets

If you try to get all properties of zvol you will realize that there is no "recordsize" and vice versa

From my experience I could suggest to use ZVOL whenever it's possible. "volblocksize" mainly depends on pool configuration and disk modelы and should be chosen after some performance tests
 

guletz

Well-Known Member
Apr 19, 2017
1,012
143
63
Brasov, Romania
From my experience I could suggest to use ZVOL whenever it's possible.

zvol is possible in only one case: if you use VM

volblocksize" mainly depends on pool configuration and disk modelы and should be chosen after some performance tests
volblocksize is not depend for anything. It will be what you will setup. By default if you do not specify a value, it will be 8 k if I remember.
 

Whatever

Member
Nov 19, 2012
225
5
18
volblocksize is not depend for anything. It will be what you will setup. By default if you do not specify a value, it will be 8 k if I remember.
Well.. the right size of volblocksize (in terms of performance) should be determined carefully via performing some measurements and tests. The same volblocksize values could vary depending on disk models, pool configuration and a workload
 
  • Like
Reactions: guletz

mir

Well-Known Member
Apr 14, 2012
3,489
97
48
Copenhagen, Denmark
Well.. the right size of volblocksize (in terms of performance) should be determined carefully via performing some measurements and tests. The same volblocksize values could vary depending on disk models, pool configuration and a workload
Another thing to take into consideration is storage efficiency. You should try to match volblock size with actual size of the written blocks. If you primarily do 4k writes, like most database systems, then favor a volblock size of 4k.
 

guletz

Well-Known Member
Apr 19, 2017
1,012
143
63
Brasov, Romania
you primarily do 4k writes, like most database systems, then favor a volblock size of 4k.
Hi @mir

It is a trick with most databases who use the ancient 4k :) (postgresql)
- using 4kn disk in any non-mirror setup (so not raid 10) you will have performance problems
- some others databases use bigger block size (16 k mysql/percona/maria, 32-64 k mssql)

Bigger block = better performance in zfs pools even for non-mirror (raid10, who is very usual for any DB engine)

And at least for mysql/percona/maria, 16 k is ok for innodb files, but for logs/jurnal any size bigger then 16 k is better.

So it is not so simple to decide what is better or optim as a general speaking.

Good luck !
 

guletz

Well-Known Member
Apr 19, 2017
1,012
143
63
Brasov, Romania
Still not quite clear. If I set the zvolblocksize to 4k, it will override the 128k recordsize set in the main pool?
The size of zvolblocksize it has nothing to do and is not corelated with any DATASET recordsize. This 2 proprieties (zvolblocksize/recordsize) are 2 different things!

ZFS datasets use an internal recordsize of 128KB by default.
Zvols have a volblocksize property that is analogous to record size. The default size is 8KB

Details here:

http://www.open-zfs.org/wiki/Performance_tuning
 
Last edited:

adamb

Well-Known Member
Mar 1, 2012
1,065
36
48
The size of zvolblocksize it has nothing to do and is not corelated with any DATASET recordsize. This 2 proprieties (zvolblocksize/recordsize) are 2 different things!
Ok, then do I need to change both zvolblocksize and recordsize to ensure my VM writes all data in 4k blocks? Im just trying to figure out what I need to do to ensure I am writing in 4k blocks.
 

guletz

Well-Known Member
Apr 19, 2017
1,012
143
63
Brasov, Romania
Ok, then do I need to change both zvolblocksize and recordsize to ensure my VM writes all data in 4k blocks?
You need only zvolblocksize to be modify /setup if you want 4k for VM(KVM guests). For a existent VM where you will make this change, only the new blocks will be with 4k zvolblocksize.
 

guletz

Well-Known Member
Apr 19, 2017
1,012
143
63
Brasov, Romania
Anyway, even with the 4k zvolblocksize, you solve only one layer(zvol layer), but you must see the rest of the storage layer(how your 4k zvolblocksize will be write on your disks who are in the zfs pool).
 

adamb

Well-Known Member
Mar 1, 2012
1,065
36
48
Anyway, even with the 4k zvolblocksize, you solve only one layer(zvol layer), but you must see the rest of the storage layer(how your 4k zvolblocksize will be write on your disks who are in the zfs pool).
They are newer 4k disks with ashift 12 so we should be good there I believe. Is that what you are referring to?
 

guletz

Well-Known Member
Apr 19, 2017
1,012
143
63
Brasov, Romania
{4k} will need to be split on all disks data from your pool.
But if you have let say 3x HDD raidz, then 4k will be split in 2 stripe(2k and 2k) any each will be landing on 2 HDD(2k/HDD). But
your disk use 4k, so it will be write 4k insted of 2k(2k of data, ant the rest of 2k with padding). So for a 4K in reality on your disk will be 8 K of data. The same will be for read!
So in this example the better option is to have {16k}!
 
Last edited:

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE and Proxmox Mail Gateway. We think our community is one of the best thanks to people like you!

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get your own in 60 seconds.

Buy now!