zfs/zvol recordsize vs zvolblocksize

[adamb]

We are trying to tune a zfs mirror running on a proxmox host for our database setup.

I am trying to get a better understanding of zfs/zvol recordsize vs zvolblocksize.

If ZFS is set to a record size of 128k, the zvol is default to 8k, which one is actually used? Does the zvolblocksize override the recordsize of the pool?

 

[Whatever]

volblocksize is used only for ZVOLs
recordsize is used for datasets

If you try to get all properties of zvol you will realize that there is no "recordsize" and vice versa

From my experience I could suggest to use ZVOL whenever it's possible. "volblocksize" mainly depends on pool configuration and disk modelы and should be chosen after some performance tests

 

[guletz]

From my experience I could suggest to use ZVOL whenever it's possible.

zvol is possible in only one case: if you use VM

volblocksize" mainly depends on pool configuration and disk modelы and should be chosen after some performance tests

volblocksize is not depend for anything. It will be what you will setup. By default if you do not specify a value, it will be 8 k if I remember.

 

[Whatever]

volblocksize is not depend for anything. It will be what you will setup. By default if you do not specify a value, it will be 8 k if I remember.

Well.. the right size of volblocksize (in terms of performance) should be determined carefully via performing some measurements and tests. The same volblocksize values could vary depending on disk models, pool configuration and a workload

 

[mir]

Well.. the right size of volblocksize (in terms of performance) should be determined carefully via performing some measurements and tests. The same volblocksize values could vary depending on disk models, pool configuration and a workload

Another thing to take into consideration is storage efficiency. You should try to match volblock size with actual size of the written blocks. If you primarily do 4k writes, like most database systems, then favor a volblock size of 4k.

 

[guletz]

you primarily do 4k writes, like most database systems, then favor a volblock size of 4k.

Hi @mir

It is a trick with most databases who use the ancient 4k (postgresql)
- using 4kn disk in any non-mirror setup (so not raid 10) you will have performance problems
- some others databases use bigger block size (16 k mysql/percona/maria, 32-64 k mssql)

Bigger block = better performance in zfs pools even for non-mirror (raid10, who is very usual for any DB engine)

And at least for mysql/percona/maria, 16 k is ok for innodb files, but for logs/jurnal any size bigger then 16 k is better.

So it is not so simple to decide what is better or optim as a general speaking.

Good luck !

 

[guletz]

The same volblocksize values could vary depending on disk models, pool configuration and a workload

100% agree with this.

 

[adamb]

Still not quite clear. If I set the zvolblocksize to 4k, it will override the 128k recordsize set in the main pool?

 

[guletz]

Still not quite clear. If I set the zvolblocksize to 4k, it will override the 128k recordsize set in the main pool?

The size of zvolblocksize it has nothing to do and is not corelated with any DATASET recordsize. This 2 proprieties (zvolblocksize/recordsize) are 2 different things!

ZFS datasets use an internal recordsize of 128KB by default.
Zvols have a volblocksize property that is analogous to record size. The default size is 8KB

Details here:

http://www.open-zfs.org/wiki/Performance_tuning

 

[adamb]

The size of zvolblocksize it has nothing to do and is not corelated with any DATASET recordsize. This 2 proprieties (zvolblocksize/recordsize) are 2 different things!

Ok, then do I need to change both zvolblocksize and recordsize to ensure my VM writes all data in 4k blocks? Im just trying to figure out what I need to do to ensure I am writing in 4k blocks.

 

[guletz]

Ok, then do I need to change both zvolblocksize and recordsize to ensure my VM writes all data in 4k blocks?

You need only zvolblocksize to be modify /setup if you want 4k for VM(KVM guests). For a existent VM where you will make this change, only the new blocks will be with 4k zvolblocksize.

 

[guletz]

Anyway, even with the 4k zvolblocksize, you solve only one layer(zvol layer), but you must see the rest of the storage layer(how your 4k zvolblocksize will be write on your disks who are in the zfs pool).

 

[adamb]

Anyway, even with the 4k zvolblocksize, you solve only one layer(zvol layer), but you must see the rest of the storage layer(how your 4k zvolblocksize will be write on your disks who are in the zfs pool).

They are newer 4k disks with ashift 12 so we should be good there I believe. Is that what you are referring to?

 

[guletz]

{4k} will need to be split on all disks data from your pool.
But if you have let say 3x HDD raidz, then 4k will be split in 2 stripe(2k and 2k) any each will be landing on 2 HDD(2k/HDD). But
your disk use 4k, so it will be write 4k insted of 2k(2k of data, ant the rest of 2k with padding). So for a 4K in reality on your disk will be 8 K of data. The same will be for read!
So in this example the better option is to have {16k}!

 

[juliokele]

{4k} will need to be split on all disks data from your pool.
But if you have let say 3x HDD raidz, then 4k will be split in 2 stripe(2k and 2k) any each will be landing on 2 HDD(2k/HDD). But
your disk use 4k, so it will be write 4k insted of 2k(2k of data, ant the rest of 2k with padding). So for a 4K in reality on your disk will be 8 K of data. The same will be for read!
So in this example the better option is to have {16k}!

4k NTFS ZVOL on RAID10 2x2Disk (4k), witch volblocksize: 8k or 16k option is better?

 

[guletz]

Hi,

I would say 16k, but ignoring ather fact(VM load/services, rate of data compression and so on).

 

[juliokele]

Hi,

I would say 16k, but ignoring ather fact(VM load/services, rate of data compression and so on).

Hi,

can you please explain why 16k and while not 8k?

 

[guletz]

can you please explain why 16k and while not 8k?

Because most of the SSDs use 16k or even bigger.

 

[juliokele]

Because most of the SSDs use 16k or even bigger.

It is Hard Disk not SSD (2x2Disk (4k))!

 

[guletz]

is Hard Disk not SSD (2x2Disk (4k))!

.... a on a hdd mirror? volblocksize depend by what os do you use and what services do you have on this VMs. See my other posts about this subject!