Improve write amplification?
- Thread starter Dunuin
- Start date
Hi,
.... my fault, sorry. After I read some other materials, I find that ZIL chain block will have less overhed if volblocksize is bigger.
Thx. a lot!
Good luck / Bafta !
Hi @Dunuin ,
Because you are on tests phase, maybe you can take in consideration to use ashift=13 ?
Good luck / Bafta !
Both cases if it is possible and you have time. I see cases when ashift=13 can be better on some particular ssds models.
Thx. a lot !
Last edited:
One interesting thing I saw in my speadsheet:
| Write amplification guest -> host | Write amplification host -> SSDs NAND | |
| sync 4K write | 9,48x | 1,25x |
| async 4K write | 4,88x | 2,36x |
I tested it here:
Still the same performance and doubled write amplification for both 128bit and 256bit AES. So looks like padding isn't the problem. Someone can explain this? Maybe someone of the staff got a deeper ZFS understanding and knows how encryption works on block level?
Last edited:
you said big volblock gives you write amplification, didnt you also increase block/cluster size in guest to match?
I would also consider doing tests with raw file on top of datasets, you get no proxmox snapshots in UI but for now you just testing performance and amplification.
Personally I am done with 4k blocks on virtualization on any new guest OS I install, I think its just too inefficient.
Here is my understanding of how writes work on recordsize, however volblocksize is different to this.
Minimum data written is ashift typically 4k.
If its a new record then no write amplification if writing less than recordsize. Only rounding up to nearest 4k, write might be smaller than actual file size due to compression, compression can only go in multiples of ashift size for data written.
If adding data to existing record, existing record has to be read either from ARC if its there or from storage, then the record is rewritten new "and" old data, so read required to do write and then having to write extra data aka amplification. Again though compression may mitigate it.
Synchronous writes as you said go to zil first, so 2 copies written, behaviour changes a little on logbias=throughput even with no slog, Adding a slog moves one of the writes away to slog device, in practice though I find usually only small data is forced sync with exception of databases and even then I dont have databases that constantly write.
Bear in mind ssd's do their own housekeeping and this alone causes write amplification, you write data, then the ssd moves it for wear levelling and this housekeeping adds to the smart write stats.
I been messing with a windows guest today, and I had to deal with some weird issues which were unexpected and after sleep now (as it pushed me back hours) I do plan to make a small virtual disk, and to test it with dataset vs zvol, different record/block sizes, and impact as well of guest cluster size. my tests are primarily for performance but I can keep an eye on the smart data as I test.
Also a note on sync=never on zfs, it is safer than you may think, write ordering is preserved when its important, zfs has another hidden setting which is what disables all flush functions, sync=never doesnt do that, but will prevent your concern of writing second copies to zil.
I would also consider doing tests with raw file on top of datasets, you get no proxmox snapshots in UI but for now you just testing performance and amplification.
Personally I am done with 4k blocks on virtualization on any new guest OS I install, I think its just too inefficient.
Here is my understanding of how writes work on recordsize, however volblocksize is different to this.
Minimum data written is ashift typically 4k.
If its a new record then no write amplification if writing less than recordsize. Only rounding up to nearest 4k, write might be smaller than actual file size due to compression, compression can only go in multiples of ashift size for data written.
If adding data to existing record, existing record has to be read either from ARC if its there or from storage, then the record is rewritten new "and" old data, so read required to do write and then having to write extra data aka amplification. Again though compression may mitigate it.
Synchronous writes as you said go to zil first, so 2 copies written, behaviour changes a little on logbias=throughput even with no slog, Adding a slog moves one of the writes away to slog device, in practice though I find usually only small data is forced sync with exception of databases and even then I dont have databases that constantly write.
Bear in mind ssd's do their own housekeeping and this alone causes write amplification, you write data, then the ssd moves it for wear levelling and this housekeeping adds to the smart write stats.
I been messing with a windows guest today, and I had to deal with some weird issues which were unexpected and after sleep now (as it pushed me back hours) I do plan to make a small virtual disk, and to test it with dataset vs zvol, different record/block sizes, and impact as well of guest cluster size. my tests are primarily for performance but I can keep an eye on the smart data as I test.
Also a note on sync=never on zfs, it is safer than you may think, write ordering is preserved when its important, zfs has another hidden setting which is what disables all flush functions, sync=never doesnt do that, but will prevent your concern of writing second copies to zil.
Last edited:
I used ext4 with "stripe-width" and xfs with "ws" to match the stripe-width of the guest fs to the blocksize of the zvol. That showed no difference.
I will test that.
Its not that easy. Atleast with linux it looks like I'm forced to use a 4K block size. I tried to increase ext4 blocksize above 4K but it told me thats not possible because the FS block size can'T be greater than the page file size of the RAM and these are 4K. So I would need to switch to hugh pages and not sure how to do that or KVM or my physical hardware is able to do that at all.
Last edited:
With ext4 use -C and also enable bigalloc feature feature. Snip below from one such partition, this was originally a 4k block partition and when I moved it to 64k clusters (alongside changing volblocksize to 64k) the performance improvement was astounding.
Code:
# tune2fs -l /dev/sdb1
tune2fs 1.44.5 (15-Dec-2018)
Filesystem volume name: <none>
Last mounted on: /home2
Filesystem UUID: 6fada182-ef45-416d-a5a0-7f85352561c2
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index sparse_super2 filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize bigalloc metadata_csum
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 128000
Block count: 131071728
Reserved block count: 0
Free blocks: 100263536
Free inodes: 125964
First block: 0
Block size: 4096
Cluster size: 65536Also look at the largefile stuff, sparse_super2, flex_bg, these reduce inodes and keep them not spread out so sequential is much more likely, I know these things most likely impact spindles, but I would expect in regards to write amplification, fragmentation would increase it, so still might be useful to you.
Thanks. I created a ext4 with "mkfs.ext4 -b 4096 -O extent -O bigalloc -O has_journal -C 32k" ontop of a 32K volblocksize zvol and ran two fio tests (32k random read/write. One sync one async) and there was only a minimal write amplification change compared to a default ext4 on a 8k volblocksize zvol. Need to do some more tests but looks like clustering will only help with large files.
Did you ran into any problems with this? Mkfs warned me that bigalloc is still a work in progress and the ext4 wiki also mentioned that this experimental and I should mount that ext4 with "nodelalloc" because there are known bugs that will corrupt your data if using delayed allocation.
Last edited:
It has been marked as experimental for a long time, I do personally use it on multiple machines where my workload is primarily large files (only on data partitions not whole OS), I have never lost data or had filesystem instability when using it.
I expect they stuck between a rock and a hard place, they need lots of people to use it to consider it not experimental, but people wont want want to use an experimental filesystem feature.
I dont use it on any commercial servers only personal ones.
Sorry to hear it had not any meaningful effect on write amplification, it did help me a lot with performance but I never analysed it for write amplification.
I expect they stuck between a rock and a hard place, they need lots of people to use it to consider it not experimental, but people wont want want to use an experimental filesystem feature.
I dont use it on any commercial servers only personal ones.
Sorry to hear it had not any meaningful effect on write amplification, it did help me a lot with performance but I never analysed it for write amplification.
Last edited:
@Dunuin do you have any tests on nvme m.2/u.2 drives ?
Most (if not all...) drives by default come with a 512 sector size lba namespace for backwards compatibility and you have to manually format them to 4096.
Since ZFS defaults to 4096 (ashift 12) I wonder how much this affects write amplification.
Here you can see it comes with Formatted LBA Size: 512
To format it (loses all data !)
After that you get
Most (if not all...) drives by default come with a 512 sector size lba namespace for backwards compatibility and you have to manually format them to 4096.
Since ZFS defaults to 4096 (ashift 12) I wonder how much this affects write amplification.
Here you can see it comes with Formatted LBA Size: 512
Code:
smartctl -a /dev/nvme0n1
...
Namespace 1 Formatted LBA Size: 512
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 2
1 - 4096 0 1To format it (loses all data !)
Code:
apt install nvme-cli
nvme format --lbaf=1 /dev/nvme0n1After that you get
Code:
smartctl -a /dev/nvme0n1
...
Namespace 1 Formatted LBA Size: 4096