Metadata for the ZFS filesystem, e.g. the pointers and checksums for each block, then the pointers and checksums for that, redundantly written, all the way up to the tree. The amplification involved in that ends up totally filling the pipeline with IOPs for every kind of underlying media, and then you end up at a fraction of the throughput. (Although in a theoretical edge case where you are way below the max throughout but need to only occasionally write independent small blocks, it could have lower latency.)Sorry, I am not sure if I can follow. By metadata do you mean parity data?
I also still don't understand if a guest with a 4k filesystem is able to write bigger than 4k.
Is it the case that, no the guest can't write bigger than its FS. Or is it something like, multiple 4k writes of the guest will get cached together if they are not sync, so there could be a single 64k write on ZFS that consists of 16x 4k writes inside the guest?
Most filesystems use 64k extents in grouping data together anyways (even though it is divided up into 4k iops) so it ends up lining up nicely with the blocks when volblocksize=64k