ZFS zvol + cache=writeback - unexpected read speed increase?

[glaeken2]

Situation:
- ZFS on NVME, 1GB/s nvme read speed
- ARC (in ram) size about 120GB - fully heated
pve-manager/8.1.4 (running kernel: 6.5.13-1-pve)

CPU: E5-2683 v4
This CPU has a very poor per-core performance. This leads to:
- Windows 10 VM, Virtio-SCSI Single, iothread, 128kb block, cache=none -> 80MB/s linear read speed (1 thread)

As I understand, QEMU talks directly to /dev/zvol/vm-1111-disk-0 when using "cache=none"
This should lead to the max speed from zfs ARC -> QEMU -> virtio-scsi. (ZVOL native read speeds are in the range of 300MB/s single thread)

Now here comes the funny part, setting cache=writeback, DOUBLES the read speed. To about 150MB/s or more.
You say the page cache helps?
I thought so, so I did cold boots, where the page cache is empty. Also tested this after flushing the page cache with: echo 3 > /proc/sys/vm/drop_caches
Still the same conclusion. Tested on a different hardware = still the same.

The speedup is very noticable inside VMs, especially windows guests on poor per-core cpus.

Even with all caches cold, pagecache and ARC, the speed is almost double.

Why would, introducing page cache between QEMU and ZFS zvol, increase speed? The data is already in the ARC cache in memory. It should actually decrease it.

 

[_gabriel]

80MB/s linear read speed (1 thread)

How did you get this value ?
if tool like Crystal Disk Mark, it needs to write data before read back and read will from the writecache.

 

[glaeken2]

How did you get this value ?
if tool like Crystal Disk Mark, it needs to write data before read back and read will from the writecache.

It's clean read from HD Tune. Also everything works faster, starting firefox, databases reads, you name it and the vm startup itself too

 

[_gabriel]

values are too slow.
nvme is also for proxmox ve os ?
what exact model of your nvme ?

 

[glaeken2]

values are too slow.
nvme is also for proxmox ve os ?
what exact model of your nvme ?

I think it's totally normal. NVME works fine:
/dev/nvme0n1:
Timing buffered disk reads: 4130 MB in 3.00 seconds = 1376.53 MB/sec

Mutil-threaded, multiqueue reads reach around [edit] 1 gigbyte/s [/edit] from a VM (when reading from the cache)
This CPU has a poor single-thread performance, and it seems that if 1 thread reads in a windows vm, also only 1 thread does the reading in qemu. Also windows viritio-scsi is slower than linux implementation. Linux reaches around 300MB/s easily on 1 thread.

On never cpus, the speed is way higher, going into 1 gigabyte/s for a single thread inside the windows vm.

 

[LnxBil]

Windows 10 VM, Virtio-SCSI Single, iothread, 128kb block, cache=none -> 80MB/s linear read speed (1 thread)
[...]
Now here comes the funny part, setting cache=writeback, DOUBLES the read speed. To about 150MB/s or more.
You say the page cache helps?

128K blocksize will be responsible for read and write amplification and the overall low numbers. Keep in mind that sequential reads on the guest does not have to correspond to sequential reads on the ZFS. Depending on the fragmentation, the throughput may vary.

 

[glaeken2]

128K blocksize will be responsible for read and write amplification and the overall low numbers. Keep in mind that sequential reads on the guest does not have to correspond to sequential reads on the ZFS. Depending on the fragmentation, the throughput may vary.

I have a way different experience with the block size. 64k and 128k have the fastest reads in my configurations. 8k and 16k perform really poor.

But the topic is not about the speed itself. It's about WHY the read speed increases when setting cache=writeback, even with ARC cache working properly and having all the data in ram.

 

[LnxBil]

It's about WHY the read speed increases when setting cache=writeback, even with ARC cache working properly and having all the data in ram.

Idea: You double cache the data, maybe that will improve the read throughput if it's not in the first cache, it may be in the second?

I have a way different experience with the block size. 64k and 128k have the fastest reads in my configurations. 8k and 16k perform really poor.

For bigger chunks I concur, yet for randomized write IO there may be exceptions.