Greetings,
I'm trying to analyze and optimize read and write performance of a VM which is performing the role of NAS and has control over some of the storage. Setup is as following
Virtual Environment 5.3-12
VM config (some parts removed):
On hardware side, there is one hard drive controlled by raid bbu controller with write back caching policy.
I did perform some tests with VM caching setting Write Through and DirectSync. I'm looking after reliability first and speed as second factor so I went for one of those two caching modes as mentioned.
For writes i did test with:
For reads I used
When performing write inside the VM it seems DirectSync perform little better on the write side even when both DirectSync and Write Through caching methods prove to be mostly inconsistent even when writing in same file (maybe CoW?). Speed can greatly vary between 39MB/s and 100MB/s for Direct Sync and 34MB/s to 47MB/s for Write Through. On the read side DirectSync and Write Throught perform the same and first read after VM reboot or reads with iflag/oflag=nocache end up at consistent 18-20MB/s. After the file been chached in ram reading numbers are through the roof and not been interesting.
When i perform same tests within the host, writes can be about 20-30MB/s better but reads are multiple times better reaching easily 65-85MB/s. I'm trying to figure out why reads inside the VM are much worse and how to improve upon that. DirectSync seems to be doing little bit better over Write Through as well as I'm thinking that host page caching - the read cache that host builds in Write Through mode, is not needed since the VM builds cache of its own anyway, caching the same stuff, so instead of wasting double the amount of ram to cache same files once in host and once in VM, I can use Direct Sync and have those files cached only inside the VM's ram. I'm thinking host's ram should rather be left free for caching stuff related to other LXCs and activities. Correct me if I'm wrong. Advices are welcome!
Thank you
I'm trying to analyze and optimize read and write performance of a VM which is performing the role of NAS and has control over some of the storage. Setup is as following
Virtual Environment 5.3-12
VM config (some parts removed):
Code:
balloon: 0
boot: dcn
bootdisk: scsi0
cores: 2
cpu: host,flags=+pcid
hookscript:
ide2: none,media=cdrom
memory: 10240
name:
net0: virtio= ,bridge=vmbr0
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,cache=directsync,size=156G
scsihw: virtio-scsi-pci
smbios1:
sockets: 1
startup: order=1,up=90,down=420
vmgenid:On hardware side, there is one hard drive controlled by raid bbu controller with write back caching policy.
I did perform some tests with VM caching setting Write Through and DirectSync. I'm looking after reliability first and speed as second factor so I went for one of those two caching modes as mentioned.
For writes i did test with:
Code:
dd bs=512 count=5120000 if=/dev/zero of=write_test.img
and
dd bs=512 iflag=nocache oflag=nocache count=5120000 if=/dev/zero of of=write_test.imgFor reads I used
Code:
dd bs=512 if=write_test.img of=/dev/null
and
dd bs=512 iflag=nocache oflag=nocache if=write_test.img of=/dev/nullWhen performing write inside the VM it seems DirectSync perform little better on the write side even when both DirectSync and Write Through caching methods prove to be mostly inconsistent even when writing in same file (maybe CoW?). Speed can greatly vary between 39MB/s and 100MB/s for Direct Sync and 34MB/s to 47MB/s for Write Through. On the read side DirectSync and Write Throught perform the same and first read after VM reboot or reads with iflag/oflag=nocache end up at consistent 18-20MB/s. After the file been chached in ram reading numbers are through the roof and not been interesting.
When i perform same tests within the host, writes can be about 20-30MB/s better but reads are multiple times better reaching easily 65-85MB/s. I'm trying to figure out why reads inside the VM are much worse and how to improve upon that. DirectSync seems to be doing little bit better over Write Through as well as I'm thinking that host page caching - the read cache that host builds in Write Through mode, is not needed since the VM builds cache of its own anyway, caching the same stuff, so instead of wasting double the amount of ram to cache same files once in host and once in VM, I can use Direct Sync and have those files cached only inside the VM's ram. I'm thinking host's ram should rather be left free for caching stuff related to other LXCs and activities. Correct me if I'm wrong. Advices are welcome!
Thank you
Last edited:
