Writing in a Linux VM is slow (storage = iSCSI + LVM)

[whinpo]

Hi,

I've noticed that scp a file between 2 linux VM didn't go faster than 100Mbits/s.

I've checked my network, my cards...but everything was fine..
I've tested on 2 VM on the same node : same speed...

then I decided to dd directly on the disk of a Linux VM...
and the speed is ugly...20MB/s

My storage is an iSCSI + LVM.
If it test a direct dd from the PVE node : i got something like 440MB/s
If i mount the filesystem of a Linux VM directly in the node : same speed 440MB/s..

The problem is only when the VM itself tries to write.

I've added the cache=none to the conf : speed went up to sometimes 40MB...but that's still very slow..

# pveversion -v
pve-manager: 1.7-11 (pve-manager/1.7/5470)
running kernel: 2.6.35-1-pve
proxmox-ve-2.6.35: 1.7-9
pve-kernel-2.6.32-2-pve: 2.6.32-8
pve-kernel-2.6.35-1-pve: 2.6.35-9
pve-kernel-2.6.24-10-pve: 2.6.24-21
qemu-server: 1.1-28
pve-firmware: 1.0-10
libpve-storage-perl: 1.0-16
vncterm: 0.9-2
vzctl: 3.0.24-1pve4
vzdump: 1.2-10
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1
pve-qemu-kvm: 0.13.0-3
ksm-control-daemon: 1.0-4

# cat /etc/qemu-server/102.conf 
name: Mars6
ide2: none,media=cdrom
vlan0: e1000=FE:B1:66:45:14:EF
vlan1: e1000=7A:FA:87:85:36:08
bootdisk: virtio0
ostype: l26
memory: 256
onboot: 1
sockets: 1
cores: 1
virtio0: Proxmox-SR:vm-102-disk-1,cache=none
description: 
virtio1: Proxmox-SR:vm-102-disk-2,cache=none,backup=no

mars6:~#  dd if=/dev/zero of=file1G.tmp bs=1M count=256
256+0 records in
256+0 records out
268435456 bytes (268 MB) copied, 10.3739 s, 25.9 MB/s

Any idea will be appreciated...

 

[dswartz]

Keep in mind that any form of ssh I/O is not very fast. What do you see if you try some other kind of I/O (just for comparison purposes...)?

 

[whinpo]

I agree SSH could be slow, that's why i did dd for my tests to note down the speed...
that's direct and that should be fast...

 

[dswartz]

Sorry about not reading carefully - I missed your dd test. I don't think I can believe the host node numbers? 440MB/sec is more than 3gb/sec to the iscsi target?

 

[whinpo]

seems to be due to the writing cache on the iSCSi target


with cache :

mwhinpo:/home/whinpo# dd if=/dev/zero of=/mnt/arf/bigfile bs=1024k count=1000 
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 3.22445 s, 325 MB/s
mwhinpo:/home/whinpo#

without cache

mwhinpo:/home/whinpo# dd if=/dev/zero of=/mnt/arf/bigfile bs=1024k count=1000 conv=fdatasync 
1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 16.9504 s, 61.9 MB/s

mwhinpo:/home/whinpo# pveperf /mnt/arf/
CPU BOGOMIPS:      10373.83
REGEX/SECOND:      816423
HD SIZE:           9.84 GB (/dev/mapper/ProxmoxTest--SR-TestPerfs)
BUFFERED READS:    49.77 MB/sec
AVERAGE SEEK TIME: 9.49 ms
FSYNCS/SECOND:     326.74

the fsyncs/sec are too small?

 

[udo]

seems to be due to the writing cache on the iSCSi target
...

Hi,
the cache is the memory on the host - not on the iSCSi target.

Udo

 

[whinpo]

you're right
thanks for the precision