Local VM slower than RBD VM?

[wahmed]

I am totally surprised by this result and i am hoping we can find some kind of explanation for this.

To eliminate my CEPH cluster as cause of VM slowdown, i have cloned a Windows XP machine on Proxmox Local SATA HDD Storage. Then i ran CrystalDiskMark program on both identical machines, one on local and the other on RBD storage. Both VM has writeback cache enabled on virtual disks.

VM Running on CEPH RBD storage
=======================


VM Running on Proxmox Local Storage
===========================


Am i seeing correctly that VM running on Local storage is little slower than RBD Storage? I triple checked that i did not mix up 2 images. I was hoping to see Local storage benchmark well above RBD benchmarks. I am almost sure i have done something wrong somewhere. Any ideas?

 

[mir]

Local contains only one SATA disk which implies a theoretical max of 120 MB/s. I bet your CEPH cluster contains many more disks and since CEPH will distribute the load over all your disks increasing the theoretical max. Though restricted to the speed of the network. If your local storage is not placed on either SSD or RAID you will never be able to outperform your CEPH cluster.

 

[wahmed]

Local contains only one SATA disk which implies a theoretical max of 120 MB/s. I bet your CEPH cluster contains many more disks and since CEPH will distribute the load over all your disks increasing the theoretical max. Though restricted to the speed of the network. If your local storage is not placed on either SSD or RAID you will never be able to outperform your CEPH cluster.

Before i say anything, may i present feast for your eyes.....


Yes, the benchmark above was achieved through the same VM, only this time i put it on a local Proxmox SSD. Soooo tempted to replace ALL hard drives with SSD throughout the network. Does not hurt to dream

Clearly Proxmox has the ability to reach that speed for a VM. You are right Mir, the CEPH cluster has 8 HDDs so it makes sense about load distribution. But local HDD should get little more than the speed i got. Perhaps it is my HDD. Have to check on that.

 

[thesubmitter]

We're using a mixed environment of SSDs, iscsiSAN, and Local HDs -

 

[wahmed]

I have bonded 3 NICs to create 3Gbps network connectivity in the 2 CEPH nodes and 2 of the Proxmox nodes.

#netperf comamnd shows the following:

cephadmin@ceph-OSD-02:~$ netperf -f G
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost (127.0.0.1) port 0 AF_INET : demo
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. GBytes/sec
87380 16384 16384 10.00 3.21

From the above it seems to me the bonding worked. But if i run iperf i get the following:

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
[ 4] local 192.101.60.2 port 5001 connected with 192.101.60.44 port 47319
[ 4] 0.0-10.0 sec 1.10 GBytes 0.11 GBytes/sec

So, netperf says i have 3gbps connection, iperf says i have 1 gbps. Did i miss something in the setup? Also file copy between is slow as usual without any sort improvement. All VMs are running on 3gbps connection.