Volunteers Wanted to test Virtual Machine Cloud Platform!

[wahmed]

Benchmark Write result from each OSD on CEPH Cluster. 1GB of data written in 4MB increment:

osd.0 = 87 MB/s
osd.1 = 83 MB/s
osd.2 = 83 MB/s
osd.4 = 89 MB/s
osd.5 = 62 MB/s
osd.6 = 76 MB/s

Any idea how these number compares with other Shared Storage? CEPH Cluster is on gigabit network. HDDs are off the shelf SATA connected to Intel RAID Controller in JBOD.

 

[riptide_wave]

Any idea how these number compares with other Shared Storage? CEPH Cluster is on gigabit network. HDDs are off the shelf SATA connected to Intel RAID Controller in JBOD.

Not sure if this helps, but here is what I get for speeds on 4x 2TB WD Green Drives in RAID 10 on a Adaptec 51645 Controler, over Gigabit Ether using NFS using UDP. (that's a mouth full)

root@Node-1-01:/mnt/pve/NFS-*****# dd if=/dev/zero of=test bs=1M count=2048
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 25.9448 s, 82.8 MB/s

 

[wahmed]

NOTE TO BETA TESTERS

With great hope to increase the I/O performance, we are making slight changes to the Test Network Infrastructure. Some changes involves putting CEPH MONs, MDSs out of Proxmox VM into their own physical machines, both Proxmox and CEPH NIC bonding to provide minimum of 3Gigabit Network, CEPH tweaks, further isolation of subnets using additional physical switches etc.
The project might take maximum of 2 days. Please run some I/O benchmarks after the weekend and let us know if you see any I/O increase. You can also run benchmarks before all the changes takes place so you can send us some side by side comparisons.

The primary goal is to make these changes without taking any Proxmox/CEPH Nodes. We will keep close eye on running VMs. But if you do see your VM went offline, you will know why.

Once again, our thanks goes out to everybody who are participating and data you have been providing to us!
Thanks Proxmox for allowing us to involve active Proxmox community to this project!

 

[wahmed]

POST NETWORK BONDING UPDATE

As you know i was going to make some major changes to the Proxmox/CEPH Cluster to increase network bandwidth. I am happy to say all changes seems to working fine, although there were some unforeseen glitches which resulted servers reboot and interruption of all VMs access. Below are some of the main changes took place:

1. Each Proxmox nodes has been upgraded to 3 gbps network bandwidth by using NIC bonding.
2. Addition of 3rd CEPH node
3. Rearranging of Proxmox network bridges.

This is what CrystalDiskMark reported before any changes:



This is what after changes took place:



This is how cluster looked while CrystalDiskMark was running:



I ran benchmark multiples times on different VMs to make sure the result was consistent. It came +/- few decimal points. Benchmarks were done on Windows 7 Pro Virtual Machines with writeback cache enabled. Could some of you please run some benchmarks and see how the result stack up? I would like to know how realistic this new numbers are
A new 3rd node was added to CEPH cluster with 3 2TB SATA HDDs. The 3rd image shows the cluster stat while one of the benchmark was running. nload and htop was used to show the stats.

This one is purely for testing. I configured one of the Windows 7 VM with 16 cores to see how it performs under Proxmox. What are you guys getting CPU scores for your machines?

 

[wahmed]

Have been running benchmark for last couple days. It seems to me that Windows 7 32 bit gives much much better I/O than Windows XP. Novabench shows higher CPU rank lower I/O on Windows XP and higher I/O and lower CPU rank for Windows 7.
Is it because SPICE is better supported on Windows 7?

 

[wahmed]

Beta Tester Global Map Update
=======================



Benchmark on a Windows 7 32 Bit
=========================

 

[wahmed]

Can somebody shed some light on the following Benchmark comparison?

Same Windows 7 Pro VM was put on CEPH and NFS storage to take this benchmark. NFS Benchmark looks somewhat unrealistic to me. And, No, there is no SSD anywhere in my setup.

Benchmark while VM on CEPH Storage:



Benchmark while VM on FreeNAS NFS Storage:


To make sure that i was not making any mistake anywhere, i ran benchmark 6 times on each storage with similar results. NFS suffered in 4k/4k QD32 Write and 4k QD32 Read but everything else insanely fast. Surely cache was playing a big role. But still.....
Would appreciate if somebody can provide some insight on this numbers.

 

[wahmed]

!! TO ALL BETA TESTER !!
===================

A huge thank you to all participated beta testers and your feedback. Your help is appreciated much more than you know.

We are going to make some changes in your assigned Virtual Machines. Even with all VMs running simultaneously, it does not seem like the Proxmox node is working too hard. Probably because quite a few VMs are not being used, they are just turned on.

So, we are going to double up the VM CPU cores from 1 to 2 to create extreme stress on the physical CPU in the Proxmox node. As of now all beta testing VMs has only one CPU core. We will keep your existing VMs so you do not lose any settings you may have done so far. This will simulate dual core VMs and should send the physical node in head spinning mode. Please let us know if extra core made any performance difference.

Thanks to you all!

 

[Amonal]

Hello symmcom
thank you four your test. I think that is very interesting.

Please, can you write all details about hard disk that you are using? Raw?

 

[wahmed]

Please, can you write all details about hard disk that you are using? Raw?

By hard disks i ssume you are asking about the virtual disks are used for all VMs. Yes all VMs uses RAW format virtio disks. I use CEPH Storage Cluster and RAW is the only thing supported. RAW format gives little peformance since it doesnt have additional layer of software as qcow2.

 

[Amonal]

Thank you, I use RAW format too, but I don't like that if you create a 200G image with only 70G of actual data, a copy operation will still need to read 200G of data.
This is a problem to backup space and for overselling space.

sorry for this offtopic, but do you recommend resize this partition to 100GB ?

Thank you, and I would like get a VM in your test environment.

 

[mir]

To solve your backup space problem you should use the following method:
To avoid huge backup files with potential lost of waste space of zeroes better split big image in a number of smaller images size 25GB and then inside vm's use LVM to assemble a number of disks into one large PV.

There are many pros to this approach for both provider (pve-host and backup) as well for consumer (VM).

 

[hotwired007]

I've just seen this thread - if you have an spare test VMs i'll be happy to use one

 

[wahmed]

Thank you, I use RAW format too, but I don't like that if you create a 200G image with only 70G of actual data, a copy operation will still need to read 200G of data.
This is a problem to backup space and for overselling space.

sorry for this offtopic, but do you recommend resize this partition to 100GB ?

Thank you, and I would like get a VM in your test environment.

I know what you mean about large RAW images. I hardly do any copy so do not feel the pain. Backup goes pretty fast though even with large image file. I believe this is due to the parsing. The size of the backup file is the size of actual data instead of full RAW image, so no space wastage there. I have few VMs which has about 500GB RAW size but with only about 120GB actual data. Whole backup does not take significantly longer than any other lower RAW size VMs.
Please email your first name, last name, city and country of origin to beta@symmcom.com for login information.

To solve your backup space problem you should use the following method:
To avoid huge backup files with potential lost of waste space of zeroes better split big image in a number of smaller images size 25GB and then inside vm's use LVM to assemble a number of disks into one large PV.

There are many pros to this approach for both provider (pve-host and backup) as well for consumer (VM).

I never thought of splitting files. Thats actually seems very clever way copy large VMs. I will have to try some VMs with split RAW image and see how it goes in my environment. Does it increase I/O performance?
Thats for the tip Mir as always!

Sent from my ASUS Transformer Pad TF700T using Tapatalk

 

[wahmed]

I've just seen this thread - if you have an spare test VMs i'll be happy to use one

Please email your first name, last name, city, country of origin to beta@symmcom.com for login info.

Sent from my ASUS Transformer Pad TF700T using Tapatalk

 

[mir]

I don't think splitting a large image up in smaller parts and add a LVM layer increases IO performance but in sure makes administration easier. Add to this that this approach means you can optimize storage use since you are able to increase or decrease storage while keeping your VM running.
For inspiration read this: http://www.microhowto.info/howto/replace_one_of_the_physical_volumes_in_an_lvm_volume_group.html
The approach is also valid for reducing disk space - of course you cannot reduce below needed space;-)

 

[wahmed]

Did a test with NFS storage setup with plain Ubuntu installation, just to see how the I/O performance is. Below is CrystalDisk benchmark. Not bad.


Got good i/o but no redundancy of any sort. At least it shows Proxmox "can" put out high performance. Its just the limitation of the shared storage itself which causes slow performance. CEPH for example in my situation.

 

[wahmed]

Map update of our wonderful Beta Testers around the world!

 

[wahmed]

We have added CephFS storage into the Beta Testing Platform. It is currently setup to only create OpenVZ Containers.
Please test it by creating Container VM at your heart content!

Although CephFS cannot be compared to the strength of NFS, but in future for a CEPH platform it has a good chance of NFS alternative. The entire CEPH Beta Testing platform has been upgraded to the latest CEPH release version of 0.72. We are hoping to see much improvement has been made by InkTank, the maker of CEPH Storage.
As always your feedback is much appreciated!

 

[wahmed]

BREAKING NEWS !!

Due to a user error during CEPH Cluster MON changes, entire storage cluster came crashing down which brought Proxmox cluster down with it. Our apologies to all Beta Tester for the service interruption. As soon as we know what we are doing, we will bring both cluster online. A full report on what went wrong and how we can prevent future crashes will be posted at later time.