How to use a block device on LVM rather than a raw file for KVM machines

Y

yatesco

Renowned Member
Sep 25, 2009
230
10
83
Hi,

The title says it all really :)

I am noticing really poor performance on the KVM machines using RAW files and I just don't trust qcow2.

Previously (before proxmox) I would use an LVM volume for each machine which was mounted as a block device. Can I do the same in proxmox? I realise I could do it manually, but can I do this through the web site - assuming the volumes were created before hand?

Thanks,

Col
 
Hi,
normaly you should not have a bad performance on local storage.
What kind of storage you have?

What is the output of
Code: 
pveperf /var/lib/vz

LVM: You can add a Volumegroup (on SAN/iSCSI) and define this at the storage-menu as lvm. After that you can add disks for VMs on this storage (each disk is a logical volume).

Udo
 
Code: 
pveperf /var/lib/vz

gives:

Code: 
root@proxmox1:~/scripts# pveperf /var/lib/vz
CPU BOGOMIPS:      22664.07
REGEX/SECOND:      439456
HD SIZE:           492.15 GB (/dev/mapper/pve-data)
BUFFERED READS:    83.12 MB/sec
AVERAGE SEEK TIME: 15.85 ms
FSYNCS/SECOND:     119.74
DNS EXT:           50.78 ms
DNS INT:           5.93 ms (XXX.com)

I am embarrassed :)

This is an LSI RAID 1 card. I haven't done anything clever like aligning boundaries when creating the LVM bits.

On the non-LVM partition:

Code: 
root@proxmox1:~/scripts# time dd if=/dev/zero bs=1M count=500 of=/tmp/500M
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 0.973306 s, 539 MB/s

real    0m0.975s
user    0m0.000s
sys     0m0.812s

on the LVM partition:

Code: 
root@proxmox1:~/scripts# time dd if=/dev/zero bs=1M count=500 of=/backup/500M
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 0.872716 s, 601 MB/s

real    0m0.898s
user    0m0.000s
sys     0m0.776s

One very weird thing(!!) is that the throughput (MB/s) is inversely proportional to the size - so 600MBs for 500M, 243MBs for 1G, 110 for 2G - what is that all about???

Col
 
No idea :( These are EG machines from OVH which are configured RAID1.

I have just received a non-RAID kimsufi proxmox 1.4 which has 250 fsyncs/second. Better - but nothing to jump at.
 
yatesco said:
Code: 
pveperf /var/lib/vz
gives:

Code: 
root@proxmox1:~/scripts# pveperf /var/lib/vz
CPU BOGOMIPS:      22664.07
REGEX/SECOND:      439456
HD SIZE:           492.15 GB (/dev/mapper/pve-data)
BUFFERED READS:    83.12 MB/sec
AVERAGE SEEK TIME: 15.85 ms
FSYNCS/SECOND:     119.74
DNS EXT:           50.78 ms
DNS INT:           5.93 ms (XXX.com)
I am embarrassed :)

This is an LSI RAID 1 card. I haven't done anything clever like aligning boundaries when creating the LVM bits.

On the non-LVM partition:

Code: 
root@proxmox1:~/scripts# time dd if=/dev/zero bs=1M count=500 of=/tmp/500M
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 0.973306 s, 539 MB/s

real    0m0.975s
user    0m0.000s
sys     0m0.812s
on the LVM partition:

Code: 
root@proxmox1:~/scripts# time dd if=/dev/zero bs=1M count=500 of=/backup/500M
500+0 records in
500+0 records out
524288000 bytes (524 MB) copied, 0.872716 s, 601 MB/s

real    0m0.898s
user    0m0.000s
sys     0m0.776s
One very weird thing(!!) is that the throughput (MB/s) is inversely proportional to the size - so 600MBs for 500M, 243MBs for 1G, 110 for 2G - what is that all about???

Col
Click to expand...

Hi,
dd with 500M shows only how fast is your RAM - Linux take the ram for caching - normaly you must write a file with double size of the ram.
If you measure also the sync, than you get normal values.
Like time (dd if=/dev/zero of=bigfile bs=1024k count=10240; sync)
Lsi-Raids have not the best linuxdriver. If you have a chance, get a other product (areca, adaptec ...). This is not easy with an onboard-raid :)

Udo
 
Ah OK.

One interesting thing - I have a new machine with software raid which gives 586 fsyncs/second on /root and under half that (223 fsyncs/second) on LVM.

Does LVM really contain that much performance penalty?
 
yatesco said:
Ah OK.

One interesting thing - I have a new machine with software raid which gives 586 fsyncs/second on /root and under half that (223 fsyncs/second) on LVM.

Does LVM really contain that much performance penalty?
Click to expand...

No!
because your /root is also on a LVM-Volume. Only /boot is /dev/sda1. Root ist on /dev/pve/root and /var/lib/vz is on /dev/pve/data.
Take a look with
Code: 
vgdisplay
lvdisplay

Udo
 
No, /root is a separate (non-LVM) partition (it isn't a stock proxmox 1.4 install, the only thing that is different is the LVM partitioning).
 
Hi Udo,

My install has a different partition scheme than the stock one I think. / is mounted on software raid1. /var/lib/vz is LVM -> software raid5.

Thanks for this help Udo. A few more questions:

I have two machines, and I don't know the relevance of the pveperf results with respect to the buffered reads/fsyncs.

machine 1:
Code: 
BUFFERED READS:    102.13 MB/sec
AVERAGE SEEK TIME: 6.76 ms
FSYNCS/SECOND:     239.10

machine 2:
Code: 
BUFFERED READS:    62.62 MB/sec
AVERAGE SEEK TIME: 12.00 ms
FSYNCS/SECOND:     711.23

which has more of an influence on the system as a whole - fsyncs or buffered reads?

(I am so out of my depth here, and I realise the sillyness of the questions I am asking, but it is a start)
 
yatesco said:
Hi Udo,

My install has a different partition scheme than the stock one I think. / is mounted on software raid1. /var/lib/vz is LVM -> software raid5.
Click to expand...
Ok, this shows, why you get different values.

yatesco said:
...
which has more of an influence on the system as a whole - fsyncs or buffered reads?
Click to expand...
Thats depends on the usage of the VMs. If you have e fileserver, which support mainly files to read, then is the throughput the bottleneck. If you have an application which has many open files for writing (or write often), then is the fsync the bottleneck.
BTW - dreamvalues looks different (both). But i think you know about that :cool:

Udo
 
yatesco said:
Does LVM really contain that much performance penalty?
Click to expand...

I guess it is the combination of sotware raid with lvm - maybe you need to fine-tune block sizes/alignment - anyway, using HW RAID is much faster, simpler, and more reliable.
 
You must log in or register to reply here.
Top Bottom