Proxmox Ve 2.3 high load on host

[dracmic]

Hi,
I have a running ProxMox Ve 2.3 server with :
pve-manager: 2.3-13 (pve-manager/2.3/7946f1f1)
running kernel: 2.6.32-19-pve
proxmox-ve-2.6.32: 2.3-93
pve-kernel-2.6.32-16-pve: 2.6.32-82
pve-kernel-2.6.32-19-pve: 2.6.32-93
lvm2: 2.02.95-1pve2
clvm: 2.02.95-1pve2
corosync-pve: 1.4.4-4
openais-pve: 1.1.4-2
libqb: 0.10.1-2
redhat-cluster-pve: 3.1.93-2
resource-agents-pve: 3.9.2-3
fence-agents-pve: 3.1.9-1
pve-cluster: 1.0-36
qemu-server: 2.3-18
pve-firmware: 1.0-21
libpve-common-perl: 1.0-49
libpve-access-control: 1.0-26
libpve-storage-perl: 2.3-6
vncterm: 1.0-3
vzctl: 4.0-1pve2
vzprocps: 2.0.11-2
vzquota: 3.1-1
pve-qemu-kvm: 1.4-8
ksm-control-daemon: 1.1-1

I`m running 7 qemu KVM machines with virtio on NIC and HDD (6 of 7 are databases server and 1 is a high intensive cpu working) , but with some guests with cpu type=host , the quests load are 3 avg on all.
But on host i have some periods of time about 70-80% of the day load over 20 avg. and cpu usage reported in proxmox interface it is about 30% .
Host has : 2xIntel(R) Xeon(R) CPU E5-2640 and 96 GB of Ram.

What i can do to drop the load off on the host because running 7 qemu vps on this type of machine it isn`t cost effective.


Thanks,
Mihai

 

[snowman66]

You can reduce idle CPU usage (and context switches) by disabling "Use Tablet for pointer".

http://pve.proxmox.com/wiki/Performance_Tweaks

 

[dracmic]

I forgot to mention that all guests are running Ubuntu 12.04.2 with kernel 3.5.

/etc/pve/qemu-server# vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
8 0 0 31741128 45712 15659204 0 0 37 145 0 0 8 1 89 2

---
later add
print screen with htop from host after changing from 6 out of 7 VM the context switch.

 

[dracmic]

Any help with this ?

 

[udo]

Hi,
such high loads are normaly IO-related. How fast is your IO-Subsystem?

Perhaps you can try to use another scheduler (deadline)?

Udo

 

[dracmic]

Hi,
such high loads are normaly IO-related. How fast is your IO-Subsystem?

Perhaps you can try to use another scheduler (deadline)?

Udo

CPU BOGOMIPS: 119697.24
REGEX/SECOND: 1096306
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 116.73 MB/sec
AVERAGE SEEK TIME: 7.55 ms
FSYNCS/SECOND: 1170.60

Deadline on host or on guests ?

on host :
host : cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]
cat /sys/block/md0/queue/scheduler
none


vmstat:
vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
7 0 0 34561468 30944 846964 0 0 38 146 0 0 8 1 88 2
9 0 0 34559732 30960 848944 0 0 788 479 16504 16289 25 1 72 1
4 5 0 34553340 30960 856144 0 0 2269 3415 15951 17484 21 1 75 3
6 1 0 34549144 30968 860044 0 0 1715 2026 14274 13125 24 1 71 4
4 0 0 34548224 30976 860508 0 0 125 370 14706 14513 20 1 78 1
6 0 0 34547540 30980 861024 0 0 949 465 13445 14058 19 1 78 1
7 1 0 34543916 30988 862300 0 0 484 3569 13857 15862 18 1 77 4
4 0 0 34539452 31000 862820 0 0 656 1945 16146 18912 21 1 71 7
6 0 0 34541852 31016 862880 0 0 69 457 13228 13277 20 1 78 2
4 0 0 34536948 31024 866196 0 0 1157 420 12937 14125 19 1 79 1
7 1 0 34538616 31032 866340 0 0 3083 2846 15046 17243 19 1 76 5
1 0 0 34538960 31052 866448 0 0 3719 2751 14698 17388 20 1 75 4
4 0 0 34538208 31064 866944 0 0 160 482 13600 14046 20 1 78 2

 

[udo]

CPU BOGOMIPS: 119697.24
REGEX/SECOND: 1096306
HD SIZE: 94.49 GB (/dev/mapper/pve-root)
BUFFERED READS: 116.73 MB/sec
AVERAGE SEEK TIME: 7.55 ms
FSYNCS/SECOND: 1170.60

Looks like single SAS-Disk (or a good SATA-disk) or? For high IO-Load is an good raidcontroller with fast disks and raid-10 recommendable.
Only to compare 4 SAS-disk raid-10:

pveperf /mnt/local_pve
CPU BOGOMIPS:      69350.00
REGEX/SECOND:      1375241
HD SIZE:           444.91 GB (/dev/mapper/pve_local-data)
BUFFERED READS:    400.58 MB/sec
AVERAGE SEEK TIME: 5.41 ms
FSYNCS/SECOND:     4847.42

Deadline on host or on guests ?

host (more important) and guest.

on host :
host : cat /sys/block/sda/queue/scheduler
noop anticipatory deadline [cfq]
cat /sys/block/md0/queue/scheduler
none


vmstat:
vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
7 0 0 34561468 30944 846964 0 0 38 146 0 0 8 1 88 2
9 0 0 34559732 30960 848944 0 0 788 479 16504 16289 25 1 72 1
4 5 0 34553340 30960 856144 0 0 2269 3415 15951 17484 21 1 75 3
6 1 0 34549144 30968 860044 0 0 1715 2026 14274 13125 24 1 71 4
4 0 0 34548224 30976 860508 0 0 125 370 14706 14513 20 1 78 1
6 0 0 34547540 30980 861024 0 0 949 465 13445 14058 19 1 78 1
7 1 0 34543916 30988 862300 0 0 484 3569 13857 15862 18 1 77 4
4 0 0 34539452 31000 862820 0 0 656 1945 16146 18912 21 1 71 7
6 0 0 34541852 31016 862880 0 0 69 457 13228 13277 20 1 78 2
4 0 0 34536948 31024 866196 0 0 1157 420 12937 14125 19 1 79 1
7 1 0 34538616 31032 866340 0 0 3083 2846 15046 17243 19 1 76 5
1 0 0 34538960 31052 866448 0 0 3719 2751 14698 17388 20 1 75 4
4 0 0 34538208 31064 866944 0 0 160 482 13600 14046 20 1 78 2

How looks the overall IO-Wait with top?
Like here

Cpu(s):  4.5%us,  2.8%sy,  0.0%ni, 92.1%id,  0.4%wa,  0.0%hi,  0.2%si,  0.0%st

Udo

 

[dracmic]

vztop:
539 processes: 538 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states: 7.4% user 3.3% system 0.0% nice 3.3% iowait 85.0% idle
CPU1 states: 36.3% user 0.4% system 0.0% nice 2.2% iowait 60.1% idle
CPU2 states: 21.1% user 0.5% system 0.0% nice 1.1% iowait 76.2% idle
CPU3 states: 16.1% user 0.1% system 0.0% nice 1.3% iowait 81.5% idle
CPU4 states: 1.4% user 0.4% system 0.0% nice 2.1% iowait 95.0% idle
CPU5 states: 2.1% user 0.1% system 0.0% nice 0.0% iowait 97.3% idle
CPU6 states: 12.1% user 0.5% system 0.0% nice 4.3% iowait 82.0% idle
CPU7 states: 42.0% user 0.5% system 0.0% nice 2.4% iowait 54.0% idle
CPU8 states: 17.5% user 0.2% system 0.0% nice 1.3% iowait 80.0% idle
CPU9 states: 20.3% user 0.2% system 0.0% nice 0.0% iowait 79.0% idle
CPU10 states: 25.1% user 0.3% system 0.0% nice 2.5% iowait 71.1% idle
CPU11 states: 7.0% user 0.2% system 0.0% nice 0.4% iowait 91.4% idle
CPU12 states: 8.1% user 0.4% system 0.0% nice 20.0% iowait 70.5% idle
CPU13 states: 9.3% user 0.4% system 0.0% nice 2.5% iowait 86.3% idle
CPU14 states: 27.4% user 1.1% system 0.0% nice 0.0% iowait 70.5% idle
CPU15 states: 12.1% user 0.3% system 0.0% nice 0.0% iowait 87.0% idle
CPU16 states: 12.2% user 0.3% system 0.0% nice 0.1% iowait 86.4% idle
CPU17 states: 0.2% user 0.1% system 0.0% nice 0.1% iowait 99.1% idle
CPU18 states: 9.0% user 0.2% system 0.0% nice 6.1% iowait 84.1% idle
CPU19 states: 1.3% user 0.2% system 0.0% nice 4.3% iowait 93.1% idle
CPU20 states: 2.1% user 0.1% system 0.0% nice 1.4% iowait 95.3% idle
CPU21 states: 7.4% user 0.4% system 0.0% nice 0.0% iowait 91.2% idle
CPU22 states: 21.2% user 0.0% system 0.0% nice 0.5% iowait 77.2% idle
CPU23 states: 0.1% user 0.1% system 0.0% nice 2.3% iowait 97.0% idle
Mem: 98944268k av, 76533668k used, 22410600k free, 0k shrd, 44508k buff
top :
Cpu(s): 23.9%us, 0.7%sy, 0.0%ni, 72.6%id, 2.6%wa, 0.0%hi, 0.2%si, 0.0%st



disks are :
1. WDC WD1002FAEX-0
2. 3 x WDC WD5000HHTZ-0

Controller: description: SATA controller
product: C600/X79 series chipset 6-Port SATA AHCI Controller
vendor: Intel Corporation

Meanwhile changed to deadline (later edit) Changed to deadline no change.
Any more sugestions ?later edit changing to other type of virtualization i was thinking at Xen ?

 

[udo]

vztop:
539 processes: 538 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states: 7.4% user 3.3% system 0.0% nice 3.3% iowait 85.0% idle
CPU1 states: 36.3% user 0.4% system 0.0% nice 2.2% iowait 60.1% idle
CPU2 states: 21.1% user 0.5% system 0.0% nice 1.1% iowait 76.2% idle
CPU3 states: 16.1% user 0.1% system 0.0% nice 1.3% iowait 81.5% idle
CPU4 states: 1.4% user 0.4% system 0.0% nice 2.1% iowait 95.0% idle
CPU5 states: 2.1% user 0.1% system 0.0% nice 0.0% iowait 97.3% idle
CPU6 states: 12.1% user 0.5% system 0.0% nice 4.3% iowait 82.0% idle
CPU7 states: 42.0% user 0.5% system 0.0% nice 2.4% iowait 54.0% idle
CPU8 states: 17.5% user 0.2% system 0.0% nice 1.3% iowait 80.0% idle
CPU9 states: 20.3% user 0.2% system 0.0% nice 0.0% iowait 79.0% idle
CPU10 states: 25.1% user 0.3% system 0.0% nice 2.5% iowait 71.1% idle
CPU11 states: 7.0% user 0.2% system 0.0% nice 0.4% iowait 91.4% idle
CPU12 states: 8.1% user 0.4% system 0.0% nice 20.0% iowait 70.5% idle
CPU13 states: 9.3% user 0.4% system 0.0% nice 2.5% iowait 86.3% idle
CPU14 states: 27.4% user 1.1% system 0.0% nice 0.0% iowait 70.5% idle
CPU15 states: 12.1% user 0.3% system 0.0% nice 0.0% iowait 87.0% idle
CPU16 states: 12.2% user 0.3% system 0.0% nice 0.1% iowait 86.4% idle
CPU17 states: 0.2% user 0.1% system 0.0% nice 0.1% iowait 99.1% idle
CPU18 states: 9.0% user 0.2% system 0.0% nice 6.1% iowait 84.1% idle
CPU19 states: 1.3% user 0.2% system 0.0% nice 4.3% iowait 93.1% idle
CPU20 states: 2.1% user 0.1% system 0.0% nice 1.4% iowait 95.3% idle
CPU21 states: 7.4% user 0.4% system 0.0% nice 0.0% iowait 91.2% idle
CPU22 states: 21.2% user 0.0% system 0.0% nice 0.5% iowait 77.2% idle
CPU23 states: 0.1% user 0.1% system 0.0% nice 2.3% iowait 97.0% idle
Mem: 98944268k av, 76533668k used, 22410600k free, 0k shrd, 44508k buff
top :
Cpu(s): 23.9%us, 0.7%sy, 0.0%ni, 72.6%id, 2.6%wa, 0.0%hi, 0.2%si, 0.0%st

Your processes needs more IO as your system can support.
Normaly iowait should be very small:

746 processes: 745 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states:    3.0% user   4.4% system    0.0% nice   0.0% iowait  92.0% idle
CPU1 states:    5.1% user   2.4% system    0.0% nice   0.0% iowait  91.4% idle
CPU2 states:    5.1% user   2.2% system    0.0% nice   0.0% iowait  92.1% idle
CPU3 states:    6.2% user   1.4% system    0.0% nice   0.0% iowait  91.3% idle
CPU4 states:    2.2% user   4.0% system    0.0% nice   0.0% iowait  93.2% idle
CPU5 states:    2.3% user   3.2% system    0.0% nice   0.0% iowait  93.4% idle
CPU6 states:    2.1% user   1.4% system    0.0% nice   0.0% iowait  95.4% idle
CPU7 states:    1.3% user   3.4% system    0.0% nice   0.0% iowait  94.2% idle
CPU8 states:    1.2% user   2.1% system    0.0% nice   0.0% iowait  96.1% idle
CPU9 states:    2.2% user   2.2% system    0.0% nice   0.0% iowait  95.0% idle
CPU10 states:   2.4% user   2.1% system    0.0% nice   0.0% iowait  94.3% idle
CPU11 states:   2.2% user   1.3% system    0.0% nice   0.0% iowait  95.4% idle
CPU12 states:   2.2% user   1.4% system    0.0% nice   0.0% iowait  95.3% idle
CPU13 states:   1.2% user   2.2% system    0.0% nice   0.0% iowait  96.1% idle
CPU14 states:   4.3% user   1.2% system    0.0% nice   0.0% iowait  93.4% idle
CPU15 states:   1.3% user   1.0% system    0.0% nice   0.0% iowait  97.0% idle
CPU16 states:   1.3% user   1.4% system    0.0% nice   0.0% iowait  96.2% idle
CPU17 states:   1.2% user   1.1% system    0.0% nice   0.0% iowait  97.1% idle
CPU18 states:   1.0% user   1.0% system    0.0% nice   0.0% iowait  97.4% idle
CPU19 states:   1.0% user   1.2% system    0.0% nice   0.0% iowait  97.2% idle
CPU20 states:   2.1% user   2.2% system    0.0% nice   0.0% iowait  95.1% idle
CPU21 states:   2.0% user   2.3% system    0.0% nice   0.0% iowait  95.1% idle
CPU22 states:   2.1% user   2.2% system    0.0% nice   0.0% iowait  95.1% idle
CPU23 states:   1.1% user   1.4% system    0.0% nice   0.0% iowait  96.4% idle
CPU24 states:   1.4% user   2.0% system    0.0% nice   0.0% iowait  96.0% idle
CPU25 states:   3.3% user   2.3% system    0.0% nice   0.0% iowait  93.2% idle
CPU26 states:   1.1% user   1.1% system    0.0% nice   0.0% iowait  97.2% idle
CPU27 states:   1.2% user   1.0% system    0.0% nice   0.0% iowait  97.2% idle
CPU28 states:   2.0% user   2.0% system    0.0% nice   0.0% iowait  95.4% idle
CPU29 states:   2.1% user   2.2% system    0.0% nice   0.0% iowait  95.1% idle
CPU30 states:   1.3% user   2.0% system    0.0% nice   0.0% iowait  96.1% idle
CPU31 states:   8.1% user   1.3% system    0.0% nice   0.0% iowait  90.0% idle

disks are :
1. WDC WD1002FAEX-0
2. 3 x WDC WD5000HHTZ-0

Controller: description: SATA controller
product: C600/X79 series chipset 6-Port SATA AHCI Controller
vendor: Intel Corporation

don't know this controller - is it an real hardware raidcontroller or software-raid?
The 20%-iowait on one cpu looks a little bit like softwareraid (that the driver is working on that core...)
I guess you use raid-5 (bad write performance).

Meanwhile changed to deadline (later edit) Changed to deadline no change.
Any more sugestions ?later edit changing to other type of virtualization i was thinking at Xen ?

I have no experiences with Xen, but I assume that your host don't get better IO... how should that work.
Years ago, I have done some test between vmware and pve - pve has an better io-performance.

Udo

 

[dracmic]

Software raid :
Raid Level : raid10
I suppose on board controller , i am new on this company and i am trying to find a solution. Do you think buying a real RAID controller would help ?

 

[udo]

Software raid :
Raid Level : raid10
I suppose on board controller , i am new on this company and i am trying to find a solution. Do you think buying a real RAID controller would help ?

Hi,
raid-10 with one 1TB disk (7200rpm) and three 500GB-disks (10000 rpm)?

A fast real hardware raid will help a lot! I have good experiances with areca-raidcontroller (get an SAS-controller also if you only have sata-disks - the controller is faster and you can change the disk to sas-drives later).
If you got one more 500GB-Raptor and an good raidcontroller you will got much more performance.

Unfortunality good raid-controller are not cheap, but normaly you got where you paid for. I have onboard lsi-controller with an very bad performance...

Udo

 

[dracmic]

Hi ,
the 1 TB isn`t included in raid , only the three 500GB-disks all at 10.000 rpm all wd velociraptor.
Any specific model ?
Thanks.

Hi,
raid-10 with one 1TB disk (7200rpm) and three 500GB-disks (10000 rpm)?

A fast real hardware raid will help a lot! I have good experiances with areca-raidcontroller (get an SAS-controller also if you only have sata-disks - the controller is faster and you can change the disk to sas-drives later).
If you got one more 500GB-Raptor and an good raidcontroller you will got much more performance.

Unfortunality good raid-controller are not cheap, but normaly you got where you paid for. I have onboard lsi-controller with an very bad performance...

Udo

 

[udo]

Hi ,
the 1 TB isn`t included in raid , only the three 500GB-disks all at 10.000 rpm all wd velociraptor.
Any specific model ?
Thanks.

Hi,
you can't build an raid-10 with 3 disks. Minimum is 4 (then 6,8...) disks. Because you need two disksets (raid-1) which are striped (raid-0) (1+0=raid10).

If you have the cance to get new disks, I prefer SAS disks like HITACHI HUS156060VLS600.

Udo

 

[dracmic]

Hi ,
I am tired i worked to find a solution for it . Yes they are 4 disk in the raid .
I was toalking about the model of the raid control the specific model.

 

[kotakomputer]

disks are :
1. WDC WD1002FAEX-0
2. 3 x WDC WD5000HHTZ-0

the 1 TB isn`t included in raid , only the three 500GB-disks all at 10.000 rpm all wd velociraptor.

As udo said you need 4 HDD to create RAID 10 (1+0). With 3 HDD you may have RAID 5 instead.

 

[dracmic]

to be complete i omitted 1 500 gb hdd extra in system are : 1x 1 tb and 4 x 500gb not 3 x 500gb how i said before. and i was asking about a recommended hw raid controller and udo said areca but what model.

 

[drott]

to be complete i omitted 1 500 gb hdd extra in system are : 1x 1 tb and 4 x 500gb not 3 x 500gb how i said before. and i was asking about a recommended hw raid controller and udo said areca but what model.

Several other considerations due to your high load and dbs tend to be disk and bus IO intensive, you will want the controller with the fastest processor, and battery backed flash write back cache ram. I would go for a gig and a good strong LSI based card. I tend to use HP 800 series controller with hotswap dual ported 6g sas 15 k drives. If you want something really fast look at the fusion io like cards, 768 gb drive with fully self healing ram is just over 5k but makes anything disk based with cache ram look like a downright sluggard and doing them in a mirror for a db environment is sweet.

The velociraptor drives just peter out on high io and i have never been able to get more than 6 esx kvms to run and there would always be one kvm that seemes to suck the io out of the system.

i would look to sd drives for the boot and logging.

If you can, move from kvm to ct for as many of the vms that you can. CTs have significantly less io overhead, raw access to io and bus devices. Dbs like postgress and mysql are very happy in cts



The key here is to not use software raid and the good cards w fastest cache and largest cache will cost 800 to 1200 depending on vendor. No matter what you do it is imperitive that the cache ram be battery backed. It better to have that with a slightly smaller cache ram.

 

[dracmic]

Hi,
The problem appeared after i moved out the newsletter system which is the guilty one in this case , the 2 postgres and 3 mysql server didn`t do so much trouble.
And the budget for Raid Controller is about 700 euros/800 dollars. I read out that is a bug in pve-2.6.32 on mdadm raid and i will put today a test system out of production to see if that is the problem.

Thanks.

Several other considerations due to your high load and dbs tend to be disk and bus IO intensive, you will want the controller with the fastest processor, and battery backed flash write back cache ram. I would go for a gig and a good strong LSI based card. I tend to use HP 800 series controller with hotswap dual ported 6g sas 15 k drives. If you want something really fast look at the fusion io like cards, 768 gb drive with fully self healing ram is just over 5k but makes anything disk based with cache ram look like a downright sluggard and doing them in a mirror for a db environment is sweet.

The velociraptor drives just peter out on high io and i have never been able to get more than 6 esx kvms to run and there would always be one kvm that seemes to suck the io out of the system.

i would look to sd drives for the boot and logging.

If you can, move from kvm to ct for as many of the vms that you can. CTs have significantly less io overhead, raw access to io and bus devices. Dbs like postgress and mysql are very happy in cts



The key here is to not use software raid and the good cards w fastest cache and largest cache will cost 800 to 1200 depending on vendor. No matter what you do it is imperitive that the cache ram be battery backed. It better to have that with a slightly smaller cache ram.

 

[udo]

Hi,
The problem appeared after i moved out the newsletter system which is the guilty one in this case , the 2 postgres and 3 mysql server didn`t do so much trouble.
And the budget for Raid Controller is about 700 euros/800 dollars. I read out that is a bug in pve-2.6.32 on mdadm raid and i will put today a test system out of production to see if that is the problem.

Thanks.

Hi,
the budget fit to this controller: http://www.amazon.com/NEW-PROFILE-S...8&qid=1369035521&sr=8-2&keywords=areca+1882-i (I have selected amazon only because of wide availablility).
I have the old one 1212, 1222, 1680 and the actual ARC-1882-ix-16 running.
The perf-values some posts ago are from the 1222 (with sas-disks).

LSI is also an option (but I don't like they stuff - just have new experiences with controller of that company (no pass through and other funny things)).

Udo