Strange slowness and micro interruptions (solved but want to share)

[mgiammarco]

I have several proxmox clusters on HP G8 and G9 servers, with Windows Server VMs.
Terminal servers users complains about "disk slowness" (I have ssd and nvme...), "micro interruptions" and so on.
It seemed an unsolvable problem until I discovered that changing in ilo4/power settings from "dynamic power savings mode" to "os control mode" all problems disappeared.
I run ceph tests and crystal disk mark tests and I have seen doubled performance!
I would like to ask if someone else has tried changing that settings to see if it happens only to me.
I would like also understand what's happening.
Thanks,
Mario

 

[abzsol]

Dude you resolved a problem going on since 6 months.

https://forum.proxmox.com/threads/c...very-night-slowing-down-vms.71573/post-358766

we have 3 HPE DL360 gen10. Changed power plan from "dynamic power savings mode" to "os control mode" on all 3 nodes and rebooted one at a time.

Thanks!!!

 

[apoc]

Also make sure you enable "performance" power plan weithin the Windows OS.

 

[mgiammarco]

Infact I got crazy for several months too. It is incredible that "dynamic power savings" is default in hp ilo. It should not have this behaviour.

 

[apoc]

Apologies. Nothing to dor with HPe.
It is default in most BIOS.
Because it is a compromise between performance and power efficiency.
It is the default in Windows OS too.
That is nothing new. It has beenn there since decades

It also has always been good / recommended / best practice to disable C-States and power saving features on virtualization hosts for a long time.

 

[Cha0s]

Has anyone noticed any difference switching to "OS control mode" from "Static High Performance"?

I am experiencing the same issue as abzsol when deleting snapshots or images on CEPH. The whole cluster slows down.

 

[abzsol]

@mgiammarco look at this pdf at page 6/7 https://support.hpe.com/hpesc/public/docDisplay?docId=a00016408en_us

I will set the workload profile to "Virtualization - Max Performance", that change also other parameters then the power regulator.

 

[mgiammarco]

I would like to explain better: here we are not choosing among several power profiles.
Here we are choosing between "let linux kernel manage powersave" or "let hp ilo manage powersave".
The second one gives problems!

 

[apoc]

Yes likely because it uses the "wrong" profile.
In BIOS disable C states and turboboost / Speedstep, switch to full perfoance plan and you should be good to go...

... Unless you going offboard with your vCPUCPU ratio

 

[mgiammarco]

Yes likely because it uses the "wrong" profile.
In BIOS disable C states and turboboost / Speedstep, switch to full perfoance plan and you should be good to go...

... Unless you going offboard with your vCPUCPU ratio

I do not understand why I have to disable C-states when if I use "os control mode" all works perfectly

 

[mgiammarco]

I would like to add another info: I have another HP Server that I use as a pfsense firewall (no proxmox, no virtualization, nothing)
The gateway monitoring algorithm gets crazy several times a day. If I put in ilom "static high performance" all works perfectly.
It seems to me a serious bug of hp bios, you cannot tell me that you have a "dynamic power savings" mode but I cannot use it because it breaks OSes.

 

[apoc]

if I use "os control mode" all works perfectly

Good for you.
My experience is different and best to fix things on the BIOS level.
The good thing about setting it in BIOS i that typically the settings can't be altered easily, by a software update for instance.

 

[velocity08]

I have several proxmox clusters on HP G8 and G9 servers, with Windows Server VMs.
Terminal servers users complains about "disk slowness" (I have ssd and nvme...), "micro interruptions" and so on.
It seemed an unsolvable problem until I discovered that changing in ilo4/power settings from "dynamic power savings mode" to "os control mode" all problems disappeared.
I run ceph tests and crystal disk mark tests and I have seen doubled performance!
I would like to ask if someone else has tried changing that settings to see if it happens only to me.
I would like also understand what's happening.
Thanks,
Mario

Hi @mgiammarco

May I please ask with your dl360 G9‘s running Ceph, what storage controller are you using?

we are looking at repurpose similar hardware into a Ceph cluster and still reviewing what the best controller from HP is.

we have both H240 and 440 controllers available.

would love some feedback on your setup, performance and stability if you don’t mind?

””Cheers
G