HPET and watchdog

[wosp]

This weekend I upgraded a Proxmox VE 3.4 cluster to Proxmox VE 4.1. Everything works as expected, but on the 3.4 cluster we had "hpet=disable" in the /etc/default/grub because if we don't, we get many (strange) log entries about hpet. After the upgrade I removed the "hpet=disable", to see if the issues is still there in 4.1. But yes, the log entries came back. So I re-added "hpet=disable" to /etc/default/grub and rebooted all nodes. After +/- 1 hour one node decided to fence (watchdogtimer expired), 3 hours later, same issue. So I have removed the "hpet=disable" again and rebooted all nodes and now there is no problem anymore (besides the log entries).

Example of log entries:

Apr 17 05:02:32 node02 kernel: [27974.504137] CE: hpet4 increased min_delta_ns to 20115 nsec
Apr 17 08:04:25 node02 kernel: [38888.003432] CE: hpet increased min_delta_ns to 20115 nsec

Any experiences with this?

Edit: FWIW: I use hardware watchdog (idrac)

 

[wosp]

Update: problem seems to be solved when disabling systemd-timesyncd and enable ntp (https://forum.proxmox.com/threads/pve-4-1-systemd-timesyncd-and-ceph-clock-skew.27043/). A couple of days ago I re-added "hpet=disable" and until now it's stable.

 

[spirit]

Hi,
I never see this on my dell servers (from old poweredge 2950 to new r630).

Do you have setup governor in bios to max performance ?

 

[wosp]

Nope, no special bios settings. We only use Dell-servers and I only see this issue on Dell PowerEdge R310 servers with X3450 CPU, not on any other Dell server (R610, R710, R320 and R420 tested). Don't know for sure if others CPU's are also affected, since I only have R310's with X3450's. But it's no problem, when I just disable hpet in grub config and switch to ntp instead of systemd-timesyncd, everything works fine.

 

[spirit]

you should really check in your bios, for "max performance" profile.
I think your hpet / clock problem, is because of energy-saving/ondemand profile, changing cpu clock speed or enabling/shutting down cores.

 

[wosp]

you should really check in your bios, for "max performance" profile.
I think your hpet / clock problem, is because of energy-saving/ondemand profile, changing cpu clock speed or enabling/shutting down cores.

I just checked 3 nodes, one R310 with the "problem", one R610 without the problem and one R710 without the problem.
On all 3 nodes I use the same settings for power management (Active Power Controller), which is the default, and I can choose between:

- Active Power Controller
- Maximum Performance
- OS Control
- Custom

But, since I don't have the problem on R610's and R710's, we can only conclude that this isn't the problem. Right?

 

[spirit]

you should really setup them to max performance.

It'll improve stability of vm. (for example,with the default, a vm do a cpu spike, to frequency or cores of cpu change, the clock drift, and sometime this can crash the vm)

 

[wosp]

Okay, I'm changing them to "Max Performance" now on all nodes on all clusters (time-consuming job ). Will also remove the hpet=disable, just to see if the error messages come back. Will let you know soon. Thanks!

 

[wosp]

After I changed to "Max Performance" power profile and removed the "hpet=disable" from GRUB config the hpet logs came back

May 19 22:22:05 node01 kernel: [ 2727.779396] CE: hpet3 increased min_delta_ns to 20115 nsec
May 20 00:20:25 node03 kernel: [ 7720.535187] CE: hpet3 increased min_delta_ns to 20115 nsec
May 20 00:51:01 node02 kernel: [10942.805762] CE: hpet increased min_delta_ns to 20115 nsec

So, the power profile wasn't the problem. However, I will keep using "Max Performance" power profile, since it seems to be a better choice. But again need to add "hpet=disable" to GRUB config. Thanks anyway.