Clocksource error when live migrating VM

[kagerito]

Hi,
We are trying out a Proxmox setup and are currently running two hosts with VE 9.1
The hosts are both Dell Poweredge 640:
56 x Intel(R) Xeon(R) Gold 5117 CPU @ 2.00GHz (2 Sockets)
and
80 x Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz (2 Sockets)

I am testing live migration of a Rocky Linux 10.1 VM (clean installed), and am encountering a clocksource error:

[root@localhost ~]# dmesg | grep -i 'clocksource.*unstable\|tsc.*watchdog'
[ 36.126271] clocksource: timekeeping watchdog on CPU0: Marking clocksource 'tsc' as unstable because the skew is too large:
[ 36.126498] clocksource: Clocksource 'tsc' skewed 3472599 ns (3 ms) over watchdog 'kvm-clock' interval of 504130422 ns (504 ms)
[ 36.126578] tsc: Marking TSC unstable due to clocksource watchdog

And it changes clocksource to kvm-clock

This happens every time I am migrating the VM between the two hosts.

The CPU mode is currently set to Skylake Server v5, 2 cores,
q35 mode
qemu guest agent installed

I've tried updating BIOS on the hosts, made sure the hosts are using NTP, turning off CPU energy savings, using different CPU modes, but nothing seems to help.

Are there more things I could try?
What's the consequence of it changing clocksource?

Thanks

 

[JohnnieKF]

Before digging into the TSC side of this, a couple of questions, because the fix is quite different depending on the answers:

• Is chronyd currently running inside the guest, or are you only syncing time on the hosts?
• Have you actually measured guest clock offset after a migration, or is the dmesg entry the only thing you've seen so far?

 

[kagerito]

Before digging into the TSC side of this, a couple of questions, because the fix is quite different depending on the answers:

• Is chronyd currently running inside the guest, or are you only syncing time on the hosts?
• Have you actually measured guest clock offset after a migration, or is the dmesg entry the only thing you've seen so far?

Chrony is running on both hosts and the guest.
I have not done any measuring, only looked at the message and log entry.

 

[JohnnieKF]

Since you haven't measured anything yet, I'd confirm there's a real problem before chasing the message. In the guest, run this before a migration and again straight after:

chronyc tracking

Watch System time and Last offset. If both stay small across the migration, the clock is fine and the dmesg entry is just noise.

It's worth checking the hosts too, since the guests inherit their time:

chronyc sources
chronyc tracking

You want a source actually selected (marked ^*) and a small offset. If the hosts disagree with each other, fix that first.

Either way, it's worth passing the guests a proper time source using the ptp_kvm module. This exposes the host's clock to the guest as a PTP device, which you point chrony at — so the guest tracks whichever host it's currently running on rather than relying on network NTP, and re-converges quickly after a migration.

Load the module in the guest. The second line makes it come back after a reboot:

modprobe ptp_kvm
echo ptp_kvm > /etc/modules-load.d/ptp_kvm.conf

Give it a stable name. The device number isn't guaranteed — if the VM has a NIC that supports hardware timestamping, that can take ptp0 and push the KVM clock to ptp1, and that ordering can change on reboot. This rule matches on the clock's name instead and gives you a consistent /dev/ptp_kvm to point chrony at, plus ownership so chronyd can read it after dropping privileges:

cat > /etc/udev/rules.d/90-ptp-kvm.rules << 'EOF'
ACTION=="add", SUBSYSTEM=="ptp", ATTR{clock_name}=="KVM virtual PTP", SYMLINK+="ptp_kvm", GROUP="chrony", MODE="0660"
EOF

Apply and check:

udevadm control --reload
udevadm trigger --subsystem-match=ptp
ls -l /dev/ptp_kvm

If /dev/ptp_kvm didn't appear, find the right device manually — it won't necessarily be ptp0 if the VM has a NIC with hardware timestamping:

cat /sys/class/ptp/ptp*/clock_name

You're looking for the one reporting KVM virtual PTP.

Then in /etc/chrony.conf:
refclock PHC /dev/ptp_kvm poll 2

After this you can restart chronyd and confirm with chronyc sources that it's selected; you can keep your existing NTP servers configured as a fallback.

• EDIT: Added udev rule I missed