Slow clock - time drift in windows guests

W

wech

Renowned Member
Sep 9, 2009
17
3
68
Hi,

we are running several virtual machines (windows and linux based hosts) and are too experiencing the time drift problems on all windows guests (x86 and x64), but none of the linux guests (also measured against ntppool time)

After reading through this forum and searching and reading a lot about kvm/qemu regarding this problem, fiddling around with several options like the infamous -td-rtc-hack (which first seemed to solve the problem) and trying different -clock settings, we still have severe problems.

Today two of our windows servers each lagged around 1 hour back.

Of course we could set up a frequent time synchronisation to the host ntp clock and probably will do so, but at least we would like to talk about some observations.

Currently both windows servers are running with the -rtc-td-hack option, but today I noticed a high load on the windows 2003 server (32 Bit, VMID=103), clogging up around 40% of the cpu spent inside kernel (according taskmanager).

Perfmon also indicated high values for “Interrupt Time” on this machine (20%-25%). The other machine seemed not be affected by this problem.

As far as I understand the td-rtc-hack, it tries to reinject lost timer interrupts into the windows hosts again and again, so windows is busy processing all lost interrupts of the last hour, which leads to the high load.

I have written a small cron job to log the time differences of all four machines and we are experiencing the time lag repatedly, sometime “only” 300-400 seconds back, but sometimes even several hours. All with the hack option active or inactive.

Today it is worse, it seems, because the host is busy copying large amounts of data since several hours.

The interesting observation now was, that when looking at the windows 2003 server clock trough a RDP connection, the hand displaying the seconds was not or very seldom moving, but when I accessed the hard disk, it was hasting to catch up – moving about 3-4 seconds in 1 real second. When I stopped accessing the hard disk (“dir /s”) it stopped again.

After doing the dir /s again, first nothing happened (I guess because the directory structure was already in the file cache), but then started running again after it was beyond the point in the directory file list where I stopped initially.

I did some more tests and it seems, accessing the network does also lead to this effect. Searching a file with windows explorer on a network drive on a different computer or a ftp get of a 2 mb file did also speed up the host clock, but the latter may be of the resulting file being written to the hard disk. Reading a file from a share on this server also results in faster clock.

Anybody any idea?

Any help would be appreciated.

Here the configuration of the most affected VM:

--103.conf:
name: VAMP
ide2: none,media=cdrom
smp: 2
ostype: w2k3
memory: 4096
onboot: 1
description: Der alte
boot: dc
freeze: 0
cpuunits: 1000
acpi: 1
kvm: 1
bootdisk: ide0
vlan0: ne2k_pci=A6:39:4E:42:27:77
hostusb: 057c:1900
ide0: vm-103-disk-0.img
ide1: vm-103-disk-1.qcow2
ide3: vm-103-disk-2.qcow2
args: -rtc-td-hack

And here infos about the host:

vmhost: ~# pveversion -v

pve-manager: 1.3-1 (pve-manager/1.3/4023)
qemu-server: 1.0-14
pve-kernel: 2.6.24-8
pve-kvm: 86-1
pve-firmware: 1
vncterm: 0.9-2
vzctl: 3.0.23-1pve3
vzdump: 1.1-2
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1

vmhost: ~# uname-a
Linux vmhost 2.6.24-7-pve #1 SMP PREEMPT Tue Jun 2 08:00:29 CEST 2009 x86_64 GNU/Linux

vmhost:~# qm showcmd 103
/usr/bin/kvm -monitor unix:/var/run/qemu-server/103.mon,server,nowait -vnc unix:/var/run/qemu-server/103.vnc,password -pidfile /var/run/qemu-server/103.pid -daemonize -usbdevice tablet -usbdevice host:057c:1900 -name VAMP -smp 2 -id 103 -cpuunits 1000 -boot dc -vga cirrus -tdf -localtime -k de -drive file=/var/lib/vz/images/103/vm-103-disk-1.qcow2,if=ide,index=1 -drive file=/var/lib/vz/images/103/vm-103-disk-0.img,if=ide,index=0,boot=on -drive file=,if=ide,index=2,media=cdrom -drive file=/var/lib/vz/images/103/vm-103-disk-2.qcow2,if=ide,index=3 -m 4096 -net tap,vlan=0,ifname=vmtab103i0,script=/var/lib/qemu-server/bridge-vlan0 -net nic,vlan=0,model=ne2k_pci,macaddr=A6:39:4E:42:27:77 -rtc-td-hack

The host is running on brand new Hardware:

Dual CPU Board with two Intel Xeon E5530 at 2.40GHz
24 GB RAM
Two 3ware 9650 RAID Controllers with eight 2,5” SATA drives attached.

It’s clock is running very stable (measured through ntppery against ntppool.org
 
Yesterday we restarted the win2k8 server (vmid 102) and set it from 4 to 1 cpu, because it did regularly fall into stasis and was only responding sporadically. But it does not solve the problems. A collegue thinks it is slightly better, but I think the improvement is neglectible.

The host system was idle most of the time, the cpu load of all kvm instances (and all other processes too) was below 3% cpu usage.

We are guesing the time drift is not the problem, but a symptom of general non-responsing of the VMs. The beforementioned load from copying large amount of data on the host does increase the problem, but it no longer seems to be the source of it.

I tried to ping -t this server and watched the seconds of the desktop clock of the server and when the second clock hand stopped moving, the ping delays also drastically increased. In this cases it is also very hard to do anything with the UI either trough the proxmox vnc console and trough a remote desktop connection.

Here the ping-log ( standard 1 second delay ):

Reply from 192.168.2.8: bytes=32 time<1ms TTL=128
Reply from 192.168.2.8: bytes=32 time<1ms TTL=128
... repeated ~50 times ...
Reply from 192.168.2.8: bytes=32 time=1ms TTL=128
Reply from 192.168.2.8: bytes=32 time=1ms TTL=128
Reply from 192.168.2.8: bytes=32 time=5ms TTL=128
Reply from 192.168.2.8: bytes=32 time=12ms TTL=128
Reply from 192.168.2.8: bytes=32 time=36ms TTL=128
Reply from 192.168.2.8: bytes=32 time=3ms TTL=128
Reply from 192.168.2.8: bytes=32 time=2ms TTL=128
Reply from 192.168.2.8: bytes=32 time=1ms TTL=128
Reply from 192.168.2.8: bytes=32 time=675ms TTL=128
Reply from 192.168.2.8: bytes=32 time=6ms TTL=128
Reply from 192.168.2.8: bytes=32 time=3ms TTL=128
Reply from 192.168.2.8: bytes=32 time=6ms TTL=128
Reply from 192.168.2.8: bytes=32 time=63ms TTL=128
Reply from 192.168.2.8: bytes=32 time=10ms TTL=128
Reply from 192.168.2.8: bytes=32 time=9ms TTL=128
Reply from 192.168.2.8: bytes=32 time=2ms TTL=128
Reply from 192.168.2.8: bytes=32 time=61ms TTL=128
Reply from 192.168.2.8: bytes=32 time=43ms TTL=128
Reply from 192.168.2.8: bytes=32 time=1ms TTL=128
Reply from 192.168.2.8: bytes=32 time=26ms TTL=128
Reply from 192.168.2.8: bytes=32 time=7ms TTL=128
Reply from 192.168.2.8: bytes=32 time=28ms TTL=128
Reply from 192.168.2.8: bytes=32 time=7ms TTL=128
Reply from 192.168.2.8: bytes=32 time=9ms TTL=128
Reply from 192.168.2.8: bytes=32 time=7ms TTL=128
Reply from 192.168.2.8: bytes=32 time=15ms TTL=128
Reply from 192.168.2.8: bytes=32 time=647ms TTL=128
Reply from 192.168.2.8: bytes=32 time=767ms TTL=128
Reply from 192.168.2.8: bytes=32 time=405ms TTL=128
Reply from 192.168.2.8: bytes=32 time=16ms TTL=128
Reply from 192.168.2.8: bytes=32 time=79ms TTL=128
Reply from 192.168.2.8: bytes=32 time=30ms TTL=128
Reply from 192.168.2.8: bytes=32 time=175ms TTL=128
Reply from 192.168.2.8: bytes=32 time=39ms TTL=128
Reply from 192.168.2.8: bytes=32 time=12ms TTL=128
Reply from 192.168.2.8: bytes=32 time=236ms TTL=128
Reply from 192.168.2.8: bytes=32 time=95ms TTL=128
Reply from 192.168.2.8: bytes=32 time=18ms TTL=128
Reply from 192.168.2.8: bytes=32 time=1ms TTL=128
Reply from 192.168.2.8: bytes=32 time=3ms TTL=128
Reply from 192.168.2.8: bytes=32 time=425ms TTL=128
Reply from 192.168.2.8: bytes=32 time<1ms TTL=128
Reply from 192.168.2.8: bytes=32 time=397ms TTL=128
Reply from 192.168.2.8: bytes=32 time=46ms TTL=128
Reply from 192.168.2.8: bytes=32 time=4ms TTL=128
Reply from 192.168.2.8: bytes=32 time=1ms TTL=128
Reply from 192.168.2.8: bytes=32 time=762ms TTL=128
Reply from 192.168.2.8: bytes=32 time=1ms TTL=128
Reply from 192.168.2.8: bytes=32 time=11ms TTL=128
Reply from 192.168.2.8: bytes=32 time=15ms TTL=128
Reply from 192.168.2.8: bytes=32 time=25ms TTL=128
Reply from 192.168.2.8: bytes=32 time<1ms TTL=128
Reply from 192.168.2.8: bytes=32 time=8ms TTL=128
Reply from 192.168.2.8: bytes=32 time=12ms TTL=128
Reply from 192.168.2.8: bytes=32 time=16ms TTL=128
Reply from 192.168.2.8: bytes=32 time=10ms TTL=128
Reply from 192.168.2.8: bytes=32 time=4ms TTL=128
Reply from 192.168.2.8: bytes=32 time=4ms TTL=128
Reply from 192.168.2.8: bytes=32 time=354ms TTL=128
Reply from 192.168.2.8: bytes=32 time=53ms TTL=128
Reply from 192.168.2.8: bytes=32 time=7ms TTL=128
Reply from 192.168.2.8: bytes=32 time=24ms TTL=128
Reply from 192.168.2.8: bytes=32 time=4ms TTL=128
Reply from 192.168.2.8: bytes=32 time=1ms TTL=128
Reply from 192.168.2.8: bytes=32 time<1ms TTL=128
Reply from 192.168.2.8: bytes=32 time<1ms TTL=128
Reply from 192.168.2.8: bytes=32 time=4ms TTL=128
Reply from 192.168.2.8: bytes=32 time=90ms TTL=128
Reply from 192.168.2.8: bytes=32 time=5ms TTL=128
Reply from 192.168.2.8: bytes=32 time=16ms TTL=128
Reply from 192.168.2.8: bytes=32 time=47ms TTL=128
Reply from 192.168.2.8: bytes=32 time=4ms TTL=128
Reply from 192.168.2.8: bytes=32 time=2ms TTL=128
Reply from 192.168.2.8: bytes=32 time=1ms TTL=128
Reply from 192.168.2.8: bytes=32 time=4ms TTL=128
Reply from 192.168.2.8: bytes=32 time=134ms TTL=128
Reply from 192.168.2.8: bytes=32 time=1ms TTL=128
Reply from 192.168.2.8: bytes=32 time=13ms TTL=128
Reply from 192.168.2.8: bytes=32 time=4ms TTL=128
Reply from 192.168.2.8: bytes=32 time=42ms TTL=128
Reply from 192.168.2.8: bytes=32 time=4ms TTL=128
Reply from 192.168.2.8: bytes=32 time=40ms TTL=128
Reply from 192.168.2.8: bytes=32 time=767ms TTL=128
Reply from 192.168.2.8: bytes=32 time<1ms TTL=128
Reply from 192.168.2.8: bytes=32 time=66ms TTL=128
Reply from 192.168.2.8: bytes=32 time=164ms TTL=128
Reply from 192.168.2.8: bytes=32 time=4ms TTL=128
Reply from 192.168.2.8: bytes=32 time=937ms TTL=128
Reply from 192.168.2.8: bytes=32 time=51ms TTL=128
Reply from 192.168.2.8: bytes=32 time=30ms TTL=128
Reply from 192.168.2.8: bytes=32 time=647ms TTL=128
Reply from 192.168.2.8: bytes=32 time=71ms TTL=128
Reply from 192.168.2.8: bytes=32 time=2ms TTL=128
Reply from 192.168.2.8: bytes=32 time=4ms TTL=128
Reply from 192.168.2.8: bytes=32 time=142ms TTL=128


Up to 900 ms ping delay for a computer on the same LAN with no load at all ...

We removed the -rtc-td-hack sometime ago, as it did not solve the problem of the VMs slowness from time to time, even as the clock was more accurate because it sped up when the general slowness was gone again.

We decided to do regular ntp-queries instead, which don't clog up irq handling on the guest, which does not have time to handle them anyway during this phases it seems.

At the moment the ping replies are down again to 1-2 ms for about 2 Minutes, then the delays started again.

Still no clue what might cause this problems - and it seems only windows guests are affected at all.
 
wech said:
...
vmhost: ~# pveversion -v

pve-manager: 1.3-1 (pve-manager/1.3/4023)
qemu-server: 1.0-14
pve-kernel: 2.6.24-8
pve-kvm: 86-1
pve-firmware: 1
vncterm: 0.9-2
vzctl: 3.0.23-1pve3
vzdump: 1.1-2
vzprocps: 2.0.11-1dso2
vzquota: 3.0.11-1

vmhost: ~# uname-a
Linux vmhost 2.6.24-7-pve #1 SMP PREEMPT Tue Jun 2 08:00:29 CEST 2009 x86_64 GNU/Linux
...
Click to expand...

go for the latest stable version (run apt-get update/upgrade). not that I see that this will fix your issue but just to make sure you got the latest version.
 
Dual CPU Board with two Intel Xeon E5530 at 2.40GHz
24 GB RAM
Two 3ware 9650 RAID Controllers with eight 2,5” SATA drives attached.
 
CPU BOGOMIPS: 38403.94
REGEX/SECOND: 441612
HD SIZE: 94.49 GB (/dev/pve/root)
BUFFERED READS: 89.32 MB/sec
AVERAGE SEEK TIME: 9.91 ms
FSYNCS/SECOND: 840.61
DNS EXT: 68.77 ms
DNS INT: 65.95 ms (mydomain.local)
 
Thanks for your assistance.

No, we are currently not using vzdump.

Concerning the usb device: this is an interesting idea. Unfortunately if I remember correctly, the problems did exist from day 1, and I attached the usb device about one week later.

The device is a AVN ISDN Modem, which is only used to dial numbers trough TAPI. I don't think it is being used very often at all.

But I'll try to disconnect it.
 
Sure :)

Must I add it to all windows-VMs or ist ist enough to add it to the one, mentioned last (id 102 / windows 2008 server)?

At the moment I've just added the parameter to this vm.

Removing the usb device (trough vm-monitor command, and also physically) yesterday, did not seem to improve the performance. Ping delays still varied from 1 ms up to 500 ms.

I'm rebooting the vm at the moment and will let you know of any changes.

Thanks again for your help.
 
wech said:
Sure :)

Must I add it to all windows-VMs or ist ist enough to add it to the one, mentioned last (id 102 / windows 2008 server)?

At the moment I've just added the parameter to this vm.
Click to expand...

just for this vm

wech said:
Removing the usb device (trough vm-monitor command, and also physically) yesterday, did not seem to improve the performance. Ping delays still varied from 1 ms up to 500 ms.

I'm rebooting the vm at the moment and will let you know of any changes.
Click to expand...

Please do a stop/start instead.
 
dietmar said:
Please do a stop/start instead.
Click to expand...

Of course. I made myself not clear. I shut down the OS and then clicked the start button in the proxmox webinterface, so the kvm process restarts.

I've just checked trough ps -lF that the command line of the running process contains the new argument.

The first two minutes (during windows boot) were not too promising. I had nearly all the time ping delays of 100-1800 ms including "not responding" replies, but at the moment it is constant at 1 ms.

I'm keeping an eye on it.
 
You also mentioned linux guest having the same problem? What kernel version does those guest have? And what clocksource do they use?

# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
 
Is there a way to give the guest a CMOS clock?

This afternoon my windows guest was approx 15 hours off. I
I may have been logged off of it for 15 hours overnight, but it didn't turn off.

Proxmox web interface's Log Search returns nothing for query "uptime".
Is there no way to see anything but it's most recent uptime measurments?

Proxmox's ntp.conf is set to use a single stratum 1 server, but it had the broadcasting line commented, so I uncommented & set it to x.x.x.255.

I've pointed Server 2008 to sync from x.x.x.255.
Changed it's max phase correction from 172800 to 31449600 seconds.
Changed it's check-in interval from 900 to 600 seconds.
And made it log.

I set the clock to a few hours behind & restarted the Windows Time service.
The clock stayed wrong & the log says:

/-- NTP Packet:
149387 18:58:50.6996590s - | LeapIndicator: 0 - no warning; VersionNumber: 3; Mode: 1 - SymmetricActive; LiVnMode: 0x19
149387 18:58:50.6996590s - | Stratum: 1 - primary reference (syncd by radio clock)
149387 18:58:50.6996590s - | Poll Interval: 15 - out of valid range; Precision: -6 - 15.625ms per tick
149387 18:58:50.6996590s - | RootDelay: 0x0000.0000s - unspecified; RootDispersion: 0x000A.0401s - 10.0156s
149387 18:58:50.6996590s - | ReferenceClockIdentifier: 0x4C4F434C - source name: "LOCL"
149387 18:58:50.6996590s - | ReferenceTimestamp: 0xCEECB8E926D4AA10 - 12907105129151682500ns - 149387 18:58:49.1516825s
149387 18:58:50.6996590s - | OriginateTimestamp: 0x0000000000000000 - unspecified
149387 18:58:50.6996590s - | ReceiveTimestamp: 0x0000000000000000 - unspecified
149387 18:58:50.6996590s - | TransmitTimestamp: 0xCEECB8EAB2DCDB37 - 12907105130698682500ns - 149387 18:58:50.6986825s
149387 18:58:50.6996590s - >-- Non-packet info:
149387 18:58:50.6996590s - | DestinationTimestamp: 149387 18:58:50.6996590s - 0xCEECB8EAB31CDA2B149387 18:58:50.6996590s - - 12907105130699659000ns149387 18:58:50.6996590s - - 149387 18:58:50.6996590s
149387 18:58:50.6996590s - | RoundtripDelay: 976500ns (0s)
149387 18:58:50.6996590s - | LocalClockOffset: -488200ns - 0:00.000488200s
149387 18:58:50.6996590s - \--

I think I wouldn't have to deal with this if I could set it's AnnounceFlag to "A" so it'd just listen to CMOS.

Anyone know?

### EDIT:

After reading this Microsoft knowlegebase article which enlightened me to the fact that there's 2 pertinent offset values to consider for positive & negative value ranges, I set both to 31449600 seconds & still no correction.
So I changed it's time source from x.x.x.255 to clock.nyc.he.net & all is well.

Any suggestions on how to verify Proxmox's ntp functionality? I'd still rather provide CMOS time to guests, but I suppose that's a rather exotic request.
 
Last edited:
the performance should be better with your hardware.....

CPU BOGOMIPS: 23940.91
REGEX/SECOND: 993583
HD SIZE: 82.50 GB (/dev/sda3)
BUFFERED READS: 175.34 MB/sec
AVERAGE SEEK TIME: 8.61 ms
FSYNCS/SECOND: 2825.50
DNS EXT: 90.10 ms
DNS INT: 20.85 ms (domain.local)

i use a single cpu x3370 8gb system with two harddisks
 
I have similar problem host proxmox 1.5 2.6.32 guest w2k8 32bit , clock in windows slows down slowly on start its 10 sec after 24h its about 1 min.

I've found option in qemu-kvm-0.12.2 :

-rtc [base=utc|localtime|date][,clock=host|vm][,driftfix=none|slew]
set the RTC base and clock, enable drift fix for clock ticks

"Switch RTC emulations to the new host_clock instead of vm_clock by
default. This has the advantage that the emulated RTC will follow
automatically the host time while it might be tuned via NTP."

Maybe it will do the magic :/

When do you plan to deploy qemu-kvm-0.12.2 in proxmox ?
 
Last edited:
Taliesin said:
I have similar problem host proxmox 1.5 2.6.32 guest w2k8 32bit , clock in windows slows down slowly on start its 10 sec after 24h its about 1 min.

I've found option in qemu-kvm-0.12.2 :

-rtc [base=utc|localtime|date][,clock=host|vm][,driftfix=none|slew]
set the RTC base and clock, enable drift fix for clock ticks

"Switch RTC emulations to the new host_clock instead of vm_clock by
default. This has the advantage that the emulated RTC will follow
automatically the host time while it might be tuned via NTP."

Maybe it will do the magic :/

When do you plan to deploy qemu-kvm-0.12.2 in proxmox ?
Click to expand...

there were some other issues with 0-12.2 so we decided to wait for the next release.
 
You must log in or register to reply here.

About

The Proxmox community has been around for many years and offers help and support for Proxmox VE, Proxmox Backup Server, and Proxmox Mail Gateway.
We think our community is one of the best thanks to people like you!

Quick Navigation

Home Get Subscription Wiki Downloads Proxmox Customer Portal About

Get your subscription!

The Proxmox team works very hard to make sure you are running the best software and getting stable updates and security enhancements, as well as quick enterprise support. Tens of thousands of happy customers have a Proxmox subscription. Get yours easily in our online shop.

Buy now!
Community platform by XenForo® © 2010-2024 XenForo Ltd.
Top Bottom
Back