Problem with time within vms

U

undefine

New Member
Dec 6, 2024
6
0
1
I have a Proxmox cluster with several Debian 12 virtual machines. These VMs use ntpd to synchronize time with local NTP servers. Under normal conditions, everything works correctly. However, when creating a disk-only snapshot, I observe that the VM’s internal time freezes for approximately 0.3–0.8 seconds. Shortly afterward, ntpd performs a time "step" to correct the offset, but I would prefer to avoid this issue entirely.

The problem occurs on VMs using both local-lvm and ceph storage. All VMs have the qemu-guest-agent installed, and no freeze-related messages appear during snapshot creation.
 
Hi,
even when creating a disk-only snapshot, the VM needs to be paused, so that the state of the disks in the snapshots is from the same time.

Do you have the guest agent enabled in the configuration? It is recommended to freeze the file systems, otherwise it cannot be guaranteed that the raw disk state (which the storage snapshot represents) is consistent from a filesystem perspective.

Do you have any actual issue with the time? If the offset is corrected right away, that sounds to me like it works as intended.
 
  • Like
Reactions: Kingneutron
Hi,
  • The guest agent is configured and working correctly. During backups, freeze and thaw operations are performed by the agent as expected.
  • When creating a snapshot, I see the message on the VM: qemu-ga: info: guest-fsfreeze called.
  • NTP does correct the time, but since it isn’t triggered by the guest agent, it can take quite a while—often several minutes.
I noticed this issue especially with database snapshots, where the "pause" in time causes the primary instance to see secondaries as being in the future.


It would be helpful to have an option to trigger an immediate time synchronization (e.g. via the guest agent) after a snapshot is taken.
 
This is still my best workaround:

Bash: 
qm guest exec $VMID /usr/sbin/hwclock -- --hctosys || true
 
Der Harry said:
This is still my best workaround:

Bash: 
qm guest exec $VMID /usr/sbin/hwclock -- --hctosys || true
Click to expand...
Thanks. Have you any idea how to integrate it with proxmox snapshots gui? Is there something simmilar to /etc/vzdump.conf with "scripts" where such code could be executed during snapshots?
 
undefine said:
Have you any idea how to integrate it with proxmox snapshots gui?
Click to expand...

I would use QEMU Agent hook scripts instead, so you can run inside the VM which ever time sync command you need when the filesystem is thawed. Some details on [1] and [2].

Out of curiosity: which DB is it? Using Percona, MySQL GTID replication or Postgresql haven't seen that snapshot or backups have caused issues with time.

[1] https://pve.proxmox.com/wiki/VM_Backup_Consistency
[2] https://forum.proxmox.com/threads/freeze-thaw-scripts.144091/post-648224
 
  • Like
Reactions: LnxBil and Der Harry
VictorSTS said:
I would use QEMU Agent hook scripts instead, so you can run inside the VM which ever time sync command you need when the filesystem is thawed. Some details on [1] and [2].
Click to expand...

Upstream really have to fix this :(

That's really sad that this bug is in Proxmox forever.
 
I think I have a solution. But it's not nice

Proxmox has Hook Scripts:

```bash
qm set <vmid> --hookscript local:snippets/hookscript.pl
pct set <ctid> --hookscript local:snippets/hookscript.pl
```

Unfortunately there are no hocks for

```text
pre-snapshot
post-snapshot
pre-rollback
post-rollback
pre-restore
post-restore
post-resume
```

But that is easy to add for Proxmox Gmbh. Then you can run a script to set the clock.
 
Yeah, is not nice as for enterprise envs.

What does your "hookscript.pl"?

In my opinion you need to catch for VM snap with RAM, then if VM is running after rollback and QEMU Guest Agent is enabled and guest agent responds to ping
then call QGA guest-set-time with host time.
 
franzz said:
What does your "hookscript.pl"?
Click to expand...

It is not "my" hookscript. Hookscripts are a general supported thing in Proxmox. They are called e.g. after / before start/stop/backup etc.

Unfortunately not for suspend, resume etc (probably I am wrong here).

When we would have this, we can use "qm exec" to run a script in the VM to set the clock.

It is easy for Proxmox GmbH to add these events to the hookscript.
 
You have to configure NTP to correct the time when the delta is large, else it won't correct this problem. How you do that varies by NTP server. For chrony you need something like "makestep 0.1 -1" in the config file.

Apparently you don't all know about that.
 
Last edited:
  • Like
Reactions: Johannes S
BobhWasatch said:
You have to configure NTP to correct the time when the delta is large, else it won't correct this problem. How you do that varies by NTP server. For chrony you need something like "makestep 0.1 -1" in the config file.
Click to expand...

With all respect.

Don't lecture about NTP. We know what it is - we know how to so called "properly configure" it.

Proxmox is lacking of a feature that RedHat explicitly mentions in the qemu guest agent and VMware also has.

That is the point - not a Linux question about NTP.

We all have mitigations. It's still a not implemented feature.
 
BobhWasatch said:
Sorry if I accidentally stepped on your bad mood. Was just trying to solve the immediate problem.
Click to expand...

This is a open ticket for many years - it has a Bug Report on Bugzilla with many ideas.

It's a bug / not implemented feature.

Not a perceived "mood issue". Not a tiny thing you can fix "in general" with NTP.

You can fix it with NTP in some situation - 1000% agreed with that. That's not the point not the question.
 
Der Harry said:
It's a bug / not implemented feature.

Not a perceived "mood issue". Not a tiny thing you can fix "in general" with NTP.

You can fix it with NTP in some situation - 1000% agreed with that. That's not the point not the question.
Click to expand...
So, what is the question then?

I mean, your post isn't providing any concrete suggestions on how to fix it eitehr. And If it did, it would probably be better placed in the existing Bugzilla report.

Complaining, "nagging", and statements such as "That should be easy for Proxmox GmbH to fix" don't usually accomplish anything. They mostly just stir up emotions. In my experience, it's better to simply avoid such comments if you actually want something from the people you're directing them at. ;)

What you definitely shouldn't do, however, is take out your frustration on people who are trying to help. After all, this is a community support forum. Its main purpose is for users to help other users, which includes finding workarounds when a feature is missing or not working the way someone would like it to.
 
Last edited:
  • Like
Reactions: Johannes S
You must log in or register to reply here.
Top Bottom
Back