[SOLVED] snapshot stopping VM
- Thread starter rcd
- Start date
I don't have any /etc/default/qemu-guest-agent on my VM:
Code:
]# rpm -ql qemu-guest-agent
/etc/qemu-ga
/etc/qemu-ga/fsfreeze-hook
/etc/qemu-ga/fsfreeze-hook.d
/etc/sysconfig/qemu-ga
/usr/bin/qemu-ga
/usr/lib/systemd/system/qemu-guest-agent.service
/usr/lib/udev/rules.d/99-qemu-guest-agent.rules
/usr/share/doc/qemu-guest-agent-2.12.0
/usr/share/doc/qemu-guest-agent-2.12.0/COPYING
/usr/share/doc/qemu-guest-agent-2.12.0/README
/usr/share/man/man8/qemu-ga.8.gz
/usr/share/qemu-kvm/qemu-ga
/usr/share/qemu-kvm/qemu-ga/fsfreeze-hook.d
/usr/share/qemu-kvm/qemu-ga/fsfreeze-hook.d/mysql-flush.sh.sample
/var/log/qemu-gaGoogle and man also does not tell me anything about this file?
Now this is the weirdest ... the only change I made was adding
To the bottom of /etc/sysconfig/qemu-ga. I did't know how to "restart" qemu so I rebooted the VM, then ran the vzdump command like before, but now it runs as it should.
Code:
[general]
verbose = 1To the bottom of /etc/sysconfig/qemu-ga. I did't know how to "restart" qemu so I rebooted the VM, then ran the vzdump command like before, but now it runs as it should.
So here is the continuation of the story. Trying to do an automatic snapshot last night I woke up to a unresponsive server, and this logfile:
The snapshot/backup was taken, the problem is just that the server was dead in the water for two hours. Off peak hours but still.
Anyway, any idea why I got the timeout on the freeze/thaw commands?
Would it be possible and/or make sense to do the freeze/thaw command first, before starting the snapshot, to make sure everything works before running the snapshot, or would that just freeze the server in the same way?
The snapshot/backup was taken, the problem is just that the server was dead in the water for two hours. Off peak hours but still.
Anyway, any idea why I got the timeout on the freeze/thaw commands?
Would it be possible and/or make sense to do the freeze/thaw command first, before starting the snapshot, to make sure everything works before running the snapshot, or would that just freeze the server in the same way?
Ok, some more details:
In the end I simply rm'd the lockfile and reset the VM, but why on earth have a --skiplock command if it doesn't work? The man page simply says "qm reset vmid --skiplock boolean" without detailing what values are defined for "boolean": I tried {TRUE, true, 1} but nothing works.
In the end I disabled the QEMU guest agent as suggested, but I think it would be nice to have the QEMU guest agent for the other things it does. Why isn't there an option in vzdump to ignore the QEMU agent?
Code:
root@server36:~# qm agent 105 fsfreeze-status
thawed
root@server36:~# qm agent 105 fsfreeze-freeze
# ... 10 minutes pass, nothing happens, 105 is dead to the world, so I decide to break this and thaw it.
^C
root@server36:~# qm agent 105 fsfreeze-thaw
QEMU guest agent is not running
root@server36:~# qm agent 105 fsfreeze-status
QEMU guest agent is not running
# ... so now the agent is gone AWOL, ok, so let's try to reset it:
root@server36:~# qm stop 105
trying to acquire lock...
can't lock file '/var/lock/qemu-server/lock-105.conf' - got timeout
# ... let's skip the lockfile
root@server36:~# qm stop 105 --skiplock true
trying to acquire lock...
can't lock file '/var/lock/qemu-server/lock-105.conf' - got timeoutIn the end I simply rm'd the lockfile and reset the VM, but why on earth have a --skiplock command if it doesn't work? The man page simply says "qm reset vmid --skiplock boolean" without detailing what values are defined for "boolean": I tried {TRUE, true, 1} but nothing works.
In the end I disabled the QEMU guest agent as suggested, but I think it would be nice to have the QEMU guest agent for the other things it does. Why isn't there an option in vzdump to ignore the QEMU agent?
Yeah, I had those timeouts on occasion too, but it was very volatile. We have over 100 VMs that are backed up, some of them daily and we saw one or two years on each backup run at most 2 that failed sporadically with the same error. Each VM lost its connection to the virtualised storage and freezed in the end too so that we needed to reset the VM and everything was up and running again. In the last year or so, we do not see this behaviour anymore, so I guess is was some underlying problem in QEMU/KVM that was fixed with of the of updates.
I believe I have the latest QEMU drivers available:
Hypervisor:
VM:
Hypervisor:
Code:
# dpkg -l | grep -i qemu
ii pve-qemu-kvm 3.0.1-4 amd64 Full virtualization on x86 hardware
ii qemu-server 5.0-54 amd64 Qemu Server ToolsVM:
Code:
# rpm -qa | grep -i qemu
qemu-guest-agent-2.12.0-3.el7.x86_64In another problem with Proxmox I realized that it is not possible to take a snapshot of a CT with bindmounts. I'm wondering if this is the same problem causing my VM snapshots. This is in fact a VM with bindmounts.
Still don't see what that has to do with freezing and thawing the container though...
Still don't see what that has to do with freezing and thawing the container though...
Is there somewhere I can open a bug report on this? It is a little frustrating that after so many months of this being a confirmed bug, there still seem to be no action taken to fix it.
I just tried again, same as always:
/var/log/vzdump/qemu-105.log
Isn't there anything I can do to try to debug this?
I just tried again, same as always:
Code:
# qm guest cmd 105 fsfreeze-status
thawed
# qm guest cmd 105 fsfreeze-freeze
^C << hangs, having to break out of the command
# qm guest cmd 105 fsfreeze-status
QEMU guest agent is not running
# qm reset 105 --skiplock
# qm guest cmd 105 fsfreeze-status
thawed/var/log/vzdump/qemu-105.log
Code:
...
2020-05-31 03:15:58 INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-105-2020_05_31-03_15_57.vma.lzo'
2020-05-31 04:15:58 ERROR: VM 105 qmp command 'guest-fsfreeze-freeze' failed - got timeout
2020-05-31 04:16:08 ERROR: VM 105 qmp command 'guest-fsfreeze-thaw' failed - got timeout
2020-05-31 04:16:08 INFO: started backup task '973576ea-d503-4800-b826-874b7b4c874b'
...Isn't there anything I can do to try to debug this?
What about the debug information I asked for over a half year ago?
Why should it be a confirmed bug? There are a lot of other people - me included - not having this problem:
I just installed a fresh CentOS 7 including all updates up to today and also installed qemu-guest-agent-2.12.0-3 and it works:
Code:
root@proxmox ~ > qm guest cmd 104 get-osinfo
{
"id" : "centos",
"kernel-release" : "3.10.0-1127.10.1.el7.x86_64",
"kernel-version" : "#1 SMP Wed Jun 3 14:28:03 UTC 2020",
"machine" : "x86_64",
"name" : "CentOS Linux",
"pretty-name" : "CentOS Linux 7 (Core)",
"version" : "7 (Core)",
"version-id" : "7"
}
root@proxmox ~ > qm guest cmd 104 fsfreeze-status
thawed
root@proxmox ~ > qm guest cmd 104 fsfreeze-freeze
2
root@proxmox ~ > qm guest cmd 104 fsfreeze-status
frozen
root@proxmox ~ > qm guest cmd 104 fsfreeze-thaw
2
root@proxmox ~ > qm guest cmd 104 fsfreeze-status
thawed
root@proxmox ~ > qm guest cmd 104 info | jq ".version"
"2.12.0"What debug information? I already added the suggested "verbose = 1" to /etc/sysconfig/qemu-ga on Nov 8, and as I explained that the immediate impression was that it fixed the problem, but the week after it did the same thing again.
The only "debugging" info I find is that now it shows in the log that the fsfreeze-freeze "fails". There are no further details. I knew that already, I need to find out WHY the freeze fails.
It was my impression from your post on Nov 22 that you were seeing the same problem sometimes, obviously I misunderstood you.
Sorry if I didn't make it clear: I have it running on all other VM's, the problem is one particular VM. Yes I understand that the problem is specific to this VM, but I still need to fix it somehow, preferably without having to reinstall the VM altogether.
Let me know exactly what debugging information you expect, and I will of course provide it.
The only "debugging" info I find is that now it shows in the log that the fsfreeze-freeze "fails". There are no further details. I knew that already, I need to find out WHY the freeze fails.
It was my impression from your post on Nov 22 that you were seeing the same problem sometimes, obviously I misunderstood you.
Sorry if I didn't make it clear: I have it running on all other VM's, the problem is one particular VM. Yes I understand that the problem is specific to this VM, but I still need to fix it somehow, preferably without having to reinstall the VM altogether.
Let me know exactly what debugging information you expect, and I will of course provide it.
Last edited:
I do have the same problem, but it sometimes works and sometimes does not. I cannot reproduce my problem. I can also not reproduce your problem, so this is hard on me to help.
That is bad, I thought there would be better or more information on why it fails. Maybe you can attach to the process with strace and have a look at the output there?