Hi guys,
I swear I posted this but I couldn't find it so here it is again.
I had a problem with one of the hosts. It seems that the server decided to shut down services and all and left the proxmox network not working. Thus server went to 100% trying to connect to the storage and then had to be rebooted. Reboot took a while due to this but it all came back up.
Here's the log
After which alot of the following repeats t16v2 is the nfs backend.
then
and becomes unresponsive.
CPU 100% and though the interface was accessible, no migration or action against the VMs were possible.
My question is what would cause those services above to go down? Just the TSC Frequency mismatch?
I swear I posted this but I couldn't find it so here it is again.
I had a problem with one of the hosts. It seems that the server decided to shut down services and all and left the proxmox network not working. Thus server went to 100% trying to connect to the storage and then had to be rebooted. Reboot took a while due to this but it all came back up.
Here's the log
Code:
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 QEMU[3114093]: kvm: warning: TSC frequency mismatch between VM (2499999 kHz) and host (2596991 kHz), and TSC scaling unavailable
Jun 7 22:51:53 imm4 systemd[1]: session-430.scope: Succeeded.
Jun 7 22:51:53 imm4 systemd[1]: session-430.scope: Consumed 1min 34.922s CPU time.
Jun 7 22:51:54 imm4 systemd[1]: Started Session 431 of user root.
Jun 7 22:51:54 imm4 systemd[1]: session-431.scope: Succeeded.
Jun 7 22:51:57 imm4 systemd[1]: Started Session 432 of user root.
Jun 7 22:51:58 imm4 systemd[1]: session-432.scope: Succeeded.
Jun 7 22:51:58 imm4 systemd[1]: session-432.scope: Consumed 1.632s CPU time.
Jun 7 22:51:58 imm4 pmxcfs[1556]: [status] notice: received log
Jun 7 22:52:00 imm4 kernel: [756723.914202] nfs: server 10.254.10.211 not responding, timed out
Jun 7 22:52:01 imm4 pvestatd[1691]: status update time (12.013 seconds)
Jun 7 22:52:08 imm4 systemd[1]: Stopping User Manager for UID 0...
Jun 7 22:52:08 imm4 systemd[3114009]: Stopped target Main User Target.
Jun 7 22:52:08 imm4 systemd[3114009]: Stopped target Basic System.
Jun 7 22:52:08 imm4 systemd[3114009]: Stopped target Paths.
Jun 7 22:52:08 imm4 systemd[3114009]: Stopped target Sockets.
Jun 7 22:52:08 imm4 systemd[3114009]: Stopped target Timers.
Jun 7 22:52:08 imm4 systemd[3114009]: dirmngr.socket: Succeeded.
Jun 7 22:52:08 imm4 systemd[3114009]: Closed GnuPG network certificate management daemon.
Jun 7 22:52:08 imm4 systemd[3114009]: gpg-agent-browser.socket: Succeeded.
Jun 7 22:52:08 imm4 systemd[3114009]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers).
Jun 7 22:52:08 imm4 systemd[3114009]: gpg-agent-extra.socket: Succeeded.
Jun 7 22:52:08 imm4 systemd[3114009]: Closed GnuPG cryptographic agent and passphrase cache (restricted).
Jun 7 22:52:08 imm4 systemd[3114009]: gpg-agent-ssh.socket: Succeeded.
Jun 7 22:52:08 imm4 systemd[3114009]: Closed GnuPG cryptographic agent (ssh-agent emulation).
Jun 7 22:52:08 imm4 systemd[3114009]: gpg-agent.socket: Succeeded.
Jun 7 22:52:08 imm4 systemd[3114009]: Closed GnuPG cryptographic agent and passphrase cache.
Jun 7 22:52:08 imm4 systemd[3114009]: Removed slice User Application Slice.
Jun 7 22:52:08 imm4 systemd[3114009]: Reached target Shutdown.
Jun 7 22:52:08 imm4 systemd[3114009]: systemd-exit.service: Succeeded.
Jun 7 22:52:08 imm4 systemd[3114009]: Finished Exit the Session.
Jun 7 22:52:08 imm4 systemd[3114009]: Reached target Exit the Session.
Jun 7 22:52:08 imm4 systemd[1]: user@0.service: Succeeded.
Jun 7 22:52:08 imm4 systemd[1]: Stopped User Manager for UID 0.
Jun 7 22:52:08 imm4 systemd[1]: Stopping User Runtime Directory /run/user/0...
Jun 7 22:52:08 imm4 systemd[1]: run-user-0.mount: Succeeded.
Jun 7 22:52:08 imm4 systemd[1]: user-runtime-dir@0.service: Succeeded.
Jun 7 22:52:08 imm4 systemd[1]: Stopped User Runtime Directory /run/user/0.
Jun 7 22:52:08 imm4 systemd[1]: Removed slice User Slice of UID 0.
Jun 7 22:52:08 imm4 systemd[1]: user-0.slice: Consumed 1min 39.806s CPU time.After which alot of the following repeats t16v2 is the nfs backend.
Code:
Jun 7 22:53:55 imm4 pvestatd[1691]: storage 't16v2' is not onlinethen
Code:
Jun 7 23:05:44 imm4 pvestatd[1691]: VM 163 qmp command failed - VM 163 qmp command 'query-proxmox-support' failed - unable to connect to VM 163 qmp socket - timeout after 31 retries
Jun 7 23:05:47 imm4 pvestatd[1691]: VM 171 qmp command failed - VM 171 qmp command 'query-proxmox-support' failed - got timeoutand becomes unresponsive.
CPU 100% and though the interface was accessible, no migration or action against the VMs were possible.
My question is what would cause those services above to go down? Just the TSC Frequency mismatch?