Detect and throttle 100% CPU on VMs

[joebaires]

It is relatively frequent that some of my VMs goes to 100% cpu usage. This time a customer got infected by one of those spywares that turns the vm into a bitcoin miner zombie.

The thing is that I have to manually detect the spike of usage in the proxmox web interface, and alert my customer, login on the VM to correct this or just turn it off.

Last time it happened, the VM were several days running at 100% cpu without me noticing it until the server itself or its PSU broke (I didn't yet have spare time to check it throughly).

I was looking for some option to a) throttle CPU usage when it exceeds certain parameters and b) trigger an alert for a prompt resolution on my side.

For a) I coundn't find any option to do it in the KVM params, somewhere I read that you should do it through cgroups (is the kvm process running under a independent cgroup?). If so, anyone figured out how to do it?

For b) it would be nice to be able to do it in proxmox, but if not, it would be fine if I were able to detect it via snmp, but coundn't find information on how to do this.

Hope to hear someone here who had this problem and knows how to fix it.

Thank you in advance!

 

[sg90]

It is relatively frequent that some of my VMs goes to 100% cpu usage. This time a customer got infected by one of those spywares that turns the vm into a bitcoin miner zombie.

The thing is that I have to manually detect the spike of usage in the proxmox web interface, and alert my customer, login on the VM to correct this or just turn it off.

Last time it happened, the VM were several days running at 100% cpu without me noticing it until the server itself or its PSU broke (I didn't yet have spare time to check it throughly).

I was looking for some option to a) throttle CPU usage when it exceeds certain parameters and b) trigger an alert for a prompt resolution on my side.

For a) I coundn't find any option to do it in the KVM params, somewhere I read that you should do it through cgroups (is the kvm process running under a independent cgroup?). If so, anyone figured out how to do it?

For b) it would be nice to be able to do it in proxmox, but if not, it would be fine if I were able to detect it via snmp, but coundn't find information on how to do this.

Hope to hear someone here who had this problem and knows how to fix it.

Thank you in advance!

Proxmox supports sending data to InfluxDB @ https://pve.proxmox.com/wiki/External_Metric_Server which can then be viewed via grafana, you can also set alerts and thresholds to be sent for any metric it receives.

I doubt Proxmox would add this kinda level internally, as there is many 3rd party solutions designed for this.