[SOLVED] Unusual logon

[krystofr]

Hi,
Please can someone point me in the right direction.

Proxmox 4.4
Today one of our VM's experienced extremely high CPU load causing many systems to fail.

Upon logging into the proxmox GUI, I could see three messages
service: pve daemon : 'successful auth for user 'validuserid@pam'

This occurred at a time when a legitimate login by that user was impossible, and co-incided with the start of the problems.

It was only one VM affected not the whole node.
I'm obviously concerned that this was a hack.

I have checked /var/log/secure and there is no log of a login with that user id at that time. Where else should I be looking to establish what occurred?

 

[krystofr]

To clarify I would like to know where I should look to establish
1. Who logged in
2. What they did

thank you in advance

 

[fabian]

you can also check /var/log/auth.log , the journal/syslog files, /var/log/pve/** and /var/log/pve/pveproxy/*. keep in mind that depending on the level of access all of the above might be unreliable/tampered with. if you already have log messages that clearly show that an account was accessed when a "legitimate login by that user was impossible", IMHO you are already in "cut off/shut down system, change all passwords of affected users/accounts, take copy for forensic investigation, start over from scratch" territory.

I'd also take a look at other systems which that account/user had access to, as well as systems which the compromised systems might have been able to access automatically (e.g., other cluster nodes, backup systems which were accessible from the compromised system via backup rotation scripts or the like, ...).

 

[krystofr]

IMHO you are already in "cut off/shut down system, change all passwords of affected users/accounts, take copy for forensic investigation, start over from scratch" territory.

Thank you Fabian, we have done as you suggested, and started from scratch. Gives much more peace of mind, then tyring to examine and repair.