Possibly hacked?

[darnokg]

Long time lurker, first time poster.... I really appreciate this forum because it really helped a lot with troubleshooting and configuration. I'm hoping someone can provide some guidance on what I should do next.
Backstory:
We're a small IT company. Couple of Windows PCs, NAS, and a Proxmox server with several Unix and Windows hosts. Some systems are outward facing, as in are not NATTED and have static/public IPs, such as a Kali, Mail-in-a-box and an another stand-alone Kali machine for Nessus scanning.
Usually once per week I conduct my routine maintenance on our work network. There isn't much, checking data is being backed up, making sure systems are up to date etc. Then I get to my Proxmox box. I open the console and press the up arrow key to run my manual backup of vm configs and see a strange command. One that I've never typed before... so I check the history and to my horror see this:
244 echo $SHELL | awk -F'/' '{print $NF}'
245 LANG=C; printf "command_start_%s" "lJTqQW96"; netstat -a -n; printf "command_done_%s" "zCxcm6t3"
246 LANG=C; printf "command_start_%s" "a2RpUMgw"; printf "command_start_%s" "TEST"; printf "command_done_%s" "r1BjwpHc"
247 LANG=C; printf "command_start_%s" "kxIV6yR_"; uname -a; printf "command_done_%s" "M8weTt9t"
248 LANG=C; printf "command_start_%s" "fbQXGRp3"; cat /opt/rsa/am/utils/etc/patchHistory.dat; printf "command_done_%s" "_lmetiEN"
249 LANG=C; printf "command_start_%s" "dvdrZJTK"; cat /opt/vmware/etc/appliance-manifest.xml; printf "command_done_%s" "Ee1a_sF8"
250 LANG=C; printf "command_start_%s" "H71sjvxi"; id; printf "command_done_%s" "f5tFDew9"
251 LANG=C; printf "command_start_%s" "LjuW5mst"; cat /etc/photon-release; printf "command_done_%s" "M1u6XQtX"
252 LANG=C; printf "command_start_%s" "UEDYfCZ7"; cat /etc/os-release; printf "command_done_%s" "BvWYUOv8"
253 LANG=C; printf "command_start_%s" "klkNA0g2"; cat /etc/lsb-release; printf "command_done_%s" "sLcgzFvM"
254 LANG=C; printf "command_start_%s" "0pUEZOLA"; cat /etc/vmware-release; printf "command_done_%s" "JRJcsCcK"
255 LANG=C; printf "command_start_%s" "2_k6Xh15"; cat /etc/gentoo-release; printf "command_done_%s" "zvISwORU"
256 LANG=C; printf "command_start_%s" "uKto_l0S"; cat /etc/slackware-version; printf "command_done_%s" "2wSBBLiy"
257 LANG=C; printf "command_start_%s" "O7S8XjnU"; cat /etc/redhat-release; printf "command_done_%s" "aNZToNWa"
258 LANG=C; printf "command_start_%s" "WcEaSGuy"; cat /etc/Eos-release; printf "command_done_%s" "ubj0iNJR"
259 LANG=C; printf "command_start_%s" "cPJYEFuE"; cat /etc/euleros-release; printf "command_done_%s" "ALoDu5qN"
260 LANG=C; printf "command_start_%s" "oiMpqoJj"; cat /etc/.productinfo; printf "command_done_%s" "Qkcp3YC4"
261 LANG=C; printf "command_start_%s" "FlMarfz7"; cat /etc/kylin-build; printf "command_done_%s" "VLSESUqU"
262 LANG=C; printf "command_start_%s" "qdf0XKIf"; cat /etc/system-release; printf "command_done_%s" "G1YUgXRk"
263 LANG=C; printf "command_start_%s" "95omG8bj"; cat /etc/SuSE-release 2>/dev/null; printf "command_done_%s" "Dlxh6uzt"
264 LANG=C; printf "command_start_%s" "8IoAiEu7"; cat /etc/debian_version; printf "command_done_%s" "WhzNsqNk"
265 LANG=C; printf "command_start_%s" "JHmOBJRF"; cat /etc/hipchat-release; printf "command_done_%s" "wMfVxsvl"
266 LANG=C; printf "command_start_%s" "_wzGSdel"; cat /etc/lsb-release; printf "command_done_%s" "3WLKhmGy"
267 LANG=C; printf "command_start_%s" "tphcaE8Y"; dpkg-query -W -f '${db:Status-Abbrev} ${Package} ${Version} ${architecture} ${binary:summary}
268 '; printf "command_done_%s" "i2ZufiF0"
269 LANG=C; printf "command_start_%s" "5Hx75yq6"; /etc/init.d/guiSvr version; printf "command_done_%s" "3YVoOVuV"
270 LANG=C; printf "command_start_%s" "SrQZJuT9"; /etc/init.d/devSvr version; printf "command_done_%s" "aiHraFh6"
271 LANG=C; printf "command_start_%s" "dGyLVKs8"; LANG=C; /sbin/ifconfig -a; printf "command_done_%s" "AQoKB3xc"
272 LANG=C; printf "command_start_%s" "7aMUESNm"; LANG=C; /sbin/ip addr show; printf "command_done_%s" "90eGBj0s"
273 LANG=C; printf "command_start_%s" "yZCMiSm0"; /bin/hostname; printf "command_done_%s" "xGlt0CYI"
274 LANG=C; printf "command_start_%s" "o9vGEPyk"; hostname -A; printf "command_done_%s" "FAt8c5Ep"
275 LANG=C; printf "command_start_%s" "PyAN9juC"; /usr/bin/last reboot 2>/dev/null; printf "command_done_%s" "kG7TdRjc"
276 LANG=C; printf "command_start_%s" "fVFxRc0P"; uname -r; printf "command_done_%s" "rHnAuObp"
277 LANG=C; printf "command_start_%s" "zDuMuI8R"; uname -m; printf "command_done_%s" "IRfA3dKb"
278 LANG=C; printf "command_start_%s" "WD3NE2hd"; uname -v; printf "command_done_%s" "xDVgcA5L"
279 LANG=C; printf "command_start_%s" "eXJXageF"; lsmod | grep -q iptable_filter && iptables -L -n -v -t filter; printf "command_done_%s" "C_FbI2ti"
280 LANG=C; printf "command_start_%s" "5tI1isFU"; lsmod | grep -q _conntrack_ipv4 && iptables -L -n -v -t nat; printf "command_done_%s" "P1uaJAna"
281 LANG=C; printf "command_start_%s" "8ht8_gZi"; lsmod | grep -q iptable_mangle && iptables -L -n -v -t mangle; printf "command_done_%s" "rttsiIyK"
282 LANG=C; printf "command_start_%s" "SOIhuHeN"; kpatch list 2>/dev/null; printf "command_done_%s" "9X2gflJx"
283 LANG=C; printf "command_start_%s" "ShHOBl5H"; uptrack-uname -a 2> /dev/null; printf "command_done_%s" "6jFkBs3C"
284 LANG=C; printf "command_start_%s" "pBotAVfO"; cat "/var/cache/uptrack/Linux/x86_64/5.13.19-6-pve/#1 SMP PVE 5.13.19-15 (Tue, 29 Mar 2022 15:59:50 +0200)/status"; printf "command_done_%s" "NbZtBCkA"
285 LANG=C; printf "command_start_%s" "hINjrBGq"; grep . /sys/devices/system/cpu/vulnerabilities/*; printf "command_done_%s" "ZU9wz0Z5"
286 echo $SHELL | awk -F'/' '{print $NF}'
287 LANG=C; printf "command_start_%s" "qFWHjvgU"; netstat -a -n; printf "command_done_%s" "x5Z2REBY"
288 LANG=C; printf "command_start_%s" "Q5zARaHT"; printf "command_start_%s" "TEST"; printf "command_done_%s" "J6tMVP00"
289 LANG=C; printf "command_start_%s" "b5hOZT42"; uname -a; printf "command_done_%s" "KTau1wVj"
290 LANG=C; printf "command_start_%s" "fHeMuToE"; cat /opt/rsa/am/utils/etc/patchHistory.dat; printf "command_done_%s" "GayNfzil"
291 LANG=C; printf "command_start_%s" "cuyrCYf3"; cat /opt/vmware/etc/appliance-manifest.xml; printf "command_done_%s" "pDzBJgpv"
292 LANG=C; printf "command_start_%s" "3YibAxKY"; id; printf "command_done_%s" "VrSQcaBm"
293 LANG=C; printf "command_start_%s" "2EOs8Pyl"; cat /etc/photon-release; printf "command_done_%s" "bvEBiSU0"
294 LANG=C; printf "command_start_%s" "naDyASn8"; cat /etc/os-release; printf "command_done_%s" "41FHtINg"
295 LANG=C; printf "command_start_%s" "QF68FmAI"; cat /etc/lsb-release; printf "command_done_%s" "yTmudyTC"
296 LANG=C; printf "command_start_%s" "bLxokMBe"; cat /etc/vmware-release; printf "command_done_%s" "fS5_4Yuk"
297 LANG=C; printf "command_start_%s" "5nK_jpHX"; cat /etc/gentoo-release; printf "command_done_%s" "iF6e5P6T"
298 LANG=C; printf "command_start_%s" "zOs2wHgU"; cat /etc/slackware-version; printf "command_done_%s" "vcwhHnH7"
299 LANG=C; printf "command_start_%s" "3SklH33A"; cat /etc/redhat-release; printf "command_done_%s" "90kOhfId"
300 LANG=C; printf "command_start_%s" "dsvv5I9u"; cat /etc/Eos-release; printf "command_done_%s" "36H0eQjz"
301 LANG=C; printf "command_start_%s" "K10axKqR"; cat /etc/euleros-release; printf "command_done_%s" "UEGnNLSQ"
302 LANG=C; printf "command_start_%s" "yHb25jYZ"; cat /etc/.productinfo; printf "command_done_%s" "dlqQ9bm7"
303 LANG=C; printf "command_start_%s" "ncdlDdCg"; cat /etc/kylin-build; printf "command_done_%s" "DdZqGQA1"
304 LANG=C; printf "command_start_%s" "0tWavLNw"; cat /etc/system-release; printf "command_done_%s" "OLH5ei7w"
305 LANG=C; printf "command_start_%s" "E5rbRN0C"; cat /etc/SuSE-release 2>/dev/null; printf "command_done_%s" "is3gmYY_"
306 LANG=C; printf "command_start_%s" "iPMBKGG1"; cat /etc/debian_version; printf "command_done_%s" "nk2NOhmb"
307 LANG=C; printf "command_start_%s" "BOD7qAnz"; cat /etc/hipchat-release; printf "command_done_%s" "9tQnKLF6"
308 LANG=C; printf "command_start_%s" "y4HXc58v"; cat /etc/lsb-release; printf "command_done_%s" "jmhH1r3U"
309 LANG=C; printf "command_start_%s" "JZjx7ywu"; dpkg-query -W -f '${db:Status-Abbrev} ${Package} ${Version} ${architecture} ${binary:summary}
310 '; printf "command_done_%s" "wUGHkDx9"
311 LANG=C; printf "command_start_%s" "a2VgoE1g"; /etc/init.d/guiSvr version; printf "command_done_%s" "Fn82YmIt"
312 LANG=C; printf "command_start_%s" "DkswpDzH"; /etc/init.d/devSvr version; printf "command_done_%s" "bGCCFfy7"
313 LANG=C; printf "command_start_%s" "xDpOfd3g"; LANG=C; /sbin/ifconfig -a; printf "command_done_%s" "usmkPYfY"
314 LANG=C; printf "command_start_%s" "F4KGlPMJ"; LANG=C; /sbin/ip addr show; printf "command_done_%s" "p7ihdMhA"
315 LANG=C; printf "command_start_%s" "_d6fynUM"; /bin/hostname; printf "command_done_%s" "bNYgeJxO"
316 LANG=C; printf "command_start_%s" "yMnCJSpG"; hostname -A; printf "command_done_%s" "hc1J8DXe"
317 LANG=C; printf "command_start_%s" "ZKIhtZtF"; /usr/bin/last reboot 2>/dev/null; printf "command_done_%s" "EjFOwOBo"
318 LANG=C; printf "command_start_%s" "JlitIKQ3"; uname -r; printf "command_done_%s" "7ZdG_2GH"
319 LANG=C; printf "command_start_%s" "I2TDJjbm"; uname -m; printf "command_done_%s" "kQoimV4n"
320 LANG=C; printf "command_start_%s" "4JuJ9IJ2"; uname -v; printf "command_done_%s" "E17TDvoP"
321 LANG=C; printf "command_start_%s" "GJrDIvIi"; lsmod | grep -q iptable_filter && iptables -L -n -v -t filter; printf "command_done_%s" "ODDFPnLX"
322 LANG=C; printf "command_start_%s" "YSR5sAN8"; lsmod | grep -q _conntrack_ipv4 && iptables -L -n -v -t nat; printf "command_done_%s" "E_SH54fn"
323 LANG=C; printf "command_start_%s" "dAiVd2vw"; lsmod | grep -q iptable_mangle && iptables -L -n -v -t mangle; printf "command_done_%s" "MgYCsfIc"
324 LANG=C; printf "command_start_%s" "f7shV3IF"; kpatch list 2>/dev/null; printf "command_done_%s" "_mlUjYiu"
325 LANG=C; printf "command_start_%s" "H4Tj0QYI"; uptrack-uname -a 2> /dev/null; printf "command_done_%s" "LhDThEcj"
326 LANG=C; printf "command_start_%s" "1pZOkMBW"; cat "/var/cache/uptrack/Linux/x86_64/5.13.19-6-pve/#1 SMP PVE 5.13.19-15 (Tue, 29 Mar 2022 15:59:50 +0200)/status"; printf "command_done_%s" "Jzc9qtX_"
327 LANG=C; printf "command_start_%s" "WkXbyw5v"; grep . /sys/devices/system/cpu/vulnerabilities/*; printf "command_done_%s" "wyrkndFR"
328 echo $SHELL | awk -F'/' '{print $NF}'
329 LANG=C; printf "command_start_%s" "nLRmR29_"; netstat -a -n; printf "command_done_%s" "bXdyRP5c"
330 LANG=C; printf "command_start_%s" "jPL5K80D"; printf "command_start_%s" "TEST"; printf "command_done_%s" "lw51_lYk"
331 LANG=C; printf "command_start_%s" "cnM5bzES"; uname -a; printf "command_done_%s" "wyOJGvFS"
332 LANG=C; printf "command_start_%s" "99rcIlcb"; cat /opt/rsa/am/utils/etc/patchHistory.dat; printf "command_done_%s" "bJOVk4FX"
333 LANG=C; printf "command_start_%s" "Xf8iGPDI"; cat /opt/vmware/etc/appliance-manifest.xml; printf "command_done_%s" "PGiIracz"
334 LANG=C; printf "command_start_%s" "osVkQper"; id; printf "command_done_%s" "mfGTX6nD"
335 LANG=C; printf "command_start_%s" "2lj1bdTL"; cat /etc/photon-release; printf "command_done_%s" "CmN_7E1x"
336 LANG=C; printf "command_start_%s" "HRF7IMaD"; cat /etc/os-release; printf "command_done_%s" "ShzDmEDQ"
337 LANG=C; printf "command_start_%s" "o0CEZVlD"; cat /etc/lsb-release; printf "command_done_%s" "8dgklCw1"
338 LANG=C; printf "command_start_%s" "lnDL3W5n"; cat /etc/vmware-release; printf "command_done_%s" "Qn52qFgz"
339 LANG=C; printf "command_start_%s" "bFZbhzrp"; cat /etc/gentoo-release; printf "command_done_%s" "k9Z_frVV"
340 LANG=C; printf "command_start_%s" "VnexgtrD"; cat /etc/slackware-version; printf "command_done_%s" "akt9wDRa"
341 LANG=C; printf "command_start_%s" "4fgUzszh"; cat /etc/redhat-release; printf "command_done_%s" "SRmQv9A3"
342 LANG=C; printf "command_start_%s" "i2iczhBe"; cat /etc/Eos-release; printf "command_done_%s" "BvR715HZ"
343 LANG=C; printf "command_start_%s" "ZtvjRaig"; cat /etc/euleros-release; printf "command_done_%s" "yGGlVV4Q"
344 LANG=C; printf "command_start_%s" "bAUbrQxy"; cat /etc/.productinfo; printf "command_done_%s" "ez6HijBo"
345 LANG=C; printf "command_start_%s" "LEvuBujl"; cat /etc/kylin-build; printf "command_done_%s" "Fp2tEeUu"
346 LANG=C; printf "command_start_%s" "P5zsw80N"; cat /etc/system-release; printf "command_done_%s" "TeaIcPkO"
347 LANG=C; printf "command_start_%s" "BFA6GFmL"; cat /etc/SuSE-release 2>/dev/null; printf "command_done_%s" "cLutPmyq"
348 LANG=C; printf "command_start_%s" "82q5K172"; cat /etc/debian_version; printf "command_done_%s" "kL2XuBO5"
349 LANG=C; printf "command_start_%s" "icSPbeHk"; cat /etc/hipchat-release; printf "command_done_%s" "Q9jHuzaL"
350 LANG=C; printf "command_start_%s" "qYuMko15"; cat /etc/lsb-release; printf "command_done_%s" "xPLuh2JP"
351 LANG=C; printf "command_start_%s" "JB0iIZtN"; dpkg-query -W -f '${db:Status-Abbrev} ${Package} ${Version} ${architecture} ${binary:summary}
352 '; printf "command_done_%s" "_DULHxeH"
353 LANG=C; printf "command_start_%s" "xvhUNGw8"; /etc/init.d/guiSvr version; printf "command_done_%s" "q1xVP7ds"
354 LANG=C; printf "command_start_%s" "lmpvfdGk"; /etc/init.d/devSvr version; printf "command_done_%s" "k2ze9BDE"
355 LANG=C; printf "command_start_%s" "JBI2n09T"; LANG=C; /sbin/ifconfig -a; printf "command_done_%s" "jJXCsVYf"
356 LANG=C; printf "command_start_%s" "AO8mz30z"; LANG=C; /sbin/ip addr show; printf "command_done_%s" "7qAWNL2a"
357 LANG=C; printf "command_start_%s" "kZyOY3gy"; /bin/hostname; printf "command_done_%s" "WwA4vHnN"
358 LANG=C; printf "command_start_%s" "HutfeRgx"; hostname -A; printf "command_done_%s" "Ktwbi4o_"
359 LANG=C; printf "command_start_%s" "ZsCGETHB"; /usr/bin/last reboot 2>/dev/null; printf "command_done_%s" "I5dEAvAF"
360 LANG=C; printf "command_start_%s" "HBfUORgC"; uname -r; printf "command_done_%s" "TYcebwyO"
361 LANG=C; printf "command_start_%s" "zDpyznz7"; uname -m; printf "command_done_%s" "tESexE3y"
362 LANG=C; printf "command_start_%s" "IAf0_XMR"; uname -v; printf "command_done_%s" "ouBx5RCH"
363 LANG=C; printf "command_start_%s" "pcUbUTSK"; lsmod | grep -q iptable_filter && iptables -L -n -v -t filter; printf "command_done_%s" "CGd7hLfN"
364 LANG=C; printf "command_start_%s" "BeiZJSZN"; lsmod | grep -q _conntrack_ipv4 && iptables -L -n -v -t nat; printf "command_done_%s" "zVVzhV80"
365 LANG=C; printf "command_start_%s" "gUHE6zLj"; lsmod | grep -q iptable_mangle && iptables -L -n -v -t mangle; printf "command_done_%s" "UJMrRMA7"
366 LANG=C; printf "command_start_%s" "5pqc56nG"; kpatch list 2>/dev/null; printf "command_done_%s" "eEx1KppC"
367 LANG=C; printf "command_start_%s" "1_wOYhvt"; uptrack-uname -a 2> /dev/null; printf "command_done_%s" "W9SC2UTg"
368 LANG=C; printf "command_start_%s" "BERqQpWe"; cat "/var/cache/uptrack/Linux/x86_64/5.13.19-6-pve/#1 SMP PVE 5.13.19-15 (Tue, 29 Mar 2022 15:59:50 +0200)/status"; printf "command_done_%s" "Y73Di9S4"
369 LANG=C; printf "command_start_%s" "nrbQ2bMF"; grep . /sys/devices/system/cpu/vulnerabilities/*; printf "command_done_%s" "RMB7T9GL"
370 echo $SHELL | awk -F'/' '{print $NF}'
------- cut short due to limited space

To my limited knowledge this seems to be some type of enumeration attempt. From what I was able to gather, is that these commands, were ran from my Windows workstation, which is even weider. Could this possibly mean someone hacked my NATed/firewalled/MCaffed workstation, and utilized an open console Shell? Seems unlikely, but entirely possible.
I looked through my Proxmox (the usual suspects: cronjobs, new users, netstat etc.) and didn't find anything SUS, expect the cmd history, which you'd think in case of an attack, that would be one of he first things an attacker would clean up.
What I also suspect, is somehow one of our external Nessus scans got messed up and scanned my Proxmox box, but how would that be even possible without specifying its IP in the scanner in the first place???? I'm trying to replicate what I did two days ago to see if these commands will show again, but so far no succes.
PLEASE! If you seen this before let me know what to make of it. If there's a print out you want to see to get a better picture etc. let me know.

 

[Stoiko Ivanov]

PLEASE! If you seen this before let me know what to make of it. If there's a print out you want to see to get a better picture etc. let me know.

It's been a while since I had to actively deal with compromised infrastructure or nessus scans - so please take care and take this with a grain of salt.

on a quick look it might very well be a misguided Nessus-scan and need not be a malicious attempt.

if it was a nessus-scan you should be able to find traces for the connections in the log/journal (maybe also if it was not nessus, but the attacker did not clean up) - so check `last`, `lastlog`, and the journal for hints when/who/from where they logged in.

Usually if you're unsure if something was compromised it's best to consider it has been.
* Meaning - if you can afford it and want to be on the safe side - restore from a backup, which is known to not have been compromised
* else make disk-dumps before continuing - in case you need it for forensics later on
* definitely check places where new accounts could have been added
** /etc/passwd, /etc/shadow (for modified passwords), pve-users ... LDAP if you have something like that
* ask around in your company - maybe it was some other department who scheduled a test/check and forgot to notify you about it?


I hope this helps!