possible hack attack?

si458 · Nov 18, 2021

Hi All,

im trying to find out if my server has been hacked or compromised but im confused how it could of happened if it has?

i was doing abit of routine maintance (updates/restarting vms etc)
and i then spotted my SSL certificate for the web port 8006 is including an external IP address i have no idea about?

i have checked the GUI and it has also listed inside the SSL cert?

but whats even more frightning is i have part scanned my whole drive for the IP address and its listed in a file for known_hosts too!?

Code:

root@pve-fth:/# grep -rnw * -e '148.70.56.219'
etc/pve/priv/known_hosts:5:148.70.56.219 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDklgCUPT3D+7frb2U+XQ29drHsJGYgnwRjoG3LEq2diRFtXeULBP0xlSPkhG0g8TeQXWnZTSeASQ1Iw6tTi8Xr5Xfj078FgjMh
qvooSd5k/Q+1xh9QzYGV9dqDBNx2nI/pvDJ+0VFllz/SGMTa6fgS4CzwrwsgFuS6WAt+stUgXiuFsHReJXfyJNDjcIGU63/6aYfXm6EaWlXxBDiFPrAcGN0XUF/vZ1rXUZ0irxM54LD+EXcJmA2Onbqa+YAB8pTupA42YW/Ce//xdrh2+
RDBZzhE/LkRi6i9XR3PUq251wguOoDwCwz03uEugAFKYYNDJ9e+jLyAPN1FppI66UC1

i have checked my interface ip address too incase it was a VPN or something but not listed there either?

Code:

root@pve-fth:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000
    link/ether 18:03:73:d7:9c:f7 brd ff:ff:ff:ff:ff:ff
    altname enp0s25
3: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 18:03:73:d7:9c:f7 brd ff:ff:ff:ff:ff:ff
    inet 192.168.88.88/24 scope global vmbr0
       valid_lft forever preferred_lft forever
    inet6 fe80::1a03:73ff:fed7:9cf7/64 scope link
       valid_lft forever preferred_lft forever
4: tap100i0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UNKNOWN group default qlen 1000
    link/ether d2:8e:cd:02:bd:e6 brd ff:ff:ff:ff:ff:ff

my server is located inside my home network with an internal ip address and no DMZ or external access?

the is only a single VM running on the server too as the server isnt powerful (dual core 6gb ram)

any suggestions would be amazing!?

Regards

Simon

oguz · Nov 18, 2021

hi,

the IP address seems to belong to tencent (chinese) cloud provider (according to whois data).

si458 said:
but whats even more frightning is i have part scanned my whole drive for the IP address and its listed in a file for known_hosts too!?

have you ever SSH'd to that IP?

also it seems pve-fth.fth.com resolves to that same IP address:

Code:

$ host fth.com
fth.com has address 172.67.216.106
fth.com has address 104.21.45.159
fth.com has IPv6 address 2606:4700:3032::ac43:d86a
fth.com has IPv6 address 2606:4700:3035::6815:2d9f
fth.com mail is handled by 5 mxbiz1.qq.com.
fth.com mail is handled by 10 mxbiz2.qq.com.
$ host pve-fth.fth.com
pve-fth.fth.com has address 148.70.56.219

and it's pingable.
do you recognize this domain name fth.com ?

si458 said:
my server is located inside my home network with an internal ip address and no DMZ or external access?

if you're not sure then in your place i would take the server off the network and start inspecting logs.

don't want to jump to conclusions just yet, but if your server is absolutely not accessible from the outside, and you're sure that you didn't add that certificate, then it could be that something else in your network is compromised (router, laptop, raspberry, something that can be routed from outside, or maybe something is infected with malware)

in any case i would start checking the logs on the server (with the server offline):
* last | head -n 100
* history
* look in the output of ls -lta / to see if there are any recent file modifications
* ps auxfw
* check cronjobs for backdoors
* netstat -anto for outgoing connections (you can check this while server has internet)

hope this helps

si458 · Nov 18, 2021

i am based in the UK and never use any chinese services/cloud servers etc

fth.com is just a random domain i picked for the server when i set it up rather than the default .localhost one

the only thing i can think of is the server has been setup since July 2021 and i think i changed the ip address of the server at somepoint so i needed to regenerate the SSL because the browser kept complaining it didnt match (i dont use dns names only ip addresses)

https://dannyda.com/2020/06/04/how-to-regenerate-self-signed-ssl-tls-certificate-for-proxmox-ve-pve/

could this effect it?

EDIT: i have checked my /etc/hosts and that shows my internal ip address?

Code:

root@pve-fth:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.88.88 pve-fth.fth.com pve

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

oguz · Nov 18, 2021

si458 said:
fth.com is just a random domain i picked for the server when i set it up rather than the default .localhost one

yeah that explains it....

si458 · Nov 18, 2021

oguz said:
yeah that explains it....

its sorted required during the setup you cant just set hostname as MYSERVER it needs to be MYSERVER.DOMAIN.COM

i believe the might be a bug with the SSL renew script as infact my hosts file is set correctly

i have checked my /etc/hosts and that shows my internal ip address
so i believe the might be a bug with the SSL renew script using external DNS and ignoring my /etc/hosts file

Code:

root@pve-fth:~# cat /etc/hosts
127.0.0.1 localhost.localdomain localhost
192.168.88.88 pve-fth.fth.com pve

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

si458 · Nov 18, 2021

i think i remember what happened!
the server crashed that morning, it lost all its config files inside /etc/pve for some reason,
so i had to try restoring them manually,
and i had to renew the SSL!
very handy hint last THANK YOU!

Code:

root     pts/0        192.168.146.222  Fri Sep 24 14:32 - 14:33  (00:00)
root     pts/0        192.168.146.222  Fri Sep 24 10:22 - 10:27  (00:05)
root     pts/0                         Fri Sep 24 07:39 - 10:22  (02:42)
reboot   system boot  5.11.22-4-pve    Fri Sep 24 07:31 - 10:34 (14+03:03)
root     pts/0                         Fri Sep 24 07:27 - down   (00:02)
root     pts/0        192.168.251.253  Tue Sep 21 21:54 - 21:56  (00:02)

t.lamprecht · Nov 18, 2021

si458 said:
192.168.88.88 pve-fth.fth.com pve

si458 said:
so i believe the might be a bug with the SSL renew script using external DNS and ignoring my /etc/hosts file

No, the private cert is generated such that the hostname, the hosts FQDN (from /etc/hosts) and IPs those resolve too are included.

As admin you must NOT use a domain that isn't under your control, that's pretty dangerous, so this is a user configuration error.

Fix your /etc/hosts, change the domain to something else, either a domain you control or .localdomain or an invalid tld (that cannot belong to any third party) and then regenerate all certs with pvecm updatecert --force

oguz · Nov 18, 2021

glad to be of help...

si458 said:
i think i remember what happened!
the server crashed that morning, it lost all its config files inside /etc/pve for some reason,
so i had to try restoring them manually,
and i had to renew the SSL!
very handy hint last THANK YOU!

Code:

root pts/0 192.168.146.222 Fri Sep 24 14:32 - 14:33 (00:00) root pts/0 192.168.146.222 Fri Sep 24 10:22 - 10:27 (00:05) root pts/0 Fri Sep 24 07:39 - 10:22 (02:42) reboot system boot 5.11.22-4-pve Fri Sep 24 07:31 - 10:34 (14+03:03) root pts/0 Fri Sep 24 07:27 - down (00:02) root pts/0 192.168.251.253 Tue Sep 21 21:54 - 21:56 (00:02)

also please follow @t.lamprecht's advice above

si458 · Nov 18, 2021

t.lamprecht said:
No, the private cert is generated such that the hostname, the FQDN and IPs those resolve too are included.

As admin you must NOT use a domain that isn't under your control, that's pretty dangerous, so this is a user configuration error.

Fix your /etc/hosts, change the domain to something else, either a domain you control or .localdomain or an invalid tld (that cannot belong to any third party) and then regenerate all certs with pvecm updatecert --force

brilliant thank you! worked a treat!

changed the subdomain to fth.local and fixed my /etc/hosts as i spotted it saying
192.168.88.88 pve-fth.fth.com pve when it should be 192.168.88.88 pve-fth.fth.local pve-fth

Search

Search

possible hack attack?

si458

Renowned Member

oguz

Proxmox Retired Staff

si458

Renowned Member

oguz

Proxmox Retired Staff

si458

Renowned Member

si458

Renowned Member

t.lamprecht

Proxmox Staff Member

oguz

Proxmox Retired Staff

si458

Renowned Member