My CT stops at 9am

[xnooztv]

Hello,

Every day at 9am, my CT 106 stop for no reason.
There is no automatic backup, no crons job, nothing at all.

Other VPS are not impacted.

How to find this problem?

Thanks.

 

[sb-jw]

What about a cronjob inside of your container? Or any other script in your container? Is the CT shutdown from inside the CT or from PVE itself?

 

[xnooztv]

Inside the CT, there is no cronjob. I have only cPanel.
I do not know where CT stops, but I think via PVE.

 

[sb-jw]

but I think via PVE.

You will see it in the GUI in the Task Log, if this is not the case, then it doesn't forced through GUI.

 

[xnooztv]

You will see it in the GUI in the Task Log, if this is not the case, then it doesn't forced through GUI.

PVE forces the stop of the CT.

 

[sb-jw]

Then you need to find out who has access and do this.
First you can try to change the password of all nodes and check the SSH Keys. If the problem is persistent, you should check Cronjobs or other task scheduler. You can try to backup the VM and restore it with another VMID.

 

[xnooztv]

Then you need to find out who has access and do this.
First you can try to change the password of all nodes and check the SSH Keys. If the problem is persistent, you should check Cronjobs or other task scheduler. You can try to backup the VM and restore it with another VMID.

It's done. As I said, I have no cron job or other task scheduler.
I will make a backup of the VM.

 

[xnooztv]

Then you need to find out who has access and do this.
First you can try to change the password of all nodes and check the SSH Keys. If the problem is persistent, you should check Cronjobs or other task scheduler. You can try to backup the VM and restore it with another VMID.

After reinstalling the machine completely, the same problem is still present. At 9:00 am, the CT goes stop by itself

I have look at the cronjobs again, there is nothing. I am the only one with this problem. Other users do not have this problem


I have this on "status task":

 

[tim]

Check the syslog around that time.

 

[xnooztv]

Check the syslog around that time.

Thank you, I tell you tomorrow.

 

[xnooztv]

Check the syslog around that time.

I have look in syslog and i get this:

Jun 03 09:00:05 sd-115048 pvedaemon[1055]: <root@pam> starting task UPID:sd-115048:00006A30:01F4F141:5CF4C575:vzstop:106:root@pam:
Jun 03 09:00:05 sd-115048 pvedaemon[27184]: stopping CT 106: UPID:sd-115048:00006A30:01F4F141:5CF4C575:vzstop:106:root@pam:
Jun 03 09:00:05 sd-115048 audit[27228]: AVC apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-106_</var/lib/lxc>" pid=27228 comm="apparmor_parser"
Jun 03 09:00:05 sd-115048 kernel: audit: type=1400 audit(1559545205.331:574): apparmor="STATUS" operation="profile_remove" profile="/usr/bin/lxc-start" name="lxc-106_</var/lib/lxc>" pid=27228 comm="apparmor_parser"
Jun 03 09:00:06 sd-115048 pvestatd[1562]: unable to get PID for CT 106 (not running?)
Jun 03 09:00:06 sd-115048 pvestatd[1562]: unable to get PID for CT 106 (not running?)
Jun 03 09:00:06 sd-115048 kernel: fwbr106i0: port 2(veth106i0) entered disabled state
Jun 03 09:00:06 sd-115048 kernel: device veth106i0 left promiscuous mode
Jun 03 09:00:06 sd-115048 kernel: fwbr106i0: port 2(veth106i0) entered disabled state
Jun 03 09:00:06 sd-115048 kernel: fwbr106i0: port 1(fwln106i0) entered disabled state
Jun 03 09:00:06 sd-115048 kernel: vmbr0: port 5(fwpr106p0) entered disabled state
Jun 03 09:00:06 sd-115048 kernel: device fwln106i0 left promiscuous mode
Jun 03 09:00:06 sd-115048 kernel: fwbr106i0: port 1(fwln106i0) entered disabled state
Jun 03 09:00:06 sd-115048 kernel: device fwpr106p0 left promiscuous mode
Jun 03 09:00:06 sd-115048 kernel: vmbr0: port 5(fwpr106p0) entered disabled state
Jun 03 09:00:07 sd-115048 pvedaemon[1055]: <root@pam> end task UPID:sd-115048:00006A30:01F4F141:5CF4C575:vzstop:106:root@pam: OK

And the config of the CT:

root@sd-115048:~# pct config 106
arch: amd64
cores: 7
cpuunits: 30000
hostname: mail.0network.net
memory: 8096
net0: name=eth0,bridge=vmbr0,firewall=1,gw=62.210.0.1,hwaddr=00:16:3e:00:1c:b9,ip=51.158.24.112/32,type=veth
onboot: 1
ostype: centos
rootfs: local:106/vm-106-disk-0.raw,size=300G
swap: 512
unprivileged: 1

 

[tim]

You just posted the stop task, which we already know. It would be more interesting what is going on before that task, in short what is in the logs before 9:00:05

 

[xnooztv]

You just posted the stop task, which we already know. It would be more interesting what is going on before that task, in short what is in the logs before 9:00:05

Oh sorry.. log from 8:40:29 to 9:00:04:
https://pastebin.com/ns61TnhB

 

[tim]

You have more than 100 failed logins in those 20 minutes and password changes every minute.
There is some api usage involved I guess or you are totally exposed to the web and someone else is doing this. You should definitely check your firewall and implement something like fail2ban or similar.

 

[xnooztv]

You have more than 100 failed logins in those 20 minutes and password changes every minute.
There is some api usage involved I guess or you are totally exposed to the web and someone else is doing this. You should definitely check your firewall and implement something like fail2ban or similar.

I activate the firewall, install and configure fail2ban. API is used in WHMCS.
Fail2ban has already banned IPs ..

 

[xnooztv]

You have more than 100 failed logins in those 20 minutes and password changes every minute.
There is some api usage involved I guess or you are totally exposed to the web and someone else is doing this. You should definitely check your firewall and implement something like fail2ban or similar.

Today, at 9am, the problem is still here. But the CT is no longer stop in the PVE.
I have firewall + fail2ban on my server.

 

[tim]

What problem? The attempted logins?
You state "Today, at 9am, the problem is still here. But the CT is no longer stop in the PVE", if the container isn't stopping anymore what problem is still here?

 

[xnooztv]

What problem? The attempted logins?
You state "Today, at 9am, the problem is still here. But the CT is no longer stop in the PVE", if the container isn't stopping anymore what problem is still here?

The problem is the CT that stops at 9am everyday. But today, it is not PVE who has stopped the CT and this is that's what I do not understand.
I have no logs on the node and no logs on the CT..

Sorry for my English, I'm French ^^

 

[James Crook]

The problem is the CT that stops at 9am everyday. But today, it is not PVE who has stopped the CT and this is that's what I do not understand.
I have no logs on the node and no logs on the CT..

Sorry for my English, I'm French ^^

From the logs it looks like someone has access to your Proxmox Host.

 

[xnooztv]

From the logs it looks like someone has access to your Proxmox Host.

This has been fixed since yesterday..