Unknown network traffic

[daves_nt_here]

Something is running on one of my PVE nodes every hour, 11 minutes past the hour. I can see data going out from PVE 1 going to PVE 2 but I have no cronjobs set up for that time.
PVE 2 was about to be decommissioned so everything's now shut down on it and that's how I noticed this little spike in traffic every hour.
I checked all the cronjobs in every LXC on PVE 1 and nothing is scheduled.
No cronjobs on either of the hosts.

I checked journalctl as well but nothing shows up there either.
I'm stumped.

 

[Onslow]

Hi, @daves_nt_here
I would do something like: log from a workstation into PVE2 with ssh, then about 10 minutes past an hour:
tcpdump -i any not host ip.address.of.workstation
You can make the filter more detailed but we don't know the source IP address of this strange traffic (there may be many addresses at PVE1).

Or at PVE2 install net-acct package, configure /etc/nacctd.conf and start the daemon. Then observe the logs (usually in /var/log/net-acct ). See man nacctd

The log should contain the epoch timestamp, protocol, src and dst addresses, src and dst port numbers, count of packets, size of data.

 

[n10ev]

Hi, @daves_nt_here,

Just a wild guess here - is it possible to have a scheduled periodic task not by cronjob but by systemd-timer(s)?

You could check it by:

# systemctl list-timers

and look for something suspicious.

Best regards,
NT

 

[daves_nt_here]

This was the answer I needed.
I ran it about a minute before the 11th minute and found a VPN address being accessed.
It was a remote server that grabs a single file off of one of the NAS servers for a remote backup. Totally forgot about that server.

Funny thing is, the file that it's grabbing was coming from PVE 1 but because I had the fstab still mounted to the NAS on PVE1 and the remote server was connecting through SSH to PVE 2 it was pulling the file from PVE1 through 2 and out to the remote server.

Thanks for the idea.

Hi, @daves_nt_here
I would do something like: log from a workstation into PVE2 with ssh, then about 10 minutes past an hour:
tcpdump -i any not host ip.address.of.workstation
You can make the filter more detailed but we don't know the source IP address of this strange traffic (there may be many addresses at PVE1).

Or at PVE2 install net-acct package, configure /etc/nacctd.conf and start the daemon. Then observe the logs (usually in /var/log/net-acct ). See man nacctd

The log should contain the epoch timestamp, protocol, src and dst addresses, src and dst port numbers, count of packets, size of data.

 

[Onslow]

@daves_nt_here
I'm glad I helped
Then you probably can mark this thread as SOLVED. By modifying the subject and selecting the SOLVED prefix from the drop-down menu. Thanks!