PVE 8.0 – fail2ban log locations missing?

speedbird · Jun 22, 2023

Hi there!
I just wanted to copy my fail2ban configs as follows:

Code:

[proxmox]
enabled = true
port = https,http,8006
filter = proxmox
logpath = /var/log/daemon.log
maxretry = 3
# 10 minutes
bantime = 600

Code:

[ssh]

enabled = true
port    = ssh
filter  = sshd
logpath  = /var/log/auth.log
maxretry = 3
bantime = 3600

But those log files are no longer present in Proxmox 8.0 on Debian bookworm.

What do I use instead to make fail2ban work again? Because currently it just fails as there are no logs to be found.

Thanks!

Dunuin · Jun 22, 2023

Debian 12 doesn't come with the rsyslog package anymore (but you can install it using apt if you want) because the systemd journal should be used now. But didn't tested it yet, if fail2ban can somehow work with journalctl.

Edit:
Looks like you could tell the fail2ban config to use systemd logging with the line backend = systemd.

Chris · Jun 22, 2023

You can use `backend = systemd`, see https://wiki.archlinux.org/title/Fail2ban#Custom_SSH_jail

speedbird · Jun 22, 2023

Alright, thanks. So if I change the files to:

Code:

[proxmox]
enabled = true
port = https,http,8006
filter = proxmox
banaction = iptables
backend   = systemd
maxretry  = 5
findtime  = 1d
bantime   = 1d

Filter:

Code:

[Definition]
failregex = pvedaemon\[.*authentication (verification )?failure; rhost=<HOST> user=.* msg=.*
ignoreregex =

and SSH to:

Code:

[sshd]
enabled   = true
filter    = sshd
banaction = iptables
backend   = systemd
maxretry  = 3
findtime  = 1d
bantime   = 1d

I'm good? I mean it starts now at least, but I'm not sure if I need to configure something else here?

Will A. Zeppeli · Jul 6, 2023

I'm having the same problem. Applying the above suggestions doesn't seem to fix it once tested

ojaksch · Jul 11, 2023

I had the same issue but solved it by investigating some 'lil time with ArchLinux' wiki as mentioned in #3 and created a file named /etc/fail2ban/jail.local with the exemplary content of

Code:

[DEFAULT]
ignoreip = 127.0.0.1/8 ::1 AND OTHER IP ADDRESSES
backend  = systemd
bantime  = 3600
findtime  = 300
maxretry = 3

You can see that I switched fail2ban to systemd in general. For my filter files I simply added a line journalmatch = VALUE to them. VALUE is depending on the type of the used systemd service to watch. For watching Kernel's log I'm using

Code:

journalmatch = _TRANSPORT=kernel

for Apache

Code:

journalmatch = _SYSTEMD_UNIT=apache2.service

and so on.

speedbird · Jul 11, 2023

That is very helpful, thanks! I'll give that a try and see if that makes fail2ban operational again. So I guess the "findtime" is the timeframe the three retrys have to be in in order to trigger an ban, correct?

ojaksch · Jul 11, 2023

Correct. Keep in mind that these are my general values and only exemplary, but a somewhat good starting point. And you can take them over and adjust them in your jails for your own sake.

MinishMan · Jul 26, 2023

I came across the same problem in Proxmox 8, got ssh working thanks to the advice here, but couldn't get my [proxmox] jail to filter failed logins.

Then I realised that as long as you're using PAM to log into Proxmox (it will say so on the login page), you can copy the default pam-generic.conf that comes with fail2ban and that will filter failed proxmox logins for you.

Code:

cd /etc/fail2ban/filter.d
cp pam-generic.conf proxmox.conf
systemctl restart fail2ban

Do a failed login, and you should now see it listed when you run:

Code:

fail2ban-client status proxmox
Status for the jail: proxmox
|- Filter
|  |- Currently failed: 1
|  |- Total failed:     1
|  `- Journal matches:
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

Problem I still have: This .conf runs without a journalmatch, which fail2ban advises against

NOTICE [proxmox] Jail started without 'journalmatch' set. Jail regexs will be checked against all journal entries, which is not advised for performance reasons.

ojaksch · Jul 26, 2023

@MinishMan Glad I could give at least some help.
What about this jail and filter? (fresh and so not yet intensively tested)

/etc/fail2ban/jail.d/proxmox.conf

Code:

[proxmox]
enabled  = true
port     = https,http,8006
filter   = proxmox
maxretry = 3
bantime  = 3600
findtime = 300

/etc/fail2ban/filter.d/proxmox.conf

Code:

[INCLUDES]
before = common.conf

[Definition]
failregex = pvedaemon\[.*authentication failure; rhost=<HOST> user=.* msg=.*

ignoreregex =
journalmatch = _SYSTEMD_UNIT=pvedaemon.service

MinishMan · Jul 26, 2023

Thank you! Looks much more elegant and has a journalmatch.

I managed to successfully jail my IP from those settings so I'll keep using them now.

I had to include backend = systemd in my /etc/fail2ban/jail.d/proxmox.conf file too, eg:

Code:

[proxmox]
enabled  = true
port     = https,http,8006
filter   = proxmox
backend  = systemd
maxretry = 3
bantime  = 3600
findtime = 300

ojaksch · Jul 27, 2023

MinishMan said:
Thank you! Looks much more elegant and has a journalmatch.
I managed to successfully jail my IP from those settings so I'll keep using them now.
I had to include backend = systemd in my /etc/fail2ban/jail.d/proxmox.conf file too

Great to hear that it's working, thanks for confirmation. I'm relatively sure that the port line can be removed, but I kept it for now for safety reasons. This line is a takeover from Bullseye.
My backend is "auto" and looks like to work. Can't remember if that was the default or not.

Inlakesh · Oct 4, 2023

Hello
I am trying to get fail2ban to work on a new proxmox 8 install.
I really would appreciate if someone can structure this up on where to put all these suggestions, I have a hard time to put it all together.

Edit:
Finally got fail2ban to work on Proxmox 8 install, here is what I did:

/etc/fail2ban/filter.d

proxmox.conf:

Code:

[INCLUDES]before = common.conf
[Definition]failregex = pvedaemon\[.authentication failure; rhost=<HOST> user=. msg=.*
ignoreregex =journalmatch = _SYSTEMD_UNIT=pvedaemon.service

/etc/fail2ban/jail.local

Add to the end:

Code:

[proxmox]
enabled = true
port = https,http,8006
filter = proxmox
backend = systemd
maxretry = 3
findtime = 300
bantime  = 3600

/etc/fail2ban/jail.d

defaults-debian.conf

Code:

[sshd]enabled = true

proxmox.conf

Code:

[proxmox]
enabled  = true
port     = https,http,8006
filter   = proxmox
backend  = systemd
maxretry = 3
bantime  = 3600
findtime = 300

sshd.local

Code:

[sshd]
enabled = true
filter = sshd
banaction = iptables
backend = systemd
maxretry = 3
bantime = 3600
findtime = 300
ignoreip = 127.0.0.1/8

Gektor · Nov 22, 2023

Inlakesh said:
Finally got fail2ban to work on Proxmox 8 install, here is what I did:

Hi, i have make all as described on Proxmox 8, and have make few tests, login on GUI with wrong password, but it didn't block IP at all. Do you test this solution?

Code:

fail2ban-client status proxmox
Status for the jail: proxmox
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- Journal matches:
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:

cglmicro · Nov 28, 2023

Me too, it log the bad IP, but never blocked that IP:

Was there a change when Debian 11 Bullseye, or Debian 12 Bookworm on how it manage the firewall (i.e.: no longer iptables) ??

Thank you.

DragonL80 · Nov 29, 2023

I think I have discovered the issue, or an issue at least.
Edit: This is on a debian 12.2 install with the latest community edition of proxmox.

So what I'm going to do is document what i've found, and what is working for me, in this post.

What all of these proxmox configs fail to take into account.. Is that if you use PAM for authentication, the failed logins wont be caught. The configs above only work for PVE authentication realm.

so I'll show you now my /etc/fail2ban/jail.d/ configs

pam-generic.conf

Code:

[pam-generic]
enabled = true
backend = systemd
banaction = iptables
findtime = 7d
bantime = 6h
maxretry = 3
ignoreip = 127.0.0.1/8 ::1

proxmox.conf

Code:

[proxmox]
enabled = true
filter = proxmox
backend = systemd
banaction = iptables
maxretry = 3
findtime = 7d
bantime = 1h

*NOTE: The port = line is unnecessary (port = https,http,8006) when using systemd backend, the auth daemon never mentions the port.

sshd.conf

Code:

[sshd]
enabled = true
filter = sshd
banaction = iptables
backend = systemd
maxretry = 3
bantime = 3600
findtime = 300
ignoreip = 127.0.0.1/8 ::1

*NOTE: You may want to consider adding your own IP to the ignoreip for both sshd and pam-generic jails as well
Edit: *NOTE: You may also want to tweak any of the *time settings to your preference.

And here is my /etc/fail2ban/filter.d/ relevant configs.

proxmox.conf

Code:

[INCLUDES]
before = common.conf


[Definition]
failregex = pvedaemon\[.*authentication failure; rhost=<HOST> user=.* msg=.*
journalmatch = _SYSTEMD_UNIT=pvedaemon.service
ignoreregex =

also here is a command you can use to test the jail for systemd

fail2ban-regex systemd-journal /etc/fail2ban/filter.d/proxmox.conf

and if you like, an alias you can put in your ~/.bashrc to create a command called showbans

alias showbans="fail2ban-client status | sed -n 's/,//g;s/.*Jail list://p' | xargs -n1 fail2ban-client status"

*NOTE: If you do decide to add this to your ~/.bashrc file, please know that you then either have to do one of the following two things to "enable" it
You can type source ~/.bashrc
Or you can log out and log in again..

Running the command showbans then gives this:

But I want to stress, if PAM is an auth realm thats available, you NEED the pam-generic jail. Otherwise pam based auth failures wont get picked up by the proxmox jail. Only if they attempt to auth with PVE realm will that jail catch them.

I really hope this helps someone. It took me two full days to figure all of this out.

Celeste

cglmicro · Dec 1, 2023

Hi Celeste.

Thank you for your reply, I manage to get the jail pam-generic, and the other one I already had.

But this is the error I receive in /var/log/fail2ban.log:


2023-11-30 19:05:32,594 fail2ban.filter         [2594148]: INFO    [sshd] Found 163.5.143.85 - 2023-11-30 19:05:32
2023-11-30 19:05:32,595 fail2ban.filter         [2594148]: INFO    [pam-generic] Found 163.5.143.85 - 2023-11-30 19:05:32
2023-11-30 19:05:34,801 fail2ban.filter         [2594148]: INFO    [sshd] Found 163.5.143.85 - 2023-11-30 19:05:34
2023-11-30 19:05:34,819 fail2ban.actions        [2594148]: NOTICE  [sshd] Ban 163.5.143.85
2023-11-30 19:05:34,826 fail2ban.utils          [2594148]: ERROR   7f9ebd0134b0 -- exec: { iptables -w -C f2b-sshd -j RETURN >/dev/null 2>&1; } || { iptables -w -N f2b-sshd || true; iptables -w -A f2b-sshd -j RETURN; }
for proto in $(echo 'tcp' | sed 's/,/ /g'); do
{ iptables -w -C INPUT -p $proto --dport ssh -j f2b-sshd >/dev/null 2>&1; } || { iptables -w -I INPUT -p $proto --dport ssh -j f2b-sshd; }
done
2023-11-30 19:05:34,826 fail2ban.utils          [2594148]: ERROR   7f9ebd0134b0 -- stderr: '/bin/sh: 1: iptables: not found'
2023-11-30 19:05:34,826 fail2ban.utils          [2594148]: ERROR   7f9ebd0134b0 -- stderr: '/bin/sh: 1: iptables: not found'
2023-11-30 19:05:34,826 fail2ban.utils          [2594148]: ERROR   7f9ebd0134b0 -- stderr: '/bin/sh: 3: iptables: not found'
2023-11-30 19:05:34,827 fail2ban.utils          [2594148]: ERROR   7f9ebd0134b0 -- returned 127
2023-11-30 19:05:34,827 fail2ban.utils          [2594148]: INFO    HINT on 127: "Command not found".  Make sure that all commands in "{ iptables -w -C f2b-sshd -j RETURN >/dev/null 2>&1; } || { iptables -w -N f2b-sshd || true; iptables -w -A f2b-sshd -j RETURN; }\nfor proto in $(echo 'tcp' | sed 's/,/ /g'); do\n{ iptables -w -C INPUT -p $proto --dport ssh -j f2b-sshd >/dev/null 2>&1; } || { iptables -w -I INPUT -p $proto --dport ssh -j f2b-sshd; }\ndone" are in the PATH of fail2ban-server process (grep -a PATH= /proc/`pidof -x fail2ban-server`/environ). You may want to start "fail2ban-server -f" separately, initiate it with "fail2ban-client reload" in another shell session and observe if additional informative error messages appear in the terminals.
2023-11-30 19:05:34,827 fail2ban.actions        [2594148]: ERROR   Failed to execute ban jail 'sshd' action 'iptables' info 'ActionInfo({'ip': '163.5.143.85', 'family': 'inet4', 'fid': <function Actions.ActionInfo.<lambda> at 0x7f9ebee6a0c0>, 'raw-ticket': <function Actions.ActionInfo.<lambda> at 0x7f9ebee6a840>})': Error starting action Jail('sshd')/iptables: 'Script error'

It really look like IPTABLES is no longer installed. I'll try to apt-get install iptables on my server snd see if it conflict with PMG, PBS or PVE.

EDIT: I tried installing iptables on one of my PMG, but it won't show the IP is banned:


root@pmg10:/etc/fail2ban/jail.d# showbans
Status for the jail: pam-generic
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 16
   |- Total banned:     16
   `- Banned IP list:   137.74.39.215 142.44.205.46 144.217.248.43 146.59.145.161 146.59.93.12 149.202.55.133 158.69.80.165 198.244.149.224 37.187.112.10 51.195.111.249 51.222.13.180 51.68.70.195 51.79.146.239 54.36.98.63 91.121.53.249 94.23.162.147
Status for the jail: proxmox
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- Journal matches:  _SYSTEMD_UNIT=pvedaemon.service
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:
Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- File list:        /var/log/auth.log
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:
root@pmg10:/etc/fail2ban/jail.d# iptables -L INPUT -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
root@pmg10:/etc/fail2ban/jail.d#

EDIT#2:
I think IPTABLES is replaced by NFTABLES, but:


root@pbs103:/etc/fail2ban/jail.d# showbans
Status for the jail: pam-generic
|- Filter
|  |- Currently failed: 4
|  |- Total failed:     488
|  `- Journal matches:
`- Actions
   |- Currently banned: 18
   |- Total banned:     18
   `- Banned IP list:   51.75.143.57 144.126.204.43 43.157.92.99 163.5.143.85 158.160.103.38 196.127.48.30 206.189.193.54 178.128.73.254 103.98.152.134 51.68.175.129 158.101.100.127 159.253.46.213 206.189.175.222 170.64.130.116 186.75.154.14 46.245.89.107 139.59.127.73 37.193.112.180
Status for the jail: proxmox
|- Filter
|  |- Currently failed: 0
|  |- Total failed:     0
|  `- Journal matches:  _SYSTEMD_UNIT=pvedaemon.service
`- Actions
   |- Currently banned: 0
   |- Total banned:     0
   `- Banned IP list:
Status for the jail: sshd
|- Filter
|  |- Currently failed: 1
|  |- Total failed:     975
|  `- Journal matches:  _SYSTEMD_UNIT=sshd.service + _COMM=sshd
`- Actions
   |- Currently banned: 21
   |- Total banned:     21
   `- Banned IP list:   141.95.207.239 51.75.143.57 54.37.205.209 178.128.73.254 51.15.171.97 170.64.130.116 103.98.152.134 43.157.92.99 46.245.89.107 144.126.204.43 186.75.154.14 163.5.143.85 158.160.103.38 196.127.48.30 51.68.175.129 206.189.193.54 158.101.100.127 37.193.112.180 159.253.46.213 206.189.175.222 139.59.127.73
root@pbs103:/etc/fail2ban/jail.d# cat sshd.conf
[sshd]
enabled = true
filter = sshd
banaction = nftables
backend = systemd
maxretry = 2
bantime = 1d
findtime = 7d
ignoreip = 127.0.0.1/8 70.50.253.148 ::1
root@pbs103:/etc/fail2ban/jail.d# nft list tables
root@pbs103:/etc/fail2ban/jail.d#

EDIT #3:
It's working now, I only had to restart fail2ban another time, and all the tables appeared in nft. Thanks for your help.

Search

Search

PVE 8.0 – fail2ban log locations missing?

speedbird

Well-Known Member

Dunuin

Distinguished Member

Chris

Proxmox Staff Member

speedbird

Well-Known Member

Will A. Zeppeli

New Member

ojaksch

Renowned Member

speedbird

Well-Known Member

ojaksch

Renowned Member

MinishMan

New Member

ojaksch

Renowned Member

MinishMan

New Member

ojaksch

Renowned Member

Inlakesh

Well-Known Member

Gektor

Member

cglmicro

Member

DragonL80

New Member

cglmicro

Member