ads' corner

The evidence

The first thing I need is to know the kind of errors I’m looking for. Go into the directory with the webserver logfiles and try the following:

grep -E '403|404|500' --exclude=error.log *.log | less

This should give you a rather long list of errors, some are valid (like from a real search engine), some are just scripts attempting to find something on your server. Pick a couple entries where you know these are invalid, but make sure you don’t accidentally pick something which will result in blocking legitimate clients. Any request accessing pictures or css files is probably a genuine error. Requests to “funny” path names like “defau1t.php” are worth investigating. Extract the path names from the requests, we need that later.

fail2ban

Install the fail2ban package. On Debian this package brings quite a few pre-configured rules, but leaves most of them deactivated. Only sshd is activated. Check out /etc/fail2ban/filter.d and /etc/fail2ban/jail.d.

New jail

In fail2ban terms, each group of service protection is named a “jail”. What we need is a new filter, and a new jail. Conveniently these can be dropped into filter.d and jail.d as new files, and I’m deploying this with Ansible.

The new filter is /etc/fail2ban/filter.d/apache-badurls.conf in my case:

# Fail2Ban configuration file
#
# Regexp to catch known spambots and software alike. Please verify
# that it is your intent to block IPs which were driven by
# above mentioned bots.


[Definition]

failregex = ^<HOST> -.*"(GET|POST|HEAD) \/shell.php.*HTTP.*
            ^<HOST> -.*"(GET|POST|HEAD) \/shells.php.*HTTP.*
            ^<HOST> -.*"(GET|POST|HEAD) \/web\/.env.*HTTP.*

ignoreregex =.*(robots.txt|favicon.ico|jpg|jpeg|png|gif|css)

datepattern = ^[^\[]*\[({DATE})
              {^LN-BEG}

failregex can hold multiple lines, each a regex. That by itself is fine, but keep in mind that complicated regex increase the logfile parsing time. I added a few examples above, my actual file is larger and I expand it from time to time when I find new invalid requests in the logfile.

ignoreregex is used to exclude certain requests, in my case I don’t care if someone can’t find a css file, an image or a robots.txt.

The new jail configuration goes into /etc/fail2ban/jail.d/apache-badurls.conf:

[apache-badurls]

port     = http,https
filter   = apache-badurls
logpath  = /var/log/apache2/*.log
maxretry = 1
findtime = 30m
bantime  = 48h
enabled  = true

• The “port” line specifies which ports should be blocked, in this case 80 (http) and 443 (https)
• The “filter” line specifies the filename in filter.d
• “logpath” is where your webserver logfile(s) live, you may have to change this
• “enabled” is important, because the default disables all jails
• “maxretry” specifies how often an entry can appear before the ban hammer hits. If you are unsure, increase this number
• And finally “bantime” is the time fail2ban will block a certain IP address

Restart the service:

service fail2ban restart

And see if the new jail is working:

fail2ban-client status apache-badurls

In parallel you might want to check the webserver logfile, and once a bad request hits there you will see that fail2ban blocks the IP address.

Summary

fail2ban is a convenient way to keep webserver logfiles clean, block scripts and bots, and reduce the overall load on a server. It's quick to deploy and the configuration is easy to understand.