Every time I peek into the webserver logfiles, I find quite a few 404 requests trying to figure out if certain exploits exist on this server. Now I get that these are automated attempts, and the number of requests coming from one IP show that they try several different exploits and path names. Nevertheless I thought that I don’t need this in my log, and on my webserver. fail2ban for the rescue.
Continue reading "Using fail2ban to block unfriendly web requests"
Found a couple more bots crawling my website, and from the look at online resources it seems I catched a few of the bad guys. Crawlers which ignore the robots.txt standard, and just crawl a website for content.
Decided to do something against it, and added a filter in Apache2.
The way I have my webserver setup is that I have templates for every website (they all have different configs), and deploy them using Ansible. Parts of the website configuration which are the same, or at least similar, are handled by includes.
Continue reading "Blog website crawlers and bots in Apache2"