Tweet0 Shares0 Tweets9 Comments
Following on from the article on fail2banand iptables this article looks at the fail2ban logfile and ways toanalyse it using simple command-line tools such as awk andgrep.
Note that the following commands have been updated now to match the new Fail2Ban 1.0.x log format which contains more fields than previous versions.
Format of the Logfile
At the simplest logging level, entries will appear in/var/log/fail2ban.log as follows (fail2ban version 1.0.2):
...2023-02-17 23:44:17,037 fail2ban.actions [992]: NOTICE [apache-auth] Ban XXX.91.244.2282023-02-17 23:44:26,259 fail2ban.actions [992]: NOTICE [apache-auth] Unban XXX.122.233.272023-02-17 23:54:15,034 fail2ban.actions [992]: NOTICE [apache-auth] Unban XXX.91.244.2282023-02-18 00:58:41,938 fail2ban.actions [992]: NOTICE [apache-noscript] Ban XXX.239.163.1262023-02-18 01:07:21,835 fail2ban.actions [992]: NOTICE [apache-noscript] Ban XXX.199.29.1382023-02-18 02:26:19,240 fail2ban.actions [992]: NOTICE [apache-noscript] Unban XXX.25.191.2472023-02-18 03:15:26,341 fail2ban.actions [992]: NOTICE [apache-noscript] Ban XXX.165.48.51
This is all very interesting, but what if you want to see a summaryreport so that you can try to identify IP addresses that regularlytrigger Fail2Ban - so that you can send a report to their ISP or blockthem using a firewall script for example?
Generating Simple Reports
All of the following commands can be run at the command-line or via ascript. They are written for Linux/UNIX systems but may work on otherplatforms.
Grouping by IP address:
awk '($(NF-1) == "Ban"){print $NF}' /var/log/fail2ban.log \ | sort | uniq -c | sort -n
Note: the variable NF equals the number offields in each row of the logfile. So $NF is the value of thelast field.
Sample output:
... 4 XXX.124.81.130 5 XXX.248.175.246 8 XXX.29.45.142
Remember that each time an IP address gets banned it's becausethey've been caught at least maxfailure times, so a total of 8represents maybe 30 matches in the relevant logfile. Once they reach10-20 you might consider them as candidates for reporting, or a morepermanent solution (see below).
To run this report for all logfiles only a slight change is needed:
zgrep -h "Ban " /var/log/fail2ban.log* \ | awk '{print $NF}' | sort | uniq -c
or, even better, we can truncate the IP addresses to identify the most problematic subnets:
zgrep -h "Ban " /var/log/fail2ban.log* \ | awk '{print $NF}' \ | awk -F\. '{print $1"."$2"."}' \ | sort | uniq -c | sort -n | tail
This is the best report for identifying problem subnets. The outputwill be the first two bytes of the most 'caught' subnets:
... 75 83.110. 90 219.95. 154 210.213.
Let's take the last one on the list (highlighted) and see what it'sbeen up to:
zgrep -c " 210.213." /var/log/fail2ban.log*
The output shows how many times those numbers appear in each logfile:
fail2ban.log:39fail2ban.log.1.gz:129fail2ban.log.2.gz:55fail2ban.log.3.gz:78fail2ban.log.4.gz:22
and which specific IP addresses are involved:
zcat -f /var/log/fail2ban.log* \ | awk '($(NF-1) == "Ban" && $NF ~ /^210\.213\./){print $NF}' \ | sort | uniq -c
The output of this will be a list of the IP addresses starting with210.213. If they look like they're part of a subnet (ormultiple subnets) you can copy the lowest and highest numbers in our Subnet Calculator to give you the subnet codewhich you can then add to your firewall rules (see below fordetails).
Grouping by IP address and Hostname:
The command for including hostnames in the list is a bit morecomplicated. You also need to insert the correct path for thelogresolve program which converts IP addresses to hostnames(the path may be something like /usr/sbin/logresolve but itvaries between systems):
awk '($(NF-1) = /Ban/){print $NF,"("$NF")"}' /var/log/fail2ban.log | sort \ | logresolve \ | uniq -c | sort -n
Note that the logresolve command can take some time to execute, especially if there are a lot of IP addresses to be processed.
The output is similar to what we've seen previously, but alsoincludes the hostname which makes it easier to identify the ISP and/orcountry of origin and to see which entries might be related:
... 4 XXX.net.pk (XXX.83.169.221) 5 XXX.248.175.246 (XXX.248.175.246) 8 XXX.example.com.au (XXX.29.45.142)
You can of course just run host, dig, nslookup or logresolve manually on the addresses that you want to identify.
Group by IP address and Fail2Ban section:
grep "Ban " /var/log/fail2ban.log \ | awk -F[\ \:] '{print $19,$17}' | sort | uniq -c | sort -n
This shows us which services each IP address has been trying toaccess/exploit:
... 4 XXX.124.81.130 [sendmail] 5 XXX.248.175.246 [sendmail] 8 XXX.29.45.142 [sendmail]
Now you know which logfiles to look in to see what they were doing toget banned. In this case it's most likely passing forged mail headers tosendmail which you can see in /var/log/mail/mail.log (or therelevant file on your system).
Reporting on 'today's activity:
Here's a report I find useful to run before midnight each day togenerate a summary of the day's activity:
grep "Ban " /var/log/fail2ban.log \ | grep $(date +%Y-%m-%d) \ | awk '{print $NF}' | sort \ | awk '{print $1,"("$1")"}' \ | logresolve \ | uniq -c | sort -n
The output will be the same as the second report above, but limitedto just today's activity rather than the whole logfile.
Grouping by Date and Fail2Ban section
This report scans all fail2ban logfiles and gives you asummary of how many ban events there were for each section on eachday:
zgrep -h "Ban " /var/log/fail2ban.log* \ | awk '{print $6,$1}' \ | sort | uniq -c
This can give you an idea of longer-term trends and the effectivenessof your firewall rules. This method of examining all logfiles ratherthan just the current one can also be applied to most of the reportsabove.
Banning an IP block or subnet
If it turns out that a significant portion of 'unwanted' trafficcomes from a single ISP then you should try sending an email to theirabuse address, but don't be too hopeful of getting a response. If the abuse continues then it's time to get strict.
First have a look at the different IP addresses that are beingcaught. See if you can identify which ones come from the same subnet. The whois reports often include this information, otherwiseyou can use our Subnet Calculator to helpyou along - just paste the lowest and highest addresses into the formand it will give you the smallest subnet that covers them both.
Once you have this value (in the form XXX.XXX.XXX.XXX/XX)you can add a firewall rule using iptables to block them fromthe server completely, or just from the port they're abusing. For asingle address you don't need to worry about subnets and the address canbe used directly.
Block a subnet from accessing SSH:
iptables -I INPUT -p tcp -s XXX.XXX.XXX.XXX/XX --dport ssh -j REJECT --reject-with tcp-reset
Block a subnet from accessing SMTP (mail):
iptables -I INPUT -p tcp -s XXX.XXX.XXX.XXX/XX --dport smtp -j REJECT --reject-with tcp-reset
Block an IP address from HTTP:
iptables -I INPUT -p tcp -s XXX.XXX.XXX.XXX --dport http -j REJECT
Block an IP address from FTP (using DROP):
iptables -I INPUT -p tcp -s XXX.XXX.XXX.XXX --dport ftp -j DROP
and so on for other services.
In the FTP example we've used the DROP policy instead of REJECT as that causes the connection to hang for a longer time rather than giving an instant notification that they've been rejected.
These rules will be added to the start of your firewall. You can also use -A (append) instead of -I (insert) to specify the end of the chain, or include a rule number to insert them into the middle of a chain. The command for removing a rule is identical, just with -D in place of -I, or again, you can specify the chain and line number.
To see what effect these rules are having - the number of packets andbytes being blocked by each rule - use the following command and look atthe values in the first two columns.
iptables -vnL INPUT --line-numbers
At some point (hopefully) the source computer will be 'fixed' or inany case stop abusing your server. You should then remove the firewallrules.
Monitoring the fail2ban log with fail2ban 1.0.2
This is something I've been meaning to investigate for some time now, and there have been a number of request for this ability. Can we use fail2ban to block for a longer time (even permanently) addresses when they've been blocked a number of times by the normal fail2ban filter.
It seems that it is possible, though you may have to set up different jails for different ports. For example, for repeat offenders according to the sendmail filter, add the following to /etc/fail2ban/jail.d/sendmail.conf:
[fail2ban-smtp]enabled = trueport = smtplogpath = /var/log/fail2ban.logmaxretry = 3findtime = 6hbantime = 1d
And then create a file /etc/fail2ban/filter.d/fail2ban-smtp.conf with the following:
[Definition]failregex = NOTICE [[]sendmail(-\w+)?[]] Unban ignoreregex = [[][-\w]+[]] Ban already banned$
Finally start the new jail:
# fail2ban-client add fail2ban-smtp# fail2ban-client start fail2ban-smtp
With these settings, fail2ban will monitor it's own logfile and if a HOST is banned three times (maxretry) in six hours (findtime) they will incur a new ban lasting a full 24 hours (bantime).
If you set the bantime value as negative then the HOST in question will never be unbanned.
Similar rules can be set up for other existing jails, and they can be combined if they share the same port. Let us know though the Feedback form below if you have any questions or comments about using it on your server.
Test new filters using fail2ban-regex
Whenever you add or change a filter you will want to test that the regular expressions are correct by running it over an existing logfile.
The tool for doing this is fail2ban-regex which can be used as follows:
fail2ban-regex --print-all-matched /var/log/fail2ban.log /etc/fail2ban/filter.d/fail2ban-smtp.conf
The first argument is the logfile to be scanned and the second argument the jail configuration file containing failregex.
The output lists first all the regular expressions that are being used followed by a tally of how many matches there are for each one. This should match what you can find manually in the logfile using grep or awk. Finally, a list of the 'caught' IP addresses is displayed.
Results=======Failregex|- Regular expressions:| [1] \[sendmail\] Ban <HOST>|`- Number of matches: [1] 46 match(es)...
If nothing is being matched, or everything is being matched that may suggest a problem with the regexp. Otherwise, if everything looks ok, you can start the new jail as described above.
You can find more details on how to check and optimise your Fail2Ban filters in this article.
- System Monitoring the fail2ban log
- System Optimising your Fail2Ban filters
- System Fail2Ban 0.8.3 Howto
- System fail2ban and sendmail
- System Using systemd to bind fail2ban to nftables
- System Blocking FTP Hacking Attempts
- System Using a Fail2Ban Jail to Whitelist a User
- System Implementing Port Knocking with knockd
- System fail2ban and iptables
< System
User Comments
Post your comment or questionLuke 13 August, 2019
Typing "zcat /var/log/fail2ban.log*" says:
gzip: /var/log/fail2ban.log: not in gzip format
gzip: /var/log/fail2ban.log.1: not in gzip format
But typing it without zcat and just cat shows nothing. Any help?
Just use "zcat -f /var/log/fail2ban.log*" to avoid this warning, but the Fail2Ban log format has also changed in recent versions, so the above commands will need re-working
John 7 August, 2014
Is there a command I can type that will show me a list of fail2ban banned IP's?
Or if I wanted to clear out all banned IP's, how would I do it?
From the command-line you can view all the iptables rules, including Fail2Ban using:
iptables -vnL --line-numbers
and remove a rule using (with caution):
iptables -D fail2ban-<JAIL> <#LINE>
You can query Fail2Ban directly:
fail2ban-client status
fail2ban-client status <JAIL>
Where '<JAIL>' is one of the jails listed in the output of the first command (e.g. 'ssh' or 'apache-overflows').
To clear out all (most) banned IP's just stop and start Fail2Ban or one particular jail. There is also a configuration option to white list specific ip addresses so they are never banned.
Robert 21 April, 2014
Actually for the fail2ban.log filter to work properly you should filter for the Unban instead of the Ban.
Else it tries to set a ban that already exists and after 10 minutes the ban will be removed like always. So by checking for the Unban you can apply that ban for 24 hours (or more) when the ip showed up to many times in your fail2ban.log
Mijo 21 October, 2013
Hello,
thank you very much for this great article, it explains it very well, while still giving solutions that are usable!
Regards...
Another Kyle 22 June, 2012
Great tutorial. I did have one minor issue setting up the new filter (/etc/fail2ban/filter.d/fail2ban-smtp.conf)
I had to add the line
[Definition]
above the
failregex = [sendmail] Ban <HOST>
ignoreregex =
lines.
Fail2Ban v0.8.4-SVN on debian
Kyle 2 June, 2011
Wow. Incredible commands. Helps me alot. With trial and error I got a nice combination:
daily list with logresolve and shown services of bans:grep "Ban " /var/log/fail2ban.log | grep `date +%Y-%m-%d` | awk -F[\ \:] '{print $10,$8,$10}' | logresolve | sort | uniq -c | sort -n
Since logresolve is not that good it might be interesting to implement Geo-Ip Service like that from maxmind.com
Jason Lynch 2 March, 2009
Does anyone know if fail2ban can be made to read gzipped logs as well? The /etc/log/fail2ban.log file only seems to contain a day or two of data. If we're looking for repeat offenders, I'd think we'd want to go back a little further if possible.
I think as long as fail2ban is running uninterrupted it will keep track of all matches within findtime. It only when it's restarted that you miss the data from rolled over log files.
Jason Lynch 23 February, 2009
I'd love to see an article on how to have fail2ban monitor it's own logs and automatically ban repeat offenders for an extended period of time (or permanently). I am currently manually grepping through those logs and adding the IP's to my blocklist.
I've just added a new section to the above article for this
Tom Klein 17 February, 2008
Thank you for the great article.
Is there any way to permanently ban IP addresses in an automatic way, which are banned e.g. 5 times before?
You can always add a rule to iptables using the command line to block a particular IP address or block of addresses:
iptables -A INPUT -p tcp -s <host> --dport <port> -j REJECT --reject-with tcp-reset
Or you could set up a Fail2Ban rule to monitor it's own logfile and block repeat offenders for a longer time period. It's something I've thought about doing and might be adding here before too long.