diff --git a/ChangeLog b/ChangeLog index 3dee9cc4..f14f5d85 100644 --- a/ChangeLog +++ b/ChangeLog @@ -13,6 +13,15 @@ TODO: implementing of options resp. other tasks from PR #1346 ### Fixes * [Grave] memory leak's fixed (gh-1277, gh-1234) +* [Grave] Misleading date patterns defined more precisely (using extended syntax + `%Ex[mdHMS]` for exact two-digit match or e. g. `%ExY` as more precise year + pattern, within same century of last year and the next 3 years) +* [Grave] extends date detector template with distance (position of match in + log-line), to prevent grave collision using (re)ordered template list (e.g. + find-spot of wrong date-match inside foreign input, misleading date patterns + by ambiguous formats, etc.) +* Distance collision check always prefers template with shortest distance + (left for right) if date pattern is not anchored * Tricky bug fix: last position of log file will be never retrieved (gh-795), because of CASCADE all log entries will be deleted from logs table together with jail, if used "INSERT OR REPLACE" statement @@ -31,6 +40,22 @@ TODO: implementing of options resp. other tasks from PR #1346 * Pyinotify-backend: stability fix for sporadically errors in multi-threaded environment (without lock) * Fixed sporadically error in testCymruInfoNxdomain, because of unsorted values +* Misleading errors logged from ignorecommand in success case on retcode 1 (gh-1194) +* fail2ban.service - systemd service updated (gh-1618): + - starting service in normal mode (without forking) + - does not restart if service exited normally (exit-code 0, e.g. stopped via fail2ban-client) + - does not restart if service can not start (exit-code 255, e.g. wrong configuration, etc.) + - service can be additionally started/stopped with commands (fail2ban-client, fail2ban-server) + - automatically creates `/var/run/fail2ban` directory before start fail2ban + (systems with virtual resp. memory-based FS for `/var/run`), see gh-1531 + - if fail2ban running as systemd-service, for logging to the systemd-journal, + the `logtarget` could be set to STDOUT + - value `logtarget` for system targets allowed also in lowercase (stdout, stderr, syslog, etc.) +* Fixed UTC/GMT named time zone, using `%Z` and `%z` patterns + (special case with 0 zone offset, see gh-1575) +* `filter.d/freeswitch.conf` + - Optional prefixes (server, daemon, dual time) if systemd daemon logs used (gh-1548) + - User part rewritten to accept IPv6 resp. domain after "@" (gh-1548) ### New Features * IPv6 support: @@ -60,6 +85,8 @@ TODO: implementing of options resp. other tasks from PR #1346 banned in this jail, if option `--unban` specified - `unban --all` - unbans all IP addresses (in all jails and database) - `unban ... ` - unbans \ (in all jails and database) (see gh-1388) + - introduced new option `-t` or `--test` to test configuration resp. start server only + if configuration is clean (fails by wrong configured jails if option `-t` specified) * New command action parameter `actionrepair` - command executed in order to restore sane environment in error case of `actioncheck`. @@ -124,6 +151,51 @@ fail2ban-client set loglevel INFO - new replacement for `` in opposition to ``, for separate usage of 2 address groups only (regardless of `usedns`), `ip4` and `ip6` together, without host (dns) +* Misconfigured jails don't prevent fail2ban from starting, server starts + nevertheless, as long as one jail was successful configured (gh-1619) + Message about wrong jail configuration logged in client log (stdout, systemd + journal etc.) and in server log with error level +* More precise date template handling (WARNING: theoretically possible incompatibilities): + - datedetector rewritten more strict as earlier; + - default templates can be specified exacter using prefix/suffix syntax (via `datepattern`); + - more as one date pattern can be specified using option `datepattern` now + (new-line separated); + - some default options like `datepattern` can be specified directly in + section `[Definition]`, that avoids contrary usage of unnecessarily `[Init]` + section, because of performance (each extra section costs time); + - option `datepattern` can be specified in jail also (e. g. jails without filters + or custom log-format, new-line separated for multiple patterns); + - if first unnamed group specified in pattern, only this will be cut out from + search log-line (e. g.: `^date:[({DATE})]` will cut out only datetime match + pattern, and leaves `date:[] ...` for searching in filter); + - faster match and fewer searching of appropriate templates + (DateDetector.matchTime calls rarer DateTemplate.matchDate now); + - several standard filters extended with exact prefixed or anchored date templates; +* Added possibility to recognize restored state of the tickets (see gh-1669). + New option `norestored` introduced, to ignore restored tickets (after restart). + To avoid execution of ban/unban for the restored tickets, `norestored = true` + could be added in definition section of action. + For conditional usage in the shell-based actions an interpolation `` + could be used also. E. g. it is enough to add following script-piece at begin + of `actionban` (or `actionunban`) to prevent execution: + `if [ '' = '1' ]; then exit 0; fi;` + Several actions extended now using `norestored` option: + - complain.conf + - dshield.conf + - mail-buffered.conf + - mail-whois-lines.conf + - mail-whois.conf + - mail.conf + - sendmail-buffered.conf + - sendmail-geoip-lines.conf + - sendmail-whois-ipjailmatches.conf + - sendmail-whois-ipmatches.conf + - sendmail-whois-lines.conf + - sendmail-whois-matches.conf + - sendmail-whois.conf + - sendmail.conf + - smtp.py + - xarf-login-attack.conf * fail2ban-testcases: - `assertLogged` extended with parameter wait (to wait up to specified timeout, before we throw assert exception) + test cases rewritten using that @@ -131,29 +203,69 @@ fail2ban-client set loglevel INFO - new `with_foreground_server_thread` decorator to test several client/server commands -ver. 0.9.6 (2016/XX/XX) - wanna-be-released +ver. 0.9.x (2016/??/??) - wanna-be-released ----------- 0.9.x line is no longer heavily developed. If you are interested in new features (e.g. IPv6 support), please consider 0.10 branch and its releases. +### Fixes +* Fixed a systemd-journal handling in fail2ban-regex (gh-1657) +* filter.d/sshd.conf + - Fixed non-anchored part of failregex (misleading match of colon inside + IPv6 address instead of `: ` in the reason-part by missing space, gh-1658) + (0.10th resp. IPv6 relevant only, amend for gh-1479) +* config/pathes-freebsd.conf + - Fixed filenames for apache and nginx log files (gh-1667) +* filter.d/sshd.conf + - new aggressive rules (gh-864): + - Connection reset by peer (multi-line rule during authorization process) + - No supported authentication methods available + - single line and multi-line expression optimized, added optional prefixes + and suffix (logged from several ssh versions), according to gh-1206; + - fixed expression received disconnect auth fail (optional space after port + part, gh-1652) + and suffix (logged from several ssh versions), according to gh-1206; +* filter.d/suhosin.conf + - greedy catch-all before `` fixed (potential vulnerability) +* Filter tests extended with check of all config-regexp, that contains greedy catch-all + before ``, that is hard-anchored at end or precise sub expression after `` + +### New Features +* New Actions: + - action.d/netscaler: Block IPs on a Citrix Netscaler ADC (gh-1663) + +* New Filters: + - filter.d/domino-smtp: IBM Domino SMTP task (gh-1603) + +### Enhancements + + +ver. 0.9.6 (2016/12/10) - stretch-is-coming +----------- + ### Fixes * Misleading add resp. enable of (already available) jail in database, that induced a subsequent error: last position of log file will be never retrieved (gh-795) * Fixed a distribution related bug within testReadStockJailConfForceEnabled (e.g. test-cases faults on Fedora, see gh-1353) -* Fixed pythonic filters and test scripts (running via wrong python version, +* Fixed pythonic filters and test scripts (running via wrong python version, uses "fail2ban-python" now); * Fixed test case "testSetupInstallRoot" for not default python version (also using direct call, out of virtualenv); * Fixed ambiguous wrong recognized date pattern resp. its optional parts (see gh-1512); * FIPS compliant, use sha1 instead of md5 if it not allowed (see gh-1540) * Monit config: scripting is not supported in path (gh-1556) +* `filter.d/apache-modsecurity.conf` + - Fixed for newer version (one space, gh-1626), optimized: non-greedy catch-all + replaced for safer match, unneeded catch-all anchoring removed, non-capturing * `filter.d/asterisk.conf` - Fixed to match different asterisk log prefix (source file: method:) +* `filter.d/dovecot.conf` + - Fixed failregex ignores failures through some not relevant info (gh-1623) * `filter.d/ignorecommands/apache-fakegooglebot` - - Fixed error within apache-fakegooglebot, that will be called + - Fixed error within apache-fakegooglebot, that will be called with wrong python version (gh-1506) * `filter.d/assp.conf` - Extended failregex and test cases to handle ASSP V1 and V2 (gh-1494) @@ -161,18 +273,27 @@ releases. - Allow for having no trailing space after 'failed:' (gh-1497) * `filter.d/vsftpd.conf` - Optional reason part in message after FAIL LOGIN (gh-1543) - +* `filter.d/sendmail-reject.conf` + - removed mandatory double space (if dns-host available, gh-1579) +* filter.d/sshd.conf + - recognized "Failed publickey for" (gh-1477); + - optimized failregex to match all of "Failed any-method for ... from " (gh-1479) + - eliminated possible complex injections (on user-name resp. auth-info, see gh-1479) + - optional port part after host (see gh-1533, gh-1581) ### New Features * New Actions: - `action.d/npf.conf` for NPF, the latest packet filter for NetBSD +* New Filters: + - `filter.d/mongodb-auth.conf` for MongoDB (document-oriented NoSQL database engine) + (gh-1586, gh-1606 and gh-1607) ### Enhancements -* DateTemplate regexp extended with the word-end boundary, additionally to +* DateTemplate regexp extended with the word-end boundary, additionally to word-start boundary -* Introduces new command "fail2ban-python", as automatically created symlink to +* Introduces new command "fail2ban-python", as automatically created symlink to python executable, where fail2ban currently installed (resp. its modules are located): - - allows to use the same version, fail2ban currently running, e.g. in + - allows to use the same version, fail2ban currently running, e.g. in external scripts just via replace python with fail2ban-python: ```diff -#!/usr/bin/env python diff --git a/FILTERS b/FILTERS index 10113dfc..e114973a 100644 --- a/FILTERS +++ b/FILTERS @@ -227,7 +227,7 @@ Regular expressions (failregex, ignoreregex) assume that the date/time has been removed from the log line (this is just how fail2ban works internally ATM). If the format is like ' error 1.2.3.4 is evil' then you need to match -the < at the start so regex should be similar to '^<> is evil$' using +the <> at the start so regex should be similar to '^<> error is evil$' using where the IP/domain name appears in the log line. The following general rules apply to regular expressions: diff --git a/MANIFEST b/MANIFEST index 56a53bec..92acb9e4 100644 --- a/MANIFEST +++ b/MANIFEST @@ -41,6 +41,7 @@ config/action.d/mynetwatchman.conf config/action.d/nftables-allports.conf config/action.d/nftables-common.conf config/action.d/nftables-multiport.conf +config/action.d/npf.conf config/action.d/nsupdate.conf config/action.d/osx-afctl.conf config/action.d/osx-ipfw.conf @@ -100,6 +101,7 @@ config/filter.d/horde.conf config/filter.d/ignorecommands/apache-fakegooglebot config/filter.d/kerio.conf config/filter.d/lighttpd-auth.conf +config/filter.d/mongodb-auth.conf config/filter.d/monit.conf config/filter.d/murmur.conf config/filter.d/mysqld-auth.conf @@ -154,6 +156,7 @@ config/paths-opensuse.conf config/paths-osx.conf CONTRIBUTING.md COPYING +.coveragerc DEVELOP fail2ban-2to3 fail2ban/client/actionreader.py @@ -214,6 +217,7 @@ fail2ban/tests/clientbeautifiertestcase.py fail2ban/tests/clientreadertestcase.py fail2ban/tests/config/action.d/brokenaction.conf fail2ban/tests/config/fail2ban.conf +fail2ban/tests/config/filter.d/common.conf fail2ban/tests/config/filter.d/simple.conf fail2ban/tests/config/filter.d/test.conf fail2ban/tests/config/filter.d/test.local @@ -289,6 +293,7 @@ fail2ban/tests/files/logs/haproxy-http-auth fail2ban/tests/files/logs/horde fail2ban/tests/files/logs/kerio fail2ban/tests/files/logs/lighttpd-auth +fail2ban/tests/files/logs/mongodb-auth fail2ban/tests/files/logs/monit fail2ban/tests/files/logs/murmur fail2ban/tests/files/logs/mysqld-auth @@ -389,6 +394,7 @@ man/fail2ban-testcases.1 man/fail2ban-testcases.h2m man/generate-man man/jail.conf.5 +.pylintrc README.md README.Solaris RELEASE diff --git a/README.md b/README.md index be0be636..ee654acb 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,9 @@ Though Fail2Ban is able to reduce the rate of incorrect authentications attempts, it cannot eliminate the risk that weak authentication presents. Configure services to use only two factor or public/private authentication mechanisms if you really want to protect services. + + | Since v0.10 fail2ban supports the matching of the IPv6 addresses. +------|------ This README is a quick introduction to Fail2ban. More documentation, FAQ, HOWTOs are available in fail2ban(1) manpage and on the website http://www.fail2ban.org diff --git a/RELEASE b/RELEASE index 1e905da1..2b2bc58e 100644 --- a/RELEASE +++ b/RELEASE @@ -53,7 +53,7 @@ Preparation or an alternative for comparison with previous release - git diff 0.9.5 | grep -B2 'index 0000000..' | grep -B1 'new file mode' | sed -n -e '/^diff /s,.* b/,,gp' >> MANIFEST + git diff 0.10.0 | grep -B2 'index 0000000..' | grep -B1 'new file mode' | sed -n -e '/^diff /s,.* b/,,gp' >> MANIFEST sort MANIFEST | uniq | sponge MANIFEST * Run:: @@ -70,7 +70,7 @@ Preparation * clean up current directory:: - diff -rul --exclude \*.pyc . /tmp/fail2ban-0.9.5/ + diff -rul --exclude \*.pyc . /tmp/fail2ban-0.10.0/ * Only differences should be files that you don't want distributed. @@ -83,7 +83,7 @@ Preparation * To generate a list of committers use e.g.:: - git shortlog -sn 0.9.5.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g' + git shortlog -sn 0.10.0.. | sed -e 's,^[ 0-9\t]*,,g' | tr '\n' '\|' | sed -e 's:|:, :g' * Ensure the top of the ChangeLog has the right version and current date. * Ensure the top entry of the ChangeLog has the right version and current date. @@ -106,7 +106,7 @@ Preparation * Tag the release by using a signed (and annotated) tag. Cut/paste release ChangeLog entry as tag annotation:: - git tag -s 0.9.5 + git tag -s 0.10.0 Pre Release =========== diff --git a/THANKS b/THANKS index 04e4cdc7..a20f9e17 100644 --- a/THANKS +++ b/THANKS @@ -110,6 +110,7 @@ SATO Kentaro Sean DuBois Sebastian Arcus Serg G. Brester (sebres) +Sergey Safarov Sireyessire silviogarbes Stefan Tatschner @@ -120,6 +121,7 @@ Thomas Mayer Tom Pike Tom Hendrikx Tomas Pihl +Thomas Skierlo (phaleas) Tony Lawrence Tomasz Ciolek Tyler diff --git a/config/action.d/complain.conf b/config/action.d/complain.conf index 9247803e..84dbaf39 100644 --- a/config/action.d/complain.conf +++ b/config/action.d/complain.conf @@ -28,8 +28,15 @@ # +[INCLUDES] + +before = helpers-common.conf + [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD @@ -54,10 +61,16 @@ actioncheck = # Tags: See jail.conf(5) man page # Values: CMD # -actionban = oifs=${IFS}; IFS=.;SEP_IP=( ); set -- ${SEP_IP}; ADDRESSES=$(dig +short -t txt -q $4.$3.$2.$1.abuse-contacts.abusix.org); IFS=${oifs} - IP= +actionban = oifs=${IFS}; + IFS=.; SEP_IP=( ); set -- ${SEP_IP}; ADDRESSES=$(dig +short -t txt -q $4.$3.$2.$1.abuse-contacts.abusix.org); + IFS=,; ADDRESSES=$(echo $ADDRESSES) + IFS=${oifs} + IP= if [ ! -z "$ADDRESSES" ]; then - (printf %%b "\n"; date '+Note: Local timezone is %%z (%%Z)'; grep -E '(^|[^0-9])([^0-9]|$)' ) | "Abuse from " ${ADDRESSES//,/\" \"} + ( printf %%b "\n"; date '+Note: Local timezone is %%z (%%Z)'; + printf %%b "\nLines containing failures of (max )\n"; + %(_grep_logs)s; + ) | "Abuse from " $ADDRESSES fi # Option: actionunban @@ -92,3 +105,7 @@ mailcmd = mail -s # mailargs = +# Number of log lines to include in the email +# +#grepmax = 1000 +#grepopts = -m diff --git a/config/action.d/dshield.conf b/config/action.d/dshield.conf index a0041986..4f2e09ca 100644 --- a/config/action.d/dshield.conf +++ b/config/action.d/dshield.conf @@ -28,6 +28,9 @@ [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD diff --git a/config/action.d/firewallcmd-rich-logging.conf b/config/action.d/firewallcmd-rich-logging.conf index d2c8fc2f..badfee83 100644 --- a/config/action.d/firewallcmd-rich-logging.conf +++ b/config/action.d/firewallcmd-rich-logging.conf @@ -35,7 +35,7 @@ actioncheck = # service name example: # firewall-cmd --zone= --add-rich-rule="rule family='' source address='' service name='' log prefix='f2b-' level='' limit value='/m' " # -# Because rich rules can only handle single or a range of ports we must split ports and execute the command for each port. Ports can be single and ranges seperated by a comma or space for an example: http, https, 22-60, 18 smtp +# Because rich rules can only handle single or a range of ports we must split ports and execute the command for each port. Ports can be single and ranges separated by a comma or space for an example: http, https, 22-60, 18 smtp actionban = ports=""; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='' source address='' port port='$p' protocol='' log prefix='f2b-' level='' limit value='/m' "; done diff --git a/config/action.d/firewallcmd-rich-rules.conf b/config/action.d/firewallcmd-rich-rules.conf index e64c3823..bed71797 100644 --- a/config/action.d/firewallcmd-rich-rules.conf +++ b/config/action.d/firewallcmd-rich-rules.conf @@ -33,7 +33,7 @@ actioncheck = # service name example: # firewall-cmd --zone= --add-rich-rule="rule family='ipv4' source address='' service name='' " # -# Because rich rules can only handle single or a range of ports we must split ports and execute the command for each port. Ports can be single and ranges seperated by a comma or space for an example: http, https, 22-60, 18 smtp +# Because rich rules can only handle single or a range of ports we must split ports and execute the command for each port. Ports can be single and ranges separated by a comma or space for an example: http, https, 22-60, 18 smtp actionban = ports=""; for p in $(echo $ports | tr ", " " "); do firewall-cmd --add-rich-rule="rule family='' source address='' port port='$p' protocol='' "; done diff --git a/config/action.d/helpers-common.conf b/config/action.d/helpers-common.conf new file mode 100644 index 00000000..b04c7f7a --- /dev/null +++ b/config/action.d/helpers-common.conf @@ -0,0 +1,16 @@ +[DEFAULT] + +# Usage: +# _grep_logs_args = 'test' +# (printf %%b "Log-excerpt contains 'test':\n"; %(_grep_logs)s; printf %%b "Log-excerpt contains 'test':\n") | mail ... +# +_grep_logs = logpath=""; grep -E %(_grep_logs_args)s $logpath | +_grep_logs_args = '(^|[^0-9])([^0-9]|$)' + +# Used for actions, that should not by executed if ticket was restored: +_bypass_if_restored = if [ '' = '1' ]; then exit 0; fi; + +[Init] +greplimit = tail -n +grepmax = 1000 +grepopts = -m \ No newline at end of file diff --git a/config/action.d/mail-buffered.conf b/config/action.d/mail-buffered.conf index 914d4a5a..e74db9cc 100644 --- a/config/action.d/mail-buffered.conf +++ b/config/action.d/mail-buffered.conf @@ -6,6 +6,9 @@ [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD diff --git a/config/action.d/mail-whois-lines.conf b/config/action.d/mail-whois-lines.conf index 6e39c605..0852ba8f 100644 --- a/config/action.d/mail-whois-lines.conf +++ b/config/action.d/mail-whois-lines.conf @@ -7,9 +7,13 @@ [INCLUDES] before = mail-whois-common.conf + helpers-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD @@ -17,7 +21,7 @@ before = mail-whois-common.conf actionstart = printf %%b "Hi,\n The jail has been started successfully.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : started on `uname -n`" + Fail2Ban" | -s "[Fail2Ban] : started on `uname -n`" # Option: actionstop # Notes.: command executed once at the end of Fail2Ban @@ -26,7 +30,7 @@ actionstart = printf %%b "Hi,\n actionstop = printf %%b "Hi,\n The jail has been stopped.\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : stopped on `uname -n`" + Fail2Ban" | -s "[Fail2Ban] : stopped on `uname -n`" # Option: actioncheck # Notes.: command executed once before each actionban command @@ -40,15 +44,19 @@ actioncheck = # Tags: See jail.conf(5) man page # Values: CMD # -actionban = printf %%b "Hi,\n + +_ban_mail_content = ( printf %%b "Hi,\n The IP has just been banned by Fail2Ban after attempts against .\n\n - Here is more information about :\n - `%(_whois_command)s`\n\n - Lines containing IP: in \n - `grep -E '(^|[^0-9])([^0-9]|$)' `\n\n + Here is more information about :\n" + %(_whois_command)s; + printf %%b "\nLines containing failures of (max )\n"; + %(_grep_logs)s; + printf %%b "\n Regards,\n - Fail2Ban"|mail -s "[Fail2Ban] : banned from `uname -n`" + Fail2Ban" ) + +actionban = %(_ban_mail_content)s | "[Fail2Ban] : banned from `uname -n`" # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the @@ -60,6 +68,12 @@ actionunban = [Init] +# Option: mailcmd +# Notes.: Your system mail command. Is passed 2 args: subject and recipient +# Values: CMD +# +mailcmd = mail -s + # Default name of the chain # name = default @@ -74,4 +88,5 @@ logpath = /dev/null # Number of log lines to include in the email # -grepopts = -m 1000 +#grepmax = 1000 +#grepopts = -m diff --git a/config/action.d/mail-whois.conf b/config/action.d/mail-whois.conf index 018c327d..553bfb69 100644 --- a/config/action.d/mail-whois.conf +++ b/config/action.d/mail-whois.conf @@ -10,6 +10,9 @@ before = mail-whois-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD diff --git a/config/action.d/mail.conf b/config/action.d/mail.conf index 7bf51a1d..4715ecc5 100644 --- a/config/action.d/mail.conf +++ b/config/action.d/mail.conf @@ -6,6 +6,9 @@ [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD diff --git a/config/action.d/netscaler.conf b/config/action.d/netscaler.conf new file mode 100644 index 00000000..87f7e7bf --- /dev/null +++ b/config/action.d/netscaler.conf @@ -0,0 +1,33 @@ +# Fail2ban Citrix Netscaler Action +# by Juliano Jeziorny +# juliano@jeziorny.eu +# +# The script will add offender IPs to a dataset on netscaler, the dataset can then be used to block the IPs at a cs/vserver or global level +# This dataset is then used to block IPs using responder policies on the netscaler. +# +# The script assumes using HTTPS with unsecure certificate to access the netscaler, +# if you have a valid certificate installed remove the -k from the curl lines, or if you want http change it accordingly (and remove the -k) +# +# This action depends on curl +# +# You need to populate the 3 options inside Init +# +# ns_host: IP or hostname of netslcaer appliance +# ns_auth: username:password, suggest base64 encoded for a little added security (echo -n "username:password" | base64) +# ns_dataset: Name of the netscaler dataset holding the IPs to be blocked. +# +# For further details on how to use it please check http://blog.ckzone.eu/2017/01/fail2ban-action-for-citrix-netscaler.html + +[Init] +ns_host = +ns_auth = +ns_dataset = + +[Definition] +actionstart = curl -kH 'Authorization: Basic ' https:///nitro/v1/config + +actioncheck = + +actionban = curl -k -H 'Authorization: Basic ' -X PUT -d '{"policydataset_value_binding":{"name":"","value":""}}' https:///nitro/v1/config/ + +actionunban = curl -H 'Authorization: Basic ' -X DELETE -k "https:///nitro/v1/config/policydataset_value_binding/?args=value:" diff --git a/config/action.d/sendmail-buffered.conf b/config/action.d/sendmail-buffered.conf index 80eb20a3..a91a6957 100644 --- a/config/action.d/sendmail-buffered.conf +++ b/config/action.d/sendmail-buffered.conf @@ -10,6 +10,9 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionstart # Notes.: command executed once at the start of Fail2Ban. # Values: CMD diff --git a/config/action.d/sendmail-geoip-lines.conf b/config/action.d/sendmail-geoip-lines.conf index 2232642c..34c3aedd 100644 --- a/config/action.d/sendmail-geoip-lines.conf +++ b/config/action.d/sendmail-geoip-lines.conf @@ -7,9 +7,13 @@ [INCLUDES] before = sendmail-common.conf + helpers-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: Command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. @@ -19,7 +23,7 @@ before = sendmail-common.conf # Tags: See jail.conf(5) man page # Values: CMD # -actionban = printf %%b "Subject: [Fail2Ban] : banned from `uname -n` +actionban = ( printf %%b "Subject: [Fail2Ban] : banned from `uname -n` Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"` From: <> To: \n @@ -33,10 +37,11 @@ actionban = printf %%b "Subject: [Fail2Ban] : banned from `uname -n` Country:`geoiplookup -f /usr/share/GeoIP/GeoIP.dat "" | cut -d':' -f2-` AS:`geoiplookup -f /usr/share/GeoIP/GeoIPASNum.dat "" | cut -d':' -f2-` hostname: `host -t A 2>&1`\n\n - Lines containing IP: in \n - `grep -E '(^|[^0-9])([^0-9]|$)' `\n\n + Lines containing failures of \n"; + %(_grep_logs)s; + printf %%b "\n Regards,\n - Fail2Ban" | /usr/sbin/sendmail -f + Fail2Ban" ) | /usr/sbin/sendmail -f [Init] @@ -50,4 +55,5 @@ logpath = /dev/null # Number of log lines to include in the email # -grepopts = -m 1000 +#grepmax = 1000 +#grepopts = -m diff --git a/config/action.d/sendmail-whois-ipjailmatches.conf b/config/action.d/sendmail-whois-ipjailmatches.conf index 689ffe45..5bcefe89 100644 --- a/config/action.d/sendmail-whois-ipjailmatches.conf +++ b/config/action.d/sendmail-whois-ipjailmatches.conf @@ -10,6 +10,9 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. diff --git a/config/action.d/sendmail-whois-ipmatches.conf b/config/action.d/sendmail-whois-ipmatches.conf index b06e6db6..4a8edcb7 100644 --- a/config/action.d/sendmail-whois-ipmatches.conf +++ b/config/action.d/sendmail-whois-ipmatches.conf @@ -10,6 +10,9 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. diff --git a/config/action.d/sendmail-whois-lines.conf b/config/action.d/sendmail-whois-lines.conf index 4156c947..e3a1c974 100644 --- a/config/action.d/sendmail-whois-lines.conf +++ b/config/action.d/sendmail-whois-lines.conf @@ -7,16 +7,20 @@ [INCLUDES] before = sendmail-common.conf + helpers-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. # Tags: See jail.conf(5) man page # Values: CMD # -actionban = printf %%b "Subject: [Fail2Ban] : banned from `uname -n` +actionban = ( printf %%b "Subject: [Fail2Ban] : banned from `uname -n` Date: `LC_ALL=C date +"%%a, %%d %%h %%Y %%T %%z"` From: <> To: \n @@ -25,10 +29,11 @@ actionban = printf %%b "Subject: [Fail2Ban] : banned from `uname -n` attempts against .\n\n Here is more information about :\n `/usr/bin/whois || echo missing whois program`\n\n - Lines containing IP: in \n - `grep -E '(^|[^0-9])([^0-9]|$)' `\n\n + Lines containing failures of \n"; + %(_grep_logs)s; + printf %%b "\n Regards,\n - Fail2Ban" | /usr/sbin/sendmail -f + Fail2Ban" ) | /usr/sbin/sendmail -f [Init] @@ -42,4 +47,5 @@ logpath = /dev/null # Number of log lines to include in the email # -grepopts = -m 1000 +#grepmax = 1000 +#grepopts = -m diff --git a/config/action.d/sendmail-whois-matches.conf b/config/action.d/sendmail-whois-matches.conf index 8bca5937..fc4ba061 100644 --- a/config/action.d/sendmail-whois-matches.conf +++ b/config/action.d/sendmail-whois-matches.conf @@ -10,6 +10,9 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. diff --git a/config/action.d/sendmail-whois.conf b/config/action.d/sendmail-whois.conf index 55b80bc5..b8d99423 100644 --- a/config/action.d/sendmail-whois.conf +++ b/config/action.d/sendmail-whois.conf @@ -10,6 +10,9 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. diff --git a/config/action.d/sendmail.conf b/config/action.d/sendmail.conf index 5f5670c3..62c94439 100644 --- a/config/action.d/sendmail.conf +++ b/config/action.d/sendmail.conf @@ -10,6 +10,9 @@ before = sendmail-common.conf [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + # Option: actionban # Notes.: command executed when banning an IP. Take care that the # command is executed with Fail2Ban user rights. diff --git a/config/action.d/smtp.py b/config/action.d/smtp.py index 2429cf48..aa85b259 100644 --- a/config/action.d/smtp.py +++ b/config/action.d/smtp.py @@ -126,6 +126,9 @@ class SMTPAction(ActionBase): bantime = self._jail.actions.getBanTime, ) + # bypass ban/unban for restored tickets + self.norestored = 1 + def _sendMessage(self, subject, text): """Sends message based on arguments and instance's properties. @@ -211,6 +214,8 @@ class SMTPAction(ActionBase): Dictionary which includes information in relation to the ban. """ + if aInfo.get('restored'): + return aInfo.update(self.message_values) message = "".join([ messages['ban']['head'], diff --git a/config/action.d/xarf-login-attack.conf b/config/action.d/xarf-login-attack.conf index 3ab73817..5274cdaf 100644 --- a/config/action.d/xarf-login-attack.conf +++ b/config/action.d/xarf-login-attack.conf @@ -32,6 +32,9 @@ [Definition] +# bypass ban/unban for restored tickets +norestored = 1 + actionstart = actionstop = diff --git a/config/filter.d/3proxy.conf b/config/filter.d/3proxy.conf index 299c3a29..76c7573c 100644 --- a/config/filter.d/3proxy.conf +++ b/config/filter.d/3proxy.conf @@ -9,6 +9,8 @@ failregex = ^\s[+-]\d{4} \S+ \d{3}0[1-9] \S+ :\d+ [\d.]+:\d+ \d+ \d+ \d+\s ignoreregex = +datepattern = {^LN-BEG} + # DEV Notes: # http://www.3proxy.ru/howtoe.asp#ERRORS indicates that 01-09 are # all authentication problems (%E field) diff --git a/config/filter.d/apache-badbots.conf b/config/filter.d/apache-badbots.conf index 48b30666..f42aa159 100644 --- a/config/filter.d/apache-badbots.conf +++ b/config/filter.d/apache-badbots.conf @@ -14,6 +14,9 @@ failregex = ^ -.*"(GET|POST|HEAD).*HTTP.*"(?:%(badbots)s|%(badbotscustom)s ignoreregex = +datepattern = ^[^\[]*\[({DATE}) + {^LN-BEG} + # DEV Notes: # List of bad bots fetched from http://www.user-agents.org # Generated on Thu Nov 7 14:23:35 PST 2013 by files/gen_badbots. diff --git a/config/filter.d/apache-common.conf b/config/filter.d/apache-common.conf index c843c92e..3d1f902b 100644 --- a/config/filter.d/apache-common.conf +++ b/config/filter.d/apache-common.conf @@ -10,6 +10,8 @@ after = apache-common.local _apache_error_client = \[\] \[(:?error|\S+:\S+)\]( \[pid \d+(:\S+ \d+)?\])? \[client (:\d{1,5})?\] +datepattern = {^LN-BEG} + # Common prefix for [error] apache messages which also would include # Depending on the version it could be # 2.2: [Sat Jun 01 11:23:08 2013] [error] [client 1.2.3.4] diff --git a/config/filter.d/apache-fakegooglebot.conf b/config/filter.d/apache-fakegooglebot.conf index b8a73504..729410ad 100644 --- a/config/filter.d/apache-fakegooglebot.conf +++ b/config/filter.d/apache-fakegooglebot.conf @@ -6,6 +6,8 @@ failregex = ^ .*Googlebot.*$ ignoreregex = +datepattern = ^[^\[]*\[({DATE}) + {^LN-BEG} # DEV Notes: # diff --git a/config/filter.d/apache-modsecurity.conf b/config/filter.d/apache-modsecurity.conf index ad7e9b24..13e9c5ea 100644 --- a/config/filter.d/apache-modsecurity.conf +++ b/config/filter.d/apache-modsecurity.conf @@ -10,9 +10,10 @@ before = apache-common.conf [Definition] -failregex = ^%(_apache_error_client)s ModSecurity: (\[.*?\] )*Access denied with code [45]\d\d.*$ +failregex = ^%(_apache_error_client)s ModSecurity:\s+(?:\[(?:\w+ \"[^\"]*\"|[^\]]*)\]\s*)*Access denied with code [45]\d\d ignoreregex = # https://github.com/SpiderLabs/ModSecurity/wiki/ModSecurity-2-Data-Formats # Author: Daniel Black +# Sergey G. Brester aka sebres (review, optimization) \ No newline at end of file diff --git a/config/filter.d/apache-pass.conf b/config/filter.d/apache-pass.conf index dd00f953..3cab87b0 100644 --- a/config/filter.d/apache-pass.conf +++ b/config/filter.d/apache-pass.conf @@ -3,16 +3,15 @@ # # The knocking request must have a referer. -[INCLUDES] - -before = apache-common.conf - [Definition] failregex = ^ - \w+ \[\] "GET HTTP/1\.[01]" 200 \d+ ".*" "[^-].*"$ ignoreregex = +datepattern = ^[^\[]*\[({DATE}) + {^LN-BEG} + [Init] knocking_url = /knocking/ diff --git a/config/filter.d/assp.conf b/config/filter.d/assp.conf index 278e25cb..9837f71d 100644 --- a/config/filter.d/assp.conf +++ b/config/filter.d/assp.conf @@ -8,7 +8,7 @@ # [Definition] -# Note: First three failregex matches below are for ASSP V1 with the remaining being designed for V2. Deleting the V1 regex is recommended but I left it in for compatibilty reasons. +# Note: First three failregex matches below are for ASSP V1 with the remaining being designed for V2. Deleting the V1 regex is recommended but I left it in for compatibility reasons. __assp_actions = (?:dropping|refusing) @@ -20,6 +20,9 @@ failregex = ^(:? \[SSL-out\])? max sender authentication errors \(\d{,3}\ ignoreregex = +datepattern = {^LN-BEG}%%b-%%d-%%Exy %%H:%%M:%%S + {^LN-BEG} + # DEV Notes: # V1 Examples matches: # Apr-27-13 02:33:09 Blocking 217.194.197.97 - too much AUTH errors (41); diff --git a/config/filter.d/asterisk.conf b/config/filter.d/asterisk.conf index 79bd1ff2..df55d288 100644 --- a/config/filter.d/asterisk.conf +++ b/config/filter.d/asterisk.conf @@ -31,6 +31,7 @@ failregex = ^%(__prefix_line)s%(log_prefix)s Registration from '[^']*' failed fo ignoreregex = +datepattern = {^LN-BEG} # Author: Xavier Devlamynck / Daniel Black # diff --git a/config/filter.d/common.conf b/config/filter.d/common.conf index 586f428a..a8cba188 100644 --- a/config/filter.d/common.conf +++ b/config/filter.d/common.conf @@ -61,4 +61,7 @@ __prefix_line = %(__date_ambit)s?\s*(?:%(__bsd_syslog_verbose)s\s+)?(?:%(__hostn # pam_ldap __pam_auth = pam_unix +# standardly all formats using prefix have line-begin anchored date: +datepattern = {^LN-BEG} + # Author: Yaroslav Halchenko diff --git a/config/filter.d/counter-strike.conf b/config/filter.d/counter-strike.conf index a896b5ca..294927bc 100644 --- a/config/filter.d/counter-strike.conf +++ b/config/filter.d/counter-strike.conf @@ -8,8 +8,6 @@ failregex = ^: Bad Rcon: "rcon \d+ "\S+" sv_contact ".*?"" from ":\d+"$ ignoreregex = -[Init] - datepattern = ^L %%d/%%m/%%Y - %%H:%%M:%%S diff --git a/config/filter.d/courier-auth.conf b/config/filter.d/courier-auth.conf index 1170a63a..87ee55b4 100644 --- a/config/filter.d/courier-auth.conf +++ b/config/filter.d/courier-auth.conf @@ -15,5 +15,7 @@ failregex = ^%(__prefix_line)sLOGIN FAILED, user=.*, ip=\[\]$ ignoreregex = +datepattern = {^LN-BEG} + # Author: Christoph Haas # Modified by: Cyril Jaquier diff --git a/config/filter.d/directadmin.conf b/config/filter.d/directadmin.conf index 7622e548..87c7802f 100644 --- a/config/filter.d/directadmin.conf +++ b/config/filter.d/directadmin.conf @@ -13,7 +13,6 @@ failregex = ^: \'\' \d{1,3} failed login attempt(s)?. \s* ignoreregex = -[Init] datepattern = ^%%Y:%%m:%%d-%%H:%%M:%%S # diff --git a/config/filter.d/domino-smtp.conf b/config/filter.d/domino-smtp.conf new file mode 100644 index 00000000..cdc17736 --- /dev/null +++ b/config/filter.d/domino-smtp.conf @@ -0,0 +1,47 @@ +# Fail2Ban configuration file for IBM Domino SMTP Server TASK to detect failed login attempts +# +# Author: Christian Brandlehner +# +# $Revision: 003 $ +# +# Configuration: +# Set the following Domino Server parameters in notes.ini: +# console_log_enabled=1 +# log_sessions=2 +# You also have to use a date and time format supported by fail2ban. Recommended notes.ini configuration is: +# DateOrder=DMY +# DateSeparator=- +# ClockType=24_Hour +# TimeSeparator=: +# +# Depending on your locale you might have to tweak the date and time format so fail2ban can read the log + +#[INCLUDES] +# Read common prefixes. If any customizations available -- read them from +# common.local +#before = common.conf + +[Definition] +# Option: failregex +# Notes.: regex to match the password failure messages in the logfile. The +# host must be matched by a group named "host". The tag "" can +# be used for standard IP/hostname matching and is only an alias for +# (?:::f{4,6}:)?(?P\S+) +# Values: TEXT +# +# Sample log entries (used different time formats and an extra sample with process info in front of date) +# 01-23-2009 19:54:51 SMTP Server: Authentication failed for user postmaster ; connecting host 1.2.3.4 +# [28325:00010-3735542592] 22-06-2014 09:56:12 smtp: postmaster [1.2.3.4] authentication failure using internet password +# 08-09-2014 06:14:27 smtp: postmaster [1.2.3.4] authentication failure using internet password +# 08-09-2014 06:14:27 SMTP Server: Authentication failed for user postmaster ; connecting host 1.2.3.4 + +__prefix = (?:\[[^\]]+\])?\s+ +failregex = ^%(__prefix)sSMTP Server: Authentication failed for user .*? \; connecting host $ + ^%(__prefix)ssmtp: (?:[^\[]+ )*\[\] authentication failure using internet password\s*$ +# Option: ignoreregex +# Notes.: regex to ignore. If this regex matches, the line is ignored. +# Values: TEXT +# + +ignoreregex = + diff --git a/config/filter.d/dovecot.conf b/config/filter.d/dovecot.conf index 136a3947..3c45a332 100644 --- a/config/filter.d/dovecot.conf +++ b/config/filter.d/dovecot.conf @@ -9,18 +9,19 @@ before = common.conf _daemon = (auth|dovecot(-auth)?|auth-worker) -failregex = ^%(__prefix_line)s(%(__pam_auth)s(\(dovecot:auth\))?:)?\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=(\s+user=\S*)?\s*$ - ^%(__prefix_line)s(pop3|imap)-login: (Info: )?(Aborted login|Disconnected)(: Inactivity)? \(((auth failed, \d+ attempts)( in \d+ secs)?|tried to use (disabled|disallowed) \S+ auth)\):( user=<\S*>,)?( method=\S+,)? rip=(, lip=(\d{1,3}\.){3}\d{1,3})?(, TLS( handshaking(: SSL_accept\(\) failed: error:[\dA-F]+:SSL routines:[TLS\d]+_GET_CLIENT_HELLO:unknown protocol)?)?(: Disconnected)?)?(, session=<\S+>)?\s*$ - ^%(__prefix_line)s(Info|dovecot: auth\(default\)|auth-worker\(\d+\)): pam\(\S+,\): pam_authenticate\(\) failed: (User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \(password mismatch\?\))\s*$ - ^%(__prefix_line)s(auth|auth-worker\(\d+\)): (pam|passwd-file)\(\S+,\): unknown user\s*$ - ^%(__prefix_line)s(auth|auth-worker\(\d+\)): Info: ldap\(\S*,,\S*\): invalid credentials\s*$ +failregex = ^%(__prefix_line)s(?:%(__pam_auth)s(?:\(dovecot:auth\))?:)?\s+authentication failure; logname=\S* uid=\S* euid=\S* tty=dovecot ruser=\S* rhost=(?:\s+user=\S*)?\s*$ + ^%(__prefix_line)s(?:pop3|imap)-login: (?:Info: )?(?:Aborted login|Disconnected)(?::(?: [^ \(]+)+)? \((?:auth failed, \d+ attempts( in \d+ secs)?|tried to use (disabled|disallowed) \S+ auth)\):( user=<[^>]+>,)?( method=\S+,)? rip=(?:, lip=\S+)?(?:, TLS(?: handshaking(?:: SSL_accept\(\) failed: error:[\dA-F]+:SSL routines:[TLS\d]+_GET_CLIENT_HELLO:unknown protocol)?)?(: Disconnected)?)?(, session=<\S+>)?\s*$ + ^%(__prefix_line)s(?:Info|dovecot: auth\(default\)|auth-worker\(\d+\)): pam\(\S+,\): pam_authenticate\(\) failed: (User not known to the underlying authentication module: \d+ Time\(s\)|Authentication failure \(password mismatch\?\))\s*$ + ^%(__prefix_line)s(?:auth|auth-worker\(\d+\)): (?:pam|passwd-file)\(\S+,\): unknown user\s*$ + ^%(__prefix_line)s(?:auth|auth-worker\(\d+\)): Info: ldap\(\S*,,\S*\): invalid credentials\s*$ ignoreregex = -[Init] - journalmatch = _SYSTEMD_UNIT=dovecot.service +datepattern = {^LN-BEG}TAI64N + {^LN-BEG} + # DEV Notes: # * the first regex is essentially a copy of pam-generic.conf # * Probably doesn't do dovecot sql/ldap backends properly (resolved in edit 21/03/2016) @@ -30,3 +31,4 @@ journalmatch = _SYSTEMD_UNIT=dovecot.service # Author: Martin Waschbuesch # Daniel Black (rewrote with begin and end anchors) # Martin O'Neal (added LDAP authentication failure regex) +# Sergey G. Brester aka sebres (reviewed, optimized, IPv6-compatibility) diff --git a/config/filter.d/ejabberd-auth.conf b/config/filter.d/ejabberd-auth.conf index 512fdb7c..56517489 100644 --- a/config/filter.d/ejabberd-auth.conf +++ b/config/filter.d/ejabberd-auth.conf @@ -25,8 +25,6 @@ failregex = ^=INFO REPORT==== ===\nI\(<0\.\d+\.0>:ejabberd_c2s:\d+\) : \([^)]+\ # ignoreregex = -[Init] - # "maxlines" is number of log lines to buffer for multi-line regex searches maxlines = 2 @@ -35,3 +33,8 @@ maxlines = 2 # Values: TEXT # journalmatch = + +#datepattern = ^(?:=[^=]+={3,} )?({DATE}) +# explicit time format using prefix =...==== and no date in second string begins with I(...)... +datepattern = ^(?:=[^=]+={3,} )?(%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)?) + ^I\(()** diff --git a/config/filter.d/freeswitch.conf b/config/filter.d/freeswitch.conf index 1ef5a256..4759fbed 100644 --- a/config/filter.d/freeswitch.conf +++ b/config/filter.d/freeswitch.conf @@ -8,13 +8,26 @@ # IP addresses on your LAN. # +[INCLUDES] + +# Read common prefixes. If any customizations available -- read them from +# common.local +before = common.conf + [Definition] -failregex = ^\.\d+ \[WARNING\] sofia_reg\.c:\d+ SIP auth (failure|challenge) \((REGISTER|INVITE)\) on sofia profile \'[^']+\' for \[.*\] from ip $ - ^\.\d+ \[WARNING\] sofia_reg\.c:\d+ Can't find user \[\d+@\d+\.\d+\.\d+\.\d+\] from $ +_daemon = freeswitch + +# Prefix contains common prefix line (server, daemon, etc.) and 2 datetimes if used systemd backend +_pref_line = ^%(__prefix_line)s(?:\d+-\d+-\d+ \d+:\d+:\d+\.\d+)? + +failregex = %(_pref_line)s \[WARNING\] sofia_reg\.c:\d+ SIP auth (failure|challenge) \((REGISTER|INVITE)\) on sofia profile \'[^']+\' for \[[^\]]*\] from ip $ + %(_pref_line)s \[WARNING\] sofia_reg\.c:\d+ Can't find user \[[^@]+@[^\]]+\] from $ ignoreregex = +datepattern = {^LN-BEG} + # Author: Rupa SChomaker, soapee01, Daniel Black # https://freeswitch.org/confluence/display/FREESWITCH/Fail2Ban # Thanks to Jim on mailing list of samples and guidance diff --git a/config/filter.d/guacamole.conf b/config/filter.d/guacamole.conf index 49cecc5a..09b4e7b0 100644 --- a/config/filter.d/guacamole.conf +++ b/config/filter.d/guacamole.conf @@ -17,6 +17,9 @@ failregex = ^.*\nWARNING: Authentication attempt from for user "[^"]*" fa # ignoreregex = -[Init] # "maxlines" is number of log lines to buffer for multi-line regex searches maxlines = 2 + +datepattern = ^%%b %%d, %%ExY %%I:%%M:%%S %%p + ^WARNING:()** + {^LN-BEG} \ No newline at end of file diff --git a/config/filter.d/kerio.conf b/config/filter.d/kerio.conf index 313c9b36..e0d94753 100644 --- a/config/filter.d/kerio.conf +++ b/config/filter.d/kerio.conf @@ -9,8 +9,6 @@ failregex = ^ SMTP Spam attack detected from , ignoreregex = -[Init] - datepattern = ^\[%%d/%%b/%%Y %%H:%%M:%%S\] # DEV NOTES: diff --git a/config/filter.d/mongodb-auth.conf b/config/filter.d/mongodb-auth.conf new file mode 100644 index 00000000..66c27abb --- /dev/null +++ b/config/filter.d/mongodb-auth.conf @@ -0,0 +1,49 @@ +# Fail2Ban filter for unsuccesfull MongoDB authentication attempts +# +# Logfile /var/log/mongodb/mongodb.log +# +# add setting in /etc/mongodb.conf +# logpath=/var/log/mongodb/mongodb.log +# +# and use of the authentication +# auth = true +# + +[Definition] +#failregex = ^\s+\[initandlisten\] connection accepted from :\d+ \#(?P<__connid>\d+) \(1 connection now open\)\s+\[conn(?P=__connid)\] Failed to authenticate\s+ +failregex = ^\s+\[conn(?P<__connid>\d+)\] Failed to authenticate [^\n]+\s+\[conn(?P=__connid)\] end connection + +ignoreregex = + + +[Init] +maxlines = 10 + +# DEV Notes: +# +# Regarding the multiline regex: +# +# There can be a nunber of non-related lines between the first and second part +# of this regex maxlines of 10 is quite generious. +# +# Note the capture __connid, includes the connection ID, used in second part of regex. +# +# The first regex is commented out (but will match also), because it is better to use +# the host from "end connection" line (uncommented above): +# - it has the same prefix, searching begins directly with failure message +# (so faster, because ignores success connections at all) +# - it is not so vulnerable in case of possible race condition +# +# Log example: +# 2016-10-20T09:54:27.108+0200 [initandlisten] connection accepted from 127.0.0.1:53276 #1 (1 connection now open) +# 2016-10-20T09:54:27.109+0200 [conn1] authenticate db: test { authenticate: 1, nonce: "xxx", user: "root", key: "xxx" } +# 2016-10-20T09:54:27.110+0200 [conn1] Failed to authenticate root@test with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@test +# 2016-11-09T09:54:27.894+0100 [conn1] end connection 127.0.0.1:53276 (0 connections now open) +# 2016-11-09T11:55:58.890+0100 [initandlisten] connection accepted from 127.0.0.1:54266 #1510 (1 connection now open) +# 2016-11-09T11:55:58.892+0100 [conn1510] authenticate db: admin { authenticate: 1, nonce: "xxx", user: "root", key: "xxx" } +# 2016-11-09T11:55:58.892+0100 [conn1510] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch +# 2016-11-09T11:55:58.894+0100 [conn1510] end connection 127.0.0.1:54266 (0 connections now open) +# +# Authors: Alexander Finkhäuser +# Sergey G. Brester (sebres) + diff --git a/config/filter.d/monit.conf b/config/filter.d/monit.conf index 599bd363..b652a1f4 100644 --- a/config/filter.d/monit.conf +++ b/config/filter.d/monit.conf @@ -13,7 +13,7 @@ before = common.conf _daemon = monit # Regexp for previous (accessing monit httpd) and new (access denied) versions -failregex = ^\[[A-Z]+\s+\]\s*error\s*:\s*Warning:\s+Client '' supplied (?:unknown user '[^']+'|wrong password for user '[^']*') accessing monit httpd$ +failregex = ^\[\s*\]\s*error\s*:\s*Warning:\s+Client '' supplied (?:unknown user '[^']+'|wrong password for user '[^']*') accessing monit httpd$ ^%(__prefix_line)s\w+: access denied -- client : (?:unknown user '[^']+'|wrong password for user '[^']*'|empty password)$ # Ignore login with empty user (first connect, no user specified) diff --git a/config/filter.d/murmur.conf b/config/filter.d/murmur.conf index 3775a9d2..507bbd2f 100644 --- a/config/filter.d/murmur.conf +++ b/config/filter.d/murmur.conf @@ -15,13 +15,14 @@ _daemon = murmurd # variable in your server config file (murmur.ini / mumble-server.ini). _usernameregex = [^>]+ -_prefix = [\n\s]*(\.\d{3})?\s+\d+ => <\d+:%(_usernameregex)s\(-1\)> Rejected connection from :\d+: +_prefix = \s+\d+ => <\d+:%(_usernameregex)s\(-1\)> Rejected connection from :\d+: failregex = ^%(_prefix)s Invalid server password$ ^%(_prefix)s Wrong certificate or password for existing user$ ignoreregex = +datepattern = ^{DATE} # DEV Notes: # diff --git a/config/filter.d/nginx-botsearch.conf b/config/filter.d/nginx-botsearch.conf index 6853e1e8..0be895b2 100644 --- a/config/filter.d/nginx-botsearch.conf +++ b/config/filter.d/nginx-botsearch.conf @@ -13,6 +13,9 @@ failregex = ^ \- \S+ \[\] \"(GET|POST|HEAD) \/ \S+\" 404 .+$ ignoreregex = +datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)? + ^[^\[]*\[({DATE}) + {^LN-BEG} # DEV Notes: # Based on apache-botsearch filter diff --git a/config/filter.d/nginx-http-auth.conf b/config/filter.d/nginx-http-auth.conf index a689f66a..e45a7f49 100644 --- a/config/filter.d/nginx-http-auth.conf +++ b/config/filter.d/nginx-http-auth.conf @@ -8,6 +8,8 @@ failregex = ^ \[error\] \d+#\d+: \*\d+ user "\S+":? (password mismatch|was not f ignoreregex = +datepattern = {^LN-BEG} + # DEV NOTES: # Based on samples in https://github.com/fail2ban/fail2ban/pull/43/files # Extensive search of all nginx auth failures not done yet. diff --git a/config/filter.d/nginx-limit-req.conf b/config/filter.d/nginx-limit-req.conf index 13cf75c2..c283849c 100644 --- a/config/filter.d/nginx-limit-req.conf +++ b/config/filter.d/nginx-limit-req.conf @@ -43,3 +43,4 @@ failregex = ^\s*\[error\] \d+#\d+: \*\d+ limiting requests, excess: [\d\.]+ by z ignoreregex = +datepattern = {^LN-BEG} diff --git a/config/filter.d/nsd.conf b/config/filter.d/nsd.conf index 8f32f7be..bfd99544 100644 --- a/config/filter.d/nsd.conf +++ b/config/filter.d/nsd.conf @@ -26,3 +26,6 @@ failregex = ^%(__prefix_line)sinfo: ratelimit block .* query TYPE255$ ^%(__prefix_line)sinfo: .* refused, no acl matches\.$ ignoreregex = + +datepattern = {^LN-BEG}Epoch + {^LN-BEG} \ No newline at end of file diff --git a/config/filter.d/openhab.conf b/config/filter.d/openhab.conf index 83857c7a..f6b96339 100644 --- a/config/filter.d/openhab.conf +++ b/config/filter.d/openhab.conf @@ -9,7 +9,6 @@ [Definition] failregex = ^\s+-\s+-\s+\[\]\s+"[A-Z]+ .*" 401 \d+\s*$ -[Init] datepattern = %%d/%%b[^/]*/%%Y:%%H:%%M:%%S %%z diff --git a/config/filter.d/oracleims.conf b/config/filter.d/oracleims.conf index e951ff5d..7d75c322 100644 --- a/config/filter.d/oracleims.conf +++ b/config/filter.d/oracleims.conf @@ -52,10 +52,12 @@ before = common.conf # Note that you MUST have LOG_FORMAT=4 for this to work! # -failregex = ^.*tr="[A-Z]+\|[0-9.]+\|\d+\|\|\d+" ap="[^"]*" mi="Bad password" us="[^"]*" di="535 5.7.8 Bad username or password( \(Authentication failed\))?\."/>$ +failregex = tr="[A-Z]+\|[0-9.]+\|\d+\|\|\d+" ap="[^"]*" mi="Bad password" us="[^"]*" di="535 5.7.8 Bad username or password( \(Authentication failed\))?\."/>$ # Option: ignoreregex # Notes.: regex to ignore. If this regex matches, the line is ignored. # Values: TEXT # ignoreregex = + +datepattern = ^ + +datepattern = ^[^\[]*\[({DATE}) + {^LN-BEG} diff --git a/config/filter.d/portsentry.conf b/config/filter.d/portsentry.conf index 27dca9b4..35ca2a3d 100644 --- a/config/filter.d/portsentry.conf +++ b/config/filter.d/portsentry.conf @@ -8,5 +8,8 @@ failregex = \/ Port\: [0-9]+ (TCP|UDP) Blocked$ ignoreregex = +datepattern = {^LN-BEG}Epoch + {^LN-BEG} + # Author: Pacop diff --git a/config/filter.d/selinux-common.conf b/config/filter.d/selinux-common.conf index 7269e8f7..b3e0ae4f 100644 --- a/config/filter.d/selinux-common.conf +++ b/config/filter.d/selinux-common.conf @@ -18,4 +18,6 @@ failregex = ^type=%(_type)s msg=audit\(:\d+\): (user )?pid=\d+ uid=%(_uid)s auid ignoreregex = +datepattern = EPOCH + # Author: Daniel Black diff --git a/config/filter.d/sendmail-reject.conf b/config/filter.d/sendmail-reject.conf index 93b8343c..2f8fd882 100644 --- a/config/filter.d/sendmail-reject.conf +++ b/config/filter.d/sendmail-reject.conf @@ -23,9 +23,9 @@ _daemon = (?:(sm-(mta|acceptingconnections)|sendmail)) failregex = ^%(__prefix_line)s\w{14}: ruleset=check_rcpt, arg1=(?P<\S+@\S+>), relay=(\S+ )?\[\]( \(may be forged\))?, reject=(550 5\.7\.1 (?P=email)\.\.\. Relaying denied\. (IP name possibly forged \[(\d+\.){3}\d+\]|Proper authentication required\.|IP name lookup failed \[(\d+\.){3}\d+\])|553 5\.1\.8 (?P=email)\.\.\. Domain of sender address \S+ does not exist|550 5\.[71]\.1 (?P=email)\.\.\. (Rejected: .*|User unknown))$ ^%(__prefix_line)sruleset=check_relay, arg1=(?P\S+), arg2=, relay=((?P=dom) )?\[(\d+\.){3}\d+\]( \(may be forged\))?, reject=421 4\.3\.2 (Connection rate limit exceeded\.|Too many open connections\.)$ - ^%(__prefix_line)s\w{14}: rejecting commands from (\S+ )?\[\] due to pre-greeting traffic after \d+ seconds$ + ^%(__prefix_line)s\w{14}: rejecting commands from (\S* )?\[\] due to pre-greeting traffic after \d+ seconds$ ^%(__prefix_line)s\w{14}: (\S+ )?\[\]: ((?i)expn|vrfy) \S+ \[rejected\]$ - ^(?P<__prefix>%(__prefix_line)s\w+: )<[^@]+@[^>]+>\.\.\. No such user here(?P=__prefix)from=<[^@]+@[^>]+>, size=\d+, class=\d+, nrcpts=\d+, bodytype=\w+, proto=E?SMTP, daemon=MTA, relay=\S+ \[\]$ + ^(?P<__prefix>%(__prefix_line)s\w+: )<[^@]+@[^>]+>\.\.\. No such user here$^(?P=__prefix)from=<[^@]+@[^>]+>, size=\d+, class=\d+, nrcpts=\d+, bodytype=\w+, proto=E?SMTP, daemon=MTA, relay=\S+ \[\]$ ignoreregex = diff --git a/config/filter.d/sogo-auth.conf b/config/filter.d/sogo-auth.conf index d56c94f7..48221dc0 100644 --- a/config/filter.d/sogo-auth.conf +++ b/config/filter.d/sogo-auth.conf @@ -6,7 +6,12 @@ failregex = ^ sogod \[\d+\]: SOGoRootPage Login from '' for user '.*' might not have worked( - password policy: \d* grace: -?\d* expire: -?\d* bound: -?\d*)?\s*$ -ignoreregex = +ignoreregex = "^" + +datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)? + {^LN-BEG}(?:%%a )?%%b %%d %%H:%%M:%%S(?:\.%%f)?(?: %%ExY)? + ^[^\[]*\[({DATE}) + {^LN-BEG} # # DEV Notes: diff --git a/config/filter.d/squid.conf b/config/filter.d/squid.conf index e26cab9c..58694c48 100644 --- a/config/filter.d/squid.conf +++ b/config/filter.d/squid.conf @@ -9,5 +9,8 @@ failregex = ^\s+\d\s\s+[A-Z_]+_DENIED/403 .*$ ignoreregex = +datepattern = {^LN-BEG}Epoch + {^LN-BEG} + # Author: Daniel Black diff --git a/config/filter.d/squirrelmail.conf b/config/filter.d/squirrelmail.conf index af0c38e7..31e922e6 100644 --- a/config/filter.d/squirrelmail.conf +++ b/config/filter.d/squirrelmail.conf @@ -5,8 +5,6 @@ failregex = ^ \[LOGIN_ERROR\].*from : Unknown user or password incorrect\. ignoreregex = -[Init] - datepattern = ^%%m/%%d/%%Y %%H:%%M:%%S # DEV NOTES: diff --git a/config/filter.d/sshd-aggressive.conf b/config/filter.d/sshd-aggressive.conf new file mode 100644 index 00000000..98175cbe --- /dev/null +++ b/config/filter.d/sshd-aggressive.conf @@ -0,0 +1,11 @@ +# Fail2Ban aggressive ssh filter for at attempted exploit +# +# Includes failregex of both sshd and sshd-ddos filters +# +[INCLUDES] + +before = sshd.conf + +[Definition] + +mode = %(aggressive)s diff --git a/config/filter.d/sshd-ddos.conf b/config/filter.d/sshd-ddos.conf index 4f71c7f3..69b42069 100644 --- a/config/filter.d/sshd-ddos.conf +++ b/config/filter.d/sshd-ddos.conf @@ -10,20 +10,8 @@ [INCLUDES] -# Read common prefixes. If any customizations available -- read them from -# common.local -before = common.conf +before = sshd.conf [Definition] -_daemon = sshd - -failregex = ^%(__prefix_line)sDid not receive identification string from \s*$ - -ignoreregex = - -[Init] - -journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd - -# Author: Yaroslav Halchenko +mode = %(ddos)s diff --git a/config/filter.d/sshd.conf b/config/filter.d/sshd.conf index eeb1518e..ebdc06ec 100644 --- a/config/filter.d/sshd.conf +++ b/config/filter.d/sshd.conf @@ -14,37 +14,64 @@ # common.local before = common.conf -[Definition] +[DEFAULT] _daemon = sshd -failregex = ^%(__prefix_line)s(?:error: PAM: )?[aA]uthentication (?:failure|error|failed) for .* from ( via \S+)?\s*$ - ^%(__prefix_line)s(?:error: PAM: )?User not known to the underlying authentication module for .* from \s*$ - ^%(__prefix_line)sFailed \S+ for .*? from (?: port \d*)?(?: ssh\d*)?(: (ruser .*|(\S+ ID \S+ \(serial \d+\) CA )?\S+ %(__md5hex)s(, client user ".*", client host ".*")?))?\s*$ - ^%(__prefix_line)sROOT LOGIN REFUSED.* FROM \s*$ - ^%(__prefix_line)s[iI](?:llegal|nvalid) user .* from \s*$ - ^%(__prefix_line)sUser .+ from not allowed because not listed in AllowUsers\s*$ - ^%(__prefix_line)sUser .+ from not allowed because listed in DenyUsers\s*$ - ^%(__prefix_line)sUser .+ from not allowed because not in any group\s*$ - ^%(__prefix_line)srefused connect from \S+ \(\)\s*$ - ^%(__prefix_line)s(?:error: )?Received disconnect from : 3: .*: Auth fail(?: \[preauth\])?$ - ^%(__prefix_line)sUser .+ from not allowed because a group is listed in DenyGroups\s*$ - ^%(__prefix_line)sUser .+ from not allowed because none of user's groups are listed in AllowGroups\s*$ - ^(?P<__prefix>%(__prefix_line)s)User .+ not allowed because account is locked(?P=__prefix)(?:error: )?Received disconnect from : 11: .+ \[preauth\]$ - ^(?P<__prefix>%(__prefix_line)s)Disconnecting: Too many authentication failures for .+? \[preauth\](?P=__prefix)(?:error: )?Connection closed by \[preauth\]$ - ^(?P<__prefix>%(__prefix_line)s)Connection from port \d+(?: on \S+ port \d+)?(?P=__prefix)Disconnecting: Too many authentication failures for .+? \[preauth\]$ - ^%(__prefix_line)s(error: )?maximum authentication attempts exceeded for .* from (?: port \d*)?(?: ssh\d*)? \[preauth\]$ - ^%(__prefix_line)spam_unix\(sshd:auth\):\s+authentication failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruser=\S*\s*rhost=\s.*$ +# optional prefix (logged from several ssh versions) like "error: ", "error: PAM: " or "fatal: " +__pref = (?:(?:error|fatal): (?:PAM: )?)? +# optional suffix (logged from several ssh versions) like " [preauth]" +__suff = (?: \[preauth\])?\s* +__on_port_opt = (?: port \d+)?(?: on \S+(?: port \d+)?)? + +# single line prefix: +__prefix_line_sl = %(__prefix_line)s%(__pref)s +# multi line prefixes (for first and second lines): +__prefix_line_ml1 = (?P<__prefix>%(__prefix_line)s)%(__pref)s +__prefix_line_ml2 = %(__suff)s$^(?P=__prefix)%(__pref)s + +mode = %(normal)s + +normal = ^%(__prefix_line_sl)s[aA]uthentication (?:failure|error|failed) for .* from ( via \S+)?\s*%(__suff)s$ + ^%(__prefix_line_sl)sUser not known to the underlying authentication module for .* from \s*%(__suff)s$ + ^%(__prefix_line_sl)sFailed \S+ for (?Pinvalid user )?(?P(?P\S+)|(?(cond_inv)(?:(?! from ).)*?|[^:]+)) from %(__on_port_opt)s(?: ssh\d*)?(?(cond_user): |(?:(?:(?! from ).)*)$) + ^%(__prefix_line_sl)sROOT LOGIN REFUSED.* FROM \s*%(__suff)s$ + ^%(__prefix_line_sl)s[iI](?:llegal|nvalid) user .*? from %(__on_port_opt)s\s*$ + ^%(__prefix_line_sl)sUser .+ from not allowed because not listed in AllowUsers\s*%(__suff)s$ + ^%(__prefix_line_sl)sUser .+ from not allowed because listed in DenyUsers\s*%(__suff)s$ + ^%(__prefix_line_sl)sUser .+ from not allowed because not in any group\s*%(__suff)s$ + ^%(__prefix_line_sl)srefused connect from \S+ \(\)\s*%(__suff)s$ + ^%(__prefix_line_sl)sReceived disconnect from %(__on_port_opt)s:\s*3: .*: Auth fail%(__suff)s$ + ^%(__prefix_line_sl)sUser .+ from not allowed because a group is listed in DenyGroups\s*%(__suff)s$ + ^%(__prefix_line_sl)sUser .+ from not allowed because none of user's groups are listed in AllowGroups\s*%(__suff)s$ + ^%(__prefix_line_sl)spam_unix\(sshd:auth\):\s+authentication failure;\s*logname=\S*\s*uid=\d*\s*euid=\d*\s*tty=\S*\s*ruser=\S*\s*rhost=\s.*%(__suff)s$ + ^%(__prefix_line_sl)s(error: )?maximum authentication attempts exceeded for .* from %(__on_port_opt)s(?: ssh\d*)? \[preauth\]$ + ^%(__prefix_line_ml1)sUser .+ not allowed because account is locked%(__prefix_line_ml2)sReceived disconnect from : 11: .+%(__suff)s$ + ^%(__prefix_line_ml1)sDisconnecting: Too many authentication failures for .+?%(__prefix_line_ml2)sConnection closed by %(__suff)s$ + ^%(__prefix_line_ml1)sConnection from %(__on_port_opt)s%(__prefix_line_ml2)sDisconnecting: Too many authentication failures for .+%(__suff)s$ + +ddos = ^%(__prefix_line_sl)sDid not receive identification string from %(__suff)s$ + ^%(__prefix_line_sl)sReceived disconnect from %(__on_port_opt)s:\s*14: No supported authentication methods available%(__suff)s$ + ^%(__prefix_line_sl)sUnable to negotiate with %(__on_port_opt)s: no matching (?:cipher|key exchange method) found. + ^%(__prefix_line_ml1)sConnection from %(__on_port_opt)s%(__prefix_line_ml2)sUnable to negotiate a (?:cipher|key exchange method)%(__suff)s$ + ^%(__prefix_line_ml1)sSSH: Server;Ltype: (?:Authname|Version|Kex);Remote: -\d+;[A-Z]\w+:.*%(__prefix_line_ml2)sRead from socket failed: Connection reset by peer%(__suff)s$ + +aggressive = %(normal)s + %(ddos)s + +[Definition] + +failregex = %(mode)s ignoreregex = -[Init] - # "maxlines" is number of log lines to buffer for multi-line regex searches maxlines = 10 journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd +datepattern = {^LN-BEG} + # DEV Notes: # # "Failed \S+ for .*? from ..." failregex uses non-greedy catch-all because diff --git a/config/filter.d/suhosin.conf b/config/filter.d/suhosin.conf index f125eadc..46fbe381 100644 --- a/config/filter.d/suhosin.conf +++ b/config/filter.d/suhosin.conf @@ -17,7 +17,7 @@ _daemon = (?:lighttpd|suhosin) _lighttpd_prefix = (?:\(mod_fastcgi\.c\.\d+\) FastCGI-stderr:\s) -failregex = ^%(__prefix_line)s%(_lighttpd_prefix)s?ALERT - .* \(attacker '', file '.*'(?:, line \d+)?\)$ +failregex = ^%(__prefix_line)s%(_lighttpd_prefix)s?ALERT - .*? \(attacker '', file '[^']*'(?:, line \d+)?\)$ ignoreregex = diff --git a/config/filter.d/tine20.conf b/config/filter.d/tine20.conf index 0fa6eccd..a80d89e8 100644 --- a/config/filter.d/tine20.conf +++ b/config/filter.d/tine20.conf @@ -10,6 +10,9 @@ failregex = ^[\da-f]{5,} [\da-f]{5,} (-- none --|.*?)( \d+(\.\d+)?(h|m|s|ms)){0 ignoreregex = +datepattern = ^[^-]+ -- [^-]+ -- - ({DATE}) + {^LN-BEG} + # Author: Mika (mkl) from Tine20.org forum: https://www.tine20.org/forum/viewtopic.php?f=2&t=15688&p=54766 # Editor: Daniel Black # Advisor: Lars Kneschke diff --git a/config/jail.conf b/config/jail.conf index 95b8e78b..1825ae92 100644 --- a/config/jail.conf +++ b/config/jail.conf @@ -261,6 +261,8 @@ action = %(action_)s [sshd] +# To use more aggressive sshd filter (inclusive sshd-ddos failregex): +#filter = sshd-aggressive port = ssh logpath = %(sshd_log)s backend = %(sshd_backend)s @@ -769,6 +771,13 @@ logpath = %(mysql_log)s backend = %(mysql_backend)s +# Log wrong MongoDB auth (for details see filter 'filter.d/mongodb-auth.conf') +[mongodb-auth] +# change port when running with "--shardsvr" or "--configsvr" runtime operation +port = 27017 +logpath = /var/log/mongodb/mongodb.log + + # Jail for more extended banning of persistent abusers # !!! WARNINGS !!! # 1. Make sure that your loglevel specified in fail2ban.conf/.local @@ -848,8 +857,9 @@ maxretry = 1 [pass2allow-ftp] # this pass2allow example allows FTP traffic after successful HTTP authentication port = ftp,ftp-data,ftps,ftps-data -# knocking_url variable must be overridden to some secret value in filter.d/apache-pass.local -filter = apache-pass +# knocking_url variable must be overridden to some secret value in jail.local +knocking_url = /knocking/ +filter = apache-pass[knocking_url="%(knocking_url)s"] # access log of the website with HTTP auth logpath = %(apache_access_log)s blocktype = RETURN @@ -883,3 +893,8 @@ logpath = /var/log/haproxy.log port = ldap,ldaps filter = slapd logpath = /var/log/slapd.log + +[domino-smtp] +port = smtp,ssmtp +filter = domino-smtp +logpath = /home/domino01/data/IBM_TECHNICAL_SUPPORT/console.log diff --git a/config/paths-freebsd.conf b/config/paths-freebsd.conf index fd788ab4..91b23636 100644 --- a/config/paths-freebsd.conf +++ b/config/paths-freebsd.conf @@ -34,13 +34,13 @@ auditd_log = /dev/null # http://svnweb.freebsd.org/ports/head/www/apache24/files/patch-config.layout # http://svnweb.freebsd.org/ports/head/www/apache22/files/patch-config.layout -apache_error_log = /usr/local/www/logs/*error[_.]log +apache_error_log = /var/log/httpd-error.log -apache_access_log = /usr/local/www/logs/*access[_.]log +apache_access_log = /var/log/httpd-access.log # http://svnweb.freebsd.org/ports/head/www/nginx/Makefile?view=markup -nginx_error_log = /var/log/nginx-error.log +nginx_error_log = /var/log/nginx/error.log -nginx_access_log = /var/log/nginx-access.log +nginx_access_log = /var/log/nginx/access.log diff --git a/config/paths-opensuse.conf b/config/paths-opensuse.conf index 0d6ad522..227a5e98 100644 --- a/config/paths-opensuse.conf +++ b/config/paths-opensuse.conf @@ -36,3 +36,15 @@ mysql_log = /var/log/mysql/mysqld.log roundcube_errors_log = /srv/www/roundcubemail/logs/errors solidpop3d_log = %(syslog_mail)s + +# These services will log to the journal via syslog, so use the journal by +# default. +syslog_backend = systemd +sshd_backend = systemd +dropbear_backend = systemd +proftpd_backend = systemd +pureftpd_backend = systemd +wuftpd_backend = systemd +postfix_backend = systemd +dovecot_backend = systemd +mysql_backend = systemd diff --git a/fail2ban/client/actionreader.py b/fail2ban/client/actionreader.py index e5025fa3..55ceda93 100644 --- a/fail2ban/client/actionreader.py +++ b/fail2ban/client/actionreader.py @@ -43,10 +43,15 @@ class ActionReader(DefinitionInitConfigReader): "actionrepair": ["string", None], "actionban": ["string", None], "actionunban": ["string", None], + "norestored": ["string", None], } def __init__(self, file_, jailName, initOpts, **kwargs): - self._name = initOpts.get("actname", file_) + actname = initOpts.get("actname") + if actname is None: + actname = file_ + initOpts["actname"] = actname + self._name = actname DefinitionInitConfigReader.__init__( self, file_, jailName, initOpts, **kwargs) @@ -64,16 +69,22 @@ class ActionReader(DefinitionInitConfigReader): return self._name def convert(self): + opts = self.getCombined(ignore=('timeout', 'bantime')) + # type-convert only after combined (otherwise boolean converting prevents substitution): + if opts.get('norestored'): + opts['norestored'] = self._convert_to_boolean(opts['norestored']) + # stream-convert: head = ["set", self._jailName] stream = list() stream.append(head + ["addaction", self._name]) multi = [] - for opt, optval in self._opts.iteritems(): + for opt, optval in opts.iteritems(): if opt in self._configOpts: multi.append([opt, optval]) if self._initOpts: for opt, optval in self._initOpts.iteritems(): - multi.append([opt, optval]) + if opt not in self._configOpts: + multi.append([opt, optval]) if len(multi) > 1: stream.append(["multi-set", self._jailName, "action", self._name, multi]) elif len(multi): diff --git a/fail2ban/client/configparserinc.py b/fail2ban/client/configparserinc.py index d6cda7f7..35fa7498 100644 --- a/fail2ban/client/configparserinc.py +++ b/fail2ban/client/configparserinc.py @@ -29,7 +29,7 @@ import re import sys from ..helpers import getLogger -if sys.version_info >= (3,2): # pragma: no cover +if sys.version_info >= (3,2): # SafeConfigParser deprecated from Python 3.2 (renamed to ConfigParser) from configparser import ConfigParser as SafeConfigParser, \ diff --git a/fail2ban/client/configreader.py b/fail2ban/client/configreader.py index 643cdf3a..7840cd12 100644 --- a/fail2ban/client/configreader.py +++ b/fail2ban/client/configreader.py @@ -28,13 +28,26 @@ import glob import os from ConfigParser import NoOptionError, NoSectionError -from .configparserinc import SafeConfigParserWithIncludes, logLevel +from .configparserinc import sys, SafeConfigParserWithIncludes, logLevel from ..helpers import getLogger +from ..server.action import CommandAction # Gets the instance of the logger. logSys = getLogger(__name__) +# if sys.version_info >= (3,5): +# def _merge_dicts(x, y): +# return {**x, **y} +# else: +def _merge_dicts(x, y): + r = x + if y: + r = x.copy() + r.update(y) + return r + + class ConfigReader(): """Generic config reader class. @@ -127,9 +140,9 @@ class ConfigReader(): return self._cfg.options(*args) return {} - def get(self, sec, opt): + def get(self, sec, opt, raw=False, vars={}): if self._cfg is not None: - return self._cfg.get(sec, opt) + return self._cfg.get(sec, opt, raw=raw, vars=vars) return None def getOptions(self, *args, **kwargs): @@ -210,6 +223,8 @@ class ConfigReaderUnshared(SafeConfigParserWithIncludes): def getOptions(self, sec, options, pOptions=None, shouldExist=False): values = dict() + if pOptions is None: + pOptions = {} for optname in options: if isinstance(options, (list,tuple)): if len(optname) > 2: @@ -218,15 +233,15 @@ class ConfigReaderUnshared(SafeConfigParserWithIncludes): (opttype, optname), optvalue = optname, None else: opttype, optvalue = options[optname] + if optname in pOptions: + continue try: if opttype == "bool": v = self.getboolean(sec, optname) elif opttype == "int": v = self.getint(sec, optname) else: - v = self.get(sec, optname) - if not pOptions is None and optname in pOptions: - continue + v = self.get(sec, optname, vars=pOptions) values[optname] = v except NoSectionError as e: if shouldExist: @@ -289,6 +304,12 @@ class DefinitionInitConfigReader(ConfigReader): return SafeConfigParserWithIncludes.read(self._cfg, self._file) def getOptions(self, pOpts): + # overwrite static definition options with init values, supplied as + # direct parameters from jail-config via action[xtra1="...", xtra2=...]: + if self._initOpts: + if not pOpts: + pOpts = dict() + pOpts = _merge_dicts(pOpts, self._initOpts) self._opts = ConfigReader.getOptions( self, "Definition", self._configOpts, pOpts) @@ -299,6 +320,28 @@ class DefinitionInitConfigReader(ConfigReader): self._initOpts['known/'+opt] = v if not opt in self._initOpts: self._initOpts[opt] = v + + def _convert_to_boolean(self, value): + return value.lower() in ("1", "yes", "true", "on") + + def getCombined(self, ignore=()): + combinedopts = self._opts + ignore = set(ignore).copy() + if self._initOpts: + combinedopts = _merge_dicts(self._opts, self._initOpts) + if not len(combinedopts): + return {} + # ignore conditional options: + for n in combinedopts: + cond = SafeConfigParserWithIncludes.CONDITIONAL_RE.match(n) + if cond: + n, cond = cond.groups() + ignore.add(n) + # substiture options already specified direct: + opts = CommandAction.substituteRecursiveTags(combinedopts, ignore=ignore) + if not opts: + raise ValueError('recursive tag definitions unable to be resolved') + return opts def convert(self): raise NotImplementedError diff --git a/fail2ban/client/configurator.py b/fail2ban/client/configurator.py index 4d28238f..e8472ac1 100644 --- a/fail2ban/client/configurator.py +++ b/fail2ban/client/configurator.py @@ -72,9 +72,9 @@ class Configurator: def getEarlyOptions(self): return self.__fail2ban.getEarlyOptions() - def getOptions(self, jail=None, updateMainOpt=None): + def getOptions(self, jail=None, updateMainOpt=None, ignoreWrong=True): self.__fail2ban.getOptions(updateMainOpt) - return self.__jails.getOptions(jail) + return self.__jails.getOptions(jail, ignoreWrong=ignoreWrong) def convertToProtocol(self): self.__streams["general"] = self.__fail2ban.convert() diff --git a/fail2ban/client/fail2banclient.py b/fail2ban/client/fail2banclient.py index 007afd57..0a1ae4f1 100755 --- a/fail2ban/client/fail2banclient.py +++ b/fail2ban/client/fail2banclient.py @@ -125,7 +125,7 @@ class Fail2banClient(Fail2banCmdLine, Thread): if client: try : client.close() - except Exception as e: + except Exception as e: # pragma: no cover if showRet or self._conf["verbose"] > 1: logSys.debug(e) if showRet or c[0] == 'echo': diff --git a/fail2ban/client/fail2bancmdline.py b/fail2ban/client/fail2bancmdline.py index 74236ab1..a9387bc6 100644 --- a/fail2ban/client/fail2bancmdline.py +++ b/fail2ban/client/fail2bancmdline.py @@ -47,6 +47,7 @@ class Fail2banCmdLine(): def __init__(self): self._argv = self._args = None self._configurator = None + self.cleanConfOnly = False self.resetConf() def resetConf(self): @@ -101,6 +102,7 @@ class Fail2banCmdLine(): output(" --logtarget |STDOUT|STDERR|SYSLOG") output(" --syslogsocket auto|") output(" -d dump configuration. For debugging") + output(" -t, --test test configuration (can be also specified with start parameters)") output(" -i interactive mode") output(" -v increase verbosity") output(" -q decrease verbosity") @@ -136,6 +138,9 @@ class Fail2banCmdLine(): self._conf[ o[2:] ] = opt[1] elif o == "-d": self._conf["dump"] = True + elif o == "-t" or o == "--test": + self.cleanConfOnly = True + self._conf["test"] = True elif o == "-v": self._conf["verbose"] += 1 elif o == "-q": @@ -173,8 +178,8 @@ class Fail2banCmdLine(): # Reads the command line options. try: - cmdOpts = 'hc:s:p:xfbdviqV' - cmdLongOpts = ['loglevel=', 'logtarget=', 'syslogsocket=', 'async', 'timeout=', 'help', 'version'] + cmdOpts = 'hc:s:p:xfbdtviqV' + cmdLongOpts = ['loglevel=', 'logtarget=', 'syslogsocket=', 'test', 'async', 'timeout=', 'help', 'version'] optList, self._args = getopt.getopt(self._argv[1:], cmdOpts, cmdLongOpts) except getopt.GetoptError: self.dispUsage() @@ -225,13 +230,30 @@ class Fail2banCmdLine(): logSys.info("Using pid file %s, [%s] logging to %s", self._conf["pidfile"], logging.getLevelName(llev), self._conf["logtarget"]) + readcfg = True if self._conf.get("dump", False): - ret, stream = self.readConfig() + if readcfg: + ret, stream = self.readConfig() + readcfg = False self.dumpConfig(stream) - return ret + if not self._conf.get("test", False): + return ret + + if self._conf.get("test", False): + if readcfg: + readcfg = False + ret, stream = self.readConfig() + if not ret: + raise ServerExecutionException("ERROR: test configuration failed") + # exit after test if no commands specified (test only): + if not len(self._args): + output("OK: configuration test is successful") + return ret # Nothing to do here, process in client/server return None + except ServerExecutionException: + raise except Exception as e: output("ERROR: %s" % (e,)) if verbose > 2: @@ -246,7 +268,8 @@ class Fail2banCmdLine(): try: self.configurator.Reload() self.configurator.readAll() - ret = self.configurator.getOptions(jail, self._conf) + ret = self.configurator.getOptions(jail, self._conf, + ignoreWrong=not self.cleanConfOnly) self.configurator.convertToProtocol() stream = self.configurator.getConfigStream() except Exception as e: @@ -274,6 +297,7 @@ class Fail2banCmdLine(): def exit(code=0): logSys.debug("Exit with code %s", code) # because of possible buffered output in python, we should flush it before exit: + logging.shutdown() sys.stdout.flush() sys.stderr.flush() # exit diff --git a/fail2ban/client/fail2banregex.py b/fail2ban/client/fail2banregex.py index 4096ac4e..d111e09c 100644 --- a/fail2ban/client/fail2banregex.py +++ b/fail2ban/client/fail2banregex.py @@ -41,12 +41,12 @@ from optparse import OptionParser, Option from ConfigParser import NoOptionError, NoSectionError, MissingSectionHeaderError try: # pragma: no cover - from systemd import journal from ..server.filtersystemd import FilterSystemd except ImportError: - journal = None + FilterSystemd = None from ..version import version +from .jailreader import JailReader from .filterreader import FilterReader from ..server.filter import Filter, FileContainer from ..server.failregex import RegexException @@ -80,7 +80,7 @@ def pprint_list(l, header=None): s = '' output( s + "| " + "\n| ".join(l) + '\n`-' ) -def journal_lines_gen(myjournal): # pragma: no cover +def journal_lines_gen(flt, myjournal): # pragma: no cover while True: try: entry = myjournal.get_next() @@ -88,7 +88,7 @@ def journal_lines_gen(myjournal): # pragma: no cover continue if not entry: break - yield FilterSystemd.formatJournalEntry(entry) + yield flt.formatJournalEntry(entry) def get_opt_parser(): # use module docstring for help output @@ -122,15 +122,15 @@ Report bugs to https://github.com/fail2ban/fail2ban/issues p.add_options([ Option("-d", "--datepattern", help="set custom pattern used to match date/times"), - Option("-e", "--encoding", + Option("-e", "--encoding", default=PREFER_ENC, help="File encoding. Default: system locale"), - Option("-r", "--raw", action='store_true', + Option("-r", "--raw", action='store_true', default=False, help="Raw hosts, don't resolve dns"), Option("--usedns", action='store', default=None, help="DNS specified replacement of tags in regexp " "('yes' - matches all form of hosts, 'no' - IP addresses only)"), Option("-L", "--maxlines", type=int, default=0, - help="maxlines for multi-line regex"), + help="maxlines for multi-line regex."), Option("-m", "--journalmatch", help="journalctl style matches overriding filter file. " "\"systemd-journal\" only"), @@ -143,6 +143,8 @@ Report bugs to https://github.com/fail2ban/fail2ban/issues help="Increase verbosity"), Option("--verbosity", action="store", dest="verbose", type=int, help="Set numerical level of verbosity (0..4)"), + Option("--verbose-date", "--VD", action='store_true', + help="Verbose date patterns/regex in output"), Option("-D", "--debuggex", action='store_true', help="Produce debuggex.com urls for debugging there"), Option("--print-no-missed", action='store_true', @@ -215,14 +217,8 @@ class LineStats(object): class Fail2banRegex(object): def __init__(self, opts): - self._verbose = opts.verbose - self._debuggex = opts.debuggex - self._maxlines = 20 - self._print_no_missed = opts.print_no_missed - self._print_no_ignored = opts.print_no_ignored - self._print_all_matched = opts.print_all_matched - self._print_all_missed = opts.print_all_missed - self._print_all_ignored = opts.print_all_ignored + # set local protected memebers from given options: + self.__dict__.update(dict(('_'+o,v) for o,v in opts.__dict__.iteritems())) self._maxlines_set = False # so we allow to override maxlines in cmdline self._datepattern_set = False self._journalmatch = None @@ -236,23 +232,23 @@ class Fail2banRegex(object): if opts.maxlines: self.setMaxLines(opts.maxlines) + else: + self._maxlines = 20 if opts.journalmatch is not None: self.setJournalMatch(opts.journalmatch.split()) if opts.datepattern: self.setDatePattern(opts.datepattern) - if opts.encoding: - self.encoding = opts.encoding - else: - self.encoding = PREFER_ENC - self.raw = True if opts.raw else False if opts.usedns: self._filter.setUseDns(opts.usedns) + self._filter.returnRawHost = opts.raw + self._filter.checkFindTime = False + self._filter.checkAllRegex = True def decode_line(self, line): - return FileContainer.decode_line('', self.encoding, line) + return FileContainer.decode_line('', self._encoding, line) def encode_line(self, line): - return line.encode(self.encoding, 'ignore') + return line.encode(self._encoding, 'ignore') def setDatePattern(self, pattern): if not self._datepattern_set: @@ -350,7 +346,8 @@ class Fail2banRegex(object): orgLineBuffer = self._filter._Filter__lineBuffer fullBuffer = len(orgLineBuffer) >= self._filter.getMaxLines() try: - line, ret = self._filter.processLine(line, date, checkAllRegex=True, returnRawHost=self.raw) + ret = self._filter.processLine(line, date) + line = self._filter.processedLine() for match in ret: # Append True/False flag depending if line was matched by # more than one regex @@ -479,8 +476,12 @@ class Fail2banRegex(object): out = [] for template in self._filter.dateDetector.templates: if self._verbose or template.hits: - out.append("[%d] %s" % ( - template.hits, template.name)) + out.append("[%d] %s" % (template.hits, template.name)) + if self._verbose_date: + out.append(" # weight: %.3f (%.3f), pattern: %s" % ( + template.weight, template.template.weight, + getattr(template, 'pattern', ''),)) + out.append(" # regex: %s" % (getattr(template, 'regex', ''),)) pprint_list(out, "[# of hits] date format") output( "\nLines: %s" % self._line_stats, ) @@ -518,30 +519,27 @@ class Fail2banRegex(object): try: hdlr = open(cmd_log, 'rb') output( "Use log file : %s" % cmd_log ) - output( "Use encoding : %s" % self.encoding ) + output( "Use encoding : %s" % self._encoding ) test_lines = self.file_lines_gen(hdlr) except IOError as e: output( e ) return False - elif cmd_log == "systemd-journal": # pragma: no cover - if not journal: + elif cmd_log.startswith("systemd-journal"): # pragma: no cover + if not FilterSystemd: output( "Error: systemd library not found. Exiting..." ) return False - myjournal = journal.Reader(converters={'__CURSOR': lambda x: x}) + output( "Use systemd journal" ) + output( "Use encoding : %s" % self._encoding ) + backend, beArgs = JailReader.extractOptions(cmd_log) + flt = FilterSystemd(None, **beArgs) + flt.setLogEncoding(self._encoding) + myjournal = flt.getJournalReader() journalmatch = self._journalmatch self.setDatePattern(None) if journalmatch: - try: - for element in journalmatch: - if element == "+": - myjournal.add_disjunction() - else: - myjournal.add_match(element) - except ValueError: - output( "Error: Invalid journalmatch: %s" % shortstr(" ".join(journalmatch)) ) - return False + flt.addJournalMatch(journalmatch) output( "Use journal match : %s" % " ".join(journalmatch) ) - test_lines = journal_lines_gen(myjournal) + test_lines = journal_lines_gen(flt, myjournal) else: output( "Use single line : %s" % shortstr(cmd_log) ) test_lines = [ cmd_log ] diff --git a/fail2ban/client/fail2banserver.py b/fail2ban/client/fail2banserver.py index dfee34d2..006a02cf 100644 --- a/fail2ban/client/fail2banserver.py +++ b/fail2ban/client/fail2banserver.py @@ -144,27 +144,27 @@ class Fail2banServer(Fail2banCmdLine): return cli def start(self, argv): - # Command line options - ret = self.initCmdLine(argv) - if ret is not None: - return ret - - # Commands - args = self._args - - cli = None - # Just start: - if len(args) == 1 and args[0] == 'start' and not self._conf.get("interactive", False): - pass - else: - # If client mode - whole processing over client: - if len(args) or self._conf.get("interactive", False): - cli = self._Fail2banClient() - return cli.start(argv) - - # Start the server: server = None try: + # Command line options + ret = self.initCmdLine(argv) + if ret is not None: + return ret + + # Commands + args = self._args + + cli = None + # Just start: + if len(args) == 1 and args[0] == 'start' and not self._conf.get("interactive", False): + pass + else: + # If client mode - whole processing over client: + if len(args) or self._conf.get("interactive", False): + cli = self._Fail2banClient() + return cli.start(argv) + + # Start the server: from ..server.utils import Utils # background = True, if should be new process running in background, otherwise start in foreground # process will be forked in daemonize, inside of Server module. diff --git a/fail2ban/client/filterreader.py b/fail2ban/client/filterreader.py index 8b30f914..d89ef5ad 100644 --- a/fail2ban/client/filterreader.py +++ b/fail2ban/client/filterreader.py @@ -28,7 +28,6 @@ import os import shlex from .configreader import DefinitionInitConfigReader -from ..server.action import CommandAction from ..helpers import getLogger # Gets the instance of the logger. @@ -40,6 +39,9 @@ class FilterReader(DefinitionInitConfigReader): _configOpts = { "ignoreregex": ["string", None], "failregex": ["string", ""], + "maxlines": ["int", None], + "datepattern": ["string", None], + "journalmatch": ["string", None], } def setFile(self, fileName): @@ -49,15 +51,6 @@ class FilterReader(DefinitionInitConfigReader): def getFile(self): return self.__file - def getCombined(self): - combinedopts = dict(list(self._opts.items()) + list(self._initOpts.items())) - if not len(combinedopts): - return {} - opts = CommandAction.substituteRecursiveTags(combinedopts) - if not opts: - raise ValueError('recursive tag definitions unable to be resolved') - return opts - def convert(self): stream = list() opts = self.getCombined() @@ -65,6 +58,7 @@ class FilterReader(DefinitionInitConfigReader): return stream for opt, value in opts.iteritems(): if opt in ("failregex", "ignoreregex"): + if value is None: continue multi = [] for regex in value.split('\n'): # Do not send a command if the rule is empty. @@ -74,16 +68,17 @@ class FilterReader(DefinitionInitConfigReader): stream.append(["multi-set", self._jailName, "add" + opt, multi]) elif len(multi): stream.append(["set", self._jailName, "add" + opt, multi[0]]) - if self._initOpts: - if 'maxlines' in self._initOpts: + elif opt == 'maxlines': # We warn when multiline regex is used without maxlines > 1 # therefore keep sure we set this option first. - stream.insert(0, ["set", self._jailName, "maxlines", self._initOpts["maxlines"]]) - if 'datepattern' in self._initOpts: - stream.append(["set", self._jailName, "datepattern", self._initOpts["datepattern"]]) + stream.insert(0, ["set", self._jailName, "maxlines", value]) + elif opt == 'datepattern': + stream.append(["set", self._jailName, "datepattern", value]) # Do not send a command if the match is empty. - if self._initOpts.get("journalmatch", '') != '': - for match in self._initOpts["journalmatch"].split("\n"): + elif opt == 'journalmatch': + if value is None: continue + for match in value.split("\n"): + if match == '': continue stream.append( ["set", self._jailName, "addjournalmatch"] + shlex.split(match)) diff --git a/fail2ban/client/jailreader.py b/fail2ban/client/jailreader.py index 2724782f..7fdcc162 100644 --- a/fail2ban/client/jailreader.py +++ b/fail2ban/client/jailreader.py @@ -43,13 +43,13 @@ logSys = getLogger(__name__) class JailReader(ConfigReader): # regex, to extract list of options: - optionCRE = re.compile("^((?:\w|-|_|\.)+)(?:\[(.*)\])?$") + optionCRE = re.compile(r"^([\w\-_\.]+)(?:\[(.*)\])?\s*$", re.DOTALL) # regex, to iterate over single option in option list, syntax: # `action = act[p1="...", p2='...', p3=...]`, where the p3=... not contains `,` or ']' # since v0.10 separator extended with `]\s*[` for support of multiple option groups, syntax # `action = act[p1=...][p2=...]` optionExtractRE = re.compile( - r'([\w\-_\.]+)=(?:"([^"]*)"|\'([^\']*)\'|([^,\]]*))(?:,|\]\s*\[|$)') + r'([\w\-_\.]+)=(?:"([^"]*)"|\'([^\']*)\'|([^,\]]*))(?:,|\]\s*\[|$)', re.DOTALL) def __init__(self, name, force_enable=False, **kwargs): ConfigReader.__init__(self, **kwargs) @@ -119,39 +119,46 @@ class JailReader(ConfigReader): ["string", "ignorecommand", None], ["string", "ignoreip", None], ["string", "filter", ""], + ["string", "datepattern", None], ["string", "action", ""]] # Before interpolation (substitution) add static options always available as default: defsec = self._cfg.get_defaults() defsec["fail2ban_version"] = version - # Read first options only needed for merge defaults ('known/...' from filter): - self.__opts = ConfigReader.getOptions(self, self.__name, opts1st, shouldExist=True) - if not self.__opts: - return False + try: + + # Read first options only needed for merge defaults ('known/...' from filter): + self.__opts = ConfigReader.getOptions(self, self.__name, opts1st, shouldExist=True) + if not self.__opts: # pragma: no cover + raise JailDefError("Init jail options failed") - if self.isEnabled(): + if not self.isEnabled(): + return True + # Read filter - if self.__opts["filter"]: - filterName, filterOpt = JailReader.extractOptions( - self.__opts["filter"]) + flt = self.__opts["filter"] + if flt: + filterName, filterOpt = JailReader.extractOptions(flt) + if not filterName: + raise JailDefError("Invalid filter definition %r" % flt) self.__filter = FilterReader( - filterName, self.__name, filterOpt, share_config=self.share_config, basedir=self.getBaseDir()) + filterName, self.__name, filterOpt, + share_config=self.share_config, basedir=self.getBaseDir()) ret = self.__filter.read() # merge options from filter as 'known/...': self.__filter.getOptions(self.__opts) ConfigReader.merge_section(self, self.__name, self.__filter.getCombined(), 'known/') if not ret: - logSys.error("Unable to read the filter") - return False + raise JailDefError("Unable to read the filter %r" % filterName) else: self.__filter = None logSys.warning("No filter set for jail %s" % self.__name) # Read second all options (so variables like %(known/param) can be interpolated): self.__opts = ConfigReader.getOptions(self, self.__name, opts) - if not self.__opts: - return False + if not self.__opts: # pragma: no cover + raise JailDefError("Read jail options failed") # cumulate filter options again (ignore given in jail): if self.__filter: @@ -163,6 +170,8 @@ class JailReader(ConfigReader): if not act: # skip empty actions continue actName, actOpt = JailReader.extractOptions(act) + if not actName: + raise JailDefError("Invalid action definition %r" % act) if actName.endswith(".py"): self.__actions.append([ "set", @@ -182,13 +191,22 @@ class JailReader(ConfigReader): action.getOptions(self.__opts) self.__actions.append(action) else: - raise AttributeError("Unable to read action") + raise JailDefError("Unable to read action %r" % actName) + except JailDefError: + raise except Exception as e: - logSys.error("Error in action definition " + act) - logSys.debug("Caught exception: %s" % (e,)) - return False + logSys.debug("Caught exception: %s", e, exc_info=True) + raise ValueError("Error in action definition %r: %r" % (act, e)) if not len(self.__actions): logSys.warning("No actions were defined for %s" % self.__name) + + except JailDefError as e: + e = str(e) + logSys.error(e) + if not self.__opts: + self.__opts = dict() + self.__opts['config-error'] = e + return False return True def convert(self, allow_no_files=False): @@ -202,6 +220,12 @@ class JailReader(ConfigReader): """ stream = [] + e = self.__opts.get('config-error') + if e: + stream.extend([['config-error', "Jail '%s' skipped, because of wrong configuration: %s" % (self.__name, e)]]) + return stream + if self.__filter: + stream.extend(self.__filter.convert()) for opt, value in self.__opts.iteritems(): if opt == "logpath" and \ not self.__opts.get('backend', None).startswith("systemd"): @@ -223,19 +247,9 @@ class JailReader(ConfigReader): stream.append(["set", self.__name, "logencoding", value]) elif opt == "backend": backend = value - elif opt == "maxretry": - stream.append(["set", self.__name, "maxretry", value]) elif opt == "ignoreip": for ip in splitwords(value): stream.append(["set", self.__name, "addignoreip", ip]) - elif opt == "findtime": - stream.append(["set", self.__name, "findtime", value]) - elif opt == "bantime": - stream.append(["set", self.__name, "bantime", value]) - elif opt.startswith("bantime."): - stream.append(["set", self.__name, opt, self.__opts[opt]]) - elif opt == "usedns": - stream.append(["set", self.__name, "usedns", value]) elif opt in ("failregex", "ignoreregex"): multi = [] for regex in value.split('\n'): @@ -246,10 +260,8 @@ class JailReader(ConfigReader): stream.append(["multi-set", self.__name, "add" + opt, multi]) elif len(multi): stream.append(["set", self.__name, "add" + opt, multi[0]]) - elif opt == "ignorecommand": - stream.append(["set", self.__name, "ignorecommand", value]) - if self.__filter: - stream.extend(self.__filter.convert()) + elif opt not in ('action', 'filter', 'enabled'): + stream.append(["set", self.__name, opt, value]) for action in self.__actions: if isinstance(action, (ConfigReaderUnshared, ConfigReader)): stream.extend(action.convert()) @@ -273,3 +285,7 @@ class JailReader(ConfigReader): val for val in optmatch.group(2,3,4) if val is not None][0] option_opts[opt.strip()] = value.strip() return option_name, option_opts + + +class JailDefError(Exception): + pass diff --git a/fail2ban/client/jailsreader.py b/fail2ban/client/jailsreader.py index 09725ec9..cd3409b4 100644 --- a/fail2ban/client/jailsreader.py +++ b/fail2ban/client/jailsreader.py @@ -54,7 +54,7 @@ class JailsReader(ConfigReader): self.__jails = list() return ConfigReader.read(self, "jail") - def getOptions(self, section=None): + def getOptions(self, section=None, ignoreWrong=True): """Reads configuration for jail(s) and adds enabled jails to __jails """ opts = [] @@ -66,7 +66,7 @@ class JailsReader(ConfigReader): sections = [ section ] # Get the options of all jails. - parse_status = True + parse_status = 0 for sec in sections: if sec == 'INCLUDES': continue @@ -77,12 +77,16 @@ class JailsReader(ConfigReader): ret = jail.getOptions() if ret: if jail.isEnabled(): + # at least one jail was successful: + parse_status |= 1 # We only add enabled jails self.__jails.append(jail) else: - logSys.error("Errors in jail %r. Skipping..." % sec) - parse_status = False - return parse_status + logSys.error("Errors in jail %r.%s", sec, " Skipping..." if ignoreWrong else "") + self.__jails.append(jail) + # at least one jail was invalid: + parse_status |= 2 + return ((ignoreWrong and parse_status & 1) or not (parse_status & 2)) def convert(self, allow_no_files=False): """Convert read before __opts and jails to the commands stream @@ -95,15 +99,13 @@ class JailsReader(ConfigReader): """ stream = list() - for opt in self.__opts: - if opt == "": - stream.append([]) # Convert jails for jail in self.__jails: stream.extend(jail.convert(allow_no_files=allow_no_files)) # Start jails for jail in self.__jails: - stream.append(["start", jail.getName()]) + if not jail.options.get('config-error'): + stream.append(["start", jail.getName()]) return stream diff --git a/fail2ban/server/action.py b/fail2ban/server/action.py index 23b60a1b..e54d296e 100644 --- a/fail2ban/server/action.py +++ b/fail2ban/server/action.py @@ -219,9 +219,9 @@ class CommandAction(ActionBase): self.timeout = 60 ## Command executed in order to initialize the system. self.actionstart = '' - ## Command executed when an IP address gets banned. + ## Command executed when ticket gets banned. self.actionban = '' - ## Command executed when an IP address gets removed. + ## Command executed when ticket gets removed. self.actionunban = '' ## Command executed in order to check requirements. self.actioncheck = '' @@ -365,7 +365,7 @@ class CommandAction(ActionBase): return self._executeOperation('', 'reloading') @classmethod - def substituteRecursiveTags(cls, inptags, conditional=''): + def substituteRecursiveTags(cls, inptags, conditional='', ignore=()): """Sort out tag definitions within other tags. Since v.0.9.2 supports embedded interpolation (see test cases for examples). @@ -387,21 +387,26 @@ class CommandAction(ActionBase): # copy return tags dict to prevent modifying of inptags: tags = inptags.copy() t = TAG_CRE + ignore = set(ignore) + done = cls._escapedTags.copy() | ignore # repeat substitution while embedded-recursive (repFlag is True) - done = cls._escapedTags.copy() while True: repFlag = False # substitute each value: for tag in tags.iterkeys(): - # ignore escaped or already done: + # ignore escaped or already done (or in ignore list): if tag in done: continue - value = str(tags[tag]) + value = orgval = str(tags[tag]) # search and replace all tags within value, that can be interpolated using other tags: m = t.search(value) refCounts = {} #logSys.log(5, 'TAG: %s, value: %s' % (tag, value)) while m: found_tag = m.group(1) + # don't replace tags that should be currently ignored (pre-replacement): + if found_tag in ignore: + m = t.search(value, m.end()) + continue #logSys.log(5, 'found: %s' % found_tag) if found_tag == tag or refCounts.get(found_tag, 1) > MAX_TAG_REPLACE_COUNT: # recursive definitions are bad @@ -429,7 +434,7 @@ class CommandAction(ActionBase): m = t.search(value, m.start()) #logSys.log(5, 'TAG: %s, newvalue: %s' % (tag, value)) # was substituted? - if tags[tag] != value: + if orgval != value: # check still contains any tag - should be repeated (possible embedded-recursive substitution): if t.search(value): repFlag = True @@ -584,7 +589,7 @@ class CommandAction(ActionBase): return self.executeCmd(realCmd, self.timeout) @staticmethod - def executeCmd(realCmd, timeout=60): + def executeCmd(realCmd, timeout=60, **kwargs): """Executes a command. Parameters @@ -612,4 +617,4 @@ class CommandAction(ActionBase): return True with _cmd_lock: - return Utils.executeCmd(realCmd, timeout, shell=True, output=False) + return Utils.executeCmd(realCmd, timeout, shell=True, output=False, **kwargs) diff --git a/fail2ban/server/actions.py b/fail2ban/server/actions.py index bda94789..1ee648df 100644 --- a/fail2ban/server/actions.py +++ b/fail2ban/server/actions.py @@ -194,7 +194,7 @@ class Actions(JailThread, Mapping): def setBanTime(self, value): value = MyTime.str2seconds(value) self.__banManager.setBanTime(value) - logSys.info("Set banTime = %s" % value) + logSys.info(" banTime: %s" % value) ## # Get the ban time. @@ -357,6 +357,8 @@ class Actions(JailThread, Mapping): aInfo["failures"] = bTicket.getAttempt() aInfo["time"] = bTicket.getTime() aInfo["matches"] = "\n".join(bTicket.getMatches()) + # to bypass actions, that should not be executed for restored tickets + aInfo["restored"] = 1 if ticket.restored else 0 # retarded merge info via twice lambdas : once for merge, once for matches/failures: if self._jail.database is not None: mi4ip = lambda overalljails=False, self=self, \ @@ -375,6 +377,8 @@ class Actions(JailThread, Mapping): # do actions : for name, action in self._actions.iteritems(): try: + if ticket.restored and getattr(action, 'norestored', False): + continue action.ban(aInfo.copy()) except Exception as e: logSys.error( @@ -466,10 +470,14 @@ class Actions(JailThread, Mapping): aInfo["failures"] = ticket.getAttempt() aInfo["time"] = ticket.getTime() aInfo["matches"] = "".join(ticket.getMatches()) + # to bypass actions, that should not be executed for restored tickets + aInfo["restored"] = 1 if ticket.restored else 0 if actions is None: logSys.notice("[%s] Unban %s", self._jail.name, aInfo["ip"]) for name, action in unbactions.iteritems(): try: + if ticket.restored and getattr(action, 'norestored', False): + continue logSys.debug("[%s] action %r: unban %s", self._jail.name, name, aInfo["ip"]) action.unban(aInfo.copy()) except Exception as e: diff --git a/fail2ban/server/asyncserver.py b/fail2ban/server/asyncserver.py index d1818d7a..9cc74658 100644 --- a/fail2ban/server/asyncserver.py +++ b/fail2ban/server/asyncserver.py @@ -241,7 +241,7 @@ class AsyncServer(asyncore.dispatcher): def _remove_sock(self): try: os.remove(self.__sock) - except OSError as e: + except OSError as e: # pragma: no cover if e.errno != errno.ENOENT: raise diff --git a/fail2ban/server/datedetector.py b/fail2ban/server/datedetector.py index 5281fc59..cccbf71a 100644 --- a/fail2ban/server/datedetector.py +++ b/fail2ban/server/datedetector.py @@ -21,11 +21,13 @@ __author__ = "Cyril Jaquier and Fail2Ban Contributors" __copyright__ = "Copyright (c) 2004 Cyril Jaquier" __license__ = "GPL" +import copy import time from threading import Lock -from .datetemplate import DatePatternRegex, DateTai64n, DateEpoch +from .datetemplate import re, DateTemplate, DatePatternRegex, DateTai64n, DateEpoch +from .utils import Utils from ..helpers import getLogger # Gets the instance of the logger. @@ -33,8 +35,52 @@ logSys = getLogger(__name__) logLevel = 6 +RE_DATE_PREMATCH = re.compile("\{DATE\}", re.IGNORECASE) +DD_patternCache = Utils.Cache(maxCount=1000, maxTime=60*60) + + +def _getPatternTemplate(pattern, key=None): + if key is None: + key = pattern + if '%' not in pattern: + key = pattern.upper() + template = DD_patternCache.get(key) + + if not template: + if key in ("EPOCH", "{^LN-BEG}EPOCH", "^EPOCH"): + template = DateEpoch(lineBeginOnly=(key != "EPOCH")) + elif key in ("TAI64N", "{^LN-BEG}TAI64N", "^TAI64N"): + template = DateTai64n(wordBegin=('start' if key != "TAI64N" else False)) + else: + template = DatePatternRegex(pattern) + + DD_patternCache.set(key, template) + return template + +def _getAnchoredTemplate(template, wrap=lambda s: '{^LN-BEG}' + s): + # wrap name: + name = wrap(template.name) + # try to find in cache (by name): + template2 = DD_patternCache.get(name) + if not template2: + # wrap pattern (or regexp if not pattern template): + regex = wrap(getattr(template, 'pattern', template.regex)) + if hasattr(template, 'pattern'): + # try to find in cache (by pattern): + template2 = DD_patternCache.get(regex) + # make duplicate and set new anchored regex: + if not template2: + if not hasattr(template, 'pattern'): + template2 = _getPatternTemplate(name) + else: + template2 = _getPatternTemplate(regex) + return template2 + + class DateDetectorCache(object): + """Implements the caching of the default templates list. + """ def __init__(self): self.__lock = Lock() self.__templates = list() @@ -43,71 +89,115 @@ class DateDetectorCache(object): def templates(self): """List of template instances managed by the detector. """ + if self.__templates: + return self.__templates with self.__lock: - if self.__templates: + if self.__templates: # pragma: no cover - race-condition + multi-threaded environment only return self.__templates self._addDefaultTemplate() return self.__templates def _cacheTemplate(self, template): """Cache Fail2Ban's default template. + """ if isinstance(template, str): - template = DatePatternRegex(template) - self.__templates.append(template) + # exact given template with word begin-end boundary: + template = _getPatternTemplate(template) + # if not already line-begin anchored, additional template, that prefers datetime + # at start of a line (safety+performance feature): + name = template.name + if not name.startswith('{^LN-BEG}') and not name.startswith('^') and hasattr(template, 'regex'): + template2 = _getAnchoredTemplate(template) + # prevent to add duplicates: + if template2.name != name: + # increase weight of such templates, because they should be always + # preferred in template sorting process (bubble up): + template2.weight = 100.0 + self.__tmpcache[0].append(template2) + # add template: + self.__tmpcache[1].append(template) def _addDefaultTemplate(self): """Add resp. cache Fail2Ban's default set of date templates. """ + self.__tmpcache = [], [] + # ISO 8601, simple date, optional subsecond and timezone: + # 2005-01-23T21:59:59.981746, 2005-01-23 21:59:59 + # simple date: 2005/01/23 21:59:59 + # custom for syslog-ng 2006.12.21 06:43:20 + self._cacheTemplate("%ExY(?P<_sep>[-/.])%m(?P=_sep)%d[T ]%H:%M:%S(?:[.,]%f)?(?:\s*%z)?") # asctime with optional day, subsecond and/or year: # Sun Jan 23 21:59:59.011 2005 - self._cacheTemplate("(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %Y)?") + self._cacheTemplate("(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?") # asctime with optional day, subsecond and/or year coming after day # http://bugs.debian.org/798923 # Sun Jan 23 2005 21:59:59.011 - self._cacheTemplate("(?:%a )?%b %d %Y %H:%M:%S(?:\.%f)?") - # simple date, optional subsecond (proftpd): - # 2005-01-23 21:59:59 - # simple date: 2005/01/23 21:59:59 - # custom for syslog-ng 2006.12.21 06:43:20 - self._cacheTemplate("%Y(?P<_sep>[-/.])%m(?P=_sep)%d %H:%M:%S(?:,%f)?") + self._cacheTemplate("(?:%a )?%b %d %ExY %H:%M:%S(?:\.%f)?") # simple date too (from x11vnc): 23/01/2005 21:59:59 # and with optional year given by 2 digits: 23/01/05 21:59:59 # (See http://bugs.debian.org/537610) # 17-07-2008 17:23:25 - self._cacheTemplate("%d(?P<_sep>[-/])%m(?P=_sep)(?:%Y|%y) %H:%M:%S") + self._cacheTemplate("%d(?P<_sep>[-/])%m(?P=_sep)(?:%ExY|%Exy) %H:%M:%S") # Apache format optional time zone: # [31/Oct/2006:09:22:55 -0000] # 26-Jul-2007 15:20:52 - self._cacheTemplate("%d(?P<_sep>[-/])%b(?P=_sep)%Y[ :]?%H:%M:%S(?:\.%f)?(?: %z)?") - # CPanel 05/20/2008:01:57:39 - self._cacheTemplate("%m/%d/%Y:%H:%M:%S") - # named 26-Jul-2007 15:20:52.252 + # named 26-Jul-2007 15:20:52.252 # roundcube 26-Jul-2007 15:20:52 +0200 + self._cacheTemplate("%d(?P<_sep>[-/])%b(?P=_sep)%ExY[ :]?%H:%M:%S(?:\.%f)?(?: %z)?") + # CPanel 05/20/2008:01:57:39 + self._cacheTemplate("%m/%d/%ExY:%H:%M:%S") # 01-27-2012 16:22:44.252 # subseconds explicit to avoid possible %m<->%d confusion - # with previous - self._cacheTemplate("%m-%d-%Y %H:%M:%S\.%f") - # TAI64N - template = DateTai64n() - template.name = "TAI64N" - self._cacheTemplate(template) + # with previous ("%d-%m-%ExY %H:%M:%S" by "%d(?P<_sep>[-/])%m(?P=_sep)(?:%ExY|%Exy) %H:%M:%S") + self._cacheTemplate("%m-%d-%ExY %H:%M:%S(?:\.%f)?") # Epoch - template = DateEpoch() - template.name = "Epoch" - self._cacheTemplate(template) - # ISO 8601 - self._cacheTemplate("%Y-%m-%d[T ]%H:%M:%S(?:\.%f)?(?:%z)?") + self._cacheTemplate('EPOCH') # Only time information in the log - self._cacheTemplate("^%H:%M:%S") + self._cacheTemplate("{^LN-BEG}%H:%M:%S") # <09/16/08@05:03:30> - self._cacheTemplate("^<%m/%d/%y@%H:%M:%S>") + self._cacheTemplate("^<%m/%d/%Exy@%H:%M:%S>") # MySQL: 130322 11:46:11 - self._cacheTemplate("^%y%m%d ?%H:%M:%S") + self._cacheTemplate("%Exy%Exm%Exd ?%H:%M:%S") # Apache Tomcat - self._cacheTemplate("%b %d, %Y %I:%M:%S %p") + self._cacheTemplate("%b %d, %ExY %I:%M:%S %p") # ASSP: Apr-27-13 02:33:06 - self._cacheTemplate("^%b-%d-%y %H:%M:%S") + self._cacheTemplate("^%b-%d-%Exy %H:%M:%S") + # 20050123T215959, 20050123 215959 + self._cacheTemplate("%ExY%Exm%Exd[T ]%ExH%ExM%ExS(?:[.,]%f)?(?:\s*%z)?") + # prefixed with optional named time zone (monit): + # PDT Apr 16 21:05:29 + self._cacheTemplate("(?:%Z )?(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?") + # +00:00 Jan 23 21:59:59.011 2005 + self._cacheTemplate("(?:%z )?(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?") + # TAI64N + self._cacheTemplate("TAI64N") + # + self.__templates = self.__tmpcache[0] + self.__tmpcache[1] + del self.__tmpcache + + +class DateDetectorTemplate(object): + """Used for "shallow copy" of the template object. + + Prevents collectively usage of hits/lastUsed in cached templates + """ + __slots__ = ('template', 'hits', 'lastUsed', 'distance') + def __init__(self, template): + self.template = template + self.hits = 0 + self.lastUsed = 0 + # the last distance to date-match within the log file: + self.distance = 0x7fffffff + + @property + def weight(self): + return self.hits * self.template.weight / max(1, self.distance) + + def __getattr__(self, name): + """ Returns attribute of template (called for parameters not in slots) + """ + return getattr(self.template, name) class DateDetector(object): @@ -120,19 +210,27 @@ class DateDetector(object): _defCache = DateDetectorCache() def __init__(self): - self.__lock = Lock() self.__templates = list() self.__known_names = set() # time the template was long unused (currently 300 == 5m): self.__unusedTime = 300 + # last known distance (bypass one char collision) and end position: + self.__lastPos = 1, None + self.__lastEndPos = 0x7fffffff, None + self.__lastTemplIdx = 0x7fffffff + # first free place: + self.__firstUnused = 0 + # pre-match pattern: + self.__preMatch = None - def _appendTemplate(self, template): + def _appendTemplate(self, template, ignoreDup=False): name = template.name if name in self.__known_names: + if ignoreDup: return raise ValueError( "There is already a template with name %s" % name) self.__known_names.add(name) - self.__templates.append(template) + self.__templates.append(DateDetectorTemplate(template)) def appendTemplate(self, template): """Add a date template to manage and use in search of dates. @@ -150,15 +248,45 @@ class DateDetector(object): If a template already exists with the same name. """ if isinstance(template, str): - template = DatePatternRegex(template) - self._appendTemplate(template) + key = pattern = template + if '%' not in pattern: + key = pattern.upper() + template = DD_patternCache.get(key) + if not template: + if key in ("{^LN-BEG}", "{DEFAULT}"): + flt = \ + lambda template: template.flags & DateTemplate.LINE_BEGIN if key == "{^LN-BEG}" else None + self.addDefaultTemplate(flt) + return + elif "{DATE}" in key: + self.addDefaultTemplate( + lambda template: not template.flags & DateTemplate.LINE_BEGIN, pattern) + return + else: + template = _getPatternTemplate(pattern, key) - def addDefaultTemplate(self): + DD_patternCache.set(key, template) + + self._appendTemplate(template) + logSys.info(" date pattern `%r`: `%s`", + getattr(template, 'pattern', ''), template.name) + logSys.debug(" date pattern regex for %r: %s", + getattr(template, 'pattern', ''), template.regex) + + def addDefaultTemplate(self, filterTemplate=None, preMatch=None): """Add Fail2Ban's default set of date templates. """ - with self.__lock: - for template in DateDetector._defCache.templates: - self._appendTemplate(template) + ignoreDup = len(self.__templates) > 0 + for template in DateDetector._defCache.templates: + # filter if specified: + if filterTemplate is not None and not filterTemplate(template): continue + # if exact pattern available - create copy of template, contains replaced {DATE} with default regex: + if preMatch is not None: + # get cached or create a copy with modified name/pattern, using preMatch replacement for {DATE}: + template = _getAnchoredTemplate(template, + wrap=lambda s: RE_DATE_PREMATCH.sub(s, preMatch)) + # append date detector template (ignore duplicate if some was added before default): + self._appendTemplate(template, ignoreDup=ignoreDup) @property def templates(self): @@ -184,22 +312,115 @@ class DateDetector(object): The regex match returned from the first successfully matched template. """ - i = 0 - with self.__lock: - for template in self.__templates: + # if no templates specified - default templates should be used: + if not len(self.__templates): + self.addDefaultTemplate() + logSys.log(logLevel-1, "try to match time for line: %.120s", line) + match = None + # first try to use last template with same start/end position: + ignoreBySearch = 0x7fffffff + i = self.__lastTemplIdx + if i < len(self.__templates): + ddtempl = self.__templates[i] + template = ddtempl.template + if template.flags & (DateTemplate.LINE_BEGIN|DateTemplate.LINE_END): + if logSys.getEffectiveLevel() <= logLevel-1: # pragma: no cover - very-heavy debug + logSys.log(logLevel-1, " try to match last anchored template #%02i ...", i) match = template.matchDate(line) - if not match is None: + ignoreBySearch = i + else: + distance, endpos = self.__lastPos[0], self.__lastEndPos[0] + if logSys.getEffectiveLevel() <= logLevel-1: + logSys.log(logLevel-1, " try to match last template #%02i (from %r to %r): ...%r==%r %s %r==%r...", + i, distance, endpos, + line[distance-1:distance], self.__lastPos[1], + line[distance:endpos], + line[endpos:endpos+1], self.__lastEndPos[1]) + # check same boundaries left/right, otherwise possible collision/pattern switch: + if (line[distance-1:distance] == self.__lastPos[1] and + line[endpos:endpos+1] == self.__lastEndPos[1] + ): + match = template.matchDate(line, distance, endpos) + if match: + distance = match.start() + endpos = match.end() + # if different position, possible collision/pattern switch: + if ( + template.flags & (DateTemplate.LINE_BEGIN|DateTemplate.LINE_END) or + (distance == self.__lastPos[0] and endpos == self.__lastEndPos[0]) + ): + logSys.log(logLevel, " matched last time template #%02i", i) + else: + logSys.log(logLevel, " ** last pattern collision - pattern change, search ...") + match = None + else: + logSys.log(logLevel, " ** last pattern not found - pattern change, search ...") + # search template and better match: + if not match: + logSys.log(logLevel, " search template (%i) ...", len(self.__templates)) + found = None, 0x7fffffff, 0x7fffffff, -1 + i = 0 + for ddtempl in self.__templates: + if logSys.getEffectiveLevel() <= logLevel-1: + logSys.log(logLevel-1, " try template #%02i: %s", i, ddtempl.name) + if i == ignoreBySearch: + i += 1 + continue + template = ddtempl.template + match = template.matchDate(line) + if match: + distance = match.start() + endpos = match.end() if logSys.getEffectiveLevel() <= logLevel: - logSys.log(logLevel, "Matched time template %s", template.name) - template.hits += 1 - template.lastUsed = time.time() - # if not first - try to reorder current template (bubble up), they will be not sorted anymore: - if i: - self._reorderTemplate(i) - # return tuple with match and template reference used for parsing: - return (match, template) + logSys.log(logLevel, " matched time template #%02i (at %r <= %r, %r) %s", + i, distance, ddtempl.distance, self.__lastPos[0], template.name) + ## last (or single) template - fast stop: + if i+1 >= len(self.__templates): + break + ## if line-begin/end anchored - stop searching: + if template.flags & (DateTemplate.LINE_BEGIN|DateTemplate.LINE_END): + break + ## stop searching if next template still unused, but we had already hits: + if (distance == 0 and ddtempl.hits) and not self.__templates[i+1].template.hits: + break + ## [grave] if distance changed, possible date-match was found somewhere + ## in body of message, so save this template, and search further: + if distance > ddtempl.distance or distance > self.__lastPos[0]: + logSys.log(logLevel, " ** distance collision - pattern change, reserve") + ## shortest of both: + if distance < found[1]: + found = match, distance, endpos, i + ## search further: + match = None + i += 1 + continue + ## winner - stop search: + break i += 1 + # check other template was found (use this one with shortest distance): + if not match and found[0]: + match, distance, endpos, i = found + logSys.log(logLevel, " use best time template #%02i", i) + ddtempl = self.__templates[i] + template = ddtempl.template + # we've winner, incr hits, set distance, usage, reorder, etc: + if match: + ddtempl.hits += 1 + ddtempl.lastUsed = time.time() + ddtempl.distance = distance + if self.__firstUnused == i: + self.__firstUnused += 1 + self.__lastPos = distance, line[distance-1:distance] + self.__lastEndPos = endpos, line[endpos:endpos+1] + # if not first - try to reorder current template (bubble up), they will be not sorted anymore: + if i and i != self.__lastTemplIdx: + i = self._reorderTemplate(i) + self.__lastTemplIdx = i + # return tuple with match and template reference used for parsing: + return (match, template) + # not found: + logSys.log(logLevel, " no template.") return (None, None) def getTime(self, line, timeMatch=None): @@ -221,31 +442,22 @@ class DateDetector(object): The Unix timestamp returned from the first successfully matched template or None if not found. """ - if timeMatch: - template = timeMatch[1] - if template is not None: - try: - date = template.getDate(line, timeMatch[0]) - if date is not None: - if logSys.getEffectiveLevel() <= logLevel: - logSys.log(logLevel, "Got time %f for %r using template %s", - date[0], date[1].group(), template.name) - return date - except ValueError: - return None - with self.__lock: - for template in self.__templates: - try: - date = template.getDate(line) - if date is None: - continue - if logSys.getEffectiveLevel() <= logLevel: - logSys.log(logLevel, "Got time %f for %r using template %s", - date[0], date[1].group(), template.name) + # search match for all specified templates: + if timeMatch is None: + timeMatch = self.matchTime(line) + # convert: + template = timeMatch[1] + if template is not None: + try: + date = template.getDate(line, timeMatch[0]) + if date is not None: + if logSys.getEffectiveLevel() <= logLevel: # pragma: no cover - heavy debug + logSys.log(logLevel, " got time %f for %r using template %s", + date[0], date[1].group(1), template.name) return date - except ValueError: # pragma: no cover - pass - return None + except ValueError: + pass + return None def _reorderTemplate(self, num): """Reorder template (bubble up) in template list if hits grows enough. @@ -257,18 +469,39 @@ class DateDetector(object): """ if num: templates = self.__templates - template = templates[num] + ddtempl = templates[num] + if logSys.getEffectiveLevel() <= logLevel: + logSys.log(logLevel, " -> reorder template #%02i, hits: %r", num, ddtempl.hits) ## current hits and time the template was long unused: - untime = template.lastUsed - self.__unusedTime - hits = template.hits + untime = ddtempl.lastUsed - self.__unusedTime + weight = ddtempl.weight + ## try to move faster (first if unused available, or half of part to current template position): + pos = self.__firstUnused if self.__firstUnused < num else num // 2 ## don't move too often (multiline logs resp. log's with different date patterns), ## if template not used too long, replace it also : - if hits > templates[num-1].hits + 5 or templates[num-1].lastUsed < untime: - ## try to move faster (half of part to current template): - pos = num // 2 - ## if not larger - move slow (exact 1 position): - if hits <= templates[pos].hits or templates[pos].lastUsed < untime: - pos = num-1 - templates[pos], templates[num] = template, templates[pos] - - + def _moveable(): + pweight = templates[pos].weight + if logSys.getEffectiveLevel() <= logLevel: + logSys.log(logLevel, " -> compare template #%02i & #%02i, weight %.3f > %.3f, hits %r > %r", + num, pos, weight, pweight, ddtempl.hits, templates[pos].hits) + return weight > pweight or untime > templates[pos].lastUsed + ## + ## if not moveable (smaller weight or target position recently used): + if not _moveable(): + ## try to move slow (exact 1 position): + if pos == num-1: + return num + pos = num-1 + ## if still smaller and template at position used, don't move: + if not _moveable(): + return num + ## move: + del templates[num] + templates[pos:0] = [ddtempl] + ## correct first unused: + while self.__firstUnused < len(templates) and templates[self.__firstUnused].hits: + self.__firstUnused += 1 + if logSys.getEffectiveLevel() <= logLevel: + logSys.log(logLevel, " -> moved template #%02i -> #%02i", num, pos) + return pos + return num diff --git a/fail2ban/server/datetemplate.py b/fail2ban/server/datetemplate.py index 4f609e78..1d0b014b 100644 --- a/fail2ban/server/datetemplate.py +++ b/fail2ban/server/datetemplate.py @@ -24,14 +24,28 @@ __author__ = "Cyril Jaquier" __copyright__ = "Copyright (c) 2004 Cyril Jaquier" __license__ = "GPL" -import re +import re, time from abc import abstractmethod -from .strptime import reGroupDictStrptime, timeRE +from .strptime import reGroupDictStrptime, timeRE, getTimePatternRE from ..helpers import getLogger logSys = getLogger(__name__) +# check already grouped contains "(", but ignores char "\(" and conditional "(?(id)...)": +RE_GROUPED = re.compile(r'(?(?<=^\[))|(?P(?<=audit\()))\d{10,11}\b(?:\.\d{3,6})?(?:(?(selinux)(?=:\d+\)))|(?(square)(?=\])))" + self.name = "Epoch" + if not lineBeginOnly: + regex = r"((?:^|(?P(?<=^\[))|(?P(?<=\baudit\()))\d{10,11}\b(?:\.\d{3,6})?)(?:(?(selinux)(?=:\d+\)))|(?(square)(?=\])))" + self.setRegex(regex, wordBegin=False) ;# already line begin resp. word begin anchored + else: + regex = r"((?P(?<=^\[))?\d{10,11}\b(?:\.\d{3,6})?)(?(square)(?=\]))" + self.setRegex(regex, wordBegin='start', wordEnd=True) def getDate(self, line, dateMatch=None): """Method to return the date for a log line. @@ -160,8 +218,7 @@ class DateEpoch(DateTemplate): dateMatch = self.matchDate(line) if dateMatch: # extract part of format which represents seconds since epoch - return (float(dateMatch.group()), dateMatch) - return None + return (float(dateMatch.group(1)), dateMatch) class DatePatternRegex(DateTemplate): @@ -178,21 +235,15 @@ class DatePatternRegex(DateTemplate): regex pattern """ - _patternRE = re.compile(r"%%(%%|[%s])" % "".join(timeRE.keys())) - _patternName = { - 'a': "DAY", 'A': "DAYNAME", 'b': "MON", 'B': "MONTH", 'd': "Day", - 'H': "24hour", 'I': "12hour", 'j': "Yearday", 'm': "Month", - 'M': "Minute", 'p': "AMPM", 'S': "Second", 'U': "Yearweek", - 'w': "Weekday", 'W': "Yearweek", 'y': 'Year2', 'Y': "Year", '%': "%", - 'z': "Zone offset", 'f': "Microseconds", 'Z': "Zone name"} - for _key in set(timeRE) - set(_patternName): # may not have them all... - _patternName[_key] = "%%%s" % _key + + _patternRE, _patternName = getTimePatternRE() + _patternRE = re.compile(_patternRE) - def __init__(self, pattern=None): + def __init__(self, pattern=None, **kwargs): super(DatePatternRegex, self).__init__() self._pattern = None if pattern is not None: - self.pattern = pattern + self.setRegex(pattern, **kwargs) @property def pattern(self): @@ -208,17 +259,23 @@ class DatePatternRegex(DateTemplate): @pattern.setter def pattern(self, pattern): + self.setRegex(pattern) + + def setRegex(self, pattern, wordBegin=True, wordEnd=True): + # original pattern: self._pattern = pattern + # if explicit given {^LN-BEG} - remove it from pattern and set 'start' in wordBegin: + if wordBegin and RE_EXLINE_BOUND_BEG.search(pattern): + pattern = RE_EXLINE_BOUND_BEG.sub('', pattern) + wordBegin = 'start' + # wrap to regex: fmt = self._patternRE.sub(r'%(\1)s', pattern) - self._name = fmt % self._patternName - super(DatePatternRegex, self).setRegex(fmt % timeRE) - - def setRegex(self, value): - raise NotImplementedError("Regex derived from pattern") - - @DateTemplate.name.setter - def name(self, value): - raise NotImplementedError("Name derived from pattern") + self.name = fmt % self._patternName + regex = fmt % timeRE + # if expected add (?iu) for "ignore case" and "unicode": + if RE_ALPHA_PATTERN.search(pattern): + regex = r'(?iu)' + regex + super(DatePatternRegex, self).setRegex(regex, wordBegin, wordEnd) def getDate(self, line, dateMatch=None): """Method to return the date for a log line. @@ -240,11 +297,7 @@ class DatePatternRegex(DateTemplate): if not dateMatch: dateMatch = self.matchDate(line) if dateMatch: - groupdict = dict( - (key, value) - for key, value in dateMatch.groupdict().iteritems() - if value is not None) - return reGroupDictStrptime(groupdict), dateMatch + return reGroupDictStrptime(dateMatch.groupdict()), dateMatch class DateTai64n(DateTemplate): @@ -256,11 +309,11 @@ class DateTai64n(DateTemplate): regex """ - def __init__(self): + def __init__(self, wordBegin=False): DateTemplate.__init__(self) + self.name = "TAI64N" # We already know the format for TAI64N - # yoh: we should not add an additional front anchor - self.setRegex("@[0-9a-f]{24}", wordBegin=False) + self.setRegex("@[0-9a-f]{24}", wordBegin=wordBegin) def getDate(self, line, dateMatch=None): """Method to return the date for a log line. @@ -280,8 +333,7 @@ class DateTai64n(DateTemplate): dateMatch = self.matchDate(line) if dateMatch: # extract part of format which represents seconds since epoch - value = dateMatch.group() + value = dateMatch.group(1) seconds_since_epoch = value[2:17] # convert seconds from HEX into local time stamp return (int(seconds_since_epoch, 16), dateMatch) - return None diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 956694af..108c103d 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -36,7 +36,6 @@ from .observer import Observers from .ticket import FailTicket from .jailthread import JailThread from .datedetector import DateDetector -from .datetemplate import DatePatternRegex, DateEpoch, DateTai64n from .mytime import MyTime from .failregex import FailRegex, Regex, RegexException from .action import CommandAction @@ -91,11 +90,16 @@ class Filter(JailThread): ## Error counter (protected, so can be used in filter implementations) ## if it reached 100 (at once), run-cycle will go idle self._errors = 0 + ## return raw host (host is not dns): + self.returnRawHost = False + ## check each regex (used for test purposes): + self.checkAllRegex = False + ## if true ignores obsolete failures (failure time < now - findTime): + self.checkFindTime = True ## Ticks counter self.ticks = 0 self.dateDetector = DateDetector() - self.dateDetector.addDefaultTemplate() logSys.debug("Created %s" % self) def __repr__(self): @@ -258,20 +262,13 @@ class Filter(JailThread): if pattern is None: self.dateDetector = None return - elif pattern.upper() == "EPOCH": - template = DateEpoch() - template.name = "Epoch" - elif pattern.upper() == "TAI64N": - template = DateTai64n() - template.name = "TAI64N" else: - template = DatePatternRegex(pattern) - self.dateDetector = DateDetector() - self.dateDetector.appendTemplate(template) - logSys.info(" date pattern `%r`: `%s`", - pattern, template.name) - logSys.debug(" date pattern regex for %r: %s", - pattern, template.regex) + dd = DateDetector() + if not isinstance(pattern, (list, tuple)): + pattern = filter(bool, map(str.strip, re.split('\n+', pattern))) + for pattern in pattern: + dd.appendTemplate(pattern) + self.dateDetector = dd ## # Get the date detector pattern, or Default Detectors if not changed @@ -281,14 +278,16 @@ class Filter(JailThread): def getDatePattern(self): if self.dateDetector is not None: templates = self.dateDetector.templates - if len(templates) > 1: + # lazy template init, by first match + if not len(templates) or len(templates) > 2: return None, "Default Detectors" - elif len(templates) == 1: + elif len(templates): if hasattr(templates[0], "pattern"): pattern = templates[0].pattern else: pattern = None return pattern, templates[0].name + return None ## # Set the maximum retry value. @@ -450,14 +449,14 @@ class Filter(JailThread): if self.__ignoreCommand: command = CommandAction.replaceTag(self.__ignoreCommand, { 'ip': ip } ) logSys.debug('ignore command: ' + command) - ret_ignore = CommandAction.executeCmd(command) + ret, ret_ignore = CommandAction.executeCmd(command, success_codes=(0, 1)) + ret_ignore = ret and ret_ignore == 0 self.logIgnoreIp(ip, log_ignore and ret_ignore, ignore_source="command") return ret_ignore return False - def processLine(self, line, date=None, returnRawHost=False, - checkAllRegex=False, checkFindTime=False): + def processLine(self, line, date=None): """Split the time portion from log msg and return findFailures on them """ if date: @@ -469,22 +468,23 @@ class Filter(JailThread): (timeMatch, template) = self.dateDetector.matchTime(l) if timeMatch: tupleLine = ( - l[:timeMatch.start()], - l[timeMatch.start():timeMatch.end()], - l[timeMatch.end():], + l[:timeMatch.start(1)], + l[timeMatch.start(1):timeMatch.end(1)], + l[timeMatch.end(1):], (timeMatch, template) ) else: tupleLine = (l, "", "", None) - return "".join(tupleLine[::2]), self.findFailure( - tupleLine, date, returnRawHost, checkAllRegex, checkFindTime) + # save last line (lazy convert of process line tuple to string on demand): + self.processedLine = lambda: "".join(tupleLine[::2]) + return self.findFailure(tupleLine, date) def processLineAndAdd(self, line, date=None): """Processes the line for failures and populates failManager """ try: - for element in self.processLine(line, date, checkFindTime=True)[1]: + for element in self.processLine(line, date): ip = element[1] unixTime = element[2] lines = element[3] @@ -543,10 +543,10 @@ class Filter(JailThread): # to find the logging time. # @return a dict with IP and timestamp. - def findFailure(self, tupleLine, date=None, returnRawHost=False, - checkAllRegex=False, checkFindTime=False): + def findFailure(self, tupleLine, date=None): failList = list() + returnRawHost = self.returnRawHost cidr = IPAddr.CIDR_UNSPEC if self.__useDns == "raw": returnRawHost = True @@ -581,7 +581,7 @@ class Filter(JailThread): timeText = self.__lastTimeText or "".join(tupleLine[::2]) date = self.__lastDate - if checkFindTime and date is not None and date < MyTime.time() - self.getFindTime(): + if self.checkFindTime and date is not None and date < MyTime.time() - self.getFindTime(): logSys.log(5, "Ignore line since time %s < %s - %s", date, MyTime.time(), self.getFindTime()) return failList @@ -602,7 +602,7 @@ class Filter(JailThread): # The ignoreregex matched. Remove ignored match. self.__lineBuffer = failRegex.getUnmatchedTupleLines() logSys.log(7, "Matched ignoreregex and was ignored") - if not checkAllRegex: + if not self.checkAllRegex: break else: continue @@ -645,7 +645,7 @@ class Filter(JailThread): ip = IPAddr(fid, IPAddr.CIDR_RAW) failList.append([failRegexIndex, ip, date, failRegex.getMatchedLines(), fail]) - if not checkAllRegex: + if not self.checkAllRegex: break else: ips = DNSUtils.textToIp(host, self.__useDns) @@ -653,7 +653,7 @@ class Filter(JailThread): for ip in ips: failList.append([failRegexIndex, ip, date, failRegex.getMatchedLines(), fail]) - if not checkAllRegex: + if not self.checkAllRegex: break except RegexException as e: # pragma: no cover - unsure if reachable logSys.error(e) diff --git a/fail2ban/server/filtersystemd.py b/fail2ban/server/filtersystemd.py index 0d720a5e..d858142b 100644 --- a/fail2ban/server/filtersystemd.py +++ b/fail2ban/server/filtersystemd.py @@ -178,6 +178,14 @@ class FilterSystemd(JournalFilter): # pragma: systemd no cover def getJournalMatch(self): return self.__matches + ## + # Get journal reader + # + # @return journal reader + + def getJournalReader(self): + return self.__journal + ## # Format journal log entry into syslog style # diff --git a/fail2ban/server/mytime.py b/fail2ban/server/mytime.py index 816b04c4..49199887 100644 --- a/fail2ban/server/mytime.py +++ b/fail2ban/server/mytime.py @@ -41,6 +41,21 @@ class MyTime: """ myTime = None + alternateNowTime = None + alternateNow = None + + @staticmethod + def setAlternateNow(t): + """Set current time. + + Use None in order to always get the real current time. + + @param t the time to set or None + """ + + MyTime.alternateNowTime = t + MyTime.alternateNow = \ + datetime.datetime.fromtimestamp(t) if t is not None else None @staticmethod def setTime(t): @@ -84,8 +99,9 @@ class MyTime: """ if MyTime.myTime is None: return datetime.datetime.now() - else: - return datetime.datetime.fromtimestamp(MyTime.myTime) + if MyTime.myTime == MyTime.alternateNowTime: + return MyTime.alternateNow + return datetime.datetime.fromtimestamp(MyTime.myTime) @staticmethod def localtime(x=None): diff --git a/fail2ban/server/server.py b/fail2ban/server/server.py index 4f11a7e3..c77246d2 100644 --- a/fail2ban/server/server.py +++ b/fail2ban/server/server.py @@ -547,17 +547,19 @@ class Server: # @param target the logging target def setLogTarget(self, target): + # check reserved targets in uppercase, don't change target, because it can be file: + systarget = target.upper() with self.__loggingLock: # don't set new handlers if already the same # or if "INHERITED" (foreground worker of the test cases, to prevent stop logging): if self.__logTarget == target: return True - if target == "INHERITED": + if systarget == "INHERITED": self.__logTarget = target return True # set a format which is simpler for console use fmt = "%(asctime)s %(name)-24s[%(process)d]: %(levelname)-7s %(message)s" - if target == "SYSLOG": + if systarget == "SYSLOG": # Syslog daemons already add date to the message. fmt = "%(name)s[%(process)d]: %(levelname)s %(message)s" facility = logging.handlers.SysLogHandler.LOG_DAEMON @@ -576,9 +578,9 @@ class Server: "Syslog socket file: %s does not exists" " or is not a socket" % self.__syslogSocket) return False - elif target == "STDOUT": + elif systarget == "STDOUT": hdlr = logging.StreamHandler(sys.stdout) - elif target == "STDERR": + elif systarget == "STDERR": hdlr = logging.StreamHandler(sys.stderr) else: # Target should be a file diff --git a/fail2ban/server/strptime.py b/fail2ban/server/strptime.py index 2e3c051c..cdfe0e0e 100644 --- a/fail2ban/server/strptime.py +++ b/fail2ban/server/strptime.py @@ -26,10 +26,59 @@ from .mytime import MyTime locale_time = LocaleTime() timeRE = TimeRE() -timeRE['z'] = r"(?PZ|[+-]\d{2}(?::?[0-5]\d)?)" +def _getYearCentRE(cent=(0,3), distance=3, now=(MyTime.now(), MyTime.alternateNow)): + """ Build century regex for last year and the next years (distance). + + Thereby respect possible run in the test-cases (alternate date used there) + """ + cent = lambda year, f=cent[0], t=cent[1]: str(year)[f:t] + exprset = set( cent(now[0].year + i) for i in (-1, distance) ) + if len(now) and now[1]: + exprset |= set( cent(now[1].year + i) for i in (-1, distance) ) + return "(?:%s)" % "|".join(exprset) if len(exprset) > 1 else "".join(exprset) -def reGroupDictStrptime(found_dict): +#todo: implement literal time zone support like CET, PST, PDT, etc (via pytz): +#timeRE['z'] = r"%s?(?PZ|[+-]\d{2}(?::?[0-5]\d)?|[A-Z]{3})?" % timeRE['Z'] +timeRE['Z'] = r"(?P[A-Z]{3,5})" +timeRE['z'] = r"(?PZ|UTC|GMT|[+-]\d{2}(?::?[0-5]\d)?)" + +# Extend build-in TimeRE with some exact patterns +# exact two-digit patterns: +timeRE['Exd'] = r"(?P3[0-1]|[1-2]\d|0[1-9])" +timeRE['Exm'] = r"(?P1[0-2]|0[1-9])" +timeRE['ExH'] = r"(?P2[0-3]|[0-1]\d)" +timeRE['ExM'] = r"(?P[0-5]\d)" +timeRE['ExS'] = r"(?P6[0-1]|[0-5]\d)" +# more precise year patterns, within same century of last year and +# the next 3 years (for possible long uptime of fail2ban); thereby +# respect possible run in the test-cases (alternate date used there): +timeRE['ExY'] = r"(?P%s\d)" % _getYearCentRE(cent=(0,3), distance=3) +timeRE['Exy'] = r"(?P%s\d)" % _getYearCentRE(cent=(2,3), distance=3) + +def getTimePatternRE(): + keys = timeRE.keys() + patt = (r"%%(%%|%s|[%s])" % ( + "|".join([k for k in keys if len(k) > 1]), + "".join([k for k in keys if len(k) == 1]), + )) + names = { + 'a': "DAY", 'A': "DAYNAME", 'b': "MON", 'B': "MONTH", 'd': "Day", + 'H': "24hour", 'I': "12hour", 'j': "Yearday", 'm': "Month", + 'M': "Minute", 'p': "AMPM", 'S': "Second", 'U': "Yearweek", + 'w': "Weekday", 'W': "Yearweek", 'y': 'Year2', 'Y': "Year", '%': "%", + 'z': "Zone offset", 'f': "Microseconds", 'Z': "Zone name", + } + for key in set(keys) - set(names): # may not have them all... + if key.startswith('Ex'): + kn = names.get(key[2:]) + if kn: + names[key] = "Ex" + kn + continue + names[key] = "%%%s" % key + return (patt, names) + +def reGroupDictStrptime(found_dict, msec=False): """Return time from dictionary of strptime fields This is tweaked from python built-in _strptime. @@ -58,14 +107,15 @@ def reGroupDictStrptime(found_dict): # weekday and julian defaulted to -1 so as to signal need to calculate # values weekday = julian = -1 - for group_key in found_dict.keys(): + for key, val in found_dict.iteritems(): + if val is None: continue # Directives not explicitly handled below: # c, x, X # handled by making out of other directives # U, W # worthless without day of the week - if group_key == 'y': - year = int(found_dict['y']) + if key == 'y': + year = int(val) # Open Group specification for strptime() states that a %y #value in the range of [00, 68] is in the century 2000, while #[69,99] is in the century 1900 @@ -73,20 +123,20 @@ def reGroupDictStrptime(found_dict): year += 2000 else: year += 1900 - elif group_key == 'Y': - year = int(found_dict['Y']) - elif group_key == 'm': - month = int(found_dict['m']) - elif group_key == 'B': - month = locale_time.f_month.index(found_dict['B'].lower()) - elif group_key == 'b': - month = locale_time.a_month.index(found_dict['b'].lower()) - elif group_key == 'd': - day = int(found_dict['d']) - elif group_key == 'H': - hour = int(found_dict['H']) - elif group_key == 'I': - hour = int(found_dict['I']) + elif key == 'Y': + year = int(val) + elif key == 'm': + month = int(val) + elif key == 'B': + month = locale_time.f_month.index(val.lower()) + elif key == 'b': + month = locale_time.a_month.index(val.lower()) + elif key == 'd': + day = int(val) + elif key == 'H': + hour = int(val) + elif key == 'I': + hour = int(val) ampm = found_dict.get('p', '').lower() # If there was no AM/PM indicator, we'll treat this like AM if ampm in ('', locale_time.am_pm[0]): @@ -101,38 +151,39 @@ def reGroupDictStrptime(found_dict): # 12 noon == 12 PM == hour 12 if hour != 12: hour += 12 - elif group_key == 'M': - minute = int(found_dict['M']) - elif group_key == 'S': - second = int(found_dict['S']) - elif group_key == 'f': - s = found_dict['f'] - # Pad to always return microseconds. - s += "0" * (6 - len(s)) - fraction = int(s) - elif group_key == 'A': - weekday = locale_time.f_weekday.index(found_dict['A'].lower()) - elif group_key == 'a': - weekday = locale_time.a_weekday.index(found_dict['a'].lower()) - elif group_key == 'w': - weekday = int(found_dict['w']) + elif key == 'M': + minute = int(val) + elif key == 'S': + second = int(val) + elif key == 'f': + if msec: + s = val + # Pad to always return microseconds. + s += "0" * (6 - len(s)) + fraction = int(s) + elif key == 'A': + weekday = locale_time.f_weekday.index(val.lower()) + elif key == 'a': + weekday = locale_time.a_weekday.index(val.lower()) + elif key == 'w': + weekday = int(val) if weekday == 0: weekday = 6 else: weekday -= 1 - elif group_key == 'j': - julian = int(found_dict['j']) - elif group_key in ('U', 'W'): - week_of_year = int(found_dict[group_key]) - if group_key == 'U': + elif key == 'j': + julian = int(val) + elif key in ('U', 'W'): + week_of_year = int(val) + if key == 'U': # U starts week on Sunday. week_of_year_start = 6 else: # W starts week on Monday. week_of_year_start = 0 - elif group_key == 'z': - z = found_dict['z'] - if z == "Z": + elif key == 'z': + z = val + if z in ("Z", "UTC", "GMT"): tzoffset = 0 else: tzoffset = int(z[1:3]) * 60 # Hours... @@ -140,6 +191,10 @@ def reGroupDictStrptime(found_dict): tzoffset += int(z[-2:]) # ...and minutes if z.startswith("-"): tzoffset = -tzoffset + elif key == 'Z': + z = val + if z in ("UTC", "GMT"): + tzoffset = 0 # Fail2Ban will assume it's this year assume_year = False @@ -176,7 +231,7 @@ def reGroupDictStrptime(found_dict): # Actully create date date_result = datetime.datetime( year, month, day, hour, minute, second, fraction) - if gmtoff: + if gmtoff is not None: date_result = date_result - datetime.timedelta(seconds=gmtoff) if date_result > now and assume_today: @@ -189,7 +244,9 @@ def reGroupDictStrptime(found_dict): year=year-1, month=month, day=day) if gmtoff is not None: - return calendar.timegm(date_result.utctimetuple()) + tm = calendar.timegm(date_result.utctimetuple()) else: - return time.mktime(date_result.timetuple()) - + tm = time.mktime(date_result.timetuple()) + if msec: + tm += fraction/1000000.0 + return tm diff --git a/fail2ban/server/transmitter.py b/fail2ban/server/transmitter.py index c5d5cfb2..108115ba 100644 --- a/fail2ban/server/transmitter.py +++ b/fail2ban/server/transmitter.py @@ -131,6 +131,9 @@ class Transmitter: return self.status(command[1:]) elif command[0] == "version": return version.version + elif command[0] == "config-error": + logSys.error(command[1]) + return None raise Exception("Invalid command") def __commandSet(self, command, multiple=False): @@ -308,7 +311,7 @@ class Transmitter: actionvalue = command[4] setattr(action, actionkey, actionvalue) return getattr(action, actionkey) - raise Exception("Invalid command (no set action or not yet implemented)") + raise Exception("Invalid command %r (no set action or not yet implemented)" % (command[1],)) def __commandGet(self, command): name = command[0] diff --git a/fail2ban/server/utils.py b/fail2ban/server/utils.py index 6ed3d8fa..57da495a 100644 --- a/fail2ban/server/utils.py +++ b/fail2ban/server/utils.py @@ -110,7 +110,7 @@ class Utils(): return flags @staticmethod - def executeCmd(realCmd, timeout=60, shell=True, output=False, tout_kill_tree=True): + def executeCmd(realCmd, timeout=60, shell=True, output=False, tout_kill_tree=True, success_codes=(0,)): """Executes a command. Parameters @@ -170,7 +170,7 @@ class Utils(): time.sleep(Utils.DEFAULT_SLEEP_INTERVAL) retcode = popen.poll() #logSys.debug("%s -- killed %s ", realCmd, retcode) - if retcode is None and not Utils.pid_exists(pgid): + if retcode is None and not Utils.pid_exists(pgid): # pragma: no cover retcode = signal.SIGKILL except OSError as e: stderr = "%s -- failed with %s" % (realCmd, e) @@ -178,7 +178,7 @@ class Utils(): if not popen: return False if not output else (False, stdout, stderr, retcode) - std_level = retcode == 0 and logging.DEBUG or logging.ERROR + std_level = logging.DEBUG if retcode in success_codes else logging.ERROR # if we need output (to return or to log it): if output or std_level >= logSys.getEffectiveLevel(): # if was timeouted (killed/terminated) - to prevent waiting, set std handles to non-blocking mode. @@ -208,8 +208,8 @@ class Utils(): popen.stderr.close() success = False - if retcode == 0: - logSys.debug("%-.40s -- returned successfully", realCmd) + if retcode in success_codes: + logSys.debug("%-.40s -- returned successfully %i", realCmd, retcode) success = True elif retcode is None: logSys.error("%-.40s -- unable to kill PID %i", realCmd, popen.pid) @@ -223,7 +223,9 @@ class Utils(): logSys.error("%-.40s -- returned %i", realCmd, retcode) if msg: logSys.info("HINT on %i: %s", retcode, msg % locals()) - return success if not output else (success, stdout, stderr, retcode) + if output: + return success, stdout, stderr, retcode + return success if len(success_codes) == 1 else (success, retcode) @staticmethod def wait_for(cond, timeout, interval=None): diff --git a/fail2ban/tests/action_d/test_badips.py b/fail2ban/tests/action_d/test_badips.py index 64db44e7..2f3b6723 100644 --- a/fail2ban/tests/action_d/test_badips.py +++ b/fail2ban/tests/action_d/test_badips.py @@ -32,6 +32,7 @@ if sys.version_info >= (2,7): # pragma: no cover - may be unavailable def setUp(self): """Call before every test case.""" + super(BadIPsActionTest, self).setUp() unittest.F2B.SkipIfNoNetwork() self.jail = DummyJail() diff --git a/fail2ban/tests/action_d/test_smtp.py b/fail2ban/tests/action_d/test_smtp.py index 86e7a368..0aeacbae 100644 --- a/fail2ban/tests/action_d/test_smtp.py +++ b/fail2ban/tests/action_d/test_smtp.py @@ -21,6 +21,7 @@ import os import smtpd import threading import unittest +import re import sys if sys.version_info >= (3, 3): import importlib @@ -41,7 +42,9 @@ class TestSMTPServer(smtpd.SMTPServer): self.peer = peer self.mailfrom = mailfrom self.rcpttos = rcpttos - self.data = data + self.org_data = data + # replace new line (with tab or space) for possible mime translations (word wrap): + self.data = re.sub(r"\n[\t ]", " ", data) self.ready = True @@ -49,6 +52,7 @@ class SMTPActionTest(unittest.TestCase): def setUp(self): """Call before every test case.""" + super(SMTPActionTest, self).setUp() self.jail = DummyJail() pythonModule = os.path.join(CONFIG_DIR, "action.d", "smtp.py") pythonModuleName = os.path.basename(pythonModule.rstrip(".py")) @@ -99,23 +103,28 @@ class SMTPActionTest(unittest.TestCase): "Subject: [Fail2Ban] %s: stopped" % self.jail.name in self.smtpd.data) - def testBan(self): + def _testBan(self, restored=False): aInfo = { 'ip': "127.0.0.2", 'failures': 3, 'matches': "Test fail 1\n", 'ipjailmatches': "Test fail 1\nTest Fail2\n", 'ipmatches': "Test fail 1\nTest Fail2\nTest Fail3\n", - } + } + if restored: + aInfo['restored'] = 1 self._exec_and_wait(lambda: self.action.ban(aInfo)) + if restored: # no mail, should raises attribute error: + self.assertRaises(AttributeError, lambda: self.smtpd.mailfrom) + return self.assertEqual(self.smtpd.mailfrom, "fail2ban") self.assertEqual(self.smtpd.rcpttos, ["root"]) subject = "Subject: [Fail2Ban] %s: banned %s" % ( self.jail.name, aInfo['ip']) - self.assertIn(subject, self.smtpd.data.replace("\n", "")) - self.assertTrue( - "%i attempts" % aInfo['failures'] in self.smtpd.data) + self.assertIn(subject, self.smtpd.data) + self.assertIn( + "%i attempts" % aInfo['failures'], self.smtpd.data) self.action.matches = "matches" self._exec_and_wait(lambda: self.action.ban(aInfo)) @@ -128,6 +137,12 @@ class SMTPActionTest(unittest.TestCase): self.action.matches = "ipmatches" self._exec_and_wait(lambda: self.action.ban(aInfo)) self.assertIn(aInfo['ipmatches'], self.smtpd.data) + + def testBan(self): + self._testBan() + + def testNOPByRestored(self): + self._testBan(restored=True) def testOptions(self): self._exec_and_wait(self.action.start) diff --git a/fail2ban/tests/banmanagertestcase.py b/fail2ban/tests/banmanagertestcase.py index b78cd488..da454ec3 100644 --- a/fail2ban/tests/banmanagertestcase.py +++ b/fail2ban/tests/banmanagertestcase.py @@ -34,6 +34,7 @@ from ..server.ticket import BanTicket class AddFailure(unittest.TestCase): def setUp(self): """Call before every test case.""" + super(AddFailure, self).setUp() setUpMyTime() self.__ticket = BanTicket('193.168.0.128', 1167605999.0) self.__banManager = BanManager() @@ -155,6 +156,7 @@ class AddFailure(unittest.TestCase): class StatusExtendedCymruInfo(unittest.TestCase): def setUp(self): """Call before every test case.""" + super(StatusExtendedCymruInfo, self).setUp() unittest.F2B.SkipIfNoNetwork() setUpMyTime() self.__ban_ip = "93.184.216.34" diff --git a/fail2ban/tests/clientbeautifiertestcase.py b/fail2ban/tests/clientbeautifiertestcase.py index 0390a8ff..7d20e84e 100644 --- a/fail2ban/tests/clientbeautifiertestcase.py +++ b/fail2ban/tests/clientbeautifiertestcase.py @@ -32,6 +32,7 @@ class BeautifierTest(unittest.TestCase): def setUp(self): """ Call before every test case """ + super(BeautifierTest, self).setUp() self.b = Beautifier() def tearDown(self): diff --git a/fail2ban/tests/clientreadertestcase.py b/fail2ban/tests/clientreadertestcase.py index e68523c2..37add795 100644 --- a/fail2ban/tests/clientreadertestcase.py +++ b/fail2ban/tests/clientreadertestcase.py @@ -55,6 +55,7 @@ class ConfigReaderTest(unittest.TestCase): def setUp(self): """Call before every test case.""" + super(ConfigReaderTest, self).setUp() self.d = tempfile.mkdtemp(prefix="f2b-temp") self.c = ConfigReaderUnshared(basedir=self.d) @@ -193,13 +194,15 @@ class JailReaderTest(LogCaptureTestCase): self.assertTrue(jail.read()) self.assertFalse(jail.getOptions()) self.assertTrue(jail.isEnabled()) - self.assertLogged('Error in action definition joho[foo') - # This unittest has been deactivated for some time... - # self.assertLogged( - # 'Caught exception: While reading action joho[foo we should have got 1 or 2 groups. Got: 0') - # let's test for what is actually logged and handle changes in the future - self.assertLogged( - "Caught exception: 'NoneType' object has no attribute 'endswith'") + self.assertLogged("Invalid action definition 'joho[foo'") + + def testJailFilterBrokenDef(self): + jail = JailReader('brokenfilterdef', basedir=IMPERFECT_CONFIG, + share_config=IMPERFECT_CONFIG_SHARE_CFG) + self.assertTrue(jail.read()) + self.assertFalse(jail.getOptions()) + self.assertTrue(jail.isEnabled()) + self.assertLogged("Invalid filter definition 'flt[test'") if STOCK: def testStockSSHJail(self): @@ -344,7 +347,7 @@ class FilterReaderTest(unittest.TestCase): ['set', 'testcase01', 'addjournalmatch', "FIELD= with spaces ", "+", "AFIELD= with + char and spaces"], ['set', 'testcase01', 'datepattern', "%Y %m %d %H:%M:%S"], - ['set', 'testcase01', 'maxlines', "1"], # Last for overide test + ['set', 'testcase01', 'maxlines', 1], # Last for overide test ] filterReader = FilterReader("testcase01", "testcase01", {}) filterReader.setBaseDir(TEST_FILES_DIR) @@ -496,7 +499,7 @@ class JailsReaderTest(LogCaptureTestCase): def testReadTestJailConf(self): jails = JailsReader(basedir=IMPERFECT_CONFIG, share_config=IMPERFECT_CONFIG_SHARE_CFG) self.assertTrue(jails.read()) - self.assertFalse(jails.getOptions()) + self.assertFalse(jails.getOptions(ignoreWrong=False)) self.assertRaises(ValueError, jails.convert) comm_commands = jails.convert(allow_no_files=True) self.maxDiff = None @@ -514,19 +517,27 @@ class JailsReaderTest(LogCaptureTestCase): ['add', 'brokenaction', 'auto'], ['set', 'brokenaction', 'addfailregex', ''], ['set', 'brokenaction', 'addaction', 'brokenaction'], - ['set', - 'brokenaction', - 'action', - 'brokenaction', - 'actionban', - 'hit with big stick '], + ['multi-set', 'brokenaction', 'action', 'brokenaction', [ + ['actionban', 'hit with big stick '], + ['actname', 'brokenaction'] + ]], ['add', 'parse_to_end_of_jail.conf', 'auto'], ['set', 'parse_to_end_of_jail.conf', 'addfailregex', ''], ['start', 'emptyaction'], ['start', 'missinglogfiles'], ['start', 'brokenaction'], - ['start', 'parse_to_end_of_jail.conf'],])) - self.assertLogged("Errors in jail 'missingbitsjail'. Skipping...") + ['start', 'parse_to_end_of_jail.conf'], + ['config-error', + "Jail 'brokenactiondef' skipped, because of wrong configuration: Invalid action definition 'joho[foo'"], + ['config-error', + "Jail 'brokenfilterdef' skipped, because of wrong configuration: Invalid filter definition 'flt[test'"], + ['config-error', + "Jail 'missingaction' skipped, because of wrong configuration: Unable to read action 'noactionfileforthisaction'"], + ['config-error', + "Jail 'missingbitsjail' skipped, because of wrong configuration: Unable to read the filter 'catchallthebadies'"], + ])) + self.assertLogged("Errors in jail 'missingbitsjail'.") + self.assertNotLogged("Skipping...") self.assertLogged("No file(s) found for glob /weapons/of/mass/destruction") if STOCK: @@ -535,7 +546,10 @@ class JailsReaderTest(LogCaptureTestCase): actionName = os.path.basename(actionConfig).replace('.conf', '') actionReader = ActionReader(actionName, "TEST", {}, basedir=CONFIG_DIR) self.assertTrue(actionReader.read()) - actionReader.getOptions({}) # populate _opts + try: + actionReader.getOptions({}) # populate _opts + except Exception as e: # pragma: no cover + self.fail("action %r\n%s: %s" % (actionName, type(e).__name__, e)) if not actionName.endswith('-common'): self.assertIn('Definition', actionReader.sections(), msg="Action file %r is lacking [Definition] section" % actionConfig) @@ -614,7 +628,7 @@ class JailsReaderTest(LogCaptureTestCase): # grab all filter names filters = set(os.path.splitext(os.path.split(a)[1])[0] for a in glob.glob(os.path.join('config', 'filter.d', '*.conf')) - if not a.endswith('common.conf')) + if not (a.endswith('common.conf') or a.endswith('-aggressive.conf'))) # get filters of all jails (filter names without options inside filter[...]) filters_jail = set( JailReader.extractOptions(jail.options['filter'])[0] for jail in jails.jails diff --git a/fail2ban/tests/config/filter.d/common.conf b/fail2ban/tests/config/filter.d/common.conf deleted file mode 100644 index 586f428a..00000000 --- a/fail2ban/tests/config/filter.d/common.conf +++ /dev/null @@ -1,64 +0,0 @@ -# Generic configuration items (to be used as interpolations) in other -# filters or actions configurations -# - -[INCLUDES] - -# Load customizations if any available -after = common.local - - -[DEFAULT] - -# Daemon definition is to be specialized (if needed) in .conf file -_daemon = \S* - -# -# Shortcuts for easier comprehension of the failregex -# -# PID. -# EXAMPLES: [123] -__pid_re = (?:\[\d+\]) - -# Daemon name (with optional source_file:line or whatever) -# EXAMPLES: pam_rhosts_auth, [sshd], pop(pam_unix) -__daemon_re = [\[\(]?%(_daemon)s(?:\(\S+\))?[\]\)]?:? - -# extra daemon info -# EXAMPLE: [ID 800047 auth.info] -__daemon_extra_re = \[ID \d+ \S+\] - -# Combinations of daemon name and PID -# EXAMPLES: sshd[31607], pop(pam_unix)[4920] -__daemon_combs_re = (?:%(__pid_re)s?:\s+%(__daemon_re)s|%(__daemon_re)s%(__pid_re)s?:?) - -# Some messages have a kernel prefix with a timestamp -# EXAMPLES: kernel: [769570.846956] -__kernel_prefix = kernel: \[ *\d+\.\d+\] - -__hostname = \S+ - -# A MD5 hex -# EXAMPLES: 07:06:27:55:b0:e3:0c:3c:5a:28:2d:7c:7e:4c:77:5f -__md5hex = (?:[\da-f]{2}:){15}[\da-f]{2} - -# bsdverbose is where syslogd is started with -v or -vv and results in <4.3> or -# appearing before the host as per testcases/files/logs/bsd/*. -__bsd_syslog_verbose = <[^.]+\.[^.]+> - -__vserver = @vserver_\S+ - -__date_ambit = (?:\[\]) - -# Common line prefixes (beginnings) which could be used in filters -# -# [bsdverbose]? [hostname] [vserver tag] daemon_id spaces -# -# This can be optional (for instance if we match named native log files) -__prefix_line = %(__date_ambit)s?\s*(?:%(__bsd_syslog_verbose)s\s+)?(?:%(__hostname)s\s+)?(?:%(__kernel_prefix)s\s+)?(?:%(__vserver)s\s+)?(?:%(__daemon_combs_re)s\s+)?(?:%(__daemon_extra_re)s\s+)? - -# PAM authentication mechanism check for failures, e.g.: pam_unix, pam_sss, -# pam_ldap -__pam_auth = pam_unix - -# Author: Yaroslav Halchenko diff --git a/fail2ban/tests/config/filter.d/zzz-generic-example.conf b/fail2ban/tests/config/filter.d/zzz-generic-example.conf index df30d725..c50f4dc6 100644 --- a/fail2ban/tests/config/filter.d/zzz-generic-example.conf +++ b/fail2ban/tests/config/filter.d/zzz-generic-example.conf @@ -8,7 +8,7 @@ # Read common prefixes. If any customizations available -- read them from # common.local. common.conf is a symlink to the original common.conf and # should be copied (dereferenced) during installation -before = common.conf +before = ../../../../config/filter.d/common.conf [Definition] @@ -20,3 +20,8 @@ failregex = ^%(__prefix_line)sF2B: failure from $ # just to test multiple ignoreregex: ignoreregex = ^%(__prefix_line)sF2B: error from 192.0.2.251$ ^%(__prefix_line)sF2B: error from 192.0.2.252$ + +# specify only exact date patterns, +1 with %%Y to test usage of last known date by wrong dates like 0000-00-00... +datepattern = {^LN-BEG}%%ExY(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)? + {^LN-BEG}(?:%%a )?%%b %%d %%H:%%M:%%S(?:\.%%f)?(?: %%ExY)? + {^LN-BEG}%%Y(?P<_sep>[-/.])%%m(?P=_sep)%%d[T ]%%H:%%M:%%S(?:[.,]%%f)?(?:\s*%%z)? diff --git a/fail2ban/tests/config/jail.conf b/fail2ban/tests/config/jail.conf index bf1dea45..659e3fd3 100644 --- a/fail2ban/tests/config/jail.conf +++ b/fail2ban/tests/config/jail.conf @@ -27,10 +27,18 @@ logpath = /weapons/of/mass/destruction enabled = true action = joho[foo +[brokenfilterdef] +enabled = true +filter = flt[test + [brokenaction] enabled = true action = brokenaction +[missingaction] +enabled = true +action = noactionfileforthisaction + [missingbitsjail] enabled = true filter = catchallthebadies diff --git a/fail2ban/tests/datedetectortestcase.py b/fail2ban/tests/datedetectortestcase.py index 0c3c306d..5b32a7e9 100644 --- a/fail2ban/tests/datedetectortestcase.py +++ b/fail2ban/tests/datedetectortestcase.py @@ -30,7 +30,7 @@ import datetime from ..server.datedetector import DateDetector from ..server import datedetector -from ..server.datetemplate import DateTemplate +from ..server.datetemplate import DatePatternRegex, DateTemplate from .utils import setUpMyTime, tearDownMyTime, LogCaptureTestCase from ..helpers import getLogger @@ -42,35 +42,40 @@ class DateDetectorTest(LogCaptureTestCase): def setUp(self): """Call before every test case.""" LogCaptureTestCase.setUp(self) - self.__old_eff_level = datedetector.logLevel - datedetector.logLevel = logSys.getEffectiveLevel() setUpMyTime() - self.__datedetector = DateDetector() - self.__datedetector.addDefaultTemplate() + self.__datedetector = None def tearDown(self): """Call after every test case.""" LogCaptureTestCase.tearDown(self) - datedetector.logLevel = self.__old_eff_level tearDownMyTime() + + @property + def datedetector(self): + if self.__datedetector is None: + self.__datedetector = DateDetector() + self.__datedetector.addDefaultTemplate() + return self.__datedetector def testGetEpochTime(self): + self.__datedetector = DateDetector() + self.__datedetector.appendTemplate('EPOCH') # correct epoch time, using all variants: for dateUnix in (1138049999, 32535244799): for date in ("%s", "[%s]", "[%s.555]", "audit(%s.555:101)"): date = date % dateUnix log = date + " [sshd] error: PAM: Authentication failure" - datelog = self.__datedetector.getTime(log) + datelog = self.datedetector.getTime(log) self.assertTrue(datelog, "Parse epoch time for %s failed" % (date,)) ( datelog, matchlog ) = datelog self.assertEqual(int(datelog), dateUnix) - self.assertIn(matchlog.group(), (str(dateUnix), str(dateUnix)+'.555')) + self.assertIn(matchlog.group(1), (str(dateUnix), str(dateUnix)+'.555')) # wrong, no epoch time (< 10 digits, more as 11 digits, begin/end of word) : for dateUnix in ('123456789', '9999999999999999', '1138049999A', 'A1138049999'): for date in ("%s", "[%s]", "[%s.555]", "audit(%s.555:101)"): date = date % dateUnix log = date + " [sshd] error: PAM: Authentication failure" - datelog = self.__datedetector.getTime(log) + datelog = self.datedetector.getTime(log) self.assertFalse(datelog) def testGetTime(self): @@ -80,109 +85,331 @@ class DateDetectorTest(LogCaptureTestCase): # is not correctly determined atm, since year is not present # in the log entry. Since this doesn't effect the operation # of fail2ban -- we just ignore incorrect day of the week - ( datelog, matchlog ) = self.__datedetector.getTime(log) + ( datelog, matchlog ) = self.datedetector.getTime(log) self.assertEqual(datelog, dateUnix) - self.assertEqual(matchlog.group(), 'Jan 23 21:59:59') + self.assertEqual(matchlog.group(1), 'Jan 23 21:59:59') def testVariousTimes(self): """Test detection of various common date/time formats f2b should understand """ dateUnix = 1106513999.0 - for anchored, sdate in ( - (False, "Jan 23 21:59:59"), - (False, "Sun Jan 23 21:59:59 2005"), - (False, "Sun Jan 23 21:59:59"), - (False, "Sun Jan 23 2005 21:59:59"), - (False, "2005/01/23 21:59:59"), - (False, "2005.01.23 21:59:59"), - (False, "23/01/2005 21:59:59"), - (False, "23/01/05 21:59:59"), - (False, "23/Jan/2005:21:59:59"), - (False, "23/Jan/2005:21:59:59 +0100"), - (False, "01/23/2005:21:59:59"), - (False, "2005-01-23 21:59:59"), - (False, "2005-01-23 21:59:59,000"), # proftpd - (False, "23-Jan-2005 21:59:59"), - (False, "23-Jan-2005 21:59:59.02"), - (False, "23-Jan-2005 21:59:59 +0100"), - (False, "23-01-2005 21:59:59"), - (True, "1106513999"), # Portsetry - (False, "01-23-2005 21:59:59.252"), # reported on f2b, causes Feb29 fix to break - (False, "@4000000041f4104f00000000"), # TAI64N - (False, "2005-01-23T20:59:59.252Z"), #ISO 8601 (UTC) - (False, "2005-01-23T15:59:59-05:00"), #ISO 8601 with TZ - (False, "2005-01-23T21:59:59"), #ISO 8601 no TZ, assume local - (True, "<01/23/05@21:59:59>"), - (True, "050123 21:59:59"), # MySQL - (True, "Jan-23-05 21:59:59"), # ASSP like - (False, "Jan 23, 2005 9:59:59 PM"), # Apache Tomcat - (True, "1106513999"), # Regular epoch - (True, "1106513999.000"), # Regular epoch with millisec - (False, "audit(1106513999.000:987)"), # SELinux + # anchored - matching expression (pattern) is anchored + # bound - pattern can be tested using word boundary (e.g. False if contains in front some optional part) + # sdate - date string used in test log-line + # rdate - if specified, the result match, which differs from sdate + for anchored, bound, sdate, rdate in ( + (False, True, "Jan 23 21:59:59", None), + (False, False, "Sun Jan 23 21:59:59 2005", None), + (False, False, "Sun Jan 23 21:59:59", None), + (False, False, "Sun Jan 23 2005 21:59:59", None), + (False, True, "2005/01/23 21:59:59", None), + (False, True, "2005.01.23 21:59:59", None), + (False, True, "23/01/2005 21:59:59", None), + (False, True, "23/01/05 21:59:59", None), + (False, True, "23/Jan/2005:21:59:59", None), + (False, True, "23/Jan/2005:21:59:59 +0100", None), + (False, True, "01/23/2005:21:59:59", None), + (False, True, "2005-01-23 21:59:59", None), + (False, True, "2005-01-23 21:59:59,000", None), # proftpd + (False, True, "23-Jan-2005 21:59:59", None), + (False, True, "23-Jan-2005 21:59:59.02", None), + (False, True, "23-Jan-2005 21:59:59 +0100", None), + (False, True, "23-01-2005 21:59:59", None), + (True, True, "1106513999", None), # Portsetry + (False, True, "01-23-2005 21:59:59.252", None), # reported on f2b, causes Feb29 fix to break + (False, False, "@4000000041f4104f00000000", None), # TAI64N + (False, True, "2005-01-23T20:59:59.252Z", None), #ISO 8601 (UTC) + (False, True, "2005-01-23T15:59:59-05:00", None), #ISO 8601 with TZ + (False, True, "2005-01-23 21:59:59", None), #ISO 8601 no TZ, assume local + (False, True, "20050123T215959", None), #Short ISO with T + (False, True, "20050123 215959", None), #Short ISO with space + (True, True, "<01/23/05@21:59:59>", None), + (False, True, "050123 21:59:59", None), # MySQL + (True, True, "Jan-23-05 21:59:59", None), # ASSP like + (False, True, "Jan 23, 2005 9:59:59 PM", None), # Apache Tomcat + (True, True, "1106513999", None), # Regular epoch + (True, True, "1106513999.000", None), # Regular epoch with millisec + (True, True, "[1106513999.000]", "1106513999.000"), # epoch squared (brackets are not in match) + (False, True, "audit(1106513999.000:987)", "1106513999.000"), # SELinux + (True, True, "no date line", None), # no date in string + ): + if rdate is None and sdate != "no date line": rdate = sdate + logSys.debug('== test %r', (anchored, bound, sdate, rdate)) + for should_match, prefix in ( + (rdate is not None, ""), + (not anchored, "bogus-prefix "), + (False, "word-boundary") ): - for should_match, prefix in ((True, ""), - (not anchored, "bogus-prefix ")): log = prefix + sdate + "[sshd] error: PAM: Authentication failure" - + # if not allowed boundary test: + if not bound and prefix == "word-boundary": continue + logSys.debug(' -- test %-5s for %r', should_match, log) # with getTime: - logtime = self.__datedetector.getTime(log) + logtime = self.datedetector.getTime(log) if should_match: - self.assertNotEqual(logtime, None, "getTime retrieved nothing: failure for %s, anchored: %r, log: %s" % ( sdate, anchored, log)) + self.assertNotEqual(logtime, None, + "getTime retrieved nothing: failure for %s by prefix %r, anchored: %r, log: %s" % ( sdate, prefix, anchored, log)) ( logUnix, logMatch ) = logtime - self.assertEqual(logUnix, dateUnix, "getTime comparison failure for %s: \"%s\" is not \"%s\"" % (sdate, logUnix, dateUnix)) - if sdate.startswith('audit('): - # yes, special case, the group only matches the number - self.assertEqual(logMatch.group(), '1106513999.000') - else: - self.assertEqual(logMatch.group(), sdate) + self.assertEqual(logUnix, dateUnix, + "getTime comparison failure for %s: by prefix %r \"%s\" is not \"%s\"" % (sdate, prefix, logUnix, dateUnix)) + self.assertEqual(logMatch.group(1), rdate) else: - self.assertEqual(logtime, None, "getTime should have not matched for %r Got: %s" % (sdate, logtime)) + self.assertEqual(logtime, None, + "getTime should have not matched for %r by prefix %r Got: %s" % (sdate, prefix, logtime)) # with getTime(matchTime) - this combination used in filter: - matchTime = self.__datedetector.matchTime(log) - logtime = self.__datedetector.getTime(log, matchTime) + (timeMatch, template) = matchTime = self.datedetector.matchTime(log) + logtime = self.datedetector.getTime(log, matchTime) + logSys.debug(' -- found - %r', template.name if timeMatch else False) if should_match: - self.assertNotEqual(logtime, None, "getTime retrieved nothing: failure for %s, anchored: %r, log: %s" % ( sdate, anchored, log)) + self.assertNotEqual(logtime, None, + "getTime retrieved nothing: failure for %s by prefix %r, anchored: %r, log: %s" % ( sdate, prefix, anchored, log)) ( logUnix, logMatch ) = logtime - self.assertEqual(logUnix, dateUnix, "getTime comparison failure for %s: \"%s\" is not \"%s\"" % (sdate, logUnix, dateUnix)) - if sdate.startswith('audit('): - # yes, special case, the group only matches the number - self.assertEqual(logMatch.group(), '1106513999.000') - else: - self.assertEqual(logMatch.group(), sdate) + self.assertEqual(logUnix, dateUnix, + "getTime comparison failure for %s by prefix %r: \"%s\" is not \"%s\"" % (sdate, prefix, logUnix, dateUnix)) + self.assertEqual(logMatch.group(1), rdate) else: - self.assertEqual(logtime, None, "getTime should have not matched for %r Got: %s" % (sdate, logtime)) + self.assertEqual(logtime, None, + "getTime should have not matched for %r by prefix %r Got: %s" % (sdate, prefix, logtime)) + logSys.debug(' -- OK') def testAllUniqueTemplateNames(self): - self.assertRaises(ValueError, self.__datedetector.appendTemplate, - self.__datedetector.templates[0]) + self.assertRaises(ValueError, self.datedetector.appendTemplate, + self.datedetector.templates[0]) def testFullYearMatch_gh130(self): # see https://github.com/fail2ban/fail2ban/pull/130 # yoh: unfortunately this test is not really effective to reproduce the # situation but left in place to assure consistent behavior mu = time.mktime(datetime.datetime(2012, 10, 11, 2, 37, 17).timetuple()) - logdate = self.__datedetector.getTime('2012/10/11 02:37:17 [error] 18434#0') + logdate = self.datedetector.getTime('2012/10/11 02:37:17 [error] 18434#0') self.assertNotEqual(logdate, None) ( logTime, logMatch ) = logdate self.assertEqual(logTime, mu) - self.assertEqual(logMatch.group(), '2012/10/11 02:37:17') + self.assertEqual(logMatch.group(1), '2012/10/11 02:37:17') # confuse it with year being at the end for i in xrange(10): - ( logTime, logMatch ) = self.__datedetector.getTime('11/10/2012 02:37:17 [error] 18434#0') + ( logTime, logMatch ) = self.datedetector.getTime('11/10/2012 02:37:17 [error] 18434#0') self.assertEqual(logTime, mu) - self.assertEqual(logMatch.group(), '11/10/2012 02:37:17') + self.assertEqual(logMatch.group(1), '11/10/2012 02:37:17') # and now back to the original - ( logTime, logMatch ) = self.__datedetector.getTime('2012/10/11 02:37:17 [error] 18434#0') + ( logTime, logMatch ) = self.datedetector.getTime('2012/10/11 02:37:17 [error] 18434#0') self.assertEqual(logTime, mu) - self.assertEqual(logMatch.group(), '2012/10/11 02:37:17') + self.assertEqual(logMatch.group(1), '2012/10/11 02:37:17') def testDateTemplate(self): - t = DateTemplate() - t.setRegex('^a{3,5}b?c*$') - self.assertEqual(t.getRegex(), '^a{3,5}b?c*$') - self.assertRaises(Exception, t.getDate, '') - self.assertEqual(t.matchDate('aaaac').group(), 'aaaac') + t = DateTemplate() + t.setRegex('^a{3,5}b?c*$') + self.assertEqual(t.regex, '^(a{3,5}b?c*)$') + self.assertRaises(Exception, t.getDate, '') + self.assertEqual(t.matchDate('aaaac').group(1), 'aaaac') + + ## no word boundaries left and right: + t = DatePatternRegex() + t.pattern = '(?iu)**time:%ExY%Exm%ExdT%ExH%ExM%ExS**' + # ** was removed from end-regex: + self.assertFalse('**' in t.regex) + # match date: + dt = 'TIME:20050102T010203' + self.assertEqual(t.matchDate('X' + dt + 'X').group(1), dt) + self.assertEqual(t.matchDate(dt).group(1), dt) + # wrong year (for exact %ExY): + dt = 'TIME:50050102T010203' + self.assertFalse(t.matchDate(dt)) + + ## start boundary left and word boundary right (automatically if not **): + t = DatePatternRegex() + t.pattern = '{^LN-BEG}time:%ExY%Exm%ExdT%ExH%ExM%ExS' + self.assertTrue('^' in t.regex) + # try match date: + dt = 'time:20050102T010203' + self.assertFalse(t.matchDate('X' + dt)) + self.assertFalse(t.matchDate(dt + 'X')) + self.assertEqual(t.matchDate('##' + dt + '...').group(1), dt) + self.assertEqual(t.matchDate(dt).group(1), dt) + # case sensitive: + dt = 'TIME:20050102T010203' + self.assertFalse(t.matchDate(dt)) + + ## auto-switching "ignore case" and "unicode" + t = DatePatternRegex() + t.pattern = '^%Y %b %d' + self.assertTrue('(?iu)' in t.regex) + dt = '2005 jun 03'; self.assertEqual(t.matchDate(dt).group(1), dt) + dt = '2005 Jun 03'; self.assertEqual(t.matchDate(dt).group(1), dt) + dt = '2005 JUN 03'; self.assertEqual(t.matchDate(dt).group(1), dt) + + def testAmbiguousInOrderedTemplates(self): + dd = self.datedetector + for (debit, line, cnt) in ( + # shortest distance to datetime should win: + ("030324 0:03:59", "some free text 030324 0:03:59 -- 2003-03-07 17:05:01 ...", 1), + # some free text with datetime: + ("2003-03-07 17:05:01", "some free text 2003-03-07 17:05:01 test ...", 15), + # distance collision detection (date from foreign input should not be found): + ("030324 0:04:00", "server mysqld[1000]: 030324 0:04:00 [Warning] Access denied ..." + " foreign-input just some free text 2003-03-07 17:05:01 test", 10), + # distance collision detection (first date should be found): + ("Sep 16 21:30:26", "server mysqld[1020]: Sep 16 21:30:26 server mysqld: 030916 21:30:26 [Warning] Access denied", 15), + # just to test sorting: + ("2005-10-07 06:09:42", "server mysqld[5906]: 2005-10-07 06:09:42 5907 [Warning] Access denied", 20), + ("2005-10-08T15:26:18.237955", "server mysqld[5906]: 2005-10-08T15:26:18.237955 6 [Note] Access denied", 20), + # date format changed again: + ("051009 10:05:30", "server mysqld[1000]: 051009 10:05:30 [Warning] Access denied ...", 50), + ): + logSys.debug('== test: %r', (debit, line, cnt)) + for i in range(cnt): + logSys.debug('Line: %s', line) + match, template = dd.matchTime(line) + self.assertTrue(match) + self.assertEqual(match.group(1), debit) + + def testLowLevelLogging(self): + # test coverage for the deep (heavy) debug messages: + try: + self.__old_eff_level = datedetector.logLevel + if datedetector.logLevel < logSys.getEffectiveLevel()+1: + datedetector.logLevel = logSys.getEffectiveLevel()+1 + dd = self.datedetector + i = 0 + for (line, cnt) in ( + ("server mysqld[5906]: 2005-10-07 06:09:%02i 5907 [Warning] Access denied", 2), + ("server mysqld[5906]: 051007 06:10:%02i 5907 [Warning] Access denied", 5), + ("server mysqld[5906]: 2005-10-07 06:09:%02i 5907 [Warning] Access denied", 10), + ): + for i in range(i, i+cnt+1): + logSys.debug('== test: %r', (line % i, cnt)) + match, template = dd.matchTime(line % i) + self.assertTrue(match) + finally: + datedetector.logLevel = self.__old_eff_level + + def testWrongTemplate(self): + t = DatePatternRegex('(%ExY%Exm%Exd') + # lazy compiling used, so try match: + self.assertRaises(Exception, t.matchDate, '(20050101') + self.assertLogged("Compile %r failed" % t.name) + # abstract: + t = DateTemplate() + self.assertRaises(Exception, t.getDate, 'no date line') + + +iso8601 = DatePatternRegex("%Y-%m-%d[T ]%H:%M:%S(?:\.%f)?%z") + +class CustomDateFormatsTest(unittest.TestCase): + + def testIso8601(self): + date = datetime.datetime.utcfromtimestamp( + iso8601.getDate("2007-01-25T12:00:00Z")[0]) + self.assertEqual( + date, + datetime.datetime(2007, 1, 25, 12, 0)) + self.assertRaises(TypeError, iso8601.getDate, None) + self.assertRaises(TypeError, iso8601.getDate, date) + + self.assertEqual(iso8601.getDate(""), None) + self.assertEqual(iso8601.getDate("Z"), None) + + self.assertEqual(iso8601.getDate("2007-01-01T120:00:00Z"), None) + self.assertEqual(iso8601.getDate("2007-13-01T12:00:00Z"), None) + date = datetime.datetime.utcfromtimestamp( + iso8601.getDate("2007-01-25T12:00:00+0400")[0]) + self.assertEqual( + date, + datetime.datetime(2007, 1, 25, 8, 0)) + date = datetime.datetime.utcfromtimestamp( + iso8601.getDate("2007-01-25T12:00:00+04:00")[0]) + self.assertEqual( + date, + datetime.datetime(2007, 1, 25, 8, 0)) + date = datetime.datetime.utcfromtimestamp( + iso8601.getDate("2007-01-25T12:00:00-0400")[0]) + self.assertEqual( + date, + datetime.datetime(2007, 1, 25, 16, 0)) + date = datetime.datetime.utcfromtimestamp( + iso8601.getDate("2007-01-25T12:00:00-04")[0]) + self.assertEqual( + date, + datetime.datetime(2007, 1, 25, 16, 0)) + + def testAmbiguousDatePattern(self): + defDD = DateDetector() + defDD.addDefaultTemplate() + for (matched, dp, line) in ( + # positive case: + ('Jan 23 21:59:59', None, 'Test failure Jan 23 21:59:59 for 192.0.2.1'), + # ambiguous "unbound" patterns (missed): + (False, None, 'Test failure TestJan 23 21:59:59.011 2015 for 192.0.2.1'), + (False, None, 'Test failure Jan 23 21:59:59123456789 for 192.0.2.1'), + # ambiguous "no optional year" patterns (matched): + ('Aug 8 11:25:50', None, 'Aug 8 11:25:50 20030f2329b8 Authentication failed from 192.0.2.1'), + ('Aug 8 11:25:50', None, '[Aug 8 11:25:50] 20030f2329b8 Authentication failed from 192.0.2.1'), + ('Aug 8 11:25:50 2014', None, 'Aug 8 11:25:50 2014 20030f2329b8 Authentication failed from 192.0.2.1'), + # direct specified patterns: + ('20:00:00 01.02.2003', r'%H:%M:%S %d.%m.%Y$', '192.0.2.1 at 20:00:00 01.02.2003'), + ('[20:00:00 01.02.2003]', r'\[%H:%M:%S %d.%m.%Y\]', '192.0.2.1[20:00:00 01.02.2003]'), + ('[20:00:00 01.02.2003]', r'\[%H:%M:%S %d.%m.%Y\]', '[20:00:00 01.02.2003]192.0.2.1'), + ('[20:00:00 01.02.2003]', r'\[%H:%M:%S %d.%m.%Y\]$', '192.0.2.1[20:00:00 01.02.2003]'), + ('[20:00:00 01.02.2003]', r'^\[%H:%M:%S %d.%m.%Y\]', '[20:00:00 01.02.2003]192.0.2.1'), + ('[17/Jun/2011 17:00:45]', r'^\[%d/%b/%Y %H:%M:%S\]', '[17/Jun/2011 17:00:45] Attempt, IP address 192.0.2.1'), + ('[17/Jun/2011 17:00:45]', r'\[%d/%b/%Y %H:%M:%S\]', 'Attempt [17/Jun/2011 17:00:45] IP address 192.0.2.1'), + ('[17/Jun/2011 17:00:45]', r'\[%d/%b/%Y %H:%M:%S\]', 'Attempt IP address 192.0.2.1, date: [17/Jun/2011 17:00:45]'), + # direct specified patterns (begin/end, missed): + (False, r'%H:%M:%S %d.%m.%Y', '192.0.2.1x20:00:00 01.02.2003'), + (False, r'%H:%M:%S %d.%m.%Y', '20:00:00 01.02.2003x192.0.2.1'), + # direct specified unbound patterns (no begin/end boundary): + ('20:00:00 01.02.2003', r'**%H:%M:%S %d.%m.%Y**', '192.0.2.1x20:00:00 01.02.2003'), + ('20:00:00 01.02.2003', r'**%H:%M:%S %d.%m.%Y**', '20:00:00 01.02.2003x192.0.2.1'), + # pattern enclosed with stars (in comparison to example above): + ('*20:00:00 01.02.2003*', r'\**%H:%M:%S %d.%m.%Y\**', 'test*20:00:00 01.02.2003*test'), + # direct specified patterns (begin/end, matched): + ('20:00:00 01.02.2003', r'%H:%M:%S %d.%m.%Y', '192.0.2.1 20:00:00 01.02.2003'), + ('20:00:00 01.02.2003', r'%H:%M:%S %d.%m.%Y', '20:00:00 01.02.2003 192.0.2.1'), + # wrong year in 1st date, so failed by convert using not precise year (filter used last known date), + # in the 2nd and 3th tests (with precise year) it should find correct the 2nd date: + (None, r'%Y-%Exm-%Exd %ExH:%ExM:%ExS', "0000-12-30 00:00:00 - 2003-12-30 00:00:00"), + ('2003-12-30 00:00:00', r'%ExY-%Exm-%Exd %ExH:%ExM:%ExS', "0000-12-30 00:00:00 - 2003-12-30 00:00:00"), + ('2003-12-30 00:00:00', None, "0000-12-30 00:00:00 - 2003-12-30 00:00:00"), + # wrong date recognized short month/day (unbounded date pattern without separator between parts), + # in the 2nd and 3th tests (with precise month and day) it should find correct the 2nd date: + ('200333 010203', r'%Y%m%d %H%M%S', "text:200333 010203 | date:20031230 010203"), + ('20031230 010203', r'%ExY%Exm%Exd %ExH%ExM%ExS', "text:200333 010203 | date:20031230 010203"), + ('20031230 010203', None, "text:200333 010203 | date:20031230 010203"), + # Explicit bound in start of the line using {^LN-BEG} key, + # (negative) in the 1st case without line begin boundary - wrong date may be found, + # (positive) in the 2nd case with line begin boundary - unexpected date / log line (not found) + # (positive) and in 3th case with line begin boundary - find the correct date + ("20030101 000000", "%ExY%Exm%Exd %ExH%ExM%ExS", "00001230 010203 - 20030101 000000"), + (None, "{^LN-BEG}%ExY%Exm%Exd %ExH%ExM%ExS", "00001230 010203 - 20030101 000000"), + ("20031230 010203", "{^LN-BEG}%ExY%Exm%Exd %ExH%ExM%ExS", "20031230 010203 - 20030101 000000"), + # Explicit bound in start of the line using {^LN-BEG} key, + # up to 2 non-alphanumeric chars front, ** - no word boundary on the right + ("20031230010203", "{^LN-BEG}%ExY%Exm%Exd%ExH%ExM%ExS**", "2003123001020320030101000000"), + ("20031230010203", "{^LN-BEG}%ExY%Exm%Exd%ExH%ExM%ExS**", "#2003123001020320030101000000"), + ("20031230010203", "{^LN-BEG}%ExY%Exm%Exd%ExH%ExM%ExS**", "##2003123001020320030101000000"), + ("20031230010203", "{^LN-BEG}%ExY%Exm%Exd%ExH%ExM%ExS", "[20031230010203]20030101000000"), + # UTC/GMT time zone offset (with %z and %Z): + (1072746123.0 - 3600, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03] server ..."), + (1072746123.0 - 3600, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03] server ..."), + (1072746123.0, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %z)?", "[2003-12-30 01:02:03 UTC] server ..."), + (1072746123.0, "{^LN-BEG}%ExY-%Exm-%Exd %ExH:%ExM:%ExS(?: %Z)?", "[2003-12-30 01:02:03 UTC] server ..."), + ): + logSys.debug('== test: %r', (matched, dp, line)) + if dp is None: + dd = defDD + else: + dd = DateDetector() + dd.appendTemplate(dp) + date = dd.getTime(line) + if matched: + self.assertTrue(date) + if isinstance(matched, basestring): + self.assertEqual(matched, date[1].group(1)) + else: + self.assertEqual(matched, date[0]) + else: + self.assertEqual(date, None) # def testDefaultTempate(self): diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index e212ff23..b8417be5 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -138,8 +138,8 @@ def _start_params(tmp, use_stock=False, logtarget="/dev/null", db=":memory:"): """Filters list of 'files' to contain only directories (under dir)""" return [f for f in files if isdir(pjoin(dir, f))] shutil.copytree(STOCK_CONF_DIR, cfg, ignore=ig_dirs) - os.symlink(pjoin(STOCK_CONF_DIR, "action.d"), pjoin(cfg, "action.d")) - os.symlink(pjoin(STOCK_CONF_DIR, "filter.d"), pjoin(cfg, "filter.d")) + os.symlink(os.path.abspath(pjoin(STOCK_CONF_DIR, "action.d")), pjoin(cfg, "action.d")) + os.symlink(os.path.abspath(pjoin(STOCK_CONF_DIR, "filter.d")), pjoin(cfg, "filter.d")) # replace fail2ban params (database with memory): r = re.compile(r'^dbfile\s*=') for line in fileinput.input(pjoin(cfg, "fail2ban.conf"), inplace=True): @@ -424,7 +424,7 @@ class Fail2banClientTest(Fail2banClientServerBase): self.execSuccess(startparams, "-vvd") self.assertLogged("Loading files") self.assertLogged("logtarget") - + @with_tmpdir @with_kill_srv def testClientStartBackgroundInside(self, tmp): @@ -675,6 +675,36 @@ class Fail2banServerTest(Fail2banClientServerBase): self.pruneLog() os.remove(pjoin(tmp, "f2b.sock")) + @with_tmpdir + @with_kill_srv + def testServerTestFailStart(self, tmp): + # started directly here, so prevent overwrite test cases logger with "INHERITED" + startparams = _start_params(tmp, logtarget="INHERITED") + cfg = pjoin(tmp, "config") + + # test configuration is correct: + self.pruneLog("[test-phase 0]") + self.execSuccess(startparams, "--test") + self.assertLogged("OK: configuration test is successful") + + # append one wrong configured jail: + _write_file(pjoin(cfg, "jail.conf"), "a", "", "[broken-jail]", + "", "filter = broken-jail-filter", "enabled = true") + + # first try test config: + self.pruneLog("[test-phase 0a]") + self.execFailed(startparams, "--test") + self.assertLogged("Unable to read the filter 'broken-jail-filter'", + "Errors in jail 'broken-jail'.", + "ERROR: test configuration failed", all=True) + + # failed to start with test config: + self.pruneLog("[test-phase 0b]") + self.execFailed(startparams, "-t", "start") + self.assertLogged("Unable to read the filter 'broken-jail-filter'", + "Errors in jail 'broken-jail'.", + "ERROR: test configuration failed", all=True) + @with_tmpdir def testKillAfterStart(self, tmp): try: @@ -725,12 +755,17 @@ class Fail2banServerTest(Fail2banClientServerBase): os.remove(fn) return _write_file(fn, "w", + "[DEFAULT]", + "_exec_once = 0", + "", "[Definition]", - "actionstart = echo '[] %s: ** start'" % actname, start, - "actionreload = echo '[] %s: .. reload'" % actname, reload, - "actionban = echo '[] %s: ++ ban '" % actname, ban, - "actionunban = echo '[] %s: -- unban '" % actname, unban, - "actionstop = echo '[] %s: __ stop'" % actname, stop, + "norestored = %(_exec_once)s", + "restore = ", + "actionstart = echo '[%(name)s] %(actname)s: ** start'", start, + "actionreload = echo '[%(name)s] %(actname)s: .. reload'", reload, + "actionban = echo '[%(name)s] %(actname)s: ++ ban %(restore)s'", ban, + "actionunban = echo '[%(name)s] %(actname)s: -- unban '", unban, + "actionstop = echo '[%(name)s] %(actname)s: __ stop'", stop, ) if unittest.F2B.log_level <= logging.DEBUG: # pragma: no cover _out_file(fn) @@ -743,20 +778,30 @@ class Fail2banServerTest(Fail2banClientServerBase): "maxretry = 3", "findtime = 10m", "failregex = ^\s*failure (401|403) from ", + "datepattern = {^LN-BEG}EPOCH", "", "[test-jail1]", "backend = " + backend, "filter =", "action = ", - " test-action1[name='%(__name__)s']" if 1 in actions else "", - " test-action2[name='%(__name__)s']" if 2 in actions else "", + " test-action1[name='%(__name__)s']" \ + if 1 in actions else "", + " test-action2[name='%(__name__)s', restore='restored: ']" \ + if 2 in actions else "", + " test-action2[name='%(__name__)s', actname=test-action3, _exec_once=1, restore='restored: ']" \ + if 3 in actions else "", "logpath = " + test1log, " " + test2log if 2 in enabled else "", " " + test3log if 2 in enabled else "", "failregex = ^\s*failure (401|403) from ", - " ^\s*error (401|403) from " if 2 in enabled else "", + " ^\s*error (401|403) from " \ + if 2 in enabled else "", "enabled = true" if 1 in enabled else "", "", "[test-jail2]", "backend = " + backend, "filter =", - "action =", + "action = ", + " test-action2[name='%(__name__)s', restore='restored: ']" \ + if 2 in actions else "", + " test-action2[name='%(__name__)s', actname=test-action3, _exec_once=1, restore='restored: ']" \ + if 3 in actions else "", "logpath = " + test2log, "enabled = true" if 2 in enabled else "", ) @@ -767,7 +812,11 @@ class Fail2banServerTest(Fail2banClientServerBase): _write_action_cfg(actname="test-action1") _write_action_cfg(actname="test-action2") - _write_jail_cfg(enabled=[1], actions=[1,2]) + _write_jail_cfg(enabled=[1], actions=[1,2,3]) + # append one wrong configured jail: + _write_file(pjoin(cfg, "jail.conf"), "a", "", "[broken-jail]", + "", "filter = broken-jail-filter", "enabled = true") + _write_file(test1log, "w", *((str(int(MyTime.time())) + " failure 401 from 192.0.2.1: test 1",) * 3)) _write_file(test2log, "w") _write_file(test3log, "w") @@ -786,6 +835,17 @@ class Fail2banServerTest(Fail2banClientServerBase): self.assertLogged( "stdout: '[test-jail1] test-action1: ** start'", "stdout: '[test-jail1] test-action2: ** start'", all=True) + # test restored is 0 (both actions available): + self.assertLogged( + "stdout: '[test-jail1] test-action2: ++ ban 192.0.2.1 restored: 0'", + "stdout: '[test-jail1] test-action3: ++ ban 192.0.2.1 restored: 0'", + all=True, wait=MID_WAITTIME) + + # broken jail was logged (in client and server log): + self.assertLogged( + "Unable to read the filter 'broken-jail-filter'", + "Errors in jail 'broken-jail'. Skipping...", + "Jail 'broken-jail' skipped, because of wrong configuration", all=True) # enable both jails, 3 logs for jail1, etc... # truncate test-log - we should not find unban/ban again by reload: @@ -841,10 +901,10 @@ class Fail2banServerTest(Fail2banClientServerBase): self.assertNotLogged( "stdout: '[test-jail1] test-action1: -- unban 192.0.2.1'") - # don't need both actions anymore: + # don't need action1 anymore: _write_action_cfg(actname="test-action1", allow=False) - _write_action_cfg(actname="test-action2", allow=False) - _write_jail_cfg(actions=[]) + # leave action2 just to test restored interpolation: + _write_jail_cfg(actions=[2,3]) # write new failures: self.pruneLog("[test-phase 2b]") @@ -872,7 +932,8 @@ class Fail2banServerTest(Fail2banClientServerBase): "[test-jail2] Found 192.0.2.2", "[test-jail2] Ban 192.0.2.2", "[test-jail2] Found 192.0.2.3", - "[test-jail2] Ban 192.0.2.3", all=True) + "[test-jail2] Ban 192.0.2.3", + all=True) # rotate logs: _write_file(test1log, "w+") @@ -895,6 +956,20 @@ class Fail2banServerTest(Fail2banClientServerBase): "[test-jail2] Restore Ban 192.0.2.4", "[test-jail2] Restore Ban 192.0.2.8", all=True ) + # test restored is 1 (only test-action2): + self.assertLogged( + "stdout: '[test-jail2] test-action2: ++ ban 192.0.2.4 restored: 1'", + "stdout: '[test-jail2] test-action2: ++ ban 192.0.2.8 restored: 1'", + all=True, wait=MID_WAITTIME) + # test test-action3 not executed at all (norestored check): + self.assertNotLogged( + "stdout: '[test-jail2] test-action3: ++ ban 192.0.2.4 restored: 1'", + "stdout: '[test-jail2] test-action3: ++ ban 192.0.2.8 restored: 1'", + all=True) + + # don't need actions anymore: + _write_action_cfg(actname="test-action2", allow=False) + _write_jail_cfg(actions=[]) # restart jail with unban all: self.pruneLog("[test-phase 2d]") diff --git a/fail2ban/tests/fail2banregextestcase.py b/fail2ban/tests/fail2banregextestcase.py index 4445fe6b..5767a3f7 100644 --- a/fail2ban/tests/fail2banregextestcase.py +++ b/fail2ban/tests/fail2banregextestcase.py @@ -101,6 +101,7 @@ class Fail2banRegexTest(LogCaptureTestCase): def testWrongIngnoreRE(self): (opts, args, fail2banRegex) = _Fail2banRegex( + "--datepattern", "{^LN-BEG}EPOCH", "test", r".*? from $", r".**" ) self.assertFalse(fail2banRegex.start(opts, args)) @@ -108,6 +109,7 @@ class Fail2banRegexTest(LogCaptureTestCase): def testDirectFound(self): (opts, args, fail2banRegex) = _Fail2banRegex( + "--datepattern", "^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", "--print-all-matched", "--print-no-missed", "Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 192.0.2.0", r"Authentication failure for .*? from $" @@ -136,6 +138,7 @@ class Fail2banRegexTest(LogCaptureTestCase): def testDirectRE_1(self): (opts, args, fail2banRegex) = _Fail2banRegex( + "--datepattern", "^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", "--print-all-matched", Fail2banRegexTest.FILENAME_01, Fail2banRegexTest.RE_00 @@ -151,6 +154,7 @@ class Fail2banRegexTest(LogCaptureTestCase): def testDirectRE_1raw(self): (opts, args, fail2banRegex) = _Fail2banRegex( + "--datepattern", "^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", "--print-all-matched", "--raw", Fail2banRegexTest.FILENAME_01, Fail2banRegexTest.RE_00 @@ -160,6 +164,7 @@ class Fail2banRegexTest(LogCaptureTestCase): def testDirectRE_1raw_noDns(self): (opts, args, fail2banRegex) = _Fail2banRegex( + "--datepattern", "^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", "--print-all-matched", "--raw", "--usedns=no", Fail2banRegexTest.FILENAME_01, Fail2banRegexTest.RE_00 @@ -169,6 +174,7 @@ class Fail2banRegexTest(LogCaptureTestCase): def testDirectRE_2(self): (opts, args, fail2banRegex) = _Fail2banRegex( + "--datepattern", "^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", "--print-all-matched", Fail2banRegexTest.FILENAME_02, Fail2banRegexTest.RE_00 @@ -178,7 +184,8 @@ class Fail2banRegexTest(LogCaptureTestCase): def testVerbose(self): (opts, args, fail2banRegex) = _Fail2banRegex( - "--verbose", "--print-no-missed", + "--datepattern", "^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", + "--verbose", "--verbose-date", "--print-no-missed", Fail2banRegexTest.FILENAME_02, Fail2banRegexTest.RE_00 ) @@ -190,6 +197,7 @@ class Fail2banRegexTest(LogCaptureTestCase): def testWronChar(self): (opts, args, fail2banRegex) = _Fail2banRegex( + "--datepattern", "^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", Fail2banRegexTest.FILENAME_WRONGCHAR, Fail2banRegexTest.FILTER_SSHD ) self.assertTrue(fail2banRegex.start(opts, args)) @@ -203,6 +211,7 @@ class Fail2banRegexTest(LogCaptureTestCase): def testWronCharDebuggex(self): (opts, args, fail2banRegex) = _Fail2banRegex( + "--datepattern", "^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?", "--debuggex", "--print-all-matched", Fail2banRegexTest.FILENAME_WRONGCHAR, Fail2banRegexTest.FILTER_SSHD ) diff --git a/fail2ban/tests/failmanagertestcase.py b/fail2ban/tests/failmanagertestcase.py index 11caf927..9de4ced6 100644 --- a/fail2ban/tests/failmanagertestcase.py +++ b/fail2ban/tests/failmanagertestcase.py @@ -36,6 +36,7 @@ class AddFailure(unittest.TestCase): def setUp(self): """Call before every test case.""" + super(AddFailure, self).setUp() self.__items = None self.__failManager = FailManager() diff --git a/fail2ban/tests/files/filter.d/testcase01.conf b/fail2ban/tests/files/filter.d/testcase01.conf index 0dbbbd2a..b3700d0b 100644 --- a/fail2ban/tests/files/filter.d/testcase01.conf +++ b/fail2ban/tests/files/filter.d/testcase01.conf @@ -33,7 +33,6 @@ failregex = ^%(__prefix_line)s(?:error: PAM: )?Authentication failure for .* fro # ignoreregex = ^.+ john from host 192.168.1.1\s*$ -[Init] # "maxlines" is number of log lines to buffer for multi-line regex searches maxlines = 1 diff --git a/fail2ban/tests/files/ignorecommand.py b/fail2ban/tests/files/ignorecommand.py index 7011b51b..da48ff91 100755 --- a/fail2ban/tests/files/ignorecommand.py +++ b/fail2ban/tests/files/ignorecommand.py @@ -1,5 +1,8 @@ #!/usr/bin/env fail2ban-python import sys +if len(sys.argv) != 2 or sys.argv[1] == "": + sys.stderr.write('usage: ignorecommand IP') + exit(10) if sys.argv[1] == "10.0.0.1": exit(0) exit(1) diff --git a/fail2ban/tests/files/logs/apache-modsecurity b/fail2ban/tests/files/logs/apache-modsecurity index d46d8ab4..3ca2e074 100644 --- a/fail2ban/tests/files/logs/apache-modsecurity +++ b/fail2ban/tests/files/logs/apache-modsecurity @@ -1,5 +1,5 @@ # failJSON: { "time": "2013-12-23T13:12:31", "match": true , "host": "173.255.225.101" } [Mon Dec 23 13:12:31 2013] [error] [client 173.255.225.101] ModSecurity: [file "/etc/httpd/modsecurity.d/activated_rules/modsecurity_crs_21_protocol_anomalies.conf"] [line "47"] [id "960015"] [rev "1"] [msg "Request Missing an Accept Header"] [severity "NOTICE"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/MISSING_HEADER_ACCEPT"] [tag "WASCTC/WASC-21"][tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"] Access denied with code 403 (phase 2). Operator EQ matched 0 at REQUEST_HEADERS. [hostname "www.mysite.net"] [uri "/"] [unique_id "Urf@f12qgHIAACrFOlgAAABA"] -# failJSON: { "time": "2013-12-28T09:18:05", "match": true , "host": "32.65.254.69" } -[Sat Dec 28 09:18:05 2013] [error] [client 32.65.254.69] ModSecurity: [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "635"] [id "340069"] [rev "4"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Web vulnerability scanner"] [severity "CRITICAL"] Access denied with code 403 (phase 2). Pattern match "(?:nessus(?:_is_probing_you_|test)|^/w00tw00t\\\\.at\\\\.)" at REQUEST_URI. [hostname "192.81.249.191"] [uri "/w00tw00t.at.blackhats.romanian.anti-sec:)"] [unique_id "4Q6RdsBR@b4AAA65LRUAAAAA"] +# failJSON: { "time": "2013-12-28T09:18:05", "match": true , "host": "32.65.254.69", "desc": "additional entry (and exact one space)" } +[Sat Dec 28 09:18:05 2013] [error] [client 32.65.254.69] ModSecurity: [file "/etc/httpd/modsecurity.d/10_asl_rules.conf"] [line "635"] [id "340069"] [rev "4"] [msg "Atomicorp.com UNSUPPORTED DELAYED Rules: Web vulnerability scanner"] [severity "CRITICAL"] Access denied with code 403 (phase 2). Pattern match "(?:nessus(?:_is_probing_you_|test)|^/w00tw00t\\\\.at\\\\.)" at REQUEST_URI. [hostname "192.81.249.191"] [uri "/w00tw00t.at.blackhats.romanian.anti-sec:)"] [unique_id "4Q6RdsBR@b4AAA65LRUAAAAA"] diff --git a/fail2ban/tests/files/logs/domino-smtp b/fail2ban/tests/files/logs/domino-smtp new file mode 100644 index 00000000..4987e7ea --- /dev/null +++ b/fail2ban/tests/files/logs/domino-smtp @@ -0,0 +1,8 @@ +# failJSON: { "time": "2005-07-03T23:07:20", "match": true , "host": "1.2.3.4" } +03-07-2005 23:07:20 SMTP Server: Authentication failed for user postmaster ; connecting host 1.2.3.4 +# failJSON: { "time": "2014-06-22T09:56:12", "match": true , "host": "1.2.3.4" } +[28325:00010-3735542592] 22-06-2014 09:56:12 smtp: postmaster [1.2.3.4] authentication failure using internet password +# failJSON: { "time": "2014-09-08T06:14:27", "match": true , "host": "1.2.3.4" } +08-09-2014 06:14:27 smtp: postmaster [1.2.3.4] authentication failure using internet password +# failJSON: { "time": "2016-11-07T22:21:20", "match": true , "host": "1.2.3.4" } +2016-11-07 22:21:20 smtp: postmaster [1.2.3.4] authentication failure using internet password diff --git a/fail2ban/tests/files/logs/dovecot b/fail2ban/tests/files/logs/dovecot index 627b8dc8..987f89f1 100644 --- a/fail2ban/tests/files/logs/dovecot +++ b/fail2ban/tests/files/logs/dovecot @@ -73,3 +73,8 @@ Jul 02 13:49:32 hostname dovecot[442]: pop3-login: Disconnected (no auth attempt # failJSON: { "time": "2005-03-23T06:10:52", "match": true , "host": "52.37.139.121" } Mar 23 06:10:52 auth: Info: ldap(dog,52.37.139.121,): invalid credentials + +# failJSON: { "time": "2005-07-26T11:11:21", "match": true , "host": "192.0.2.1" } +Jul 26 11:11:21 hostname dovecot: imap-login: Disconnected: Too many invalid commands (tried to use disallowed plaintext auth): user=, rip=192.0.2.1, lip=192.168.1.1, session= +# failJSON: { "time": "2005-07-26T11:12:19", "match": true , "host": "192.0.2.2" } +Jul 26 11:12:19 hostname dovecot: imap-login: Disconnected: Too many invalid commands (auth failed, 1 attempts in 17 secs): user=, method=PLAIN, rip=192.0.2.2, lip=192.168.1.1, TLS, session= diff --git a/fail2ban/tests/files/logs/freeswitch b/fail2ban/tests/files/logs/freeswitch index f151995c..2579e6f4 100644 --- a/fail2ban/tests/files/logs/freeswitch +++ b/fail2ban/tests/files/logs/freeswitch @@ -9,3 +9,8 @@ 2013-12-31 17:39:54.767815 [WARNING] sofia_reg.c:2531 Can't find user [1001@192.168.2.51] from 5.11.47.236 # failJSON: { "time": "2013-12-31T17:39:54", "match": true, "host": "185.24.234.141" } 2013-12-31 17:39:54.767815 [WARNING] sofia_reg.c:2531 Can't find user [100@192.168.2.51] from 185.24.234.141 + +# failJSON: { "time": "2016-09-25T18:57:58", "match": true, "host": "192.0.2.1", "desc": "Systemd dual time with prefix - 1st expr" } +2016-09-25T18:57:58.150982 www.srv.tld freeswitch[122921]: 2016-09-25 18:57:58.150982 [WARNING] sofia_reg.c:2889 Can't find user [201@::1] from 192.0.2.1 +# failJSON: { "time": "2016-09-25T18:57:58", "match": true, "host": "192.0.2.2", "desc": "Systemd dual time with prefix - 2nd expr" } +2016-09-25T18:57:58.150982 www.srv.tld freeswitch[122921]: 2016-09-25 18:57:58.150982 [WARNING] sofia_reg.c:1720 SIP auth failure (INVITE) on sofia profile 'sipinterface_1' for [9810972597751739@::1] from ip 192.0.2.2 \ No newline at end of file diff --git a/fail2ban/tests/files/logs/mongodb-auth b/fail2ban/tests/files/logs/mongodb-auth new file mode 100644 index 00000000..8a308892 --- /dev/null +++ b/fail2ban/tests/files/logs/mongodb-auth @@ -0,0 +1,30 @@ +# failJSON: { "match": false } +2016-11-20T00:04:00.110+0100 [conn1] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@admin +# failJSON: { "time": "2016-11-20T00:04:00", "match": true , "host": "192.0.2.35" } +2016-11-20T00:04:00.111+0100 [conn1] end connection 192.0.2.35:53276 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T00:24:00.110+0100 [conn5] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed UserNotFound Could not find user root@admin +# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.171" } +2016-11-20T00:24:00.111+0100 [conn5] end connection 192.0.2.171:53276 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T00:24:00.110+0100 [conn334] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch +# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.176" } +2016-11-20T00:24:00.111+0100 [conn334] end connection 192.0.2.176:53276 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T00:24:00.110+0100 [conn56] Failed to authenticate root@admin with mechanism MONGODB-CR: AuthenticationFailed key mismatch +# failJSON: { "time": "2016-11-20T00:24:00", "match": true , "host": "192.0.2.1" } +2016-11-20T00:24:00.111+0100 [conn56] end connection 192.0.2.1:53276 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T12:54:02.370+0100 [initandlisten] connection accepted from 127.0.0.1:58774 #2261 (1 connection now open) +# failJSON: { "match": false } +2016-11-20T12:54:02.370+0100 [conn2261] end connection 127.0.0.1:58774 (0 connections now open) + +# failJSON: { "match": false } +2016-11-20T13:07:49.781+0100 [conn2271] authenticate db: admin { authenticate: 1, nonce: "xxx", user: "root", key: "xxx" } +# failJSON: { "time": "2016-11-20T13:07:49", "match": false , "host": "192.0.2.178" } +2016-11-20T13:07:49.834+0100 [conn2271] end connection 192.0.2.178:60268 (3 connections now open) + diff --git a/fail2ban/tests/files/logs/sendmail-reject b/fail2ban/tests/files/logs/sendmail-reject index b326cf43..70d4dde6 100644 --- a/fail2ban/tests/files/logs/sendmail-reject +++ b/fail2ban/tests/files/logs/sendmail-reject @@ -40,6 +40,8 @@ Feb 19 18:01:50 batman sm-mta[78152]: ruleset=check_relay, arg1=[196.213.73.146] # failJSON: { "time": "2005-02-27T10:53:06", "match": true , "host": "209.15.212.253" } Feb 27 10:53:06 batman sm-mta[44307]: s1R9r60D044307: rejecting commands from [209.15.212.253] due to pre-greeting traffic after 0 seconds +# failJSON: { "time": "2005-02-27T10:53:07", "match": true , "host": "1.2.3.4" } +Feb 27 10:53:07 strange sm-mta[18001]: u9A0GtpL018001: rejecting commands from example.com [1.2.3.4] due to pre-greeting traffic after 6 seconds # failJSON: { "time": "2005-02-27T15:44:18", "match": true , "host": "41.204.78.137" } Feb 27 15:44:18 batman sm-mta[87838]: s1REiHdq087838: ruleset=check_rcpt, arg1=, relay=[41.204.78.137], reject=550 5.7.1 ... Relaying denied. IP name lookup failed [41.204.78.137] diff --git a/fail2ban/tests/files/logs/sshd b/fail2ban/tests/files/logs/sshd index 89e4a3e2..e2c5f537 100644 --- a/fail2ban/tests/files/logs/sshd +++ b/fail2ban/tests/files/logs/sshd @@ -9,8 +9,14 @@ May 29 20:56:56 imago sshd[28732]: error: PAM: Authentication failure for test-i #2 # failJSON: { "time": "2005-02-25T14:34:10", "match": true , "host": "194.117.26.69" } Feb 25 14:34:10 belka sshd[31602]: Failed password for invalid user ROOT from 194.117.26.69 port 50273 ssh2 +# failJSON: { "time": "2005-02-25T14:34:10", "match": true , "host": "aaaa:bbbb:cccc:1234::1:1" } +Feb 25 14:34:10 belka sshd[31603]: Failed password for invalid user ROOT from aaaa:bbbb:cccc:1234::1:1 port 50273 ssh2 # failJSON: { "time": "2005-02-25T14:34:10", "match": true , "host": "194.117.26.70" } Feb 25 14:34:10 belka sshd[31602]: Failed password for invalid user ROOT from 194.117.26.70 port 12345 +# failJSON: { "time": "2005-02-25T14:34:10", "match": true , "host": "aaaa:bbbb:cccc:1234::1:1" } +Feb 25 14:34:10 belka sshd[31603]: Failed password for invalid user ROOT from aaaa:bbbb:cccc:1234::1:1 port 12345 +# failJSON: { "time": "2005-02-25T14:34:11", "match": true , "host": "aaaa:bbbb:cccc:1234::1:1" } +Feb 25 14:34:11 belka sshd[31603]: Failed password for invalid user ROOT from aaaa:bbbb:cccc:1234::1:1 #3 # failJSON: { "time": "2005-01-05T01:31:41", "match": true , "host": "1.2.3.4" } @@ -19,8 +25,10 @@ Jan 5 01:31:41 www sshd[1643]: ROOT LOGIN REFUSED FROM 1.2.3.4 Jan 5 01:31:41 www sshd[1643]: ROOT LOGIN REFUSED FROM ::ffff:1.2.3.4 #4 -# failJSON: { "time": "2005-07-20T14:42:11", "match": true , "host": "211.114.51.213" } -Jul 20 14:42:11 localhost sshd[22708]: Invalid user ftp from 211.114.51.213 +# failJSON: { "time": "2005-07-20T14:42:11", "match": true , "host": "192.0.2.1", "desc": "Invalid user" } +Jul 20 14:42:11 localhost sshd[22708]: Invalid user ftp from 192.0.2.1 +# failJSON: { "time": "2005-07-20T14:42:12", "match": true , "host": "192.0.2.2", "desc": "Invalid user with port" } +Jul 20 14:42:12 localhost sshd[22708]: Invalid user ftp from 192.0.2.2 port 37220 #5 new filter introduced after looking at 44087D8C.9090407@bluewin.ch # yoh: added ':' after [sshd] since the case without is not really common any more @@ -116,10 +124,20 @@ Sep 29 17:15:02 spaceman sshd[12946]: Failed password for user from 127.0.0.1 po # failJSON: { "time": "2004-09-29T17:15:02", "match": true , "host": "127.0.0.1", "desc": "Injecting while exhausting initially present {0,100} match length limits set for ruser etc" } Sep 29 17:15:02 spaceman sshd[12946]: Failed password for user from 127.0.0.1 port 20000 ssh1: ruser XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX from 1.2.3.4 +# failJSON: { "time": "2004-09-29T17:15:03", "match": true , "host": "aaaa:bbbb:cccc:1234::1:1", "desc": "Injecting while exhausting initially present {0,100} match length limits set for ruser etc" } +Sep 29 17:15:03 spaceman sshd[12946]: Failed password for user from aaaa:bbbb:cccc:1234::1:1 port 20000 ssh1: ruser XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX from 1.2.3.4 # failJSON: { "time": "2004-11-11T08:04:51", "match": true , "host": "127.0.0.1", "desc": "Injecting on username ssh 'from 10.10.1.1'@localhost" } Nov 11 08:04:51 redbamboo sshd[2737]: Failed password for invalid user from 10.10.1.1 from 127.0.0.1 port 58946 ssh2 +# failJSON: { "time": "2004-11-11T08:04:52", "match": true , "host": "127.0.0.1", "desc": "More complex injecting on username ssh 'test from 10.10.1.2 port 55555 ssh2'@localhost" } +Nov 11 08:04:52 redbamboo sshd[2737]: Failed password for invalid user test from 10.10.1.2 port 55555 ssh2 from 127.0.0.1 port 58946 ssh2 +# failJSON: { "time": "2004-11-11T08:04:52", "match": true , "host": "127.0.0.1", "desc": "More complex injecting on auth-info ssh test@localhost, auth-info: ' from 10.10.1.2 port 55555 ssh2'" } +Nov 11 08:04:52 redbamboo sshd[2737]: Failed password for invalid user test from 127.0.0.1 port 58946 ssh2: from 10.10.1.2 port 55555 ssh2 +# failJSON: { "time": "2005-07-05T18:22:44", "match": true , "host": "127.0.0.1", "desc": "Failed publickey for ..." } +Jul 05 18:22:44 mercury sshd[4669]: Failed publickey for graysky from 127.0.0.1 port 37954 ssh2: RSA SHA256:v3dpapGleDaUKf$4V1vKyR9ZyUgjaJAmoCTcb2PLljI +# failJSON: { "time": "2005-07-05T18:22:45", "match": true , "host": "aaaa:bbbb:cccc:1234::1:1", "desc": "Failed publickey for ..." } +Jul 05 18:22:45 mercury sshd[4670]: Failed publickey for graysky from aaaa:bbbb:cccc:1234::1:1 port 37955 ssh2: RSA SHA256:v3dpapGleDaUKf$4V1vKyR9ZyUgjaJAmoCTcb2PLljI # failJSON: { "match": false } Nov 23 21:50:19 sshd[8148]: Disconnecting: Too many authentication failures for root [preauth] @@ -133,6 +151,8 @@ Nov 23 21:50:37 sshd[7148]: Connection closed by 61.0.0.1 [preauth] # failJSON: { "time": "2005-07-13T18:44:28", "match": true , "host": "89.24.13.192", "desc": "from gh-289" } Jul 13 18:44:28 mdop sshd[4931]: Received disconnect from 89.24.13.192: 3: com.jcraft.jsch.JSchException: Auth fail +# failJSON: { "time": "2005-01-02T01:18:41", "match": true , "host": "10.0.0.1", "desc": "space after port is optional (gh-1652)" } +Jan 2 01:18:41 host sshd[11808]: error: Received disconnect from 10.0.0.1 port 7736:3: com.jcraft.jsch.JSchException: Auth fail [preauth] # failJSON: { "time": "2004-10-01T17:27:44", "match": true , "host": "94.249.236.6", "desc": "newer format per commit 36919d9f" } Oct 1 17:27:44 localhost sshd[24077]: error: Received disconnect from 94.249.236.6: 3: com.jcraft.jsch.JSchException: Auth fail [preauth] @@ -163,4 +183,3 @@ Apr 27 13:02:04 host sshd[29116]: Received disconnect from 1.2.3.4: 11: Normal S # Match sshd auth errors on OpenSUSE systems # failJSON: { "time": "2015-04-16T20:02:50", "match": true , "host": "222.186.21.217", "desc": "Authentication for user failed" } 2015-04-16T18:02:50.321974+00:00 host sshd[2716]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=222.186.21.217 user=root - diff --git a/fail2ban/tests/files/logs/sshd-aggressive b/fail2ban/tests/files/logs/sshd-aggressive new file mode 100644 index 00000000..5b4d3a12 --- /dev/null +++ b/fail2ban/tests/files/logs/sshd-aggressive @@ -0,0 +1,3 @@ +# sshd-aggressive includes sshd and sshd-ddos failregex's: +# addFILE: "sshd" +# addFILE: "sshd-ddos" \ No newline at end of file diff --git a/fail2ban/tests/files/logs/sshd-ddos b/fail2ban/tests/files/logs/sshd-ddos index 452abbde..f0a787a1 100644 --- a/fail2ban/tests/files/logs/sshd-ddos +++ b/fail2ban/tests/files/logs/sshd-ddos @@ -1,3 +1,41 @@ # http://forums.powervps.com/showthread.php?t=1667 # failJSON: { "time": "2005-06-07T01:10:56", "match": true , "host": "69.61.56.114" } Jun 7 01:10:56 host sshd[5937]: Did not receive identification string from 69.61.56.114 + +# gh-864(1): +# failJSON: { "match": false } +Nov 24 23:46:39 host sshd[32686]: SSH: Server;Ltype: Version;Remote: 127.0.0.1-1780;Protocol: 2.0;Client: libssh2_1.4.3 +# failJSON: { "time": "2004-11-24T23:46:43", "match": true , "host": "127.0.0.1", "desc": "Multiline for connection reset by peer (1)" } +Nov 24 23:46:43 host sshd[32686]: fatal: Read from socket failed: Connection reset by peer [preauth] + +# gh-864(2): +# failJSON: { "match": false } +Nov 24 23:46:40 host sshd[32686]: SSH: Server;Ltype: Kex;Remote: 127.0.0.1-1780;Enc: aes128-ctr;MAC: hmac-sha1;Comp: none [preauth] +# failJSON: { "time": "2004-11-24T23:46:43", "match": true , "host": "127.0.0.1", "desc": "Multiline for connection reset by peer (2)" } +Nov 24 23:46:43 host sshd[32686]: fatal: Read from socket failed: Connection reset by peer [preauth] + +# gh-864(3): +# failJSON: { "match": false } +Nov 24 23:46:41 host sshd[32686]: SSH: Server;Ltype: Authname;Remote: 127.0.0.1-1780;Name: root [preauth] +# failJSON: { "time": "2004-11-24T23:46:43", "match": true , "host": "127.0.0.1", "desc": "Multiline for connection reset by peer (3)" } +Nov 24 23:46:43 host sshd[32686]: fatal: Read from socket failed: Connection reset by peer [preauth] + +# several other cases from gh-864: +# failJSON: { "time": "2004-11-25T01:34:12", "match": true , "host": "127.0.0.1", "desc": "No supported authentication methods" } +Nov 25 01:34:12 srv sshd[123]: Received disconnect from 127.0.0.1: 14: No supported authentication methods available [preauth] +# failJSON: { "time": "2004-11-25T01:35:13", "match": true , "host": "127.0.0.1", "desc": "No supported authentication methods" } +Nov 25 01:35:13 srv sshd[123]: error: Received disconnect from 127.0.0.1: 14: No supported authentication methods available [preauth] +# failJSON: { "time": "2004-11-25T01:35:14", "match": true , "host": "192.168.2.92", "desc": "Optional space after port" } +Nov 25 01:35:14 srv sshd[3625]: error: Received disconnect from 192.168.2.92 port 1684:14: No supported authentication methods available [preauth] + +# gh-1545: +# failJSON: { "time": "2004-11-26T13:03:29", "match": true , "host": "192.0.2.1", "desc": "No matching cipher" } +Nov 26 13:03:29 srv sshd[45]: Unable to negotiate with 192.0.2.1 port 55419: no matching cipher found. Their offer: aes256-cbc,rijndael-cbc@lysator.liu.se,aes192-cbc,aes128-cbc,arcfour128,arcfour,3des-cbc,none [preauth] + +# gh-1117: +# failJSON: { "time": "2004-11-26T13:03:30", "match": true , "host": "192.0.2.2", "desc": "No matching key exchange method" } +Nov 26 13:03:30 srv sshd[45]: fatal: Unable to negotiate with 192.0.2.2 port 55419: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 +# failJSON: { "match": false } +Nov 26 15:03:30 host sshd[22440]: Connection from 192.0.2.3 port 39678 on 192.168.1.9 port 22 +# failJSON: { "time": "2004-11-26T15:03:31", "match": true , "host": "192.0.2.3", "desc": "Multiline - no matching key exchange method" } +Nov 26 15:03:31 host sshd[22440]: fatal: Unable to negotiate a key exchange method [preauth] \ No newline at end of file diff --git a/fail2ban/tests/files/logs/zzz-generic-example b/fail2ban/tests/files/logs/zzz-generic-example index 2044c387..d0c31740 100644 --- a/fail2ban/tests/files/logs/zzz-generic-example +++ b/fail2ban/tests/files/logs/zzz-generic-example @@ -46,6 +46,14 @@ Jun 22 20:37:04 server test-demo[402]: writeToStorage plist={ # failJSON: { "time": "2005-06-22T20:37:04", "match": true , "host": "192.0.2.2" } 0000-12-30 00:00:00 server test-demo[47831]: F2B: failure from 192.0.2.2 +# -- test no zone and UTC/GMT named zone "2005-06-21T14:55:10 UTC" == "2005-06-21T16:55:10 CEST" (diff +2h in CEST): +# failJSON: { "time": "2005-06-21T16:55:09", "match": true , "host": "192.0.2.09" } +2005-06-21 16:55:09 machine test-demo(pam_unix)[13709] F2B: error from 192.0.2.09 +# failJSON: { "time": "2005-06-21T16:55:10", "match": true , "host": "192.0.2.10" } +2005-06-21 14:55:10 UTC machine test-demo(pam_unix)[13709] F2B: error from 192.0.2.10 +# failJSON: { "time": "2005-06-21T16:55:11", "match": true , "host": "192.0.2.11" } +2005-06-21 14:55:11 GMT machine test-demo(pam_unix)[13709] F2B: error from 192.0.2.11 + # failJSON: { "time": "2005-06-21T16:56:02", "match": true , "host": "192.0.2.250" } [Jun 21 16:56:02] machine test-demo(pam_unix)[13709] F2B: error from 192.0.2.250 # failJSON: { "match": false, "desc": "test 1st ignoreregex" } diff --git a/fail2ban/tests/files/testcase01a.log b/fail2ban/tests/files/testcase01a.log new file mode 100644 index 00000000..203f0517 --- /dev/null +++ b/fail2ban/tests/files/testcase01a.log @@ -0,0 +1,4 @@ +Dec 31 11:55:01 [sshd] error: PAM: Authentication failure for test from 87.142.124.10 +Dec 31 11:55:02 [sshd] error: PAM: Authentication failure for test from 87.142.124.10 +Dec 31 11:55:03 [sshd] error: PAM: Authentication failure for test from 87.142.124.10 +Dec 31 11:55:04 [sshd] error: PAM: Authentication failure for test from 87.142.124.10 diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 2b57ce47..986cf1f0 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -270,6 +270,7 @@ def _copy_lines_to_journal(in_, fields={},n=None, skip=0, terminal_line=""): # p class BasicFilter(unittest.TestCase): def setUp(self): + super(BasicFilter, self).setUp() self.filter = Filter('name') def testGetSetUseDNS(self): @@ -283,10 +284,10 @@ class BasicFilter(unittest.TestCase): def testGetSetDatePattern(self): self.assertEqual(self.filter.getDatePattern(), (None, "Default Detectors")) - self.filter.setDatePattern("^%Y-%m-%d-%H%M%S.%f %z") + self.filter.setDatePattern("^%Y-%m-%d-%H%M%S.%f %z **") self.assertEqual(self.filter.getDatePattern(), - ("^%Y-%m-%d-%H%M%S.%f %z", - "^Year-Month-Day-24hourMinuteSecond.Microseconds Zone offset")) + ("^%Y-%m-%d-%H%M%S.%f %z **", + "^Year-Month-Day-24hourMinuteSecond.Microseconds Zone offset **")) def testAssertWrongTime(self): self.assertRaises(AssertionError, @@ -363,6 +364,7 @@ class IgnoreIP(LogCaptureTestCase): setUpMyTime() self.filter.addIgnoreIP('192.168.1.0/25') self.filter.addFailRegex('') + self.filter.setDatePattern('{^LN-BEG}EPOCH') self.filter.processLineAndAdd('1387203300.222 192.168.1.32') self.assertLogged('Ignore 192.168.1.32') tearDownMyTime() @@ -377,6 +379,10 @@ class IgnoreIP(LogCaptureTestCase): self.filter.setIgnoreCommand(sys.executable + ' ' + os.path.join(TEST_FILES_DIR, "ignorecommand.py ")) self.assertTrue(self.filter.inIgnoreIPList("10.0.0.1")) self.assertFalse(self.filter.inIgnoreIPList("10.0.0.0")) + self.assertLogged("returned successfully 0", "returned successfully 1", all=True) + self.pruneLog() + self.assertFalse(self.filter.inIgnoreIPList("")) + self.assertLogged("usage: ignorecommand IP", "returned 10", all=True) def testIgnoreCauseOK(self): ip = "93.184.216.34" @@ -461,6 +467,7 @@ class LogFileFilterPoll(unittest.TestCase): def setUp(self): """Call before every test case.""" + super(LogFileFilterPoll, self).setUp() self.filter = FilterPoll(DummyJail()) self.filter.addLogPath(LogFileFilterPoll.FILENAME) @@ -476,6 +483,8 @@ class LogFileFilterPoll(unittest.TestCase): self.assertFalse(self.filter.isModified(LogFileFilterPoll.FILENAME)) def testSeekToTimeSmallFile(self): + # speedup search using exact date pattern: + self.filter.setDatePattern('^%ExY-%Exm-%Exd %ExH:%ExM:%ExS') fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='.log') time = 1417512352 f = open(fname, 'w') @@ -560,6 +569,8 @@ class LogFileFilterPoll(unittest.TestCase): _killfile(f, fname) def testSeekToTimeLargeFile(self): + # speedup search using exact date pattern: + self.filter.setDatePattern('^%ExY-%Exm-%Exd %ExH:%ExM:%ExS') fname = tempfile.mktemp(prefix='tmp_fail2ban', suffix='.log') time = 1417512352 f = open(fname, 'w') @@ -649,6 +660,8 @@ class LogFileMonitor(LogCaptureTestCase): self.assertLogged('Unable to open %s' % self.name) def testErrorProcessLine(self): + # speedup search using exact date pattern: + self.filter.setDatePattern('^%ExY-%Exm-%Exd %ExH:%ExM:%ExS') self.filter.sleeptime /= 1000.0 ## produce error with not callable processLine: _org_processLine = self.filter.processLine @@ -711,6 +724,8 @@ class LogFileMonitor(LogCaptureTestCase): pass def testNewChangeViaGetFailures_simple(self): + # speedup search using exact date pattern: + self.filter.setDatePattern('^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?') # suck in lines from this sample log file self.filter.getFailures(self.name) self.assertRaises(FailManagerEmpty, self.filter.failManager.toBan) @@ -726,6 +741,8 @@ class LogFileMonitor(LogCaptureTestCase): _assert_correct_last_attempt(self, self.filter, GetFailures.FAILURES_01) def testNewChangeViaGetFailures_rewrite(self): + # speedup search using exact date pattern: + self.filter.setDatePattern('^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?') # # if we rewrite the file at once self.file.close() @@ -744,6 +761,8 @@ class LogFileMonitor(LogCaptureTestCase): _assert_correct_last_attempt(self, self.filter, GetFailures.FAILURES_01) def testNewChangeViaGetFailures_move(self): + # speedup search using exact date pattern: + self.filter.setDatePattern('^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?') # # if we move file into a new location while it has been open already self.file.close() @@ -765,6 +784,7 @@ class CommonMonitorTestCase(unittest.TestCase): def setUp(self): """Call before every test case.""" + super(CommonMonitorTestCase, self).setUp() self._failTotal = 0 def waitFailTotal(self, count, delay=1.): @@ -815,6 +835,8 @@ def get_monitor_failures_testcase(Filter_): self.jail = DummyJail() self.filter = Filter_(self.jail) self.filter.addLogPath(self.name, autoSeek=False) + # speedup search using exact date pattern: + self.filter.setDatePattern('^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?') self.filter.active = True self.filter.addFailRegex("(?:(?:Authentication failure|Failed [-/\w+]+) for(?: [iI](?:llegal|nvalid) user)?|[Ii](?:llegal|nvalid) user|ROOT LOGIN REFUSED) .*(?: from|FROM) ") self.filter.start() @@ -1219,6 +1241,8 @@ class GetFailures(LogCaptureTestCase): self.jail = DummyJail() self.filter = FileFilter(self.jail) self.filter.active = True + # speedup search using exact date pattern: + self.filter.setDatePattern('^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?') # TODO Test this #self.filter.setTimeRegex("\S{3}\s{1,2}\d{1,2} \d{2}:\d{2}:\d{2}") #self.filter.setTimePattern("%b %d %H:%M:%S") @@ -1325,6 +1349,11 @@ class GetFailures(LogCaptureTestCase): output = (('212.41.96.186', 4, 1124013600.0), ('212.41.96.185', 2, 1124013598.0)) + # speedup search using exact date pattern: + self.filter.setDatePattern(('^%ExY(?P<_sep>[-/.])%m(?P=_sep)%d[T ]%H:%M:%S(?:[.,]%f)?(?:\s*%z)?', + '^(?:%a )?%b %d %H:%M:%S(?:\.%f)?(?: %ExY)?', + '^EPOCH' + )) self.filter.setMaxRetry(2) self.filter.addLogPath(GetFailures.FILENAME_04, autoSeek=0) self.filter.addFailRegex("Invalid user .* ") @@ -1354,6 +1383,8 @@ class GetFailures(LogCaptureTestCase): if enc is not None: self.tearDown();self.setUp(); self.filter.setLogEncoding(enc); + # speedup search using exact date pattern: + self.filter.setDatePattern('^%ExY-%Exm-%Exd %ExH:%ExM:%ExS') self.assertNotLogged('Error decoding line'); self.filter.addLogPath(fname) self.filter.addFailRegex(failregex) @@ -1529,6 +1560,7 @@ class DNSUtilsNetworkTests(unittest.TestCase): def setUp(self): """Call before every test case.""" + super(DNSUtilsNetworkTests, self).setUp() unittest.F2B.SkipIfNoNetwork() def test_IPAddr(self): diff --git a/fail2ban/tests/misctestcase.py b/fail2ban/tests/misctestcase.py index 450904d5..cd841a0f 100644 --- a/fail2ban/tests/misctestcase.py +++ b/fail2ban/tests/misctestcase.py @@ -23,13 +23,11 @@ __license__ = "GPL" import logging import os -import re import sys import unittest import tempfile import shutil import fnmatch -import datetime from glob import glob from StringIO import StringIO @@ -37,8 +35,6 @@ from utils import LogCaptureTestCase, logSys as DefLogSys from ..helpers import formatExceptionInfo, mbasename, TraceBack, FormatterWithTraceBack, getLogger, uni_decode from ..helpers import splitwords -from ..server.datedetector import DateDetector -from ..server.datetemplate import DatePatternRegex from ..server.mytime import MyTime @@ -90,6 +86,7 @@ def _getSysPythonVersion(): class SetupTest(unittest.TestCase): def setUp(self): + super(SetupTest, self).setUp() unittest.F2B.SkipIfFast() setup = os.path.join(os.path.dirname(__file__), '..', '..', 'setup.py') self.setup = os.path.exists(setup) and setup or None @@ -320,91 +317,6 @@ class TestsUtilsTest(LogCaptureTestCase): self.assertRaisesRegexp(Exception, 'not all arguments converted', lambda: logSys.debug('test', 1, 2, 3)) -iso8601 = DatePatternRegex("%Y-%m-%d[T ]%H:%M:%S(?:\.%f)?%z") - - -class CustomDateFormatsTest(unittest.TestCase): - - def testIso8601(self): - date = datetime.datetime.utcfromtimestamp( - iso8601.getDate("2007-01-25T12:00:00Z")[0]) - self.assertEqual( - date, - datetime.datetime(2007, 1, 25, 12, 0)) - self.assertRaises(TypeError, iso8601.getDate, None) - self.assertRaises(TypeError, iso8601.getDate, date) - - self.assertEqual(iso8601.getDate(""), None) - self.assertEqual(iso8601.getDate("Z"), None) - - self.assertEqual(iso8601.getDate("2007-01-01T120:00:00Z"), None) - self.assertEqual(iso8601.getDate("2007-13-01T12:00:00Z"), None) - date = datetime.datetime.utcfromtimestamp( - iso8601.getDate("2007-01-25T12:00:00+0400")[0]) - self.assertEqual( - date, - datetime.datetime(2007, 1, 25, 8, 0)) - date = datetime.datetime.utcfromtimestamp( - iso8601.getDate("2007-01-25T12:00:00+04:00")[0]) - self.assertEqual( - date, - datetime.datetime(2007, 1, 25, 8, 0)) - date = datetime.datetime.utcfromtimestamp( - iso8601.getDate("2007-01-25T12:00:00-0400")[0]) - self.assertEqual( - date, - datetime.datetime(2007, 1, 25, 16, 0)) - date = datetime.datetime.utcfromtimestamp( - iso8601.getDate("2007-01-25T12:00:00-04")[0]) - self.assertEqual( - date, - datetime.datetime(2007, 1, 25, 16, 0)) - - def testAmbiguousDatePattern(self): - defDD = DateDetector() - defDD.addDefaultTemplate() - logSys = DefLogSys - for (matched, dp, line) in ( - # positive case: - ('Jan 23 21:59:59', None, 'Test failure Jan 23 21:59:59 for 192.0.2.1'), - # ambiguous "unbound" patterns (missed): - (False, None, 'Test failure TestJan 23 21:59:59.011 2015 for 192.0.2.1'), - (False, None, 'Test failure Jan 23 21:59:59123456789 for 192.0.2.1'), - # ambiguous "no optional year" patterns (matched): - ('Aug 8 11:25:50', None, 'Aug 8 11:25:50 14430f2329b8 Authentication failed from 192.0.2.1'), - ('Aug 8 11:25:50', None, '[Aug 8 11:25:50] 14430f2329b8 Authentication failed from 192.0.2.1'), - ('Aug 8 11:25:50 2014', None, 'Aug 8 11:25:50 2014 14430f2329b8 Authentication failed from 192.0.2.1'), - # direct specified patterns: - ('20:00:00 01.02.2003', r'%H:%M:%S %d.%m.%Y$', '192.0.2.1 at 20:00:00 01.02.2003'), - ('[20:00:00 01.02.2003]', r'\[%H:%M:%S %d.%m.%Y\]', '192.0.2.1[20:00:00 01.02.2003]'), - ('[20:00:00 01.02.2003]', r'\[%H:%M:%S %d.%m.%Y\]', '[20:00:00 01.02.2003]192.0.2.1'), - ('[20:00:00 01.02.2003]', r'\[%H:%M:%S %d.%m.%Y\]$', '192.0.2.1[20:00:00 01.02.2003]'), - ('[20:00:00 01.02.2003]', r'^\[%H:%M:%S %d.%m.%Y\]', '[20:00:00 01.02.2003]192.0.2.1'), - ('[17/Jun/2011 17:00:45]', r'^\[%d/%b/%Y %H:%M:%S\]', '[17/Jun/2011 17:00:45] Attempt, IP address 192.0.2.1'), - ('[17/Jun/2011 17:00:45]', r'\[%d/%b/%Y %H:%M:%S\]', 'Attempt [17/Jun/2011 17:00:45] IP address 192.0.2.1'), - ('[17/Jun/2011 17:00:45]', r'\[%d/%b/%Y %H:%M:%S\]', 'Attempt IP address 192.0.2.1, date: [17/Jun/2011 17:00:45]'), - # direct specified patterns (begin/end, missed): - (False, r'%H:%M:%S %d.%m.%Y', '192.0.2.1x20:00:00 01.02.2003'), - (False, r'%H:%M:%S %d.%m.%Y', '20:00:00 01.02.2003x192.0.2.1'), - # direct specified patterns (begin/end, matched): - ('20:00:00 01.02.2003', r'%H:%M:%S %d.%m.%Y', '192.0.2.1 20:00:00 01.02.2003'), - ('20:00:00 01.02.2003', r'%H:%M:%S %d.%m.%Y', '20:00:00 01.02.2003 192.0.2.1'), - ): - logSys.debug('== test: %r', (matched, dp, line)) - if dp is None: - dd = defDD - else: - dp = DatePatternRegex(dp) - dd = DateDetector() - dd.appendTemplate(dp) - date = dd.getTime(line) - if matched: - self.assertTrue(date) - self.assertEqual(matched, date[1].group()) - else: - self.assertEqual(date, None) - - class MyTimeTest(unittest.TestCase): def testStr2Seconds(self): diff --git a/fail2ban/tests/samplestestcase.py b/fail2ban/tests/samplestestcase.py index 31b1812e..3fdc50c6 100644 --- a/fail2ban/tests/samplestestcase.py +++ b/fail2ban/tests/samplestestcase.py @@ -31,6 +31,7 @@ import re import sys import time import unittest +from ..server.failregex import Regex from ..server.filter import Filter from ..client.filterreader import FilterReader from .utils import setUpMyTime, tearDownMyTime, CONFIG_DIR @@ -38,12 +39,20 @@ from .utils import setUpMyTime, tearDownMyTime, CONFIG_DIR TEST_CONFIG_DIR = os.path.join(os.path.dirname(__file__), "config") TEST_FILES_DIR = os.path.join(os.path.dirname(__file__), "files") +# regexp to test greedy catch-all should be not-greedy: +RE_HOST = Regex('').getRegex() +RE_WRONG_GREED = re.compile(r'\.[+\*](?!\?).*' + re.escape(RE_HOST) + r'.*(?:\.[+\*].*|[^\$])$') + class FilterSamplesRegex(unittest.TestCase): def setUp(self): """Call before every test case.""" + super(FilterSamplesRegex, self).setUp() self.filter = Filter(None) + self.filter.returnRawHost = True + self.filter.checkAllRegex = True + self.filter.checkFindTime = False self.filter.active = True setUpMyTime() @@ -60,6 +69,19 @@ class FilterSamplesRegex(unittest.TestCase): >= 10, "Expected more FilterSampleRegexs tests") + def testReWrongGreedyCatchAll(self): + """Tests regexp RE_WRONG_GREED is intact (positive/negative)""" + self.assertTrue( + RE_WRONG_GREED.search('greedy .* test' + RE_HOST + ' test not hard-anchored')) + self.assertTrue( + RE_WRONG_GREED.search('greedy .+ test' + RE_HOST + ' test vary .* anchored$')) + self.assertFalse( + RE_WRONG_GREED.search('greedy .* test' + RE_HOST + ' test no catch-all, hard-anchored$')) + self.assertFalse( + RE_WRONG_GREED.search('non-greedy .*? test' + RE_HOST + ' test not hard-anchored')) + self.assertFalse( + RE_WRONG_GREED.search('non-greedy .+? test' + RE_HOST + ' test vary catch-all .* anchored$')) + def testSampleRegexsFactory(name, basedir): def testFilter(self): @@ -93,60 +115,74 @@ def testSampleRegexsFactory(name, basedir): os.path.isfile(os.path.join(TEST_FILES_DIR, "logs", name)), "No sample log file available for '%s' filter" % name) - logFile = fileinput.FileInput( - os.path.join(TEST_FILES_DIR, "logs", name)) - regexsUsed = set() - for line in logFile: - jsonREMatch = re.match("^# ?failJSON:(.+)$", line) - if jsonREMatch: - try: - faildata = json.loads(jsonREMatch.group(1)) - except ValueError as e: - raise ValueError("%s: %s:%i" % - (e, logFile.filename(), logFile.filelineno())) - line = next(logFile) - elif line.startswith("#") or not line.strip(): - continue - else: - faildata = {} + filenames = [name] + i = 0 + while i < len(filenames): + filename = filenames[i]; i += 1; + logFile = fileinput.FileInput(os.path.join(TEST_FILES_DIR, "logs", + filename)) - ret = self.filter.processLine( - line, returnRawHost=True, checkAllRegex=True)[1] - if not ret: - # Check line is flagged as none match - self.assertFalse(faildata.get('match', True), - "Line not matched when should have: %s:%i %r" % - (logFile.filename(), logFile.filelineno(), line)) - elif ret: - # Check line is flagged to match - self.assertTrue(faildata.get('match', False), - "Line matched when shouldn't have: %s:%i %r" % - (logFile.filename(), logFile.filelineno(), line)) - self.assertEqual(len(ret), 1, "Multiple regexs matched %r - %s:%i" % - (map(lambda x: x[0], ret),logFile.filename(), logFile.filelineno())) + # test regexp contains greedy catch-all before , that is + # not hard-anchored at end or has not precise sub expression after : + for fr in self.filter.getFailRegex(): + if RE_WRONG_GREED.search(fr): #pragma: no cover + raise AssertionError("Following regexp of \"%s\" contains greedy catch-all before , " + "that is not hard-anchored at end or has not precise sub expression after :\n%s" % + (name, str(fr).replace(RE_HOST, ''))) - # Verify timestamp and host as expected - failregex, host, fail2banTime, lines, fail = ret[0] - self.assertEqual(host, faildata.get("host", None)) + for line in logFile: + jsonREMatch = re.match("^# ?(failJSON|addFILE):(.+)$", line) + if jsonREMatch: + try: + faildata = json.loads(jsonREMatch.group(2)) + if jsonREMatch.group(1) == 'addFILE': + filenames.append(faildata) + continue + except ValueError as e: + raise ValueError("%s: %s:%i" % + (e, logFile.filename(), logFile.filelineno())) + line = next(logFile) + elif line.startswith("#") or not line.strip(): + continue + else: + faildata = {} - t = faildata.get("time", None) - try: - jsonTimeLocal = datetime.datetime.strptime(t, "%Y-%m-%dT%H:%M:%S") - except ValueError: - jsonTimeLocal = datetime.datetime.strptime(t, "%Y-%m-%dT%H:%M:%S.%f") + ret = self.filter.processLine(line) + if not ret: + # Check line is flagged as none match + self.assertFalse(faildata.get('match', True), + "Line not matched when should have: %s:%i %r" % + (logFile.filename(), logFile.filelineno(), line)) + elif ret: + # Check line is flagged to match + self.assertTrue(faildata.get('match', False), + "Line matched when shouldn't have: %s:%i %r" % + (logFile.filename(), logFile.filelineno(), line)) + self.assertEqual(len(ret), 1, "Multiple regexs matched %r - %s:%i" % + (map(lambda x: x[0], ret),logFile.filename(), logFile.filelineno())) - jsonTime = time.mktime(jsonTimeLocal.timetuple()) - - jsonTime += jsonTimeLocal.microsecond / 1000000 + # Verify timestamp and host as expected + failregex, host, fail2banTime, lines, fail = ret[0] + self.assertEqual(host, faildata.get("host", None)) - self.assertEqual(fail2banTime, jsonTime, - "UTC Time mismatch fail2ban %s (%s) != failJson %s (%s) (diff %.3f seconds) on: %s:%i %r:" % - (fail2banTime, time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(fail2banTime)), - jsonTime, time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(jsonTime)), - fail2banTime - jsonTime, logFile.filename(), logFile.filelineno(), line ) ) + t = faildata.get("time", None) + try: + jsonTimeLocal = datetime.datetime.strptime(t, "%Y-%m-%dT%H:%M:%S") + except ValueError: + jsonTimeLocal = datetime.datetime.strptime(t, "%Y-%m-%dT%H:%M:%S.%f") - regexsUsed.add(failregex) + jsonTime = time.mktime(jsonTimeLocal.timetuple()) + + jsonTime += jsonTimeLocal.microsecond / 1000000 + + self.assertEqual(fail2banTime, jsonTime, + "UTC Time mismatch fail2ban %s (%s) != failJson %s (%s) (diff %.3f seconds) on: %s:%i %r:" % + (fail2banTime, time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(fail2banTime)), + jsonTime, time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(jsonTime)), + fail2banTime - jsonTime, logFile.filename(), logFile.filelineno(), line ) ) + + regexsUsed.add(failregex) for failRegexIndex, failRegex in enumerate(self.filter.getFailRegex()): self.assertTrue( diff --git a/fail2ban/tests/servertestcase.py b/fail2ban/tests/servertestcase.py index 3125f605..83e91831 100644 --- a/fail2ban/tests/servertestcase.py +++ b/fail2ban/tests/servertestcase.py @@ -28,6 +28,7 @@ import unittest import time import tempfile import os +import re import sys import platform @@ -65,7 +66,7 @@ class TransmitterBase(unittest.TestCase): def setUp(self): """Call before every test case.""" - #super(TransmitterBase, self).setUp() + super(TransmitterBase, self).setUp() self.transm = self.server._Server__transm # To test thransmitter we don't need to start server... #self.server.start('/dev/null', '/dev/null', force=False) @@ -302,6 +303,8 @@ class Transmitter(TransmitterBase): jail=self.jailName) self.setGetTest( "datepattern", "Epoch", (None, "Epoch"), jail=self.jailName) + self.setGetTest( + "datepattern", "^Epoch", (None, "{^LN-BEG}Epoch"), jail=self.jailName) self.setGetTest( "datepattern", "TAI64N", (None, "TAI64N"), jail=self.jailName) self.setGetTestNOK("datepattern", "%Cat%a%%%g", jail=self.jailName) @@ -1117,7 +1120,7 @@ class ServerConfigReaderTests(LogCaptureTestCase): # (we don't use it in this test at all): elif unittest.F2B.fast and ( len(cmd) > 3 and cmd[0] in ('set', 'multi-set') and cmd[2] == 'addfailregex' - ): + ): # pragma: no cover cmd[0] = "set" cmd[3] = "DUMMY-REGEX " # command to server, use cmdHandler direct instead of `transm.proceed(cmd)`: @@ -1619,31 +1622,114 @@ class ServerConfigReaderTests(LogCaptureTestCase): # wrap default command processor: action.executeCmd = self._executeCmd # test start : - logSys.debug('# === start ==='); self.pruneLog() + self.pruneLog('# === start ===') action.start() self.assertLogged(*tests['start'], all=True) # test ban ip4 : - logSys.debug('# === ban-ipv4 ==='); self.pruneLog() + self.pruneLog('# === ban-ipv4 ===') action.ban({'ip': IPAddr('192.0.2.1')}) self.assertLogged(*tests['ip4-check']+tests['ip4-ban'], all=True) self.assertNotLogged(*tests['ip6'], all=True) # test unban ip4 : - logSys.debug('# === unban ipv4 ==='); self.pruneLog() + self.pruneLog('# === unban ipv4 ===') action.unban({'ip': IPAddr('192.0.2.1')}) self.assertLogged(*tests['ip4-check']+tests['ip4-unban'], all=True) self.assertNotLogged(*tests['ip6'], all=True) # test ban ip6 : - logSys.debug('# === ban ipv6 ==='); self.pruneLog() + self.pruneLog('# === ban ipv6 ===') action.ban({'ip': IPAddr('2001:DB8::')}) self.assertLogged(*tests['ip6-check']+tests['ip6-ban'], all=True) self.assertNotLogged(*tests['ip4'], all=True) # test unban ip6 : - logSys.debug('# === unban ipv6 ==='); self.pruneLog() + self.pruneLog('# === unban ipv6 ===') action.unban({'ip': IPAddr('2001:DB8::')}) self.assertLogged(*tests['ip6-check']+tests['ip6-unban'], all=True) self.assertNotLogged(*tests['ip4'], all=True) # test stop : - logSys.debug('# === stop ==='); self.pruneLog() + self.pruneLog('# === stop ===') action.stop() self.assertLogged(*tests['stop'], all=True) + def _executeMailCmd(self, realCmd, timeout=60): + # replace pipe to mail with pipe to cat: + realCmd = re.sub(r'\)\s*\|\s*mail\b([^\n]*)', + r' echo mail \1 ) | cat', realCmd) + # replace abuse retrieving (possible no-network): + realCmd = re.sub(r'[^\n]+\bADDRESSES=\$\(dig\s[^\n]+', + 'ADDRESSES="abuse-1@abuse-test-server, abuse-2@abuse-test-server"', realCmd) + # execute action: + return _actions.CommandAction.executeCmd(realCmd, timeout=timeout) + + def testComplexMailActionMultiLog(self): + testJailsActions = ( + # mail-whois-lines -- + ('j-mail-whois-lines', + 'mail-whois-lines[' + 'name=%(__name__)s, grepopts="-m 1", grepmax=2, mailcmd="mail -s", ' + + # 2 logs to test grep from multiple logs: + 'logpath="' + os.path.join(TEST_FILES_DIR, "testcase01.log") + '\n' + + ' ' + os.path.join(TEST_FILES_DIR, "testcase01a.log") + '", ' + '_whois_command="echo \'-- information about --\'"' + ']', + { + 'ip4-ban': ( + 'The IP 87.142.124.10 has just been banned by Fail2Ban after', + '100 attempts against j-mail-whois-lines.', + 'Here is more information about 87.142.124.10 :', + '-- information about 87.142.124.10 --', + 'Lines containing failures of 87.142.124.10 (max 2)', + 'testcase01.log:Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 87.142.124.10', + 'testcase01a.log:Dec 31 11:55:01 [sshd] error: PAM: Authentication failure for test from 87.142.124.10', + ), + }), + # complain -- + ('j-complain-abuse', + 'complain[' + 'name=%(__name__)s, grepopts="-m 1", grepmax=2, mailcmd="mail -s",' + + # 2 logs to test grep from multiple logs: + 'logpath="' + os.path.join(TEST_FILES_DIR, "testcase01.log") + '\n' + + ' ' + os.path.join(TEST_FILES_DIR, "testcase01a.log") + '", ' + ']', + { + 'ip4-ban': ( + 'Lines containing failures of 87.142.124.10 (max 2)', + 'testcase01.log:Dec 31 11:59:59 [sshd] error: PAM: Authentication failure for kevin from 87.142.124.10', + 'testcase01a.log:Dec 31 11:55:01 [sshd] error: PAM: Authentication failure for test from 87.142.124.10', + # both abuse mails should be separated with space: + 'mail -s Abuse from 87.142.124.10 abuse-1@abuse-test-server abuse-2@abuse-test-server', + ), + }), + ) + server = TestServer() + transm = server._Server__transm + cmdHandler = transm._Transmitter__commandHandler + + for jail, act, tests in testJailsActions: + stream = self.getDefaultJailStream(jail, act) + + # for cmd in stream: + # print(cmd) + + # transmit jail to the server: + for cmd in stream: + # command to server: + ret, res = transm.proceed(cmd) + self.assertEqual(ret, 0) + + jails = server._Server__jails + + for jail, act, tests in testJailsActions: + # print(jail, jails[jail]) + for a in jails[jail].actions: + action = jails[jail].actions[a] + logSys.debug('# ' + ('=' * 50)) + logSys.debug('# == %-44s ==', jail + ' - ' + action._name) + logSys.debug('# ' + ('=' * 50)) + # wrap default command processor: + action.executeCmd = self._executeMailCmd + # test ban : + self.pruneLog('# === ban ===') + action.ban({'ip': IPAddr('87.142.124.10'), + 'failures': 100, + }) + self.assertLogged(*tests['ip4-ban'], all=True) diff --git a/fail2ban/tests/sockettestcase.py b/fail2ban/tests/sockettestcase.py index 5bf0be57..1a94a952 100644 --- a/fail2ban/tests/sockettestcase.py +++ b/fail2ban/tests/sockettestcase.py @@ -41,6 +41,7 @@ class Socket(unittest.TestCase): def setUp(self): """Call before every test case.""" + super(Socket, self).setUp() self.server = AsyncServer(self) sock_fd, sock_name = tempfile.mkstemp('fail2ban.sock', 'socket') os.close(sock_fd) diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py index c336cf84..7c411eee 100644 --- a/fail2ban/tests/utils.py +++ b/fail2ban/tests/utils.py @@ -48,6 +48,8 @@ from ..version import version logSys = getLogger(__name__) +TEST_NOW = 1124013600 + CONFIG_DIR = os.environ.get('FAIL2BAN_CONFIG_DIR', None) if not CONFIG_DIR: @@ -257,6 +259,14 @@ def initTests(opts): def F2B_SkipIfNoNetwork(): raise unittest.SkipTest('Skip test because of "--no-network"') unittest.F2B.SkipIfNoNetwork = F2B_SkipIfNoNetwork + + # persistently set time zone to CET (used in zone-related test-cases), + # yoh: we need to adjust TZ to match the one used by Cyril so all the timestamps match + os.environ['TZ'] = 'Europe/Zurich' + time.tzset() + # set alternate now for time related test cases: + MyTime.setAlternateNow(TEST_NOW) + # precache all invalid ip's (TEST-NET-1, ..., TEST-NET-3 according to RFC 5737): c = DNSUtils.CACHE_ipToName for i in xrange(255): @@ -286,17 +296,10 @@ old_TZ = os.environ.get('TZ', None) def setUpMyTime(): # Set the time to a fixed, known value # Sun Aug 14 12:00:00 CEST 2005 - # yoh: we need to adjust TZ to match the one used by Cyril so all the timestamps match - os.environ['TZ'] = 'Europe/Zurich' - time.tzset() - MyTime.setTime(1124013600) + MyTime.setTime(TEST_NOW) def tearDownMyTime(): - os.environ.pop('TZ') - if old_TZ: # pragma: no cover - os.environ['TZ'] = old_TZ - time.tzset() MyTime.myTime = None @@ -384,7 +387,6 @@ def gatherTests(regexps=None, opts=None): tests.addTest(unittest.makeSuite(misctestcase.HelpersTest)) tests.addTest(unittest.makeSuite(misctestcase.SetupTest)) tests.addTest(unittest.makeSuite(misctestcase.TestsUtilsTest)) - tests.addTest(unittest.makeSuite(misctestcase.CustomDateFormatsTest)) tests.addTest(unittest.makeSuite(misctestcase.MyTimeTest)) # Database tests.addTest(unittest.makeSuite(databasetestcase.DatabaseTest)) @@ -408,6 +410,7 @@ def gatherTests(regexps=None, opts=None): # DateDetector tests.addTest(unittest.makeSuite(datedetectortestcase.DateDetectorTest)) + tests.addTest(unittest.makeSuite(datedetectortestcase.CustomDateFormatsTest)) # Filter Regex tests with sample logs tests.addTest(unittest.makeSuite(samplestestcase.FilterSamplesRegex)) @@ -524,6 +527,16 @@ if True: ## if not hasattr(unittest.TestCase, 'assertIn'): self.fail(msg) unittest.TestCase.assertNotIn = assertNotIn +_org_setUp = unittest.TestCase.setUp +def _customSetUp(self): + # print('=='*10, self) + if unittest.F2B.log_level <= logging.DEBUG: # so if DEBUG etc -- show them (and log it in travis)! + print("") + logSys.debug('='*10 + ' %s ' + '='*20, self.id()) + _org_setUp(self) + +unittest.TestCase.setUp = _customSetUp + class LogCaptureTestCase(unittest.TestCase): @@ -599,12 +612,11 @@ class LogCaptureTestCase(unittest.TestCase): # Let's log everything into a string self._log = LogCaptureTestCase._MemHandler(unittest.F2B.log_lazy) logSys.handlers = [self._log] - if self._old_level <= logging.DEBUG: # so if DEBUG etc -- show them (and log it in travis)! - print("") + if self._old_level <= logging.DEBUG: logSys.handlers += self._old_handlers - logSys.debug('='*10 + ' %s ' + '='*20, self.id()) - else: + else: # lowest log level to capture messages logSys.setLevel(logging.DEBUG) + super(LogCaptureTestCase, self).setUp() def tearDown(self): """Call after every test case.""" @@ -613,6 +625,7 @@ class LogCaptureTestCase(unittest.TestCase): logSys = getLogger("fail2ban") logSys.handlers = self._old_handlers logSys.level = self._old_level + super(LogCaptureTestCase, self).tearDown() def _is_logged(self, *s, **kwargs): logged = self._log.getvalue() diff --git a/files/debian-initd b/files/debian-initd index 0d2bed9a..d6660215 100755 --- a/files/debian-initd +++ b/files/debian-initd @@ -22,7 +22,7 @@ # rename this file: (sudo) mv /etc/init.d/fail2ban.init /etc/init.d/fail2ban # same with the logrotate file: (sudo) mv /etc/logrotate.d/fail2ban.logrotate /etc/logrotate.d/fail2ban # -PATH=/usr/sbin:/usr/bin:/sbin:/bin +PATH=/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/bin DESC="authentication failure monitor" NAME=fail2ban diff --git a/files/fail2ban.service b/files/fail2ban.service index e79faed1..6eeba957 100644 --- a/files/fail2ban.service +++ b/files/fail2ban.service @@ -5,12 +5,16 @@ After=network.target iptables.service firewalld.service PartOf=iptables.service firewalld.service [Service] -Type=forking -ExecStart=/usr/bin/fail2ban-client -x start +Type=simple +ExecStartPre=/bin/mkdir -p /var/run/fail2ban +ExecStart=/usr/bin/fail2ban-server -xf start +# if should be logged in systemd journal, use following line or set logtarget to stdout in fail2ban.local +# ExecStart=/usr/bin/fail2ban-server -xf --logtarget=stdout start ExecStop=/usr/bin/fail2ban-client stop ExecReload=/usr/bin/fail2ban-client reload PIDFile=/var/run/fail2ban/fail2ban.pid -Restart=always +Restart=on-failure +RestartPreventExitStatus=0 255 [Install] WantedBy=multi-user.target diff --git a/man/fail2ban-server.1 b/man/fail2ban-server.1 index de8ba6a4..09dcd65a 100644 --- a/man/fail2ban-server.1 +++ b/man/fail2ban-server.1 @@ -23,6 +23,9 @@ pidfile path logging level .HP \fB\-\-logtarget\fR |STDOUT|STDERR|SYSLOG +logging target +.br +Note. If fail2ban running as systemd-service, for logging to the systemd-journal, the logtarget could be set to STDOUT .HP \fB\-\-syslogsocket\fR auto| .TP diff --git a/man/jail.conf.5 b/man/jail.conf.5 index 51a00cdc..2e333e5a 100644 --- a/man/jail.conf.5 +++ b/man/jail.conf.5 @@ -130,7 +130,9 @@ The items that can be set are: verbosity level of log output: CRITICAL, ERROR, WARNING, NOTICE, INFO, DEBUG, TRACEDEBUG, HEAVYDEBUG or corresponding numeric value (50-5). Default: ERROR (equal 40) .TP .B logtarget -log target: filename, SYSLOG, STDERR or STDOUT. Default: STDERR +log target: filename, SYSLOG, STDERR or STDOUT. Default: STDOUT if not set in fail2ban.conf/fail2ban.local +.br +Note. If fail2ban running as systemd-service, for logging to the systemd-journal, the logtarget could be set to STDOUT .br Only a single log target can be specified. If you change logtarget from the default value and you are using logrotate -- also adjust or disable rotation in the