From fe9e85c71d73d7ec57048113768f2a02fa110ba6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Allan=20Nordh=C3=B8y?= Date: Fri, 10 Nov 2017 23:56:10 +0100 Subject: [PATCH 01/33] "Fail2Ban", other language improvements --- README.md | 34 +++++++++++++++++----------------- 1 file changed, 17 insertions(+), 17 deletions(-) diff --git a/README.md b/README.md index 1138a3ac..71997c98 100644 --- a/README.md +++ b/README.md @@ -6,45 +6,45 @@ ## Fail2Ban: ban hosts that cause multiple authentication errors -Fail2Ban scans log files like `/var/log/auth.log` and bans IP addresses having +Fail2Ban scans log files like `/var/log/auth.log` and bans IP addresses conducting too many failed login attempts. It does this by updating system firewall rules to reject new connections from those IP addresses, for a configurable amount of time. Fail2Ban comes out-of-the-box ready to read many standard log files, -such as those for sshd and Apache, and is easy to configure to read any log -file you choose, for any error you choose. +such as those for sshd and Apache, and is easily configured to read any log +file of your choosing, for any error you wish. -Though Fail2Ban is able to reduce the rate of incorrect authentications -attempts, it cannot eliminate the risk that weak authentication presents. -Configure services to use only two factor or public/private authentication +Though Fail2Ban is able to reduce the rate of incorrect authentication +attempts, it cannot eliminate the risk presented by weak authentication. +Set up services to use only two factor, or public/private authentication mechanisms if you really want to protect services. -This README is a quick introduction to Fail2ban. More documentation, FAQ, HOWTOs -are available in fail2ban(1) manpage and on the website http://www.fail2ban.org +This README is a quick introduction to Fail2Ban. More documentation, FAQ, and HOWTOs +available in the fail2ban(1) manpage and on the website https://www.fail2ban.org Installation: ------------- -**It is possible that Fail2ban is already packaged for your distribution. In -this case, you should use it instead.** +**It is possible that Fail2Ban is already packaged for your distribution. In +this case, you should use that instead.** Required: -- [Python2 >= 2.6 or Python >= 3.2](http://www.python.org) or [PyPy](http://pypy.org) +- [Python2 >= 2.6 or Python >= 3.2](https://www.python.org) or [PyPy](https://pypy.org) Optional: - [pyinotify >= 0.8.3](https://github.com/seb-m/pyinotify) - - Linux >= 2.6.13 -- [gamin >= 0.0.21](http://www.gnome.org/~veillard/gamin) -- [systemd >= 204](http://www.freedesktop.org/wiki/Software/systemd) +- Linux >= 2.6.13 +- [gamin >= 0.0.21](https://www.gnome.org/~veillard/gamin) +- [systemd >= 204](https://www.freedesktop.org/wiki/Software/systemd) - [dnspython](http://www.dnspython.org/) -To install, just do: +To install: tar xvfj fail2ban-0.9.7.tar.bz2 cd fail2ban-0.9.7 python setup.py install This will install Fail2Ban into the python library directory. The executable -scripts are placed into `/usr/bin`, and configuration under `/etc/fail2ban`. +scripts are placed into `/usr/bin`, and configuration in `/etc/fail2ban`. Fail2Ban should be correctly installed now. Just type: @@ -86,7 +86,7 @@ Contact: See [CONTRIBUTING.md](https://github.com/fail2ban/fail2ban/blob/master/CONTRIBUTING.md) ### You just appreciate this program: -send kudos to the original author ([Cyril Jaquier](mailto: Cyril Jaquier )) +Send kudos to the original author ([Cyril Jaquier](mailto: Cyril Jaquier )) or *better* to the [mailing list](https://lists.sourceforge.net/lists/listinfo/fail2ban-users) since Fail2Ban is "community-driven" for years now. From 855f5d0ced70247ec80823ae4c8507adbd14554b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Allan=20Nordh=C3=B8y?= Date: Sat, 11 Nov 2017 14:03:15 +0100 Subject: [PATCH 02/33] to be found --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 71997c98..d97cdf38 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ Set up services to use only two factor, or public/private authentication mechanisms if you really want to protect services. This README is a quick introduction to Fail2Ban. More documentation, FAQ, and HOWTOs -available in the fail2ban(1) manpage and on the website https://www.fail2ban.org +to be found on fail2ban(1) manpage and the website: https://www.fail2ban.org Installation: ------------- From d7e320b96d1339d19331d4cad05e77493e1ce669 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Allan=20Nordh=C3=B8y?= Date: Tue, 23 Jan 2018 21:09:53 +0100 Subject: [PATCH 03/33] reverting linux indentation --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d97cdf38..0f08b3e6 100644 --- a/README.md +++ b/README.md @@ -32,7 +32,7 @@ Required: Optional: - [pyinotify >= 0.8.3](https://github.com/seb-m/pyinotify) -- Linux >= 2.6.13 +- ↓ Linux >= 2.6.13 - [gamin >= 0.0.21](https://www.gnome.org/~veillard/gamin) - [systemd >= 204](https://www.freedesktop.org/wiki/Software/systemd) - [dnspython](http://www.dnspython.org/) From b34ae5999e0d8ee1af8939527305c13152844b3d Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 5 Mar 2018 19:35:10 +0100 Subject: [PATCH 04/33] action.d/hostdeny.conf: fixes IPv6 syntax differentiate the IPv4 and IPv6 syntax (where it is enclosed in square brackets) --- config/action.d/hostsdeny.conf | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/config/action.d/hostsdeny.conf b/config/action.d/hostsdeny.conf index 5cca6529..4277fed8 100644 --- a/config/action.d/hostsdeny.conf +++ b/config/action.d/hostsdeny.conf @@ -31,7 +31,7 @@ actioncheck = # Tags: See jail.conf(5) man page # Values: CMD # -actionban = IP= && printf %%b ": $IP\n" >> +actionban = printf %%b ": <_ip_value>\n" >> # Option: actionunban # Notes.: command executed when unbanning an IP. Take care that the @@ -39,7 +39,7 @@ actionban = IP= && printf %%b ": $IP\n" >> # Tags: See jail.conf(5) man page # Values: CMD # -actionunban = IP=$(echo | sed 's/\./\\./g') && sed -i "/^: $IP$/d" +actionunban = IP=$(echo "<_ip_value>" | sed 's/[][\.]/\\\0/g') && sed -i "/^: $IP$/d" [Init] @@ -54,3 +54,9 @@ file = /etc/hosts.deny # for hosts.deny/hosts_access. Default is all services. # Values: STR Default: ALL daemon_list = ALL + +# internal variable IP (to differentiate the IPv4 and IPv6 syntax, where it is enclosed in brackets): +_ip_value = + +[Init?family=inet6] +_ip_value = [] From b16aafe2334e7897c2048a9c48205340af9e80c9 Mon Sep 17 00:00:00 2001 From: "Sergey G. Brester" Date: Mon, 5 Mar 2018 19:42:05 +0100 Subject: [PATCH 05/33] Update ChangeLog --- ChangeLog | 1 + 1 file changed, 1 insertion(+) diff --git a/ChangeLog b/ChangeLog index abbd254b..82fb8668 100644 --- a/ChangeLog +++ b/ChangeLog @@ -41,6 +41,7 @@ ver. 0.10.3-dev-1 (20??/??/??) - development edition - failregex got an optional space in order to match new log-format (see gh-2061); - fixed ddos-mode regex to match refactored message (some versions can contain port now, see gh-2062); * `action.d/badips.py`: implicit convert IPAddr to str, solves an issue "expected string, IPAddr found" (gh-2059); +* `action.d/hostsdeny.conf`: fixed IPv6 syntax (enclosed in square brackets, gh-2066); * (Free)BSD ipfw actionban fixed to allow same rule added several times (gh-2054); ### New Features From 5b63ad17c674084a9dd796065307bdda13327625 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 5 Mar 2018 21:54:18 +0100 Subject: [PATCH 06/33] stability of the test-cases: avoid echoing of server-ready in configure thread, if heavy-debug (only answer from new internal command "server-status"). --- fail2ban/client/fail2banclient.py | 6 +++--- fail2ban/server/transmitter.py | 3 +++ 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/fail2ban/client/fail2banclient.py b/fail2ban/client/fail2banclient.py index 13ebcdef..f80193ee 100755 --- a/fail2ban/client/fail2banclient.py +++ b/fail2ban/client/fail2banclient.py @@ -99,7 +99,7 @@ class Fail2banClient(Fail2banCmdLine, Thread): ret = client.send(c) if ret[0] == 0: logSys.log(5, "OK : %r", ret[1]) - if showRet or c[0] == 'echo': + if showRet or c[0] in ('echo', 'server-status'): output(beautifier.beautify(ret[1])) else: logSys.error("NOK: %r", ret[1].args) @@ -128,7 +128,7 @@ class Fail2banClient(Fail2banCmdLine, Thread): except Exception as e: # pragma: no cover if showRet or self._conf["verbose"] > 1: logSys.debug(e) - if showRet or c[0] == 'echo': + if showRet or c[0] in ('echo', 'server-status'): sys.stdout.flush() return streamRet @@ -186,7 +186,7 @@ class Fail2banClient(Fail2banCmdLine, Thread): logSys.error("Fail2ban seems to be in unexpected state (not running but the socket exists)") return None - stream.append(['echo', 'Server ready']) + stream.append(['server-status']) return stream ## diff --git a/fail2ban/server/transmitter.py b/fail2ban/server/transmitter.py index ecc2a138..1aa7f3e2 100644 --- a/fail2ban/server/transmitter.py +++ b/fail2ban/server/transmitter.py @@ -115,6 +115,9 @@ class Transmitter: return cnt elif command[0] == "echo": return command[1:] + elif command[0] == "server-status": + logSys.debug("Server ready") + return "Server ready" elif command[0] == "sleep": value = command[1] time.sleep(float(value)) From 71b19d9eba8547070c5080023829bc4989405dad Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 7 Mar 2018 15:25:27 +0100 Subject: [PATCH 07/33] stability of time-related test-cases: a bit increased timeouts; code normalization, review and coverage --- fail2ban/tests/fail2banclienttestcase.py | 4 ++-- fail2ban/tests/filtertestcase.py | 15 +++++-------- fail2ban/tests/misctestcase.py | 18 +++++++++++++++ fail2ban/tests/utils.py | 28 ++++++++++++++++-------- 4 files changed, 45 insertions(+), 20 deletions(-) diff --git a/fail2ban/tests/fail2banclienttestcase.py b/fail2ban/tests/fail2banclienttestcase.py index ed3d9868..5dcbdef5 100644 --- a/fail2ban/tests/fail2banclienttestcase.py +++ b/fail2ban/tests/fail2banclienttestcase.py @@ -55,8 +55,8 @@ CLIENT = "fail2ban-client" SERVER = "fail2ban-server" BIN = dirname(Fail2banServer.getServerPath()) -MAX_WAITTIME = 30 if not unittest.F2B.fast else 5 -MID_WAITTIME = MAX_WAITTIME +MAX_WAITTIME = unittest.F2B.maxWaitTime(unittest.F2B.MAX_WAITTIME) +MID_WAITTIME = unittest.F2B.maxWaitTime(unittest.F2B.MID_WAITTIME) ## # Several wrappers and settings for proper testing: diff --git a/fail2ban/tests/filtertestcase.py b/fail2ban/tests/filtertestcase.py index 2bbfcd9d..ec839585 100644 --- a/fail2ban/tests/filtertestcase.py +++ b/fail2ban/tests/filtertestcase.py @@ -82,10 +82,7 @@ def _killfile(f, name): _killfile(None, name + '.bak') -def _maxWaitTime(wtime): - if unittest.F2B.fast: # pragma: no cover - wtime /= 10.0 - return wtime +_maxWaitTime = unittest.F2B.maxWaitTime class _tmSerial(): @@ -657,12 +654,12 @@ class LogFileMonitor(LogCaptureTestCase): _killfile(self.file, self.name) pass - def isModified(self, delay=2.): + def isModified(self, delay=2): """Wait up to `delay` sec to assure that it was modified or not """ return Utils.wait_for(lambda: self.filter.isModified(self.name), _maxWaitTime(delay)) - def notModified(self, delay=2.): + def notModified(self, delay=2): """Wait up to `delay` sec as long as it was not modified """ return Utils.wait_for(lambda: not self.filter.isModified(self.name), _maxWaitTime(delay)) @@ -817,7 +814,7 @@ class CommonMonitorTestCase(unittest.TestCase): super(CommonMonitorTestCase, self).setUp() self._failTotal = 0 - def waitFailTotal(self, count, delay=1.): + def waitFailTotal(self, count, delay=1): """Wait up to `delay` sec to assure that expected failure `count` reached """ ret = Utils.wait_for( @@ -826,7 +823,7 @@ class CommonMonitorTestCase(unittest.TestCase): self._failTotal += count return ret - def isFilled(self, delay=1.): + def isFilled(self, delay=1): """Wait up to `delay` sec to assure that it was modified or not """ return Utils.wait_for(self.jail.isFilled, _maxWaitTime(delay)) @@ -836,7 +833,7 @@ class CommonMonitorTestCase(unittest.TestCase): """ return Utils.wait_for(self.jail.isEmpty, _maxWaitTime(delay)) - def waitForTicks(self, ticks, delay=2.): + def waitForTicks(self, ticks, delay=2): """Wait up to `delay` sec to assure that it was modified or not """ last_ticks = self.filter.ticks diff --git a/fail2ban/tests/misctestcase.py b/fail2ban/tests/misctestcase.py index fda1a5cd..adf7886b 100644 --- a/fail2ban/tests/misctestcase.py +++ b/fail2ban/tests/misctestcase.py @@ -287,6 +287,20 @@ class TestsUtilsTest(LogCaptureTestCase): self.assertNotLogged('test "xyz"') self.assertNotLogged('test', 'xyz', all=False) self.assertNotLogged('test', 'xyz', 'zyx', all=True) + ## maxWaitTime: + orgfast, unittest.F2B.fast = unittest.F2B.fast, False + self.assertFalse(isinstance(unittest.F2B.maxWaitTime(True), bool)) + self.assertEqual(unittest.F2B.maxWaitTime(lambda: 50)(), 50) + self.assertEqual(unittest.F2B.maxWaitTime(25), 25) + self.assertEqual(unittest.F2B.maxWaitTime(25.), 25.0) + unittest.F2B.fast = True + try: + self.assertEqual(unittest.F2B.maxWaitTime(lambda: 50)(), 50) + self.assertEqual(unittest.F2B.maxWaitTime(25), 2.5) + self.assertEqual(unittest.F2B.maxWaitTime(25.), 25.0) + finally: + unittest.F2B.fast = orgfast + self.assertFalse(unittest.F2B.maxWaitTime(False)) ## assertLogged, assertNotLogged negative case: self.pruneLog() logSys.debug('test "xyz"') @@ -296,8 +310,12 @@ class TestsUtilsTest(LogCaptureTestCase): self.assertNotLogged, 'test', 'xyz', all=True) self._testAssertionErrorRE(r"was not found in the log", self.assertLogged, 'test', 'zyx', all=True) + self._testAssertionErrorRE(r"was not found in the log, waited 1e-06", + self.assertLogged, 'test', 'zyx', all=True, wait=1e-6) self._testAssertionErrorRE(r"None among .* was found in the log", self.assertLogged, 'test_zyx', 'zyx', all=False) + self._testAssertionErrorRE(r"None among .* was found in the log, waited 1e-06", + self.assertLogged, 'test_zyx', 'zyx', all=False, wait=1e-6) self._testAssertionErrorRE(r"All of the .* were found present in the log", self.assertNotLogged, 'test', 'xyz', all=False) ## assertDictEqual: diff --git a/fail2ban/tests/utils.py b/fail2ban/tests/utils.py index 3eeb8eb4..901952eb 100644 --- a/fail2ban/tests/utils.py +++ b/fail2ban/tests/utils.py @@ -180,6 +180,10 @@ def initProcess(opts): class F2B(DefaultTestOptions): + + MAX_WAITTIME = 60 + MID_WAITTIME = 30 + def __init__(self, opts): self.__dict__ = opts.__dict__ if self.fast: @@ -215,8 +219,12 @@ class F2B(DefaultTestOptions): return wrapper return _deco_wrapper - def maxWaitTime(self,wtime): - if self.fast: + def maxWaitTime(self, wtime=True): + if isinstance(wtime, bool) and wtime: + wtime = self.MAX_WAITTIME + # short only integer interval (avoid by conditional wait with callable, and dual + # wrapping in some routines, if it will be called twice): + if self.fast and isinstance(wtime, int): wtime = float(wtime) / 10 return wtime @@ -761,21 +769,24 @@ class LogCaptureTestCase(unittest.TestCase): """ wait = kwargs.get('wait', None) if wait: + wait = unittest.F2B.maxWaitTime(wait) res = Utils.wait_for(lambda: self._is_logged(*s, **kwargs), wait) else: res = self._is_logged(*s, **kwargs) if not kwargs.get('all', False): # at least one entry should be found: - if not res: # pragma: no cover + if not res: logged = self._log.getvalue() - self.fail("None among %r was found in the log: ===\n%s===" % (s, logged)) + self.fail("None among %r was found in the log%s: ===\n%s===" % (s, + ((', waited %s' % wait) if wait else ''), logged)) else: # each entry should be found: - if not res: # pragma: no cover + if not res: logged = self._log.getvalue() for s_ in s: if s_ not in logged: - self.fail("%r was not found in the log: ===\n%s===" % (s_, logged)) + self.fail("%r was not found in the log%s: ===\n%s===" % (s_, + ((', waited %s' % wait) if wait else ''), logged)) def assertNotLogged(self, *s, **kwargs): """Assert that strings were not logged @@ -792,11 +803,10 @@ class LogCaptureTestCase(unittest.TestCase): for s_ in s: if s_ not in logged: return - if True: # pragma: no cover - self.fail("All of the %r were found present in the log: ===\n%s===" % (s, logged)) + self.fail("All of the %r were found present in the log: ===\n%s===" % (s, logged)) else: for s_ in s: - if s_ in logged: # pragma: no cover + if s_ in logged: self.fail("%r was found in the log: ===\n%s===" % (s_, logged)) def pruneLog(self, logphase=None): From a3739bbf27c391d89814abc9cf53aa4f5ca814bf Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 7 Mar 2018 16:25:54 +0100 Subject: [PATCH 08/33] trim name and add one space after padding --- fail2ban/server/server.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fail2ban/server/server.py b/fail2ban/server/server.py index b166b94a..039dd879 100644 --- a/fail2ban/server/server.py +++ b/fail2ban/server/server.py @@ -566,7 +566,7 @@ class Server: self.__logTarget = target return True # set a format which is simpler for console use - fmt = "%(name)-24s[%(process)d]: %(levelname)-7s %(message)s" + fmt = "%(name)-23.23s [%(process)d]: %(levelname)-7s %(message)s" if systarget == "SYSLOG": facility = logOptions.get('facility', 'DAEMON').upper() try: From ce6ca0029a79425976e60cc66aef3761f6c2008e Mon Sep 17 00:00:00 2001 From: sebres Date: Wed, 7 Mar 2018 16:27:42 +0100 Subject: [PATCH 09/33] minimize log output in trace case (index instead of full-regexp by "matched" log-line) --- fail2ban/server/filter.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/fail2ban/server/filter.py b/fail2ban/server/filter.py index 9494b869..5b9125ed 100644 --- a/fail2ban/server/filter.py +++ b/fail2ban/server/filter.py @@ -693,12 +693,12 @@ class Filter(JailThread): # Iterates over all the regular expressions. for failRegexIndex, failRegex in enumerate(self.__failRegex): if logSys.getEffectiveLevel() <= logging.HEAVYDEBUG: # pragma: no cover - logSys.log(5, " Looking for failregex %r", failRegex.getRegex()) + logSys.log(5, " Looking for failregex %d - %r", failRegexIndex, failRegex.getRegex()) failRegex.search(self.__lineBuffer, orgBuffer) if not failRegex.hasMatched(): continue # The failregex matched. - logSys.log(7, " Matched %s", failRegex) + logSys.log(7, " Matched failregex %d: %s", failRegexIndex, failRegex.getGroups()) # Checks if we must ignore this match. if self.ignoreLine(failRegex.getMatchedTupleLines()) \ is not None: From 2e533a3a3a4f9eda488f84d370c4731fa5f54dd0 Mon Sep 17 00:00:00 2001 From: sebres Date: Fri, 9 Mar 2018 13:54:04 +0100 Subject: [PATCH 10/33] better handling of default date templates (bounds, replacement using own expressions `...{DATE}...`, etc.) --- fail2ban/server/datedetector.py | 70 +++++++++++++++++++-------------- fail2ban/server/datetemplate.py | 14 ++++++- 2 files changed, 53 insertions(+), 31 deletions(-) diff --git a/fail2ban/server/datedetector.py b/fail2ban/server/datedetector.py index 13d70699..2e85b940 100644 --- a/fail2ban/server/datedetector.py +++ b/fail2ban/server/datedetector.py @@ -109,9 +109,6 @@ class DateDetectorCache(object): """Cache Fail2Ban's default template. """ - if isinstance(template, str): - # exact given template with word begin-end boundary: - template = _getPatternTemplate(template) # if not already line-begin anchored, additional template, that prefers datetime # at start of a line (safety+performance feature): name = template.name @@ -126,60 +123,74 @@ class DateDetectorCache(object): # add template: self.__tmpcache[1].append(template) - def _addDefaultTemplate(self): - """Add resp. cache Fail2Ban's default set of date templates. - """ - self.__tmpcache = [], [] + DEFAULT_TEMPLATES = [ # ISO 8601, simple date, optional subsecond and timezone: # 2005-01-23T21:59:59.981746, 2005-01-23 21:59:59, 2005-01-23 8:59:59 # simple date: 2005/01/23 21:59:59 # custom for syslog-ng 2006.12.21 06:43:20 - self._cacheTemplate("%ExY(?P<_sep>[-/.])%m(?P=_sep)%d(?:T| ?)%H:%M:%S(?:[.,]%f)?(?:\s*%z)?") + "%ExY(?P<_sep>[-/.])%m(?P=_sep)%d(?:T| ?)%H:%M:%S(?:[.,]%f)?(?:\s*%z)?", # asctime with optional day, subsecond and/or year: # Sun Jan 23 21:59:59.011 2005 - self._cacheTemplate("(?:%a )?%b %d %k:%M:%S(?:\.%f)?(?: %ExY)?") + "(?:%a )?%b %d %k:%M:%S(?:\.%f)?(?: %ExY)?", # asctime with optional day, subsecond and/or year coming after day # http://bugs.debian.org/798923 # Sun Jan 23 2005 21:59:59.011 - self._cacheTemplate("(?:%a )?%b %d %ExY %k:%M:%S(?:\.%f)?") + "(?:%a )?%b %d %ExY %k:%M:%S(?:\.%f)?", # simple date too (from x11vnc): 23/01/2005 21:59:59 # and with optional year given by 2 digits: 23/01/05 21:59:59 # (See http://bugs.debian.org/537610) # 17-07-2008 17:23:25 - self._cacheTemplate("%d(?P<_sep>[-/])%m(?P=_sep)(?:%ExY|%Exy) %k:%M:%S") + "%d(?P<_sep>[-/])%m(?P=_sep)(?:%ExY|%Exy) %k:%M:%S", # Apache format optional time zone: # [31/Oct/2006:09:22:55 -0000] # 26-Jul-2007 15:20:52 # named 26-Jul-2007 15:20:52.252 # roundcube 26-Jul-2007 15:20:52 +0200 - self._cacheTemplate("%d(?P<_sep>[-/])%b(?P=_sep)%ExY[ :]?%H:%M:%S(?:\.%f)?(?: %z)?") + "%d(?P<_sep>[-/])%b(?P=_sep)%ExY[ :]?%H:%M:%S(?:\.%f)?(?: %z)?", # CPanel 05/20/2008:01:57:39 - self._cacheTemplate("%m/%d/%ExY:%H:%M:%S") + "%m/%d/%ExY:%H:%M:%S", # 01-27-2012 16:22:44.252 # subseconds explicit to avoid possible %m<->%d confusion # with previous ("%d-%m-%ExY %k:%M:%S" by "%d(?P<_sep>[-/])%m(?P=_sep)(?:%ExY|%Exy) %k:%M:%S") - self._cacheTemplate("%m-%d-%ExY %k:%M:%S(?:\.%f)?") + "%m-%d-%ExY %k:%M:%S(?:\.%f)?", # Epoch - self._cacheTemplate('EPOCH') + "EPOCH", # Only time information in the log - self._cacheTemplate("{^LN-BEG}%H:%M:%S") + "{^LN-BEG}%H:%M:%S", # <09/16/08@05:03:30> - self._cacheTemplate("^<%m/%d/%Exy@%H:%M:%S>") + "^<%m/%d/%Exy@%H:%M:%S>", # MySQL: 130322 11:46:11 - self._cacheTemplate("%Exy%Exm%Exd ?%H:%M:%S") + "%Exy%Exm%Exd ?%H:%M:%S", # Apache Tomcat - self._cacheTemplate("%b %d, %ExY %I:%M:%S %p") + "%b %d, %ExY %I:%M:%S %p", # ASSP: Apr-27-13 02:33:06 - self._cacheTemplate("^%b-%d-%Exy %k:%M:%S") + "^%b-%d-%Exy %k:%M:%S", # 20050123T215959, 20050123 215959, 20050123 85959 - self._cacheTemplate("%ExY%Exm%Exd(?:T| ?)%ExH%ExM%ExS(?:[.,]%f)?(?:\s*%z)?") + "%ExY%Exm%Exd(?:T| ?)%ExH%ExM%ExS(?:[.,]%f)?(?:\s*%z)?", # prefixed with optional named time zone (monit): # PDT Apr 16 21:05:29 - self._cacheTemplate("(?:%Z )?(?:%a )?%b %d %k:%M:%S(?:\.%f)?(?: %ExY)?") + "(?:%Z )?(?:%a )?%b %d %k:%M:%S(?:\.%f)?(?: %ExY)?", # +00:00 Jan 23 21:59:59.011 2005 - self._cacheTemplate("(?:%z )?(?:%a )?%b %d %k:%M:%S(?:\.%f)?(?: %ExY)?") + "(?:%z )?(?:%a )?%b %d %k:%M:%S(?:\.%f)?(?: %ExY)?", # TAI64N - self._cacheTemplate("TAI64N") + "TAI64N", + ] + + @property + def defaultTemplates(self): + if isinstance(DateDetectorCache.DEFAULT_TEMPLATES[0], str): + for i, dt in enumerate(DateDetectorCache.DEFAULT_TEMPLATES): + dt = _getPatternTemplate(dt) + DateDetectorCache.DEFAULT_TEMPLATES[i] = dt + return DateDetectorCache.DEFAULT_TEMPLATES + + def _addDefaultTemplate(self): + """Add resp. cache Fail2Ban's default set of date templates. + """ + self.__tmpcache = [], [] + # cache default templates: + for dt in self.defaultTemplates: + self._cacheTemplate(dt) # self.__templates = self.__tmpcache[0] + self.__tmpcache[1] del self.__tmpcache @@ -269,8 +280,7 @@ class DateDetector(object): self.addDefaultTemplate(flt) return elif "{DATE}" in key: - self.addDefaultTemplate( - lambda template: not template.flags & DateTemplate.LINE_BEGIN, pattern) + self.addDefaultTemplate(preMatch=pattern, allDefaults=False) return else: template = _getPatternTemplate(pattern, key) @@ -283,18 +293,20 @@ class DateDetector(object): logSys.debug(" date pattern regex for %r: %s", getattr(template, 'pattern', ''), template.regex) - def addDefaultTemplate(self, filterTemplate=None, preMatch=None): + def addDefaultTemplate(self, filterTemplate=None, preMatch=None, allDefaults=True): """Add Fail2Ban's default set of date templates. """ ignoreDup = len(self.__templates) > 0 - for template in DateDetector._defCache.templates: + for template in ( + DateDetector._defCache.templates if allDefaults else DateDetector._defCache.defaultTemplates + ): # filter if specified: if filterTemplate is not None and not filterTemplate(template): continue # if exact pattern available - create copy of template, contains replaced {DATE} with default regex: if preMatch is not None: # get cached or create a copy with modified name/pattern, using preMatch replacement for {DATE}: template = _getAnchoredTemplate(template, - wrap=lambda s: RE_DATE_PREMATCH.sub(lambda m: s, preMatch)) + wrap=lambda s: RE_DATE_PREMATCH.sub(lambda m: DateTemplate.unboundPattern(s), preMatch)) # append date detector template (ignore duplicate if some was added before default): self._appendTemplate(template, ignoreDup=ignoreDup) diff --git a/fail2ban/server/datetemplate.py b/fail2ban/server/datetemplate.py index 49fa0c66..606ba6ec 100644 --- a/fail2ban/server/datetemplate.py +++ b/fail2ban/server/datetemplate.py @@ -37,8 +37,10 @@ RE_GROUPED = re.compile(r'(? Date: Fri, 9 Mar 2018 13:56:38 +0100 Subject: [PATCH 11/33] filter.d/recidive.conf: fixed if logging into systemd-journal (SYSLOG) with daemon name in prefix, gh-2069 --- ChangeLog | 1 + config/filter.d/recidive.conf | 12 ++++++------ fail2ban/tests/files/logs/recidive | 5 +++++ 3 files changed, 12 insertions(+), 6 deletions(-) diff --git a/ChangeLog b/ChangeLog index 82fb8668..ceb3c822 100644 --- a/ChangeLog +++ b/ChangeLog @@ -37,6 +37,7 @@ ver. 0.10.3-dev-1 (20??/??/??) - development edition ### Fixes * `filter.d/asterisk.conf`: fixed failregex prefix by log over remote syslog server (gh-2060); * `filter.d/exim.conf`: failregex extended - SMTP call dropped: too many syntax or protocol errors (gh-2048); +* `filter.d/recidive.conf`: fixed if logging into systemd-journal (SYSLOG) with daemon name in prefix, gh-2069; * `filter.d/sshd.conf`: - failregex got an optional space in order to match new log-format (see gh-2061); - fixed ddos-mode regex to match refactored message (some versions can contain port now, see gh-2062); diff --git a/config/filter.d/recidive.conf b/config/filter.d/recidive.conf index e2501cf6..63833cab 100644 --- a/config/filter.d/recidive.conf +++ b/config/filter.d/recidive.conf @@ -21,18 +21,18 @@ before = common.conf [Definition] -_daemon = fail2ban\.actions\s* +_daemon = (?:fail2ban(?:-server|\.actions)\s*) -# The name of the jail that this filter is used for. In jail.conf, name the -# jail using this filter 'recidive', or change this line! +# The name of the jail that this filter is used for. In jail.conf, name the jail using +# this filter 'recidive', or supply another name with `filter = recidive[_jailname="jail"]` _jailname = recidive -failregex = ^(%(__prefix_line)s| %(_daemon)s%(__pid_re)s?:\s+)NOTICE\s+\[(?!%(_jailname)s\])(?:.*)\]\s+Ban\s+\s*$ +failregex = ^%(__prefix_line)s(?:\s*fail2ban\.actions\s*%(__pid_re)s?:\s+)?NOTICE\s+\[(?!%(_jailname)s\])(?:.*)\]\s+Ban\s+\s*$ + +datepattern = ^{DATE} ignoreregex = -[Init] - journalmatch = _SYSTEMD_UNIT=fail2ban.service PRIORITY=5 # Author: Tom Hendrikx, modifications by Amir Caspi diff --git a/fail2ban/tests/files/logs/recidive b/fail2ban/tests/files/logs/recidive index edb3a155..5746a55d 100644 --- a/fail2ban/tests/files/logs/recidive +++ b/fail2ban/tests/files/logs/recidive @@ -12,3 +12,8 @@ Sep 16 00:44:55 spaceman fail2ban.actions: NOTICE [jail] Ban 10.0.0.7 # failJSON: { "time": "2006-02-13T15:52:30", "match": true , "host": "1.2.3.4", "desc": "Extended with [PID] and padding" } 2006-02-13 15:52:30,388 fail2ban.actions [123]: NOTICE [sendmail] Ban 1.2.3.4 + +# failJSON: { "time": "2005-01-16T17:11:25", "match": true , "host": "192.0.2.1", "desc": "SYSLOG / systemd-journal without daemon-name" } +Jan 16 17:11:25 testorg fail2ban.actions[6605]: NOTICE [postfix-auth] Ban 192.0.2.1 +# failJSON: { "time": "2005-03-05T08:41:28", "match": true , "host": "192.0.2.2", "desc": "SYSLOG / systemd-journal with daemon-name" } +Mar 05 08:41:28 test.org fail2ban-server[11524]: fail2ban.actions [11524]: NOTICE [postfix-auth] Ban 192.0.2.2 From e8ffab28fb8c50c576f0422c726b0b1c04829397 Mon Sep 17 00:00:00 2001 From: sebres Date: Mon, 19 Mar 2018 13:18:55 +0100 Subject: [PATCH 12/33] filter.d/apache-noscript.conf: extended to match "Primary script unknown", got from php-fpm module. --- config/filter.d/apache-noscript.conf | 9 +++++++-- fail2ban/tests/files/logs/apache-noscript | 2 ++ 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/config/filter.d/apache-noscript.conf b/config/filter.d/apache-noscript.conf index fbc1af64..abc083a6 100644 --- a/config/filter.d/apache-noscript.conf +++ b/config/filter.d/apache-noscript.conf @@ -17,8 +17,13 @@ before = apache-common.conf [Definition] -failregex = ^%(_apache_error_client)s ((AH001(28|30): )?File does not exist|(AH01264: )?script not found or unable to stat): /\S*(php([45]|[.-]cgi)?|\.asp|\.exe|\.pl)(, referer: \S+)?\s*$ - ^%(_apache_error_client)s script '/\S*(php([45]|[.-]cgi)?|\.asp|\.exe|\.pl)\S*' not found or unable to stat(, referer: \S+)?\s*$ +script = /\S*(?:php(?:[45]|[.-]cgi)?|\.asp|\.exe|\.pl) + +prefregex = ^%(_apache_error_client)s .+$ + +failregex = ^(?:(?:AH001(?:28|30): )?File does not exist|(AH01264: )?script not found or unable to stat):