Commit graph

6088 commits

Author SHA1 Message Date
sebres
fc3e8a5d37 remove help command from protocol (the command was never supported);
closes gh-3241
2025-03-31 02:29:51 +02:00
sebres
1d6ff06856 amend to a0093b557e: filter only readable journal files by retrieving non-rotated files (if user is not root) 2025-03-31 02:28:40 +02:00
sebres
767c89f863 satisfy spellcheck 2025-03-31 01:27:52 +02:00
sebres
a0093b557e Merge branch 'systemd-review'
Large set of fixes and enhancements for `systemd` and `auto` backends:
* fixes `systemd` bug with missing journal descriptor after rotation by reopening of journal if it is recognized as not alive (gh-3929)
* improve threaded clean-up of all filters, new thread functions `afterStop` (to force clean-up after stop) and `done`, invoking `afterStop` once
* ensure journal-reader is always closed (additional prevention against leaks and "too many open files"), thereby avoid sporadic segfault in systemd module (see https://github.com/systemd/python-systemd/issues/143)
* fixes `systemd` causing "too many open files" error for a lot of journal files and large amout of systemd jails (see new parameter `rotated` below, gh-3391);
* backend `systemd` extended with new parameter `rotated` (default `false`, as prevention against "too many open files"),
  that allows to monitor only actual journals and ignore now a lot of rotated files by default; so can drastically reduce
  amount of used file descriptors, normally to 1 or 2 descriptors per jail (gh-3391)
* implements automatic switch `backend = auto` to backend `systemd`, when the following is true (RFE gh-3768):
  - no files matching `logpath` found for this jail;
  - no `systemd_if_nologs = false` is specified for the jail (`true` by default);
  - option `journalmatch` is set for the jail or its filter (otherwise it'd be too heavy to allow all auto-jails,
    even if they have never been foreseen for journal monitoring);
  (option `skip_if_nologs` will be ignored if we could switch backend to `systemd`)
2025-03-31 01:18:53 +02:00
sebres
d5718503ad update changelog and documentation (new features and handling) 2025-03-31 01:13:02 +02:00
sebres
6b56259f9a amend, obtain argument namespace before we'll use it 2025-03-31 01:11:05 +02:00
sebres
b2352f113e implements the feature of automatic switch backend = auto to backend systemd, when:
- no files matching `logpath` found for this jail;
- no `systemd_if_nologs = false` (`true` by default) is specified for the jail;
- option `journalmatch` is set for the jail or its filter (otherwise it'd be too heavy to allow all auto-jails, even if they have never been foreseen for journal);
- option `skip_if_nologs` will be ignored if we could switch backend to `systemd`;
closes gh-3768
2025-03-30 22:31:44 +02:00
sebres
5a2fd9b31c split new test to 2 tests (allows to cover _globJournalFiles even if system-journal is not available) 2025-03-30 20:13:39 +02:00
sebres
4eef68b3d3 backend systemd extended with new parameter rotated (default false, as prevention against "too many open files"), that allows to monitor only actual journals and ignore a lot of rotated files by default; so can drastically reduce amount of used file descriptors (to 1 or 2 per jail);
closes #3391
2025-03-30 19:03:32 +02:00
sebres
7a4985178f amend 2025-03-30 18:59:18 +02:00
sebres
786d5b7e9e test-suite: increase wait-time for fast-mode for long waiting intervals (stability, avoid sporadic errors) 2025-03-30 06:07:17 +02:00
sebres
191d1e9533 improve threaded clean-up of filters, new functions afterStop (to force clean-up after stop) and done, invoking afterStop once; ensure journal-reader is always closed (prevention against "too many open files"), thereby avoid sporadic segfault in systemd module (https://github.com/systemd/python-systemd/issues/143) 2025-03-30 06:04:49 +02:00
sebres
9f0b6382bf idle must be before anything else in loop (to avoid endless errors if something continuously fails and filter will be placed to idle state after 100 unhandled errors) 2025-03-30 06:04:47 +02:00
sebres
f49d50b8fd ensure the reader is really closed before reopen (preventing leaks if some handles or whatever are still open) 2025-03-30 06:04:44 +02:00
sebres
994a0b69da fixes systemd bug with missing journal descriptor after rotation by reopening of journal if it is recognized (it is not alive);
closes gh-3929
2025-03-30 00:53:27 +01:00
Sergey G. Brester
16ae53e888
Update main.yml
GHA: update python, 3.14.0-alpha.6 and pypy3.11
2025-03-28 23:07:27 +01:00
sebres
ee421dfbd6 filter.d/apache-noscript.conf - consider new log-format with "AH02811: stderr from /...";
closes gh-3900
2025-03-28 22:52:51 +01:00
sebres
b0d4eb07e5 command-line: test config shall output error directly and not using logger 2025-03-19 02:44:32 +01:00
sebres
d02a613e89 configreaders: don't swallow return code by decoding error (whole jail or fail2ban config failed to read due to some error like encoding etc), so dump or test of config would get an error at end (and coverage for #3971) 2025-03-19 02:19:16 +01:00
sebres
8ae6eaf39a filter.d/postfix.conf - default _daemon in prefix-line is loosened - can match everything starting with word postfix, like postfix-example.com/smtpd;
closes gh-3297
2025-03-10 22:35:26 +01:00
Sergey G. Brester
505d51fd5d
Update PULL_REQUEST_TEMPLATE.md 2025-03-04 19:19:57 +01:00
sebres
4bb1fd519d test-suite: if failed, sample regexs factory would show responsible header line (failJSON) together with the error line 2025-03-04 14:39:24 +01:00
sebres
cf9c8f1e9b test-suite: fixed sample regexs factory counting of line number (if it errors, the line number showing in error line was incorrect, because of missing increment) 2025-03-04 14:27:21 +01:00
Sergey G. Brester
c035428535
Merge pull request #3954 from luckylittle/feature/systemd-journal-vsftpd
`filter.d/vsftpd.conf` - fixed regex (if failures generated by systemd-journal)
2025-03-04 14:20:01 +01:00
sebres
79346e4f2c updated ChangeLog 2025-03-04 14:15:14 +01:00
sebres
94fe9cf4a8 more fixes, capture user names, more tests...
since line 7 matches successfully now (it was disabled in gh-358 because of obsolete format), it is marked as match:true (line can be removed later if unneeded)
2025-03-04 14:13:07 +01:00
sebres
1e06ab68b4 fixed filter (new regex is unneeded), tests format of failures produced by system journal 2025-03-04 13:47:59 +01:00
Sergey G. Brester
e9a42847bc
Merge pull request #3955 from luckylittle/feature/systemd-journal-lighttpd
`filter.d/lighttpd-auth.conf` - fixed regex (if failures generated by systemd-journal), bypass several prefixes now
2025-03-04 13:21:43 +01:00
Sergey G. Brester
3e9a4b4a48
Update ChangeLog 2025-03-04 13:20:54 +01:00
Sergey G. Brester
95cdf553f5
fixes test in lighttpd-auth: added failJSON to match the line 2025-03-04 13:09:21 +01:00
Sergey G. Brester
13a74feaad
2nd RE unneeded, fix single RE - bypass everything before open parenthesis 2025-03-04 13:02:50 +01:00
Lucian Maly
6e3bfd800c
Added author 2025-03-04 12:26:14 +11:00
Lucian Maly
9d7646e6c0
Added author 2025-03-04 12:25:27 +11:00
Lucian Maly
f5ba525cd2
Added sample log line 2025-03-04 12:22:35 +11:00
Lucian Maly
fd1d0d25a8
Added regex for systemd-journal matches of lighttpd-auth 2025-03-04 12:20:24 +11:00
Lucian Maly
bd4cb606e5
Added sample log line 2025-03-04 11:47:49 +11:00
Lucian Maly
65d473fc8e
Added regex for systemd-journal matches of vsftpd 2025-03-04 11:43:38 +11:00
sebres
e3ab969047 increase interval for up-to-date check (to 1 minute) after error, to avoid continuous flood in log on further possible errors 2025-03-04 00:07:31 +01:00
sebres
9145db8de3 small code review of FileIPAddrSet: encapsulate check for changed logic to _isModified and slightly increase coverage for it (latency, changed, unchanged) 2025-03-03 23:59:36 +01:00
sebres
7233edd0bf amend ChangeLog updated: ignoreip extended with file:... syntax to ignore IPs from file-ip-set;
+ silence codespell
2025-03-03 20:07:05 +01:00
sebres
c54f1a4603 Merge branch 'ignore-file-ip-addr-set':
configuration `ignoreip` and fail2ban-client commands `addignoreip`/`delignoreip` extended with `file:...` syntax to ignore IPs from file-ip-set (containing IP, subnet, dns/fqdn or raw strings);
the file would be read lazy on demand, by first ban (and automatically reloaded by update after small latency to avoid expensive stats check on every compare);
the entries inside the file can be separated by comma, space or new line with optional comments (text following chars # or ; after space or newline would be ignored up to next newline)
2025-03-03 20:00:32 +01:00
sebres
5bea1c87f1 add few comments to test-ign-ips-file for the sake of completeness and coverage 2025-03-03 19:52:23 +01:00
sebres
6efa3a3144 man extended (ignoreip supports file://path/file-with-ip-set) 2025-03-03 19:19:21 +01:00
sebres
fe37047061 test coverage for FileIPAddrSet and ignoreip for file://... 2025-03-03 19:06:08 +01:00
sebres
81a5b1596b filter and configuration ignoreip extended with file:... to ignore IPs from file-ip-set (containing IP, subnet, dns/fqdn or raw strings); the file would be read lazy on demand, by first ban (and automatically reloaded by update after small latency) 2025-03-03 19:03:48 +01:00
sebres
d684339edd allow comments in file with ip-set: text followed # or ; chars after space or newline would be ignored 2025-03-03 19:00:09 +01:00
sebres
bdae15b522 ipdns.py: implemented FileIPAddrSet supporting file with IP-set, what may contain IP, subnet, or dns, with lazy load and dynamically reloaded by changes (with small latency to avoid expensive stats check on every compare) 2025-03-03 18:40:15 +01:00
Sergey G. Brester
c9b5e845ba
action.d/cloudflare-token.conf: fixes actionunban retrieving of CF-ID from IP:
force adding parameters to URL as query string (add `-G` to curl);
closes gh-3952
2025-03-01 20:19:35 +01:00
Sergey G. Brester
e5199aee92
action.d/ufw.conf: update comment:
fix syntax in example, because `dst` as command parameter doesn't have precedence over or-expression, so second `sport` would ignore `dst` and kill any connection for https regardless the IP
2025-03-01 00:23:55 +01:00
sebres
1c61836169 main.yml: merge branch 'gha-try-new-runner':
Some checks failed
Codespell / Check for spelling errors (push) Has been cancelled
CI / build (3.10) (push) Has been cancelled
CI / build (3.11) (push) Has been cancelled
CI / build (3.12) (push) Has been cancelled
CI / build (3.13) (push) Has been cancelled
CI / build (3.14.0-alpha.5) (push) Has been cancelled
CI / build (3.8) (push) Has been cancelled
CI / build (3.9) (push) Has been cancelled
CI / build (pypy3.10) (push) Has been cancelled
- update runner image (20.04 gets end of date)
- update python versions (v.3.7 is unsupported for 24.04, bump v.3.14 to next alpha)
2025-02-25 18:38:19 +01:00