Support for paths allows to narrow down specific static extended
filters to specific webpages on a given site.
Examples of usage:
example.com/toto##h1
/example\.com\/toto\d+/#@#h1
For the sake of convenience for filter list maintainers, this commit
add ability to quote static network option values, so as to avoid the
need to escape commas when parser ambiguity arises.
The quotes can be `"`, `'`, or backticks.
Example, the following filter requires escaping commas:
example.com$xhr,replace=/"loremIpsum.*?([A-Z]"\}|"\}{2\,4})\}\]\,//,1p
Can be now rewritten with no need to escape when using quotes:
example.com$xhr,replace='/"loremIpsum.*?([A-Z]"\}|"\}{2,4})\}\],//',1p
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/3161
- https://github.com/uBlockOrigin/uBlock-issues/discussions/2895#discussioncomment-8504374
Two checkboxes have been added to the "My filters "pane:
1. A checkbox to wholly disable/enable "My filters". This is equivalent
to the checkbox for "My filters" in "Filter lists" pane.
2. A checkbox to enable/disable the trustworthiness of the content
of "My filters". Default to untrusted.
Since toggling these checkboxes requires reloading all filter lists,
their new state must be committed through the "Apply changes" button.
Additionally: a "book" icon has been added to the top-right of the
dashboard, which is a link to the wiki according to whichever pane is
currently active.
Related discussion:
https://github.com/uBlockOrigin/uBlock-issues/discussions/2895
Changes:
The _content of the My filters_ pane is now considered untrusted by
default, and only uBO's own lists are now trusted by default.
It has been observed that too many people will readily copy-paste
filters from random sources. Copy-pasting filters which require trust
represents a security risk to users with no understanding of how the
filters work and their potential abuse.
Using a filter which requires trust in a filter list from an untrusted
source will cause the filter to be invalid, i.e. shown as an error.
A new advanced setting has been added to control which lists are
considered trustworthy: `trustedListPrefixes`, which is a space-
separated list of tokens. Examples of possible values:
- `ublock-`: trust only uBO lists, exclude everything else including
content of _My filters_ (default value)
- `ublock- user-`: trust uBO lists and content of _My filters_
- `-`: trust no list, essentially disabling all filters requiring
trust (admins or people who don't trust us may want to use this)
One can also decide to trust lists maintained elsewhere. For example,
for stock AdGuard lists add ` adguard-`. To trust stock EasyList lists,
add ` easylist-`.
To trust a specific regional stock list, look-up its token in
assets.json and add to `trustedListPrefixes`.
The matching is made with String.startsWith(), hence why `ublock-`
matches all uBO's own filter lists.
This also allows to trust imported lists, for example add
` https://filters.adtidy.org/extension/ublock/filters/` to trust all
non-stock AdGuard lists.
Add the complete URL of a given imported list to trust only that one
list.
URLs not starting with `https://` or `file:///` will be rejected,
i.e. `http://example.org` will be ignored.
Invalid URLs are rejected.
The `urltransform` option allows to redirect a non-blocked network
request to another URL. There are restrictions on its usage:
- require a trusted source -- thus uBO-maintained lists or user
filters
- the `urltransform` value must start with a `/`
If at least one of these conditions is not fulfilled, the filter
will be invalid and rejected.
The requirement to start with `/` is to enforce that only the path
part of a URL can be modified, thus ensuring the network request
is redirected to the same scheme and authority (as defined at
https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Syntax).
Usage example (redirect requests for CSS resources to a non-existing
resource, for demonstration purpose):
||iana.org^$css,urltransform=/notfound.css
Name of this option is inspired from DNR API:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/declarativeNetRequest/URLTransform
This commit required to bring the concept of "trusted source" to
the static network filtering engine.
Reference:
https://adguard.com/kb/general/ad-filtering/create-own-filters/#conditions-directive
This commit should make uBO fully compatible with the `!#if`
directives found throughout AdGuard's filter lists.
Additionally, added the new `!#else` directive for convenience
to filter list authors:
!#if cap_html_filtering
example.com##^script:has-text(fakeAd)
!#else
example.com##+js(rmnt, script, fakeAd)
!#endif
The syntax highlighter could throw with some invalid static
network filter patterns. This was caused by the syntax
highlighter still drilling down the pattern parts after
having told codemirror to style the whole pattern as an
error, thus causing the codemirror stream position to go
backward.
This allows to reduce the horizontal size of the gutter; more efficient
management of folding since we already need to parse each line for the
linter; and eventually this will also allows to detect unbalanced
!#if-!#endif directives and report such cases as errors.
Additionally, keep incrementally improving error reporting details
in the linter.
As per CodeMirror's documentation, eachLine() iterator is
faster, so use this. Also no need to keep track or marked
lines, we can just find them on demand, this makes the code
simpler.
This commit is a rewrite of the static filtering parser into a
tree-based data structure, for easier maintenance and better
abstraction of parsed filters.
This simplifies greatly syntax coloring of filters and also
simplify extending filter syntax.
The minimum version of Chromium-based browsers has been raised
to version 73 because of usage of String.matchAll().
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1791
The following case of invalid syntax was not reported as
invalid by the syntax highlighter:
... example.com image ...
With dynamic filtering, there can't be a specific
hostname when a specific type is used, or a
specific type when a specific hostname is used, one
or the other must be `*`.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1690
New procedural operator: `:matches-path(...)`
Description: this is a all-or-nothing passthrough operator, which
on/off behavior is dictated by whether the argument match the
path of the current location. The argument can be either plain
text to be found at any position in the path, or a literal regex
against which the path is tested.
Whereas cosmetic filters can be made specific to whole domain,
the new `:matches-path()` operator allows to further narrow
the specificity according to the path of the current document
lcoation.
Typically this procedural operator is used as first operator in
a procedural cosmetic filter, so as to ensure that no further
matching work is performed should there be no match against the
current path of the current document location.
Example of usage:
example.com##:matches-path(/shop) p
Will hide all `p` elements when visiting `https://example.com/shop/stuff`,
but not when visiting `https://example.com/` or any other page
on `example.com` which has no instance of `/shop` in the path part
of the URL.
The code exported to nodejs package was revised to use modern
JavaScript syntax. A few issues were fixed at the same time.
The exported classes are:
- DynamicHostRuleFiltering
- DynamicURLRuleFiltering
- DynamicSwitchRuleFiltering
These related to the content the of "My rules" pane in the
uBlock Origin extension.
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/1664
The changes are enough to fulfill the related issue.
A new platform has been added in order to allow for building
a NodeJS package. From the root of the project:
./tools/make-nodejs
This will create new uBlock0.nodejs directory in the
./dist/build directory, which is a valid NodeJS package.
From the root of the package, you can try:
node test
This will instantiate a static network filtering engine,
populated by easylist and easyprivacy, which can be used
to match network requests by filling the appropriate
filtering context object.
The test.js file contains code which is typical example
of usage of the package.
Limitations: the NodeJS package can't execute the WASM
versions of the code since the WASM module requires the
use of fetch(), which is not available in NodeJS.
This is a first pass at modularizing the codebase, and
while at it a number of opportunistic small rewrites
have also been made.
This commit requires the minimum supported version for
Chromium and Firefox be raised to 61 and 60 respectively.
The syntax to remove response header is a special case
of HTML filtering, whereas the response headers are
targeted, rather than the response body:
example.com##^responseheader(header-name)
Where `header-name` is the name of the header to
remove, and must always be lowercase.
The removal of response headers can only be applied to
document resources, i.e. main- or sub-frames.
Only a limited set of headers can be targeted for
removal:
location
refresh
report-to
set-cookie
This limitation is to ensure that uBO never lowers the
security profile of web pages, i.e. we wouldn't want to
remove `content-security-policy`.
Given that the header removal occurs at onHeaderReceived
time, this new ability works for all browsers.
The motivation for this new filtering ability is instance
of website using a `refresh` header to redirect a visitor
to an undesirable destination after a few seconds.
This commit fixes mouse double-click-and-drag operations,
which was broken due to the implementation of a custom
word selection in the filter list editor/viewer.
Regex-based static network filters are those most likely to
cause performance degradation, and as such the best guard
against undue performance degradation caused by regex-based
filters is the ability to extract valid and good tokens
from regex patterns.
This commit introduces a complete regex parser so that the
static network filtering engine can now safely extract
tokens regardless of the complexity of the regex pattern.
The regex parser is a library imported from:
https://github.com/foo123/RegexAnalyzer
The syntax highlighter adds an underline to regex-based
filters as a visual aid to filter authors so as to avoid
mistakenly creating regex-based filters. This commit
further colors the underline as a warning when a regex-based
filter is found to be untokenizable.
Filter list authors are invited to spot these untokenizable
regex-based filters in their lists to verify that no
mistake were made for those filters, causing them to be
untokenizabke. For example, what appears to be a mistake:
/^https?:\/\/.*\/sw.js?.[a-zA-Z0-9%]{50,}/
Though the mistake is minor, the regex-based filter above
is untokenizable as a result, and become tokenizable when
the `.` is properly escaped:
/^https?:\/\/.*\/sw\.js?.[a-zA-Z0-9%]{50,}/
Filter list authors can use this search expression in the
asset viewer to find instances of regex-based filters:
/^(@@)?\/[^\n]+\/(\$|$)/
Related issue:
- https://github.com/uBlockOrigin/uBlock-issues/issues/857
The recognized resources are:
- abp-resource:blank-mp3
- abp-resource:blank-js
ABP's tokens are excluded from auto-complete so as to not
get in the way of uBO's filter list maintainers.
