If an IP address can be extracted from the hostname portion of
a URL, the IP address matching will be performed at onBeforeRequest()
time.
Regardless, IP address matching will subsequently always be performed
at onHeadersReceived() time as the request details at that point
contain a reliable IP address value on supported platforms (Firefox-
only as of now).
The `cap_ipaddress` now evaluates to `true` in Chromium-based
browsers. Even though these browsers are unable to provide reliable
IP address value at onHeadersReceived() time, they can still
perform IP address matching for IP address extracted from hostname
portion of a URL.
Properly report unsupported `header=` and `ipaddress=` option in
log file.
`header=` support may become possible soon with Chromium 128
introducing blocking according to response headers content.
The purpose is to block according to the ip address of a network
request. In the current implementation, the filter option can only
be enforced at onHeadersReceived time.
The new filter option cannot be enforced in Chromium-based browsers
since the ip address of network requests is available only at
onResponseStarted time, which is not blocking.
The value assigned to `ipaddress` can either be a plain string which
must match exactly a given ip address, or a regex which will be
matched against the ip address.
The `ipaddress` option can only be enforced when the extension
framework does provide a valid ip address in a onHeadersReceived
listener. For instance, cached resources do not have a valid ip
address and thus can't be a match to `ipaddress` option.
Example:
*$script,ipaddress=93.184.215.14
For the sake of convenience for filter list maintainers, this commit
add ability to quote static network option values, so as to avoid the
need to escape commas when parser ambiguity arises.
The quotes can be `"`, `'`, or backticks.
Example, the following filter requires escaping commas:
example.com$xhr,replace=/"loremIpsum.*?([A-Z]"\}|"\}{2\,4})\}\]\,//,1p
Can be now rewritten with no need to escape when using quotes:
example.com$xhr,replace='/"loremIpsum.*?([A-Z]"\}|"\}{2,4})\}\],//',1p
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/3362
There used to be a way to test URL against the network filtering engine,
but this was removed in a distant past during refactoring.
The ability has been brought back through uBO's own developer tools,
accessible through the _More_ button in the _Support_ pane in the
dashboard.
To query the static network filtering engine, enter the following
in the text editor:
snfe?url-to-test [type] [url-of-context]
`snfe?` is a prompt indicating the intent to query the static network
filtering engine.
At a minimum there must be a URL to test.
Optionally the type of the resource to match, default to `xhr` if
none specified. Also optionally, the context from within which the
request is made. Example:
Enter:
snfe?https://www.google-analytics.com/analytics.js
Result:
url: https://www.google-analytics.com/analytics.js
blocked: ||google-analytics.com^
Enter:
snfe?https://www.google-analytics.com/analytics.js script
Result:
url: https://www.google-analytics.com/analytics.js
type: script
blocked: ||google-analytics.com^
modified: ||google-analytics.com/analytics.js$script,redirect-rule=google-analytics_analytics.js:5
Enter:
snfe?https://example.com/
Result:
url: https://example.com/
not blocked
Enter:
snfe?https://example.com/ ping
Result:
url: https://example.com/
type: ping
blocked: *$ping,3p
A new icon has been added to the popup panel, to open a popup
window with a terse list of DNR events for the current tab, in
reverse chronological order (most recent DNR event appears at
the top).
The new ability is available only when the extension is sideloaded,
as per `declarativeNetRequestFeedback` documentation. Ref:
https://developer.chrome.com/docs/extensions/reference/api/declarativeNetRequest#event-onRuleMatchedDebug
Purposefully minimal, so as to have something rather than nothing
when having to diagnose filtering issue with the DNR API. Example:
https://github.com/uBlockOrigin/uBOL-home/issues/156
The content of the popup window does not dynamically update, force
a refresh (F5) to get the most recent DNR events. This might be
improved in the future.
The DNR event buffer is not persisted, so the buffer is empty when
service worker is restarted. This might be improved in the future
by using session storage API.
There is no output filtering ability in this first draft. This
might be improved in the future.
DNR rules are reported. The filter from which a DNR rule
originates is not reported. Given that the rulesets are optimized
after conversion from original filter lists to reduce the DNR rule
count, this is unlikely to ever be possible.
`subject:matches-prop(arg)`
Description: Allows to select an element by a property name (or chain of
properties), and optionally the property value.
Chainable: Yes.
`subject`: Can be a plain CSS selector, or a procedural cosmetic filter.
`arg`: A declaration in the form `chain=value`, where `chain` is a dot-
separated string for the target property, and `value` is the optional
property value to match. `value` can be literal text or literal regular
expression. When no `value` is declared, the operator only tests for
the presence of the target property
Example:
example.org##div:matches-prop(imanad)
example.org##img:matches-prop(naturalWidth=160)
Corrpution would occur when reading back serialized data which
contained multiple references to same instance of an object.
The issue could manifest when reading cache storage-related
data from the browser storage API, since the serializer is not
used when reading from indexedDB. Private/incognito mode
fall back on using browser storage API as cache storage.
Off the top of my head, I think the following conditions all
together could result in high likelihood of malfunction caused
by improperly deserializing data at launch time:
- Load from a selfie
- Selfie created after uBO ran for a while
- Selfie loaded from browser storage API (not the case by
default)
Possibly related to reports of uBO malfunctioning:
https://github.com/uBlockOrigin/uBlock-issues/issues/3217#event-12686416838
For internal use by filter list maintainers, do not open issues
about this. Left undocumented on purpose.
This new procedural operator allows to target elements in the
shadow root of an element.
subject:shadow(arg)
- Description: Look-up matching elements inside the shadow root (if
present) of _subject_.
- Chainable: Yes
- _subject_: Can be a plain or procedural selector.
- _arg_: A plain or a procedural selector for the elements to target
inside the shadowroot.
Example:
..##body > div:not([class]):shadow(div[style]):has(:shadow([data-i18n^="#ad"]))
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/3161
- https://github.com/uBlockOrigin/uBlock-issues/discussions/2895#discussioncomment-8504374
Two checkboxes have been added to the "My filters "pane:
1. A checkbox to wholly disable/enable "My filters". This is equivalent
to the checkbox for "My filters" in "Filter lists" pane.
2. A checkbox to enable/disable the trustworthiness of the content
of "My filters". Default to untrusted.
Since toggling these checkboxes requires reloading all filter lists,
their new state must be committed through the "Apply changes" button.
Additionally: a "book" icon has been added to the top-right of the
dashboard, which is a link to the wiki according to whichever pane is
currently active.
This will make HTML filtering and `replace=` filter option less
likely to be bypassed by uBO, as the body response filterer
previously required an encoding to be expressly declared before
acting on the response body.
UTF-8 usage is currently reported as ~98.2%:
https://w3techs.com/technologies/history_overview/character_encoding
To disable collating global blocked/allowed counts.
Boolean, default to `false`.
Setting to `true` will prevent uBO from loading/saving global
blocked/allowed counts, and in such case the "Blocked since
install" count instead reflects the count since uBO launched.
Setting back to `false` will cause the counts to resume from
last time they were saved.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/3100
Turns out it's currently the fastest among the three currently
implemented (Cache, browser.storage.session, indexedDB). Possibly
because indexedDB can natively persist structure-cloneable data,
something uBO can now benefit with the work on abstracting away
the limitations of various storages being limited to persist only
text or JSON data.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2969
In uBO, the "cache storage" is used to save resources which can
be safely discarded, though at the cost of having to fetch or
recompute them again.
Extension storage (browser.storage.local) is now always used as
cache storage backend. This has always been the default for
Chromium-based browsers.
For Firefox-based browsers, IndexedDB was used as backend for
cache storage, with fallback to extension storage when using
Firefox in private mode by default.
Extension storage is reliable since it works in all contexts,
though it may not be the most performant one.
To speed-up loading of resources from extension storage, uBO will
now make use of Cache API storage, which will mirror content of
key assets saved to extension storage. Typically loading resources
from Cache API is faster than loading the same resources from
the extension storage.
Only resources which must be loaded in memory as fast as possible
will make use of the Cache API storage layered on top of the
extension storage.
Compiled filter lists and memory snapshot of filtering engines
(aka "selfies") will be mirrored to the Cache API storage, since
these must be loaded into memory as fast as possible, and reloading
filter lists from their compiled counterpart is a common
operation.
This new design makes it now seamless to work in permanent private
mode for Firefox-based browsers, since extension storage now
always contains cache-related assets.
Support for IndexedDB is removed for the time being, except to
support migration of cached assets the first time uBO runs with
the new cache storage design.
In order to easily support all choices of storage, a new serializer
has been introduced, which is capable of serializing/deserializing
structure-cloneable data to/from a JS string.
Because of this new serializer, JS data structures can be stored
directly from their native representation, and deserialized
directly to their native representation from uBO's point of view,
since the serialization occurs (if needed) only at the storage
interface level.
This new serializer simplifies many code paths where data
structures such as Set, Map, TypedArray, RegExp, etc. had to be
converted in a disparate manner to be able to persist them to
extension storage.
The new serializer supports workers and LZ4 compression. These
can be configured through advanced settings.
With this new layered design, it's possible to introduce more
storage layers if measured as beneficial (i.e. maybe
browser.storage.session)
References:
- https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/local
- https://developer.mozilla.org/en-US/docs/Web/API/Cache
- https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm
As reported internally to ubo-security by https://github.com/distinctmondaylila
One issue is a regression from the rewriting of the static filtering
parser in version 1.47.0, specifically the following commit:
https://github.com/gorhill/uBlock/commit/8ea3b0f64c
The existing regex was no longer suitable to properly detect
some usage of `report-xxx` in the rwritten parser.
Another issue which predates 1.47.0 is that the regex used for
validation was case-sensititive, while the `report-uri` directive
can be written using uppercase letters, i.e. `Report-uri`.
Avoid race conditions between isolated world-side broadcast channel
and main-side broadcast channel, so as to not lose logging
information if the isolated world-side is not yet ready to
receive through its broadcast channel.
Additionally, added new scriptlet: `trusted-replace-argument`.
[...]##+js(trusted-replace-argument, fn, argpos, argval [,condition, pattern])
Where:
- `fn` is the function we want to proxy through an `apply` handler.
This can also be a class, in which case the scriptlet will proxy
through `construct` handler. At the moment, `fn` must exist at the
time the scriptlet executes.
- `argpos` is the 0-based position of the argument we want to change
- `argval` is the value we want to have for the argument -- the value
is interpreted the same way the value for `set-constant` is
interpreted.
- `condition, pattern` is a vararg which tells the scriptlet to act
only if `pattern` is found in the argument to overwrite.
Example of usage:
alliptvlinks.com##+js(trusted-replace-argument, MutationObserver, 0, noopFunc)
This commit brings the following changes to the logger:
All logging output generated by injected scriptlets are now sent to
the logger, the developer console will no longer be used to log
scriptlet logging information.
When the logger is not opened, the scriplets will not output any
logging information.
The goal with this new approach is to allow filter authors to
more easily assess the working of scriptlets without having to
go through scriptlet parameters to enable logging.
Consequently all the previous ways to tell scriptlets to log
information are now obsolete: if the logger is opened, the
scriptlets will log information to the logger.
Another benefit of this approach is that the dev tools do not
need to be open to obtain scriptlets logging information.
Accordingly, new filter expressions have been added to the logger:
"info" and "error". Selecting the "scriptlet" expression will also
keep the logging information from scriptlets.
A new button has been added to the logger (not yet i18n-ed): a
"volume" icon, which allows to enable verbose mode. When verbose
mode is enabled, the scriptlets may choose to output more
information regarding their inner working.
The entries in the logger will automatically expand on mouse hover.
This allows to scroll through entries which text does not fit into
a single row.
Clicking anywhere on an entry in the logger will open the detailed
view when applicable.
Generic information/errors will now be rendered regardless of which
tab is currently selected in the logger (similar to how tabless
entries are already being rendered).
Procedural filters with `:xpath` operator were silently rejected
at conversion time because the parser was failing to evaluate the
xpath expression due to the absence of a `document` object in
nodejs.
If `document` object is not present, the parser will assume the
xpath expression is valid.
Scriptlets parameters which are quoted must be re-quoted when
output to the logger to be sure they can be properly looked up
in the list, and that they can be used through copy-paste
operations.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/3083
This will not completely eliminate the issue but it should
lower the likelihood it will occur -- so at least uBO can
still benefit from reliable scriptlet execution in Firefox.
The idea is to remove as many dependencies as possible for
low-level ScriptletFilteringEngine in order to make it easier
to reuse the module outside uBO itself.
The high-level derived class takes care of caching and
injection of scriptlets into documents, which requires
more knowledge about the environment in which scriptlets
are to be used.
Also improve scriptlet cache usage to minimize overhead of
retrieving scriptlets.
If the `manual` parameter is assigned a date in the form of
`YYMMDD`, this will tell uBO to update lists from origin sources
when the current time is within the range of the specified date,
otherwise lists will be updated from CDNs. Updating from CDNs
is always strongly recommended since this enables differential
updates.
For the time being, `manual=1` will always cause to update lists
from origin, but this form will be deprecated once next stable
release is widespread. The idea is to not leave behind stale
and obsolete links which would be detrimental to differential
updates should someone click on one of these old links left
behind.
Purging all the lists from cache storage is detrimental to
differential update, and cause filter lists to be updated less
often and consequently to be less up to date then when letting
differential updater do its work.
Possibly related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2136
Also reported internally, steps to reproduce the issue fixed here:
- Open uBO's dashboard through 3-dot > Add-ons > uBO > Settings
- Bring forth "Filter lists" pane
We want the tab to be already opened at next launch
- Quit Firefox for Android
- Launch Firefox for Android
Result:
Very long launch time, lists marked as out of date.
In Support pane's troubleshooting information, a delta sign will
be added to list update information when the list was last updated
through differential update.
Related commit:
https://github.com/uBlockOrigin/uAssets/commit/20312c2178
To work around incompatibity with `urltransform` in 1.53.0.
With this commit support for `urltransform` is dropped in
favor of `uritransform`. Since `uritransform` won't be parsed
by older versions of uBO, the change in syntax for `urltransform`
between 1.53.0 and 1.54.0 will no longer cause issue.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2969
Changes:
Use browser.alarms to trigger selfie creation. Presence of a selfie
improve markedly time to readiness when uBO is unsuspended.
Mirror content of storage.local to (in-memory) storage.session for
faster load to readiness when uBO is ususpended.
Broadcast channels are more suited to uBO than DOM events to dispatch
notifications to different parts of uBO.
DOM events can only be dispatched to local context, broadcast channels
dispatch to all contexts (i.e. background process, workers, auxiliary
pages) -- this last behavior is better suited to uBO to communicate
internal changes to all potential listeners, not just those in the local
context.
Additionally, broadcasting to content scripts is now done through
tabs.sendMessage() instead of through potentially opened message
ports, this simplifies broadcasting to content scripts, and this
doesn't require to have long-lived message ports in content
scripts.
Fortunately, getAvailableLists() filters out non-filter
list resources thus preventing those mistakenly selected
resources from being loaded as filter lists.
There is a lot asynchronicity in the auto-update code, and
the fix here is to detect then fix instances of out of sync
state between a cached filter list and its metadata stored
separately.
When done compiling, force CSSTree to parse an empty string, so
as to ensure it doesn't keep a reference to that string.
Typically, the string passed to CSSTree is a small slice of a
larger string which is a whole filter list. This means that
holding a reference to the sliced string causes the JS engine
to hold in memory to the whole filter list last parsed.
Reference documentation:
https://adguard.com/kb/general/ad-filtering/create-own-filters/#replace-modifier
This is a network filter option which can only be loaded from a
trusted source.
Since this filter is about modifying the response body, it currently
only works in Firefox.
As discussed with filter list maintainers.
Manual update of one or more lists will cause the most recent version
of these lists to be fetched from the "origin" server, and since the
lists from "origin" servers cannot be updated through differential
update, the lists will be subsequently updated according to their
`Expires` directive.
When the lists are auto-updated, the "CDN" servers will be used,
and as a result the lists will start to be updated trhough
differential updates every 6-hour (currently).
Thus it is recommended and optimal to let the lists auto-update,
since you will benefit from a much shorter delay to get up-to-date
lists (i.e. every 6-hour instead of every 6-day).
You can force the auto-updater to fetch all the lists by clicking
"Purge all caches", then restart uBO without clicking "Update".
This will cause uBO to perform an emergency auto-update at restart
time, after which you will have all the lists which are candidates
for differential update.
The "Update now" button in the "Support" pane will also cause lists
to be fetched from their "origin" server.
Related discussion:
https://github.com/ameshkov/diffupdates
The benefits of diff-patching filter lists is much shorter update
schedule and significantly less bandwidth consumed.
At the moment, only default filter lists are subject to be
diff-patched.
External filter lists can make their lists diff-patchable by
following the specification link above.
Only filter lists fetched by the auto-updater are candidate for
diff-patching.
Forcing a manual update of the filter lists will prevent the
diff-patcher from kicking in until one or more lists are
auto-updated.
Some back-of-the-envelop calculations regarding the load on free
CDN solutions used by uBO to distribute its own filter lists:
Currently, for each CDN (with lists updating after days):
~560 M req/month, ~78 TB/month
With diff-patching lists on a 6-hour schedule:
~390 M req/month, 1 TB/month
Those estimates were done according to statistics shown by
jsDelivr, which is one of 4 CDNs picked randomly when a list
updates:
https://www.jsdelivr.com/package/gh/uBlockOrigin/uAssetsCDN?tab=stats
Related discussion:
https://github.com/uBlockOrigin/uBlock-issues/discussions/2895
Changes:
The _content of the My filters_ pane is now considered untrusted by
default, and only uBO's own lists are now trusted by default.
It has been observed that too many people will readily copy-paste
filters from random sources. Copy-pasting filters which require trust
represents a security risk to users with no understanding of how the
filters work and their potential abuse.
Using a filter which requires trust in a filter list from an untrusted
source will cause the filter to be invalid, i.e. shown as an error.
A new advanced setting has been added to control which lists are
considered trustworthy: `trustedListPrefixes`, which is a space-
separated list of tokens. Examples of possible values:
- `ublock-`: trust only uBO lists, exclude everything else including
content of _My filters_ (default value)
- `ublock- user-`: trust uBO lists and content of _My filters_
- `-`: trust no list, essentially disabling all filters requiring
trust (admins or people who don't trust us may want to use this)
One can also decide to trust lists maintained elsewhere. For example,
for stock AdGuard lists add ` adguard-`. To trust stock EasyList lists,
add ` easylist-`.
To trust a specific regional stock list, look-up its token in
assets.json and add to `trustedListPrefixes`.
The matching is made with String.startsWith(), hence why `ublock-`
matches all uBO's own filter lists.
This also allows to trust imported lists, for example add
` https://filters.adtidy.org/extension/ublock/filters/` to trust all
non-stock AdGuard lists.
Add the complete URL of a given imported list to trust only that one
list.
URLs not starting with `https://` or `file:///` will be rejected,
i.e. `http://example.org` will be ignored.
Invalid URLs are rejected.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2896
TODO: Eventually, distinguish between filtering profile increasing
or decreasing so as to avoid flushing caches when increasing
filtering, which should not affect the scriptlets cache.
(In addition to in already supported single- and double-quote).
The parsing of (optionally) quoted arguments from an argument
list has been spinned off into a standalone helper in order to
be reused in other parts of the parser eventually.
The `urltransform` option allows to redirect a non-blocked network
request to another URL. There are restrictions on its usage:
- require a trusted source -- thus uBO-maintained lists or user
filters
- the `urltransform` value must start with a `/`
If at least one of these conditions is not fulfilled, the filter
will be invalid and rejected.
The requirement to start with `/` is to enforce that only the path
part of a URL can be modified, thus ensuring the network request
is redirected to the same scheme and authority (as defined at
https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Syntax).
Usage example (redirect requests for CSS resources to a non-existing
resource, for demonstration purpose):
||iana.org^$css,urltransform=/notfound.css
Name of this option is inspired from DNR API:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/declarativeNetRequest/URLTransform
This commit required to bring the concept of "trusted source" to
the static network filtering engine.
As per discussion with uBO volunteers.
Volunteers offering support for uBO will be able to craft links with
specially formed URLs, which once clicked will cause uBO to automatically
force an update of specified filter lists.
The URL must be crafted as shown in the example below:
https://ublockorigin.github.io/uAssets/update-lists.html?listkeys=ublock-filters,easylist
Where the `listkeys` parameter is a comma-separated list of tokens
corresponding to filter lists. If a token does not match an enabled
filter list, it will be ignored.
The ability to update filter lists through a specially crafted link
is available only on uBO's own support sites:
- https://github.com/uBlockOrigin/
- https://reddit.com/r/uBlockOrigin/
- https://ublockorigin.github.io/
Additionally, a visual cue has been added in the "Filter lists" pane
to easily spot the filter lists which have been recently updated, where
"recently" is currently defined as less than an hour ago.
