DNR conversion module is a dev tool, no point loading it by
default. This was done this way because in a distant past uBO
had to be compatible with browsers not supporting dynamically
loaded modules.
Currently all supported browser versions support dynamically
loaded modules.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/3362
There used to be a way to test URL against the network filtering engine,
but this was removed in a distant past during refactoring.
The ability has been brought back through uBO's own developer tools,
accessible through the _More_ button in the _Support_ pane in the
dashboard.
To query the static network filtering engine, enter the following
in the text editor:
snfe?url-to-test [type] [url-of-context]
`snfe?` is a prompt indicating the intent to query the static network
filtering engine.
At a minimum there must be a URL to test.
Optionally the type of the resource to match, default to `xhr` if
none specified. Also optionally, the context from within which the
request is made. Example:
Enter:
snfe?https://www.google-analytics.com/analytics.js
Result:
url: https://www.google-analytics.com/analytics.js
blocked: ||google-analytics.com^
Enter:
snfe?https://www.google-analytics.com/analytics.js script
Result:
url: https://www.google-analytics.com/analytics.js
type: script
blocked: ||google-analytics.com^
modified: ||google-analytics.com/analytics.js$script,redirect-rule=google-analytics_analytics.js:5
Enter:
snfe?https://example.com/
Result:
url: https://example.com/
not blocked
Enter:
snfe?https://example.com/ ping
Result:
url: https://example.com/
type: ping
blocked: *$ping,3p
Related issues:
- https://github.com/uBlockOrigin/uBlock-issues/issues/3161
- https://github.com/uBlockOrigin/uBlock-issues/discussions/2895#discussioncomment-8504374
Two checkboxes have been added to the "My filters "pane:
1. A checkbox to wholly disable/enable "My filters". This is equivalent
to the checkbox for "My filters" in "Filter lists" pane.
2. A checkbox to enable/disable the trustworthiness of the content
of "My filters". Default to untrusted.
Since toggling these checkboxes requires reloading all filter lists,
their new state must be committed through the "Apply changes" button.
Additionally: a "book" icon has been added to the top-right of the
dashboard, which is a link to the wiki according to whichever pane is
currently active.
In uBO, the "cache storage" is used to save resources which can
be safely discarded, though at the cost of having to fetch or
recompute them again.
Extension storage (browser.storage.local) is now always used as
cache storage backend. This has always been the default for
Chromium-based browsers.
For Firefox-based browsers, IndexedDB was used as backend for
cache storage, with fallback to extension storage when using
Firefox in private mode by default.
Extension storage is reliable since it works in all contexts,
though it may not be the most performant one.
To speed-up loading of resources from extension storage, uBO will
now make use of Cache API storage, which will mirror content of
key assets saved to extension storage. Typically loading resources
from Cache API is faster than loading the same resources from
the extension storage.
Only resources which must be loaded in memory as fast as possible
will make use of the Cache API storage layered on top of the
extension storage.
Compiled filter lists and memory snapshot of filtering engines
(aka "selfies") will be mirrored to the Cache API storage, since
these must be loaded into memory as fast as possible, and reloading
filter lists from their compiled counterpart is a common
operation.
This new design makes it now seamless to work in permanent private
mode for Firefox-based browsers, since extension storage now
always contains cache-related assets.
Support for IndexedDB is removed for the time being, except to
support migration of cached assets the first time uBO runs with
the new cache storage design.
In order to easily support all choices of storage, a new serializer
has been introduced, which is capable of serializing/deserializing
structure-cloneable data to/from a JS string.
Because of this new serializer, JS data structures can be stored
directly from their native representation, and deserialized
directly to their native representation from uBO's point of view,
since the serialization occurs (if needed) only at the storage
interface level.
This new serializer simplifies many code paths where data
structures such as Set, Map, TypedArray, RegExp, etc. had to be
converted in a disparate manner to be able to persist them to
extension storage.
The new serializer supports workers and LZ4 compression. These
can be configured through advanced settings.
With this new layered design, it's possible to introduce more
storage layers if measured as beneficial (i.e. maybe
browser.storage.session)
References:
- https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage/local
- https://developer.mozilla.org/en-US/docs/Web/API/Cache
- https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm
This commit brings the following changes to the logger:
All logging output generated by injected scriptlets are now sent to
the logger, the developer console will no longer be used to log
scriptlet logging information.
When the logger is not opened, the scriplets will not output any
logging information.
The goal with this new approach is to allow filter authors to
more easily assess the working of scriptlets without having to
go through scriptlet parameters to enable logging.
Consequently all the previous ways to tell scriptlets to log
information are now obsolete: if the logger is opened, the
scriptlets will log information to the logger.
Another benefit of this approach is that the dev tools do not
need to be open to obtain scriptlets logging information.
Accordingly, new filter expressions have been added to the logger:
"info" and "error". Selecting the "scriptlet" expression will also
keep the logging information from scriptlets.
A new button has been added to the logger (not yet i18n-ed): a
"volume" icon, which allows to enable verbose mode. When verbose
mode is enabled, the scriptlets may choose to output more
information regarding their inner working.
The entries in the logger will automatically expand on mouse hover.
This allows to scroll through entries which text does not fit into
a single row.
Clicking anywhere on an entry in the logger will open the detailed
view when applicable.
Generic information/errors will now be rendered regardless of which
tab is currently selected in the logger (similar to how tabless
entries are already being rendered).
The idea is to remove as many dependencies as possible for
low-level ScriptletFilteringEngine in order to make it easier
to reuse the module outside uBO itself.
The high-level derived class takes care of caching and
injection of scriptlets into documents, which requires
more knowledge about the environment in which scriptlets
are to be used.
Also improve scriptlet cache usage to minimize overhead of
retrieving scriptlets.
If the `manual` parameter is assigned a date in the form of
`YYMMDD`, this will tell uBO to update lists from origin sources
when the current time is within the range of the specified date,
otherwise lists will be updated from CDNs. Updating from CDNs
is always strongly recommended since this enables differential
updates.
For the time being, `manual=1` will always cause to update lists
from origin, but this form will be deprecated once next stable
release is widespread. The idea is to not leave behind stale
and obsolete links which would be detrimental to differential
updates should someone click on one of these old links left
behind.
Purging all the lists from cache storage is detrimental to
differential update, and cause filter lists to be updated less
often and consequently to be less up to date then when letting
differential updater do its work.
Possibly related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2136
Also reported internally, steps to reproduce the issue fixed here:
- Open uBO's dashboard through 3-dot > Add-ons > uBO > Settings
- Bring forth "Filter lists" pane
We want the tab to be already opened at next launch
- Quit Firefox for Android
- Launch Firefox for Android
Result:
Very long launch time, lists marked as out of date.
In Support pane's troubleshooting information, a delta sign will
be added to list update information when the list was last updated
through differential update.
Broadcast channels are more suited to uBO than DOM events to dispatch
notifications to different parts of uBO.
DOM events can only be dispatched to local context, broadcast channels
dispatch to all contexts (i.e. background process, workers, auxiliary
pages) -- this last behavior is better suited to uBO to communicate
internal changes to all potential listeners, not just those in the local
context.
Additionally, broadcasting to content scripts is now done through
tabs.sendMessage() instead of through potentially opened message
ports, this simplifies broadcasting to content scripts, and this
doesn't require to have long-lived message ports in content
scripts.
Related discussion:
https://github.com/uBlockOrigin/uBlock-issues/discussions/2895
Changes:
The _content of the My filters_ pane is now considered untrusted by
default, and only uBO's own lists are now trusted by default.
It has been observed that too many people will readily copy-paste
filters from random sources. Copy-pasting filters which require trust
represents a security risk to users with no understanding of how the
filters work and their potential abuse.
Using a filter which requires trust in a filter list from an untrusted
source will cause the filter to be invalid, i.e. shown as an error.
A new advanced setting has been added to control which lists are
considered trustworthy: `trustedListPrefixes`, which is a space-
separated list of tokens. Examples of possible values:
- `ublock-`: trust only uBO lists, exclude everything else including
content of _My filters_ (default value)
- `ublock- user-`: trust uBO lists and content of _My filters_
- `-`: trust no list, essentially disabling all filters requiring
trust (admins or people who don't trust us may want to use this)
One can also decide to trust lists maintained elsewhere. For example,
for stock AdGuard lists add ` adguard-`. To trust stock EasyList lists,
add ` easylist-`.
To trust a specific regional stock list, look-up its token in
assets.json and add to `trustedListPrefixes`.
The matching is made with String.startsWith(), hence why `ublock-`
matches all uBO's own filter lists.
This also allows to trust imported lists, for example add
` https://filters.adtidy.org/extension/ublock/filters/` to trust all
non-stock AdGuard lists.
Add the complete URL of a given imported list to trust only that one
list.
URLs not starting with `https://` or `file:///` will be rejected,
i.e. `http://example.org` will be ignored.
Invalid URLs are rejected.
The `urltransform` option allows to redirect a non-blocked network
request to another URL. There are restrictions on its usage:
- require a trusted source -- thus uBO-maintained lists or user
filters
- the `urltransform` value must start with a `/`
If at least one of these conditions is not fulfilled, the filter
will be invalid and rejected.
The requirement to start with `/` is to enforce that only the path
part of a URL can be modified, thus ensuring the network request
is redirected to the same scheme and authority (as defined at
https://en.wikipedia.org/wiki/Uniform_Resource_Identifier#Syntax).
Usage example (redirect requests for CSS resources to a non-existing
resource, for demonstration purpose):
||iana.org^$css,urltransform=/notfound.css
Name of this option is inspired from DNR API:
https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/declarativeNetRequest/URLTransform
This commit required to bring the concept of "trusted source" to
the static network filtering engine.
As per discussion with uBO volunteers.
Volunteers offering support for uBO will be able to craft links with
specially formed URLs, which once clicked will cause uBO to automatically
force an update of specified filter lists.
The URL must be crafted as shown in the example below:
https://ublockorigin.github.io/uAssets/update-lists.html?listkeys=ublock-filters,easylist
Where the `listkeys` parameter is a comma-separated list of tokens
corresponding to filter lists. If a token does not match an enabled
filter list, it will be ignored.
The ability to update filter lists through a specially crafted link
is available only on uBO's own support sites:
- https://github.com/uBlockOrigin/
- https://reddit.com/r/uBlockOrigin/
- https://ublockorigin.github.io/
Additionally, a visual cue has been added in the "Filter lists" pane
to easily spot the filter lists which have been recently updated, where
"recently" is currently defined as less than an hour ago.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2773
The `randomize` paramater introduced in https://github.com/gorhill/uBlock/commit/418087de9c
is now named `directive`, and beside the `true` value which is meant
to respond with a random 10-character string, it can now take the
following value:
war:[web_accessible_resource name]
In order to mock the XHR response with a web accessible resource. For
example:
piquark6046.github.io##+js(no-xhr-if, adsbygoogle.js, war:googlesyndication_adsbygoogle.js)
Will cause the XHR performed by the webpage to resolve to the content
of `/web_accessible_resources/googlesyndication_adsbygoogle.js`.
Should the resource not exist, the empty string will be returned.
Reference:
https://adguard.com/kb/general/ad-filtering/create-own-filters/#conditions-directive
This commit should make uBO fully compatible with the `!#if`
directives found throughout AdGuard's filter lists.
Additionally, added the new `!#else` directive for convenience
to filter list authors:
!#if cap_html_filtering
example.com##^script:has-text(fakeAd)
!#else
example.com##+js(rmnt, script, fakeAd)
!#endif
As discussed internally with filter list maintainers.
Additionally, added a search field to filter out lists. This
is still a work in progress, no need to open issues about this,
I am aware of what is missing (i18n, more tags, etc.)
Related discussion:
- https://github.com/uBlockOrigin/uBlock-issues/discussions/2582
If there exist any built-in filter list which last update time
is older than 2 hours, the "Report a filter issue" page will ask
the user to update their filter lists then verify that the issue
still exists.
Once filter lists are updated, the troubleshooting information
will reflect the change in update time.
