This feature is hidden behind the "Developer mode" setting in
the dashboard. When "Developer mode" is enabled, a tab named
"Develop" will become available in the dashboard. This tab is
meant to contain tools for technical users.
At the moment, the "Develop" pane allows to create custom DNR
rules through a (CodeMirror-based) editor.
For the sake of convenience, the DNR rule must be entered in
YAML-like format. The format is not really full compliant YAML,
just YAML-like, and very strict in order to ensure the parser
stays simple enough.
Lines starting with `#` are comments and will be ignored by the
parser.
Any line which do not match the parser's expectation will be
marked as invalid, and the whole DNR rule containing such invalid
lines will be discarded.
There must not be empty lines inside a rule definition.
Each DNR rule must be separated with a `---` line, which is
known as a YAML document separator.
String values must not be quoted, otherwise the quotes will be
considered part of the value. There is one exception: `''` will
be parsed as "an empty string".
The editor will attempt to auto-complete known DNR keywords. That
feature will improve over time.
Though the parser will identify some errors, not all invalid DNR
rules are currently identified by the parser, and these will be
reported when the rules are registered through the DNR API. Better
identifying invalid DNR rules at edit time will improve over time.
The editor will report `regexFilter` values which are not
supported by the DNR engine on the current platform.
The editor reacts to instances of `regexFilter: ...` to report
whether a regex value is supported. This means you can test for
a regex value by using `# regexFilter: ...` so that you do not
have to create an actual DNR rules just for the sake of testing.
Custom DNR rules can be exported into a JSON file (a format
known by the DNR API as a "static ruleset").
JSON-based ruleset can be imported, the content will be converted
to YAML-like syntax.
The editor will attempt to convert to YAML pasted content which
can be JSON-parsed. It's possible to paste partially or wholly
JSON-based rulesets.
When disabling "Developer mode", all custom DNR rules will be
unregistered from the DNR API. The DNR rules content will be left
intact in such case. Existing DNR rules will be registered into
the DNR API when re-enabling "Developer mode".
Administrators can prevent "Developer mode" from being enabled
by adding `develop` token to `disabledFeatures` setting.
Related discussion:
https://github.com/uBlockOrigin/uBOL-home/discussions/323
The main motivation is to give list maintainers a tool to assist
with resolving filter issues. Custom DNR rules can assist in
crafting and validating filters meant to work with uBOL.
A secondary motivation is to provide technical users the ability
to further customize their content blocker.
More conveniences will be added over time, this is a first version.
Related issue:
https://github.com/uBlockOrigin/uBOL-home/issues/157
The `header=` option will be converted into DNR's `responseHeaders`
condition.
There will be an attempt to convert regex-based values into DNR-
compatible syntax. Not all regex-based patterns can be converted to
use DNR's patterns with `*` and `?` special characters.
The implementation of `header=` option in uBO has been revisited to
improve compatibility with DNR syntax to minimize burden for list
maintainers when creating `header=` filters compatible with both
uBO and uBOL.
The changes:
- Header names are now case-insensitive by default
- Occurrences of `*` in non-regex-based header values now mean
"matches any number of characters"
- Occurrences of `?` in non-regex-based header values now mean
"matches zero or one character"
At time of commit, and as per MDN, only Chromium-based browsers
currently support filtering on repsonse headers:
https://developer.mozilla.org/docs/Mozilla/Add-ons/WebExtensions/API/declarativeNetRequest/HeaderInfo
Also as per MDN, Chromium 121-127 silently ignore the `responseHeaders`
condition, potentially causing undue blocking of network requests.
Currently uBOL support Chromium 122 and later, meaning we need to mind
potential false positives in Chromium 122-127 for filters using
`header=` option.
Related issue:
https://github.com/uBlockOrigin/uAssets/issues/28408
Stable Safari is known to not block certain network requests due to
the current state of the DNR engine in Safari. The warning is to
ensure volunteers are not burdened by issues originating from the
browser.
Eventually Safari's DNR implementation will catch up with the
specification, at which time the warning will be removed.
This is a first step to integrate CodeMirror6 into the project.
As a side effect, this should take care of:
https://github.com/uBlockOrigin/uBOL-home/issues/297
Though most likely the list of no-filtering websites will probably
move to its own pane as in uBO in some future.
Most of the time the URL doesn't change and just forcing a reload
of the page is sufficient. When a document is strict-blocked, the
URL must be updated however.
Excerpt from <https://openphish.com/terms.html>:
> Except as expressly permitted by OpenPhish in writing, you agree not
> to license, sell, rent, lease, transfer, assign, distribute, display,
> disclose, create derivative works or otherwise make all or any portion
> of the information obtained through the Services available to any
> third party.
onMessage() is now a listener installed synchrnously when uBOL
executes, so no longer need to deal with failure to send messages.
Related commit:
ab458b492a
Setting name: "defaultFiltering"
Type: string
Valid values: "none", "basic", "optimal", "complete"
Additionally, fix default "basic" mode toggling back to "optimal"
upon extension restart when uBOL has broad host permissions. The
default mode will toggle automatically only when there is a change
in broad host permissions status.
Related issue:
https://github.com/uBlockOrigin/uBOL-home/issues/326
uBOL now asks broad host permissions by default. Users can still
choose narrow host permissions by using their browser's controls
for this.
For instance in Chromium, those host permissions controls are found
in the "Site access" section in the detailed view of an extension.
One can set "Site access" to "On click" to revoke broad host
permissions, and grant host permissions to only specific site.
In such mode, uBOL will still block through the DNR API, but no
cosmetic or scriptlet filtering will occurs, as these requires
permission to "read and change data" on websites for which higher
filtering mode is desired.
Some browsers do not automatically grant broad host permissions
even when an extension asks for broad permissions at install time,
and going forward all browsers will likely adopt this approach, and
thus it no longer made sense for uBOL to default to no broad hosts
permissions at install time, especially given this leads to issues
with no solution -- issues solved with the new approach (e.g. like
the ability to deploy uBOL in Optimal mode by default).
Support for paths allows to narrow down specific static extended
filters to specific webpages on a given site.
Examples of usage:
example.com/toto##h1
/example\.com\/toto\d+/#@#h1
Call to postMessage was failing in the zapper when using `use_dynamic_url` in
manifest. Manually overwriting the per-session dynamic hostname with the real
extension id fixes this.
A bad test prevented the "no filtering" by default mode to not take
effect immediately when activated while no site had yet been set to
"no filtering". It would however take effect as soon as a specific
site would be excluded from "no filtering".
`generichide` option is implicitly enforced on all sites unless an
exception overrides `generichide`. Though rare, sometimes a
`generichide` exception needs to be overridden so that generic
cosmetic filtering is made possible on a specific site.
This commit is to add support for restoring generic cosmetic
filtering on sites which were excluded through a `generichide`
exception.
Concretely, this is needed to ensure the test suite can properly
verify that generic cosmetic filtering is working when the
filtering mode is set to "complete":
||ublockorigin.github.io^$generichide,important
Microsoft Edge for Android requires a "link to the core function
test cases for your Edge Android extension" to verify that the
extension function properly.
This extends coverage of strict-blocking to pattern-based filters
with `doc` filter option.
When proceeding with a URL-skip URL present, no temporary bypass
will be created when the "Don't warn me again about this site" is
left unchecked. The idea is to avoid the intermediate redirects
if we navigate again on the same strict-blocked site, while a
temporary bypass would prevent this.
uBO's "Badware risks" list has been spinned off as its own list.
The idea is that should a site be strict-blocked from that list,
we would want to know the strict-block is due to the
"Badware risks" list.
Submission to Edge Addons store fails with:
----
The following checks failed:
Package acceptance validation error:
Manifest file reference 'urlhaus.full' does not exist in the zip archive.
(Note: File locations are case-sensitive)
Package acceptance validation error:
Manifest file reference 'openphish.domains' does not exist in the zip archive.
(Note: File locations are case-sensitive)
----
Guessing the use of period in ruleset file path is causing the
issue.
Related issue:
https://github.com/uBlockOrigin/uBOL-home/issues/254
Cosmetic filters with only negated hostnames would end up being
classified as specific, while in reality the filter is generic, with
specific exceptions.
This commit fixes the improper classification. Enforcing specific
exception filters in the generic realm is still an issue to
address.
Add information about which ruleset caused a page to be strict-
blocked.
Whenever possible, add ability to URL-skip an incoming redirect
in a strict-blocked page.
Add new default list: "30-day Phishing Domain List"
Related issue:
https://github.com/uBlockOrigin/uBOL-home/issues/214
This implements basic functionality for strict blocking, i.e. the
ability to block navigation to undesirable websites. This is a
first implementation, which converts only filters that are plain
hostnames.
Unlike with uBO, it is not possible to know from which ruleset a
blocking rule originates. Nonetheless, users will have to make a
choice as to whether navigation should proceed or not.
A setting has been added to the dashboard to wholly enable/disable
strict blocking. It is enabled by default.
Potential future improvements, pending investigation on feasability
in an MV3 framework:
- Extend coverage to explicit `document` filters
- Leverage and use `urlskip=` filters in the blocking page in order
to proceed while bypassing unwanted redirects.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/3452
Use blob-based injection only when direct injection fails because
of a page's CSP. This is a mitigation until a better approach is
devised.
Such future better approach to investigate:
- Use `MAIN` world injection supported by contentScript.register()
since Firefox 128
- Investigate registering script to inject ahead of time thru
some heuristic
New managed setting:
"disabledFeatures": {
"title": "User interface features to disable",
"description": "A list of tokens, each of which correspond to a user interface feature to disable.",
"type": "array",
"items": { "type": "string" }
}
Supported tokens:
- `dashboard`: Prevent access to all dashboard settings
- `filteringMode`: Prevent changes to the default filtering mode,
or the current filtering mode of any site
Related feedback:
https://github.com/uBlockOrigin/uBOL-home/discussions/35#discussioncomment-11326086
Collecting matched rules when the extension is side-loaded is now
opt-in, by enabling "Developer mode" in the dashboard.
The reason is to allow the extension to behave same as the official
released version when side-loaded.
Specifically, as side-loaded extension, uBOL's service worker would
wake up due to matched-rule listener even though it would not wake
up the worker with same configuration in stable release.
Not all matching scriptlets were injected on a given site in Optimal
or Complete mode when default mode was set to Basic or less.
A high profile manifestation of this bug was that Youtube ads were
not being blocked when using Optimal on Youtube while default mode
was Basic.
Related issue:
https://github.com/uBlockOrigin/uBOL-home/issues/229
Add "Filter lists" pane in dashboard
The DNR API now supports enabling 50 static rulesets put of a
maximum of 100 (instead of 10 out of 50 originally). Thus given
the potentially growing number of static rulesets, the available
stock rulesets has been moved to its own pane, with the following
improvements:
- Support sublists
- Support search
Aditionally, "RU AdList: Counter" has been added as a stock
ruleset.
Other changes:
- Do not re-evaluate regexes which failed validation
- Better reduce `removeparam` rules
Related bugzilla issue:
https://bugzilla.mozilla.org/show_bug.cgi?id=1736575
The issue was fixed months ago.
The removed code block is causing uBOL to be flagged as "including
remotely hosted code".
To be clear, the removed obsolete code block was not related to
executing remote code. The referenced code was in the file itself,
not remote, and this was a workaround for when Firefox was not
supporting injecting script in the `MAIN` world.
The issue was fixed months ago in Firefox, so there is no point for
the workaround.
Related discussion:
https://github.com/uBlockOrigin/uBOL-home/discussions/35#discussioncomment-11157444
New policy setting: `rulesets`
Type: array
Type of array items: string
Each item in the list is a list id (as seen in `rulesets/ruleset-details.json`),
prefixed with either `+` to enable the ruleset, or `-` to disable the ruleset.
Users will not be able to enable or disable rulesets present in the `rulesets`
policy. Disabled rulesets will not appear in the dashboard.
Use `-*` to remove all non-default rulesets, except for those added
using `+[ruleset_id]`.
Additionally, some work has been done to properly handle policy changes in
a non-blocking and deferred manner, as I observed that it often takes long
for calls to `storage.manage.get` to resolve. This potentailly takes care
of the following issue:
https://github.com/uBlockOrigin/uBOL-home/issues/174
Ensure serialization returns copy of data rather than live
references to data. This allows to immediately deserialize() the
result of serialize().
Also, adjust code to modified behavior of filterQuery().
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/3394
When the "No large media elements" per-site switch is toggled on,
it will also act to prevent autoplay of video/audio media, regardless
of their size. This also works for xhr-based media streaming.
If blocking by size is not desirable while blocking autoplay is
desired, one can toggle on "No large media elements" switch while
setting "Block media elements larger than ..." to a very high value.
Default to `true`.
Set to `false` to wholly disable calls to `dns.resolve()` (Firefox-
only). Disabling calls to `dns.resolve()` will prevent cname-uncloaking
and will limit ability to enforce `ipaddress` filter option.
This commit makes the DNS resolution code better suited for both
filtering on cname and ip address. The change allows early availability
of ip address so that `ipaddress=` option can be matched at
onBeforeRequest time.
As a result, it is now possible to block root document using
`ipaddress=` option -- so long as an ip address can be extracted
before first onBeforeRequest() call.
Related issue:
https://github.com/uBlockOrigin/uBlock-issues/issues/2792
Caveat
------
the ip address used is the first one among the list of ip
addresses returned by dns.resolve() method. There is no way for uBO
to know which exact ip address will be used by the browser when
sending the request, so this is at most a best guess. The exact IP
address used by the browser is available at onHeadersReceived time,
and uBO will also filter according to this value, but by then the
network request has already been sent to the remote server.
Possibly a future improvement would make available the whole list
of ip addresses to the filtering engine, but even then it's impossible
to know with certainty which ip address will ultimately be used by the
browser -- it is entirely possible that the ip address used by the
browser might not be in the list received through dns.resolve().
If an IP address can be extracted from the hostname portion of
a URL, the IP address matching will be performed at onBeforeRequest()
time.
Regardless, IP address matching will subsequently always be performed
at onHeadersReceived() time as the request details at that point
contain a reliable IP address value on supported platforms (Firefox-
only as of now).
The `cap_ipaddress` now evaluates to `true` in Chromium-based
browsers. Even though these browsers are unable to provide reliable
IP address value at onHeadersReceived() time, they can still
perform IP address matching for IP address extracted from hostname
portion of a URL.
A new icon has been added to the popup panel, to open a popup
window with a terse list of DNR events for the current tab, in
reverse chronological order (most recent DNR event appears at
the top).
The new ability is available only when the extension is sideloaded,
as per `declarativeNetRequestFeedback` documentation. Ref:
https://developer.chrome.com/docs/extensions/reference/api/declarativeNetRequest#event-onRuleMatchedDebug
Purposefully minimal, so as to have something rather than nothing
when having to diagnose filtering issue with the DNR API. Example:
https://github.com/uBlockOrigin/uBOL-home/issues/156
The content of the popup window does not dynamically update, force
a refresh (F5) to get the most recent DNR events. This might be
improved in the future.
The DNR event buffer is not persisted, so the buffer is empty when
service worker is restarted. This might be improved in the future
by using session storage API.
There is no output filtering ability in this first draft. This
might be improved in the future.
DNR rules are reported. The filter from which a DNR rule
originates is not reported. Given that the rulesets are optimized
after conversion from original filter lists to reduce the DNR rule
count, this is unlikely to ever be possible.
