Domains were previously not validated before being handled, leading to a
potential scenario where someone could pass something like
"element_url=127.0.0.1:<port>/<resource>" to access other resources on a
machine running Whoogle. This change ensures that the resource used in
both endpoints is a valid domain.
This also includes validation of config names to prevent names from
including path values such as "../../(etc)".
When starting whoogle from another directory, the path to the calculator
widget was previously invalid. It now specifies the path relative to the widget
loader file.
The calculator was previously triggered for partial matches with words
like "calc", which meant searches containing the word "calcium" would
cause the calculator widget to appear.
Redirects to alternative frontends can now be defined using the
WHOOGLE_REDIRECTS environment variable. Usage is documented in the
readme, but is basically defined as <parent>:<new>.
Closes#988
This relates to an issue with an unknown cause (unable to reproduce on
my end) where the preferences string does not contain the correct amount
of padding on a base64 encoded value. This is mediated by appending
padding to the end of the encoded value, since any extra padding is
removed anyways.
Fixes#987
Defines separate environment variables for setting mobile vs desktop user
agents
Defines an environment variable for using the client's User-Agent
Co-authored-by: Ben Busby <contact@benbusby.com>
More MarkupResemblesLocatorWarning warnings have been appearing. This
seems to be caused by parsing HTML content that contains a URL.
This new change suppresses the warning at the root level of the app
before any content has been parsed, so this error shouldn't appear
again.
Fixes#968
* Add translation for new strings from 7041b43db9
Use same terms as Google's zh-tw interface.
* Fix missing period
* Sync string order with en (easier for future updates)
Fix the exception `AttributeError: 'Filter' object has no attribute 'block_url'`
introduced in this commit [1].
`self.block_title` and `self.block_url` were members of the Filter
object[2], but not anymore after commit [1].
This bug can be reproduced with setting WHOOGLE_CONFIG_BLOCK_URL to a
non-empty string.
[1] 10a15e06e1
[2] 284a8102c8
An invalid parsing warning was being thrown by the latest version of the
bs4 library. This suppresses that warning from being shown in the
console.
A 404 handler was added to move logging from the console to the error
template, since a lot of users assumed that 404 errors from the result
page were problems with Whoogle itself.
Fixes#967
When a browser adds a search engine using the opensearch template, it
does not have the correct context necessary to autofill the
`preferences` arg with the user's session prefs. As a result, queries
made using the browser bar will have the instance's default preferences
filled into the template.
Removing this shouldn't have any side effects, since queries made on the
same machine will have the correct session associated with the user.
Fixes#929
Some distributions require manually installing Python 3.10, which makes
it less convenient than just using whatever version of Python3.X the
package manager supports. Since the only 3.10 feature being used was
"match", and it was a very small change, it's been replaced with an
if/else statement to ensure compatibility with older versions of Python
3.
Navigating between pages of results now includes the user's preferences
string, which allows them to retain their config for a particular
instance between result pages.
Fixes#960
This adds a simple calculator widget, somewhat similar to the one presented
when searching calculator on Google.
Also, it adds somewhat of a template for making the addition of new widgets
easier via the app/utils/widgets.py file. My eventual plan is to use this to
create more widgets that appear in Google, such as a color picker, timer, etc.
---------
Co-authored-by: Ben Busby <contact@benbusby.com>
Medium redirects needed further cleanup to account for instances where a
link contains a subdomain that would not make sense in a Farside
redirect link.
Fixes#947
The url prefix was not included when reconstructing the root url using
X-Forwarded-* headers, causing some elements to fail to load properly.
Fixes#937
Add a function to check if target_word contains CJK characters
If a search term contains Chinese, Japanese, or Korean characters,
the term is bolded in search results regardless of whitespace.
CJK characters: Chinese, Japanese (hiragana, katakana, kanji),
and Korean (hangul syllables, hangul jamo)
Co-authored-by: Ben Busby <contact@benbusby.com>
The whoogle.env file previously needed to be created and enabled using
the WHOOGLE_DOTENV var. This removes the second step and loads the env
file if it's found during app init.
The Dockerfile has also been updated to copy in whoogle.env if it
exists.
Fixes#909
Moved the cleaner functions to app/utils/escaper.py
Removed unused import 're'
Moved the cleaner functionalities to the "search.py" and "routes.py"
Making sure escaped chars stay escaped during process
Replaced "<" and ">" with "andlt;" and "andgt;", respectively. This way,
when the 'response' object get loaded to bsoup (which happens several times
throughout the process between search.py and routes.py), bsoup will not
unescape them.
Introduces the ability to refine searches by time period:
- Past hour
- Past 24 hours
- Past week
- Past month
- Past year
Co-authored-by: Ben Busby <contact@benbusby.com>
Proxies that only support HTTP were causing request timeouts due to an
invalid upgrade to HTTPS when creating the request. This update restores
the ability to have an HTTP-only proxy for all requests.
Fixes#906
Parent sites using a 'www' subdomain or something similar were not
redirecting properly. This updates the hostname check to only validate
against the primary domain, except for Wikipedia since the subdomain is
used for interface translation in that case.
Fixes#901
Replacing result links and text when site alts are enabled is now part
of its own function, and handles replacement of link location and link
description separately.
Fixes#880
This can be updated later to allow users with cookies enabled to use a
key that is unique to their session (if they want, not mandatory), but
for now it makes more sense to just use a single key for all queries
from all users. This should eliminate a lot of issues that users have
reported where they are unable to decrypt queries or page elements due
to an expired/renewed session key.
The change made to whoogle-contrast-text in #873 wasn't the right
decision, since whoogle-contrast-text is meant to contrast with darker
UI elements. whoogle-text already contrasts with the default white
background.
* Sync setup.cfg with requirements.txt
* Include tests in PyPI tarballs
And exclude them from setuptools
* Set version number only once
Switch to PEP517 standard (pyproject.toml) for builds
New changes to google search now include ads prefixed with the keyword
"sponsored". This update should remove these from appearing in search
results.
Fixes#871
Changed the implementation to work if the bang is at anyplace in the query.
Added a check to not spend time looking for an operator if a "!" is not present
in the query.
No longer allowed to have the bang at the "!" char at the end, since this may
cause some conflicts like the issue cited before, where the ! is after a word
in the query, which is natural in most languages.
Adds support for encoding (and optionally encrypting) user config values as
a single string that can be passed to any endpoint with the "preferences" url
param.
Co-authored-by: Ben Busby <contact@benbusby.com>
Sessions are no longer validated using the "/session/..." route. This
created a lot of problems due to buggy/unexpected behavior coming from
the Flask-Session dependency, which is (more or less) no longer
maintained.
Sessions are also no longer strictly server-side-only. The majority of
information that was being stored in user sessions was aesthetic only,
aside from the session specific key used to encrypt URLs. This key is
still unique per user, but is not (or shouldn't be) in anyone's threat
model to keep absolutely 100% private from everyone. Especially paranoid
users of Whoogle can easily modify the code to use a randomly generated
encryption key that is reset on session invalidation (and set
invalidation time to a short enough period for their liking).
Ultimately, this should result in much more stable sessions per client.
There shouldn't be decryption issues with element URLs or queries
during result page navigation.
Adds a new advanced search icon alongside the result tabs for switching
to a different country from the result page.
This will obviously get populated with other methods of filtering
results, but for now it's just the country selector.
Session validation includes a method for determining the proxy host url,
but previously did not include the path for the initial request. This
caused a situation where users with a new session would not be able to
complete their first search, since the session validation follow-through
url did not include the actual path for their search query.
The method now includes a flag for only extracting the root url, which
is needed for creating full urls in the content filter.
Fixes#708
This adds a new "temporary" config section of the results view, where a
user can now change the country that their results come from without
changing their default config settings.
Closes#322
Queries performed in a different language than what is configured
contain a result div that prompts the user to configure their language
preferences using google's preferences page.
Since we want all language configuration to occur on Whoogle only, we
can safely remove this result div.
Fixes#444Fixes#386
Farside can now redirect quora links to querte instances and imdb links
to libremdb instances. This updates Whoogle to perform link replacements
for both services when site alts are configured.
For users running local instances of service alternatives such as
invidious, the alt replacement procedure broke if the scheme of the
original service (almost always https) didn't match the scheme of their
defined local service (likely http).
This adds a small check to see if the alt has a defined scheme, and if
so, removes the original scheme for that result.
Fixes#806
Wikipedia -> Wikiless redirects always result in an english language
result, even if the Wikipedia result would've been in a non-english
language. This is due to Wikipedia using language specific subdomains
(i.e. de.wikipedia.org, en.wikipedia.org, etc) whereas Wikiless uses a
"lang" url param.
This has been fixed by inspecting the subdomain of the wikipedia link
and passing that value to Wikiless as the lang param if it's determined
to be a language specific value (currently just looking for a 2-char
subdomain).
See #805
If Whoogle is accessed on a non-standard port _and_ proxied,
this port is lost to the application and `element['src']`s are
incorrectly formed (omitting port).
HTTP x-Forwarded-Host will contain this front port number in
a typical Nginx reverse proxy configuration.
The majority of image links and links that are not handle by whoogle are not
opening in new tabs, this allow links that are not related to the application
to open in new tabs.
Due to how instances installed with pip seem to have issues storing
unrelated files in the same directory as sessions, exception handling
during session validation has been expanded to blindly ignore all
exceptions. This portion of the code is more for maintainers of large
public instances with a bunch of users who block cookies anyways, so
having basic app functionality break down as a result shouldn't be the
default.
Country config value should be checked against the valid value when
updating the home page config, not the other way around. This can lead
to a state where a user sets up an invalid country value, but can still
be matched against a correct value that is part of the invalid value
(i.e. "countryUK" is invalid, but would match against the correct value,
"UK")
Also minor refactor of where the session file size validation occurs.
For pip installed instances of Whoogle, there seems to be an issue where
files other than sessions are being stored in the same directory as the
sessions. From a brief investigation, this does not seem to be caused by
Whoogle, since Flask-Session objects are the only files stored in that
directory. It could be an issue with the library that is being used for
sessions, however.
Regardless, the app shouldn't crash when trying to validate and remove
invalid sessions, so a file size limit of 4KB was imposed during
validation. Any file found in the session directory that exceeds this
size limit will be ignored.
Fixes#777Fixes#793
Added password authentication for tor control port.
For user configuration of access to tor control port. This file should be
heavily restricted in file system.
Co-authored-by: MadcowOG <madcowog@Arch-Main.localdomain>
