prehistoric-systems/FilterLists
1
0
Fork 0
mirror of https://github.com/collinbarrett/FilterLists.git synced 2026-03-11 09:04:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

refactor: 🔥 delete all

Browse source
This commit is contained in:
Collin M. Barrett 2020-08-22 05:27:48 -05:00
parent 24afedc12f
commit 8ae9d9cdfa
522 changed files with 0 additions and 4613055 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

35
.dockerignore
View file

@ -1,35 +0,0 @@
# From default Visual Studio .NET Core template
**/.classpath
**/.dockerignore
**/.env
**/.git
**/.gitignore
**/.project
**/.settings
**/.toolstarget
**/.vs
**/.vscode
**/*.*proj.user
**/*.dbmdl
**/*.jfm
**/azds.yaml
**/bin
**/charts
**/docker-compose*
**/Dockerfile*
**/node_modules
**/npm-debug.log
**/obj
**/secrets.dev.yaml
**/values.dev.yaml
LICENSE
README.md
# Other common
**/.gitattributes
**/.github
**/.sln
**/*.DotSettings*
**/appsettings*.json
**/build
**/*.md

17
.env
View file

@ -1,17 +0,0 @@
API_CONNECTION_STRING=Server=mariadb;Database=filterlists;Uid=filterlists;Pwd=filterlists;
API_MYSQL_ROOT_PASSWORD=filterlists
API_MYSQL_DATABASE=filterlists
API_MYSQL_USER=filterlists
API_MYSQL_PASSWORD=filterlists
AGENT_APPLICATION_INSIGHTS_INSTRUMENTATION_KEY=
AGENT_ARCHIVE_GIT_REPOSITORY_DIRECTORY=archives
AGENT_ARCHIVE_GIT_SIGNATURE_NAME=
AGENT_ARCHIVE_GIT_SIGNATURE_EMAIL=
AGENT_ARCHIVE_GIT_COMMIT_MESSAGE_SUFFIX= FilterLists archive by FilterLists.Agent
AGENT_FILTERLISTS_API_BASE_URL=https://filterlists.com/api/v1
AGENT_FILTERLISTS_API_CLIENT_USER_AGENT=
AGENT_GITHUB_PRODUCT_HEADER_VALUE=
AGENT_GITHUB_PERSONAL_ACCESS_TOKEN=
AGENT_GITHUB_REPOSITORY_OWNER=collinbarrett
AGENT_GITHUB_REPOSITORY=FilterLists

32
.github/CONTRIBUTING.md vendored
View file

@ -1,32 +0,0 @@
So you know of some lists you want to add, but you are very unsure about how to do so? Fear no more. Here's how to do it:
### Add/update lists, licenses, subscription links, homepages, and similar:
1. Right-click on the following link and choose "Save link": https://raw.githubusercontent.com/collinbarrett/FilterLists/master/data/FilterList.json
2. Open the downloaded _FilterList.json_ file in Sublime Text, Notepad++, or any similar raw-text editor who can handle large files and JSON indentations.
3. Add the desired changes. For new lists, "id", "name", "[syntaxId](https://github.com/collinbarrett/FilterLists/blob/master/data/Syntax.json)" and "viewUrl" are mandatory. "homeUrl" and "[licenseId](https://github.com/collinbarrett/FilterLists/blob/master/data/License.json)" are also important if they are applicable to the list. Other ones in active use are "description", "donateUrl", "emailAddress", "issuesUrl", "viewUrlMirror1" and "viewUrlMirror2".
4. Once you've performed the changes, save the file.
5. Before you can upload the changes to this repo, you must first create a branch of this repo. The easiest way to do so by far, is to take care of assigning tags to the list first (Explained below).
6. Once you're on a new branch, go to `https://github.com/[Your GitHub username]/FilterLists/tree/master/data` (The URL may be different under many circumstances, such as if you've contributed to FilterLists.com before), click "Upload" in the upper right, and choose the `FilterList.json` file that you saved.
7. Submit a PR, and we'll double-check it from there.
### Assign tags to lists (Mandatory for new lists):
1. Go to https://github.com/collinbarrett/FilterLists/edit/master/data/FilterListTag.json, and edit it online.
2. "filterListId" is the list's ID in FilterList.json. "tagId" is [one of these tags](https://github.com/collinbarrett/FilterLists/blob/master/data/Tag.json).
3. If you are 100% certain that none of the existing tags are suitable for a particular list, choose tagId 15 (Topical).
4. Save the file. This will create a new branch and/or fork, which will be convenient for list additions in general.
### Assign languages to lists:
_(This presumes you've already assigned tags to the lists.)_
1. Go to the branch you created after assigning tags.
2. Go to `.../data/FilterListLanguage.json` and begin editing it.
3. "languageId" is [one of a randomly numbered list of languages](https://github.com/collinbarrett/FilterLists/blob/master/data/Language.json). For instance, German is 167, while Japanese is 141.
4. Save the file.
### Assign contributors to lists:
_(This presumes you've already assigned tags to the lists.)_
1. Go to the branch you created after assigning tags.
2. Go to `.../data/FilterListMaintainer.json` and begin editing it.
3. "maintainerId" is [one of various filterlist makers](https://github.com/collinbarrett/FilterLists/blob/master/data/Maintainer.json). If you wish to assign someone who is not mentioned in that file, you are merrily welcome to add maintainer profiles to _Maintainer.json_
4. Save.

30
data/Dependent.json
View file

@ -1,30 +0,0 @@
[
{
"dependencyFilterListId": 301,
"dependentFilterListId": 344
},
{
"dependencyFilterListId": 479,
"dependentFilterListId": 344
},
{
"dependencyFilterListId": 97,
"dependentFilterListId": 819
},
{
"dependencyFilterListId": 97,
"dependentFilterListId": 852
},
{
"dependencyFilterListId": 97,
"dependentFilterListId": 853
},
{
"dependencyFilterListId": 60,
"dependentFilterListId": 1816
},
{
"dependencyFilterListId": 347,
"dependentFilterListId": 1817
}
]

24201
data/FilterList.json
View file

File diff suppressed because it is too large Load diff

3626
data/FilterListLanguage.json
View file

File diff suppressed because it is too large Load diff

5534
data/FilterListMaintainer.json
View file

File diff suppressed because it is too large Load diff

12150
data/FilterListTag.json
View file

File diff suppressed because it is too large Load diff

54
data/Fork.json
View file

@ -1,54 +0,0 @@
[
{
"forkFilterListId": 350,
"upstreamFilterListId": 12
},
{
"forkFilterListId": 563,
"upstreamFilterListId": 16
},
{
"forkFilterListId": 16,
"upstreamFilterListId": 59
},
{
"forkFilterListId": 16,
"upstreamFilterListId": 99
},
{
"forkFilterListId": 158,
"upstreamFilterListId": 258
},
{
"forkFilterListId": 163,
"upstreamFilterListId": 262
},
{
"forkFilterListId": 16,
"upstreamFilterListId": 278
},
{
"forkFilterListId": 164,
"upstreamFilterListId": 293
},
{
"forkFilterListId": 167,
"upstreamFilterListId": 295
},
{
"forkFilterListId": 166,
"upstreamFilterListId": 301
},
{
"forkFilterListId": 565,
"upstreamFilterListId": 301
},
{
"forkFilterListId": 312,
"upstreamFilterListId": 311
},
{
"forkFilterListId": 313,
"upstreamFilterListId": 311
}
]

1802
data/Language.json
View file

File diff suppressed because it is too large Load diff

304
data/License.json
View file

@ -1,304 +0,0 @@
[
{
"descriptionUrl": "https://creativecommons.org/licenses/by-nc-nd/4.0/",
"id": 1,
"name": "CC BY-NC-ND 4.0",
"permissiveAdaptation": false,
"permissiveCommercial": false
},
{
"descriptionUrl": "https://opensource.org/licenses/MIT",
"id": 2,
"name": "The MIT License",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"id": 3,
"name": "Restricted",
"permissiveAdaptation": false,
"permissiveCommercial": false
},
{
"descriptionUrl": "https://www.gnu.org/licenses/gpl-3.0.en.html",
"id": 4,
"name": "GPLv3",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://en.wikipedia.org/wiki/All_rights_reserved",
"id": 5,
"name": "All Rights Reserved",
"permissiveAdaptation": false,
"permissiveCommercial": false
},
{
"descriptionUrl": "https://creativecommons.org/licenses/by-nc-sa/3.0/",
"id": 6,
"name": "CC BY-NC-SA 3.0",
"permissiveAdaptation": true,
"permissiveCommercial": false
},
{
"id": 7,
"name": "Permissive",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://creativecommons.org/licenses/by-sa/4.0/",
"id": 8,
"name": "CC BY-SA 4.0",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://creativecommons.org/licenses/by-nc-sa/4.0/",
"id": 9,
"name": "CC BY-NC-SA 4.0",
"permissiveAdaptation": true,
"permissiveCommercial": false
},
{
"descriptionUrl": "https://opensource.org/licenses/BSD-3-Clause",
"id": 10,
"name": "BSD-3-Clause",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://unlicense.org/",
"id": 11,
"name": "The Unlicense",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://creativecommons.org/licenses/by-sa/3.0/",
"id": 12,
"name": "CC BY-SA 3.0",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://creativecommons.org/licenses/by-nc/4.0/",
"id": 13,
"name": "CC BY-NC 4.0",
"permissiveAdaptation": true,
"permissiveCommercial": false
},
{
"descriptionUrl": "http://www.wtfpl.net/",
"id": 14,
"name": "WTFPL",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://creativecommons.org/licenses/by-nd/4.0/",
"id": 15,
"name": "CC BY-ND 4.0",
"permissiveAdaptation": false,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://creativecommons.org/licenses/by/3.0/",
"id": 16,
"name": "CC BY 3.0",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://opensource.org/licenses/ISC",
"id": 17,
"name": "ISC",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://www.apache.org/licenses/LICENSE-2.0",
"id": 18,
"name": "Apache 2.0",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://www.gnu.org/licenses/old-licenses/gpl-2.0.html",
"id": 19,
"name": "GPLv2",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"id": 20,
"name": "Public Domain",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://osint.bambenekconsulting.com/license.txt",
"id": 21,
"name": "OSINT",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://raw.githubusercontent.com/googlehosts/hosts/master/LICENSE",
"id": 22,
"name": "GoogleHosts",
"permissiveAdaptation": false,
"permissiveCommercial": false
},
{
"descriptionUrl": "https://dbad-license.org/",
"id": 23,
"name": "\"Don't Be A Dick\" Public License",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://creativecommons.org/licenses/by/4.0/",
"id": 24,
"name": "CC BY 4.0",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://creativecommons.org/licenses/by-nc-sa/2.5/",
"id": 25,
"name": "CC BY-NC-SA 2.5",
"permissiveAdaptation": true,
"permissiveCommercial": false
},
{
"descriptionUrl": "https://creativecommons.org/licenses/by-nc-nd/3.0/",
"id": 26,
"name": "CC BY-NC-ND 3.0",
"permissiveAdaptation": false,
"permissiveCommercial": false
},
{
"descriptionUrl": "https://opensource.org/licenses/BSD-2-Clause",
"id": 27,
"name": "BSD-2-Clause",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://creativecommons.org/publicdomain/zero/1.0/",
"id": 28,
"name": "CC0 1.0 Universal",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://spdx.org/licenses/Beerware.html",
"id": 29,
"name": "The Beer-Ware License",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://creativecommons.org/licenses/by-sa/2.1/jp/deed.en",
"id": 30,
"name": "CC BY-SA 2.1 JP",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://creativecommons.org/licenses/by-nc/3.0/",
"id": 31,
"name": "CC BY-NC 3.0",
"permissiveAdaptation": true,
"permissiveCommercial": false
},
{
"id": 32,
"name": "Permissive non-commercial",
"permissiveAdaptation": true,
"permissiveCommercial": false
},
{
"descriptionUrl": "https://www.mozilla.org/MPL/2.0/",
"id": 33,
"name": "Mozilla Public License v2.0",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://www.gnu.org/licenses/lgpl-3.0.en.html",
"id": 34,
"name": "Lesser GPL v3",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://github.com/DandelionSprout/Dandelicence/blob/master/DandelicenceNewestVersion.md",
"id": 35,
"name": "Dandelicence",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://creativecommons.org/licenses/by-nd/3.0/",
"id": 36,
"name": "CC BY-ND 3.0",
"permissiveAdaptation": false,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://www.gnu.org/licenses/old-licenses/lgpl-2.1.en.html",
"id": 37,
"name": "Lesser GPL v2.1",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"id": 38,
"name": "Generic copyfree",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"id": 39,
"name": "Generic copyleft",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://pgl.yoyo.org/license/",
"id": 40,
"name": "McRae General Public License",
"permissiveAdaptation": true,
"permissiveCommercial": false
},
{
"descriptionUrl": "https://directory.fsf.org/wiki/License:AGPL-1.0",
"id": 41,
"name": "AGPLv1",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://www.vicidial.com/?page_id=260",
"id": 42,
"name": "AGPLv2",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://www.gnu.org/licenses/agpl-3.0.html",
"id": 43,
"name": "AGPLv3",
"permissiveAdaptation": true,
"permissiveCommercial": true
},
{
"descriptionUrl": "https://opensource.org/licenses/CPAL-1.0",
"id": 44,
"name": "CPAL-1.0",
"permissiveAdaptation": true,
"permissiveCommercial": true
}
]

825
data/Maintainer.json
View file

@ -1,825 +0,0 @@
[
{
"id": 2,
"name": "AppliliZ",
"twitterHandle": "280blocker"
},
{
"id": 3,
"name": "Brad Conte",
"twitterHandle": "bradconte"
},
{
"homeUrl": "https://forum.xda-developers.com/member.php?u=4877037",
"id": 4,
"name": "BSDgeek_Jake"
},
{
"id": 5,
"name": "Michael Trimm"
},
{
"emailAddress": "info@eyeo.com",
"homeUrl": "https://eyeo.com/",
"id": 6,
"name": "eyeo GmbH",
"twitterHandle": "eyeo"
},
{
"homeUrl": "https://easylist.to/",
"id": 7,
"name": "The EasyList Authors"
},
{
"homeUrl": "https://github.com/mayve",
"id": 8,
"name": "mayve"
},
{
"homeUrl": "https://www.malwaredomainlist.com/",
"id": 10,
"name": "Malware Domain List Community",
"twitterHandle": "_MDL_"
},
{
"homeUrl": "https://forums.lanik.us/memberlist.php?mode=viewprofile&u=542",
"id": 11,
"name": "Lian"
},
{
"homeUrl": "https://forums.lanik.us/memberlist.php?mode=viewprofile&u=2266",
"id": 12,
"name": "Crits"
},
{
"homeUrl": "https://forums.lanik.us/memberlist.php?mode=viewprofile&u=1260",
"id": 13,
"name": "smed79"
},
{
"emailAddress": "mp3geek@gmail.com",
"homeUrl": "https://www.fanboy.co.nz/",
"id": 14,
"name": "Ryan Brown",
"twitterHandle": "fanboynz"
},
{
"homeUrl": "https://forums.lanik.us/memberlist.php?mode=viewprofile&u=333",
"id": 15,
"name": "MonztA"
},
{
"homeUrl": "https://forums.lanik.us/memberlist.php?mode=viewprofile&u=1488",
"id": 16,
"name": "Famlam"
},
{
"homeUrl": "https://forums.lanik.us/memberlist.php?mode=viewprofile&u=1400",
"id": 17,
"name": "Khrin"
},
{
"homeUrl": "https://github.com/SlowMemory",
"id": 18,
"name": "SlowMemory"
},
{
"homeUrl": "https://www.blogger.com/profile/09662737599549871224",
"id": 19,
"name": "Mike Burgess"
},
{
"homeUrl": "https://github.com/taylr",
"id": 20,
"name": "Matt Taylor"
},
{
"homeUrl": "https://github.com/hoshsadiq/adblock-nocoin-list",
"id": 21,
"name": "Hosh"
},
{
"emailAddress": "imreeil42@gmail.com",
"homeUrl": "https://sproutsluckycorner.wordpress.com/",
"id": 22,
"name": "Imre Kristoffer Eilertsen"
},
{
"emailAddress": "cosntacptamem@abuse.ch",
"homeUrl": "https://abuse.ch/",
"id": 23,
"name": "abuse.ch",
"twitterHandle": "abuse_ch"
},
{
"homeUrl": "https://github.com/metaphoricgiraffe",
"id": 24,
"name": "metaphoricgiraffe"
},
{
"emailAddress": "badmojr@gmail.com",
"homeUrl": "https://forum.xda-developers.com/member.php?u=8758244",
"id": 25,
"name": "badmojr"
},
{
"homeUrl": "https://github.com/elypter",
"id": 26,
"name": "elypter"
},
{
"homeUrl": "https://ker.af/",
"id": 27,
"name": "Rafael Keramidas",
"twitterHandle": "iamkeraf"
},
{
"emailAddress": "blablabla@alleblock.pl",
"homeUrl": "https://github.com/maciejtarmas",
"id": 28,
"name": "Maciej Tarmas"
},
{
"homeUrl": "https://github.com/bkrucarci",
"id": 29,
"name": "Bekir"
},
{
"emailAddress": "anudeep@protonmail.com",
"homeUrl": "https://github.com/anudeepND",
"id": 30,
"name": "Anudeep",
"twitterHandle": "anudeepND"
},
{
"homeUrl": "https://github.com/arcetera",
"id": 31,
"name": "arcetera"
},
{
"homeUrl": "https://github.com/MajkiIT",
"id": 32,
"name": "MajkiIT"
},
{
"emailAddress": "kadrep@outlook.com",
"homeUrl": "https://github.com/azet12/",
"id": 33,
"name": "azet12"
},
{
"emailAddress": "polishjarvis@gmail.com",
"homeUrl": "https://github.com/PolishFiltersTeam/",
"id": 34,
"name": "Polish Filters Team"
},
{
"emailAddress": "bjorn@bjornstar.com",
"homeUrl": "https://bjornstar.com/",
"id": 35,
"name": "Bjorn Stromberg"
},
{
"emailAddress": "martin@herndl.org",
"homeUrl": "https://github.com/monojp",
"id": 36,
"name": "Martin Herndl"
},
{
"homeUrl": "https://github.com/vokins",
"id": 37,
"name": "vokins"
},
{
"homeUrl": "https://kowith337.github.io/",
"id": 38,
"name": "Kowith Singkornkeeree",
"twitterHandle": "kowith337"
},
{
"emailAddress": "me@pureapp.in.th",
"homeUrl": "https://www.pureapp.in.th/",
"id": 39,
"name": "Pakkapon Phongthawee"
},
{
"emailAddress": "gato@intaa.net",
"homeUrl": "https://warui.intaa.net/",
"id": 40,
"name": "Warui",
"twitterHandle": "gatolabo"
},
{
"emailAddress": "multiverse2011@gmail.com",
"homeUrl": "https://github.com/multiverse2011",
"id": 41,
"name": "Multiverse2011",
"twitterHandle": "multiverse2011"
},
{
"homeUrl": "https://cosmonote.blogspot.com",
"id": 42,
"name": "Cosmonote"
},
{
"homeUrl": "https://github.com/Lerist",
"id": 43,
"name": "Lerist"
},
{
"homeUrl": "http://vxvault.net/",
"id": 44,
"name": "VXVault",
"twitterHandle": "siri_urz"
},
{
"emailAddress": "support@disconnect.me",
"homeUrl": "https://disconnect.me",
"id": 45,
"name": "Disconnect",
"twitterHandle": "disconnectme"
},
{
"homeUrl": "https://adguard.com/",
"id": 46,
"name": "AdGuard",
"twitterHandle": "adguard"
},
{
"homeUrl": "https://github.com/reek",
"id": 47,
"name": "Reek"
},
{
"homeUrl": "https://github.com/eladkarako",
"id": 48,
"name": "Elad Karako"
},
{
"emailAddress": "zerodot1@bk.ru",
"homeUrl": "https://github.com/ZeroDot1",
"id": 49,
"name": "ZeroDot1",
"twitterHandle": "hobbygrafix"
},
{
"emailAddress": "lolipopplus@protonmail.com",
"homeUrl": "https://github.com/Rictusempra",
"id": 51,
"name": "Rictusempra"
},
{
"emailAddress": "tomas@getadblock.com",
"homeUrl": "https://github.com/tomasko126",
"id": 52,
"name": "Tomáš Taro"
},
{
"homeUrl": "https://github.com/Hubird-au",
"id": 53,
"name": "Hubird-au"
},
{
"homeUrl": "https://github.com/hl2guide",
"id": 54,
"name": "deanoman"
},
{
"homeUrl": "https://github.com/haykam821",
"id": 55,
"name": "haykam821"
},
{
"homeUrl": "https://github.com/cb-software",
"id": 56,
"name": "cb-software"
},
{
"homeUrl": "https://github.com/gorhill",
"id": 57,
"name": "Raymond Hill",
"twitterHandle": "gorhill"
},
{
"homeUrl": "https://github.com/toshiya44",
"id": 58,
"name": "Toshiya"
},
{
"emailAddress": "jamie@jamiedubs.com",
"homeUrl": "https://jamiedubs.com/",
"id": 59,
"name": "Jamie Wilkinson",
"twitterHandle": "jamiew"
},
{
"homeUrl": "https://github.com/raghavdua1995",
"id": 62,
"name": "Raghav Dua"
},
{
"homeUrl": "https://github.com/jspenguin2017",
"id": 63,
"name": "Hugo Xu"
},
{
"homeUrl": "https://github.com/lassekongo83",
"id": 64,
"name": "Mattias"
},
{
"homeUrl": "https://gist.github.com/BBcan177",
"id": 66,
"name": "BBcan177",
"twitterHandle": "BBcan177"
},
{
"homeUrl": "https://github.com/betterwebleon/",
"id": 68,
"name": "betterwebleon"
},
{
"homeUrl": "https://github.com/notracking",
"id": 69,
"name": "notracking"
},
{
"homeUrl": "https://github.com/KonoromiHimaries",
"id": 70,
"name": "KonoromiHimaries"
},
{
"homeUrl": "https://forums.lanik.us/memberlist.php?mode=viewprofile&u=9300",
"id": 71,
"name": "invisible666"
},
{
"emailAddress": "endolith@gmail.com",
"homeUrl": "https://github.com/endolith",
"id": 72,
"name": "endolith"
},
{
"homeUrl": "https://github.com/UnluckyLuke",
"id": 73,
"name": "UnluckyLuke"
},
{
"homeUrl": "https://github.com/xd4rker",
"id": 75,
"name": "xd4rker",
"twitterHandle": "xd4rker"
},
{
"homeUrl": "https://github.com/hit3shjain",
"id": 76,
"name": "hit3shjain"
},
{
"homeUrl": "https://github.com/SkeletalDemise",
"id": 77,
"name": "SkeletalDemise"
},
{
"emailAddress": "github@paulbutler.org",
"homeUrl": "https://paulbutler.org",
"id": 78,
"name": "Paul Butler",
"twitterHandle": "paulgb"
},
{
"emailAddress": "info@dshield.org",
"homeUrl": "https://dshield.org/",
"id": 79,
"name": "DShield"
},
{
"emailAddress": "email@kowabit.de",
"homeUrl": "https://www.kowabit.de/",
"id": 80,
"name": "kowabit"
},
{
"homeUrl": "https://github.com/chadmayfield",
"id": 81,
"name": "Chad Mayfield",
"twitterHandle": "chadrmayfield"
},
{
"homeUrl": "https://getadblock.com/",
"id": 82,
"name": "AdBlock"
},
{
"homeUrl": "https://forum.xda-developers.com/member.php?u=4546389",
"id": 83,
"name": "jerryn70"
},
{
"emailAddress": "mail.energized@protonmail.com",
"homeUrl": "https://github.com/EnergizedProtection",
"id": 84,
"name": "Team Boltz",
"twitterHandle": "adroitadorkhan"
},
{
"homeUrl": "https://github.com/FadeMind",
"id": 85,
"name": "FadeMind"
},
{
"homeUrl": "http://rlwpx.free.fr/",
"id": 86,
"name": "Airelle"
},
{
"id": 87,
"name": "Aegis Team",
"twitterHandle": "TeamAEGISgg"
},
{
"emailAddress": "support@riskanalytics.com",
"homeUrl": "https://www.riskanalytics.com/",
"id": 88,
"name": "RiskAnalytics"
},
{
"emailAddress": "support@it-mate.co.uk",
"homeUrl": "https://hosts-file.net/",
"id": 89,
"name": "Hosts-File Dot Net"
},
{
"emailAddress": "jmdugan_pubprofile_1522347973@biocontact.org",
"homeUrl": "https://biocontact.org/jmdugan/",
"id": 90,
"name": "Jonathan Dugan"
},
{
"emailAddress": "pgl@yoyo.org",
"homeUrl": "https://pgl.yoyo.org/",
"id": 91,
"name": "Peter Lowe"
},
{
"homeUrl": "https://forums.lanik.us/viewforum.php?f=102",
"id": 92,
"name": "RU AdList"
},
{
"emailAddress": "steveb@stevenblack.com",
"homeUrl": "http://stevenblack.com/",
"id": 93,
"name": "Steven Black",
"twitterHandle": "steveblack"
},
{
"homeUrl": "https://github.com/yourduskquibbles",
"id": 94,
"name": "yourduskquibbles"
},
{
"homeUrl": "https://github.com/k2jp",
"id": 95,
"name": "k2jp"
},
{
"emailAddress": "blog@zoso.ro",
"homeUrl": "https://www.zoso.ro/",
"id": 96,
"name": "Zoso",
"twitterHandle": "zoso"
},
{
"emailAddress": "easylisthebrew@gmail.com",
"homeUrl": "https://github.com/easylist/EasyListHebrew",
"id": 97,
"name": "EasyListHebrew"
},
{
"emailAddress": "easylist.china@gmail.com",
"homeUrl": "https://abpchina.org/forum/",
"id": 98,
"name": "EasyList China"
},
{
"homeUrl": "https://margevicius.lt/",
"id": 99,
"name": "gymka"
},
{
"homeUrl": "https://stanev.org/",
"id": 100,
"name": "Alex Stanev"
},
{
"id": 101,
"name": "BLOGRAM"
},
{
"homeUrl": "https://notabug.org/anonymous74100?tab=activity",
"id": 102,
"name": "anonymous74100"
},
{
"emailAddress": "costa@firehol.org",
"homeUrl": "https://firehol.org/",
"id": 103,
"name": "FireHOL"
},
{
"emailAddress": "mail@perflyst.de",
"homeUrl": "https://github.com/Perflyst",
"id": 104,
"name": "Perflyst"
},
{
"homeUrl": "https://github.com/crazy-max",
"id": 105,
"name": "CrazyMax"
},
{
"homeUrl": "https://github.com/sipp11",
"id": 106,
"name": "sipp11"
},
{
"emailAddress": "bogachenkove@gmail.com",
"homeUrl": "https://github.com/bogachenko",
"id": 107,
"name": "Bogachenko Vyacheslav"
},
{
"emailAddress": "nolamiller1203+githubmochifilter@gmail.com",
"homeUrl": "https://pokapoka.shoooter.net/",
"id": 108,
"name": "eEIi0A5L",
"twitterHandle": "mochi_adfilter"
},
{
"homeUrl": "https://github.com/deathbybandaid",
"id": 109,
"name": "Deathbybandaid"
},
{
"homeUrl": "https://akamaru.de/",
"id": 110,
"name": "Akamaru",
"twitterHandle": "Akamaru_Pie"
},
{
"homeUrl": "https://github.com/MrBukLau",
"id": 111,
"name": "Buk Lau"
},
{
"homeUrl": "https://github.com/olegwukr",
"id": 112,
"name": "olegwukr"
},
{
"homeUrl": "https://github.com/koenrh",
"id": 113,
"name": "Koen Rouwhorst"
},
{
"emailAddress": "neofelhz@gmail.com",
"homeUrl": "https://nfz.moe/",
"id": 114,
"name": "neo Felhz",
"twitterHandle": "neoFelhz"
},
{
"homeUrl": "https://www.stopforumspam.com/",
"id": 115,
"name": "StopForumSpam.com",
"twitterHandle": "StopForumSpam"
},
{
"emailAddress": "github@chrisbuijs.com",
"homeUrl": "https://github.com/cbuijs/",
"id": 116,
"name": "Chris Buijs"
},
{
"emailAddress": "admin@intr0.com",
"homeUrl": "https://intr0.com/",
"id": 117,
"name": "intr0"
},
{
"homeUrl": "https://github.com/ookangzheng",
"id": 118,
"name": "ookangzheng"
},
{
"emailAddress": "hector@molinero.dev",
"homeUrl": "https://github.com/hectorm",
"id": 119,
"name": "Héctor Molinero Fernández"
},
{
"homeUrl": "https://github.com/ngosang",
"id": 120,
"name": "Diego Heras"
},
{
"emailAddress": "mitchellkrog@gmail.com",
"homeUrl": "https://github.com/mitchellkrogza",
"id": 121,
"name": "mitchellkrogza"
},
{
"emailAddress": "dev@oz.nu",
"homeUrl": "https://github.com/oznu",
"id": 122,
"name": "oznu"
},
{
"emailAddress": "reddestdream@gmail.com",
"homeUrl": "https://github.com/ReddestDream",
"id": 123,
"name": "ReddestDream"
},
{
"emailAddress": "jcb@bambenekconsulting.com",
"homeUrl": "https://www.bambenekconsulting.com",
"id": 124,
"name": "John Bambenek"
},
{
"emailAddress": "jwspamspy@pobox.com",
"homeUrl": "https://joewein.de/",
"id": 125,
"name": "Joe Wein"
},
{
"emailAddress": "radicalbotanical@aol.com",
"homeUrl": "https://github.com/bongochong",
"id": 126,
"name": "Chongo Bong"
},
{
"homeUrl": "https://quidsup.net/",
"id": 127,
"name": "QuidsUp"
},
{
"homeUrl": "https://github.com/kano1",
"id": 128,
"name": "kano1"
},
{
"homeUrl": "https://hostsfile.mine.nu/",
"id": 129,
"name": "Andy Short"
},
{
"homeUrl": "https://github.com/RooneyMcNibNug",
"id": 130,
"name": "RooneyMcNibNug",
"twitterHandle": "rooneymcnibnug"
},
{
"homeUrl": "http://columwalsh.ie/",
"id": 131,
"name": "Colum Walsh"
},
{
"homeUrl": "https://github.com/mtxadmin",
"id": 132,
"name": "mtxadmin"
},
{
"homeUrl": "https://gioxx.org/",
"id": 133,
"name": "Giovanni Francesco Solone",
"twitterHandle": "gioxx"
},
{
"emailAddress": "elias.ojala+github@kapsi.fi",
"homeUrl": "https://github.com/theel0ja",
"id": 134,
"name": "Elias Ojala"
},
{
"homeUrl": "https://github.com/CitizenXVIL",
"id": 135,
"name": "CitizenXVIL"
},
{
"emailAddress": "author@i-dont-care-about-cookies.eu",
"homeUrl": "https://www.i-dont-care-about-cookies.eu/",
"id": 136,
"name": "Daniel Kladnik"
},
{
"homeUrl": "https://github.com/Sinfonietta",
"id": 137,
"name": "Sinfonietta"
},
{
"homeUrl": "https://github.com/HankAviator",
"id": 138,
"name": "HankAviator"
},
{
"homeUrl": "https://www.github.developerdan.com/hosts/",
"id": 139,
"name": "lightswitch05"
},
{
"homeUrl": "https://github.com/mmotti",
"id": 140,
"name": "mmotti"
},
{
"homeUrl": "https://github.com/hawkeye116477",
"id": 141,
"name": "hawkeye116477"
},
{
"homeUrl": "https://github.com/caffeinewriter",
"id": 142,
"name": "caffeinewriter"
},
{
"homeUrl": "https://github.com/Nebula-Mechanica",
"id": 143,
"name": "Nebula-Mechanica"
},
{
"homeUrl": "https://github.com/Cybo1927",
"id": 144,
"name": "Cybo1927"
},
{
"emailAddress": "iurcaberciudarius@gmail.com",
"homeUrl": "https://github.com/DariusIurca",
"id": 145,
"name": "DariusIurca",
"twitterHandle": "dariusiurca"
},
{
"emailAddress": "geoffrey+website@frogeye.fr",
"homeUrl": "https://geoffrey.frogeye.fr/",
"id": 146,
"name": "Geoffrey Frogeye",
"twitterHandle": "GeoffreyFrogeye"
},
{
"homeUrl": "https://github.com/LanikSJ",
"id": 147,
"name": "LanikSJ"
},
{
"emailAddress": "me@deletescape.ch",
"homeUrl": "https://deletescape.ch/",
"id": 148,
"name": "deletescape",
"twitterHandle": "deletescape"
},
{
"homeUrl": "https://github.com/InnoScorpio",
"id": 149,
"name": "InnoScorpio"
},
{
"homeUrl": "https://github.com/Yuki2718",
"id": 150,
"name": "Yuki2718"
},
{
"homeUrl": "https://nou.si/",
"id": 151,
"name": "Soteria nou"
},
{
"emailAddress": "daniel@danielmiessler.com",
"homeUrl": "https://danielmiessler.com/",
"id": 152,
"name": "Daniel Miessler",
"twitterHandle": "danielmiessler"
},
{
"emailAddress": "fabrice.prigent@ut-capitole.fr",
"homeUrl": "https://dsi.ut-capitole.fr/blacklists/",
"id": 153,
"name": "Fabrice Prigent"
},
{
"homeUrl": "https://blocklist.site/",
"id": 154,
"name": "Block Lists Project"
},
{
"homeUrl": "https://github.com/bongochong",
"id": 155,
"name": "Chongo Bong"
},
{
"homeUrl": "https://www.ftpihole.com/",
"id": 156,
"name": "FTPiHole"
}
]

270
data/Merge.json
View file

@ -1,270 +0,0 @@
[
{
"mergeFilterListId": 11,
"upstreamFilterListId": 12
},
{
"mergeFilterListId": 11,
"upstreamFilterListId": 301
},
{
"mergeFilterListId": 13,
"upstreamFilterListId": 12
},
{
"mergeFilterListId": 13,
"upstreamFilterListId": 14
},
{
"mergeFilterListId": 13,
"upstreamFilterListId": 301
},
{
"mergeFilterListId": 168,
"upstreamFilterListId": 157
},
{
"mergeFilterListId": 168,
"upstreamFilterListId": 159
},
{
"mergeFilterListId": 168,
"upstreamFilterListId": 162
},
{
"mergeFilterListId": 168,
"upstreamFilterListId": 166
},
{
"mergeFilterListId": 168,
"upstreamFilterListId": 242
},
{
"mergeFilterListId": 168,
"upstreamFilterListId": 301
},
{
"mergeFilterListId": 255,
"upstreamFilterListId": 564
},
{
"mergeFilterListId": 265,
"upstreamFilterListId": 564
},
{
"mergeFilterListId": 266,
"upstreamFilterListId": 564
},
{
"mergeFilterListId": 271,
"upstreamFilterListId": 338
},
{
"mergeFilterListId": 274,
"upstreamFilterListId": 275
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 17
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 32
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 54
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 80
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 81
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 99
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 138
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 170
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 177
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 179
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 206
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 226
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 255
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 276
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 282
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 316
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 317
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 324
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 325
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 329
},
{
"mergeFilterListId": 309,
"upstreamFilterListId": 340
},
{
"mergeFilterListId": 331,
"upstreamFilterListId": 199
},
{
"mergeFilterListId": 331,
"upstreamFilterListId": 330
},
{
"mergeFilterListId": 331,
"upstreamFilterListId": 332
},
{
"mergeFilterListId": 331,
"upstreamFilterListId": 333
},
{
"mergeFilterListId": 371,
"upstreamFilterListId": 110
},
{
"mergeFilterListId": 371,
"upstreamFilterListId": 232
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 2
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 18
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 19
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 27
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 38
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 70
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 71
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 97
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 99
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 183
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 184
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 214
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 232
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 247
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 279
},
{
"mergeFilterListId": 375,
"upstreamFilterListId": 307
},
{
"mergeFilterListId": 566,
"upstreamFilterListId": 569
},
{
"mergeFilterListId": 566,
"upstreamFilterListId": 570
},
{
"mergeFilterListId": 566,
"upstreamFilterListId": 573
},
{
"mergeFilterListId": 566,
"upstreamFilterListId": 574
},
{
"mergeFilterListId": 566,
"upstreamFilterListId": 575
},
{
"mergeFilterListId": 566,
"upstreamFilterListId": 583
},
{
"mergeFilterListId": 566,
"upstreamFilterListId": 585
},
{
"mergeFilterListId": 566,
"upstreamFilterListId": 586
}
]

283
data/Software.json
View file

@ -1,283 +0,0 @@
[
{
"downloadUrl": "https://github.com/gorhill/uBlock#installation",
"homeUrl": "https://github.com/gorhill/uBlock",
"id": 1,
"isAbpSubscribable": true,
"name": "uBlock Origin"
},
{
"downloadUrl": "https://adblockplus.org/",
"homeUrl": "https://adblockplus.org/",
"id": 2,
"isAbpSubscribable": true,
"name": "Adblock Plus"
},
{
"downloadUrl": "https://adguard.com/en/download.html",
"homeUrl": "https://adguard.com/",
"id": 3,
"isAbpSubscribable": true,
"name": "AdGuard"
},
{
"downloadUrl": "https://f-droid.org/packages/org.jak_linux.dns66/",
"homeUrl": "https://github.com/julian-klode/dns66",
"id": 4,
"isAbpSubscribable": false,
"name": "DNS66"
},
{
"downloadUrl": "https://github.com/NanoAdblocker/NanoCore#nano-adblocker-core",
"homeUrl": "https://github.com/NanoAdblocker/NanoCore",
"id": 5,
"isAbpSubscribable": true,
"name": "Nano Adblocker"
},
{
"downloadUrl": "https://getadblock.com/",
"homeUrl": "https://getadblock.com/",
"id": 6,
"isAbpSubscribable": true,
"name": "AdBlock"
},
{
"downloadUrl": "https://f-droid.org/packages/org.adaway/",
"homeUrl": "https://adaway.org/",
"id": 7,
"isAbpSubscribable": false,
"name": "AdAway"
},
{
"downloadUrl": "https://addons.mozilla.org/firefox/addon/personal-blocklist/",
"homeUrl": "https://addons.mozilla.org/firefox/addon/personal-blocklist/",
"id": 8,
"isAbpSubscribable": false,
"name": "Personal Blocklist for Firefox"
},
{
"downloadUrl": "http://einaregilsson.com/redirector/",
"homeUrl": "http://einaregilsson.com/redirector/",
"id": 10,
"isAbpSubscribable": false,
"name": "Redirector"
},
{
"downloadUrl": "https://github.com/scottlerch/HostsFileEditor/releases",
"homeUrl": "https://hostsfileeditor.com/",
"id": 11,
"isAbpSubscribable": false,
"name": "Hosts File Editor"
},
{
"downloadUrl": "https://github.com/2ndalpha/gasmask/releases",
"homeUrl": "https://github.com/2ndalpha/gasmask",
"id": 12,
"isAbpSubscribable": false,
"name": "Gas Mask"
},
{
"downloadUrl": "https://github.com/xd4rker/MinerBlock/releases",
"homeUrl": "https://github.com/xd4rker/MinerBlock",
"id": 13,
"isAbpSubscribable": false,
"name": "MinerBlock"
},
{
"downloadUrl": "https://pi-hole.net/",
"homeUrl": "https://pi-hole.net/",
"id": 14,
"isAbpSubscribable": false,
"name": "Pi-hole"
},
{
"downloadUrl": "https://support.microsoft.com/en-gb/help/18520/download-internet-explorer-11-offline-installer",
"homeUrl": "https://docs.microsoft.com/en-us/previous-versions/windows/internet-explorer/ie-developer/platform-apis/hh273399(v=vs.85)",
"id": 16,
"isAbpSubscribable": false,
"name": "Internet Explorer (TPL)"
},
{
"downloadUrl": "https://greasyfork.org/scripts/1682-google-hit-hider-by-domain-search-filter-block-sites",
"homeUrl": "https://www.jeffersonscher.com/gm/google-hit-hider/",
"id": 17,
"isAbpSubscribable": false,
"name": "Google Hit Hider by Domain"
},
{
"downloadUrl": "https://firehol.org/installing/",
"homeUrl": "https://firehol.org/",
"id": 18,
"isAbpSubscribable": false,
"name": "FireHOL"
},
{
"homeUrl": "https://www.samsungknox.com",
"id": 19,
"isAbpSubscribable": false,
"name": "Samsung Knox"
},
{
"downloadUrl": "https://www.obdev.at/products/littlesnitch/download.html",
"homeUrl": "https://www.obdev.at/products/littlesnitch/index.html",
"id": 20,
"isAbpSubscribable": false,
"name": "Little Snitch"
},
{
"downloadUrl": "https://www.privoxy.org/sf-download-mirror/",
"homeUrl": "https://www.privoxy.org/",
"id": 21,
"isAbpSubscribable": false,
"name": "Privoxy"
},
{
"downloadUrl": "https://diversion.ch/diversion/installation.html",
"homeUrl": "https://diversion.ch/",
"id": 22,
"isAbpSubscribable": false,
"name": "Diversion"
},
{
"downloadUrl": "http://www.thekelleys.org.uk/dnsmasq/",
"homeUrl": "http://www.thekelleys.org.uk/dnsmasq/doc.html",
"id": 23,
"isAbpSubscribable": false,
"name": "dnsmasq"
},
{
"downloadUrl": "https://www.slimjet.com/en/dlpage.php",
"homeUrl": "https://www.slimjet.com",
"id": 24,
"isAbpSubscribable": true,
"name": "Slimjet"
},
{
"downloadUrl": "https://github.com/gorhill/uMatrix#umatrix",
"homeUrl": "https://github.com/gorhill/uMatrix",
"id": 25,
"isAbpSubscribable": true,
"name": "uMatrix"
},
{
"downloadUrl": "https://f-droid.org/packages/org.blokada.alarm/",
"homeUrl": "https://blokada.org",
"id": 26,
"isAbpSubscribable": false,
"name": "Blokada"
},
{
"downloadUrl": "https://github.com/henrypp/hostsmgr/releases",
"homeUrl": "https://www.henrypp.org/product/hostsmgr",
"id": 27,
"isAbpSubscribable": false,
"name": "hostsmgr"
},
{
"downloadUrl": "https://zenz-solutions.de/personaldnsfilter/",
"homeUrl": "https://zenz-solutions.de/personaldnsfilter/",
"id": 28,
"isAbpSubscribable": false,
"name": "personalDNSfilter"
},
{
"downloadUrl": "https://nlnetlabs.nl/projects/unbound/download/",
"homeUrl": "https://nlnetlabs.nl/projects/unbound/about/",
"id": 29,
"isAbpSubscribable": false,
"name": "Unbound"
},
{
"downloadUrl": "https://www.isc.org/bind/",
"homeUrl": "https://www.isc.org/bind/",
"id": 30,
"isAbpSubscribable": false,
"name": "BIND"
},
{
"downloadUrl": "https://github.com/AdguardTeam/AdGuardHome/releases",
"homeUrl": "https://github.com/AdguardTeam/AdGuardHome",
"id": 31,
"isAbpSubscribable": false,
"name": "AdGuard Home"
},
{
"downloadUrl": "https://adnauseam.io/",
"homeUrl": "https://adnauseam.io/",
"id": 32,
"isAbpSubscribable": true,
"name": "AdNauseam"
},
{
"id": 33,
"isAbpSubscribable": false,
"name": "Legacy Unix derivatives"
},
{
"id": 34,
"isAbpSubscribable": false,
"name": "Windows command line"
},
{
"downloadUrl": "https://shadowsocks.org/en/download/clients.html",
"homeUrl": "https://shadowsocks.org/en/index.html",
"id": 35,
"isAbpSubscribable": false,
"name": "Shadowsocks"
},
{
"downloadUrl": "https://github.com/shadowsocksrr/shadowsocksr-csharp/releases",
"homeUrl": "https://github.com/shadowsocksrr/shadowsocksr-csharp",
"id": 36,
"isAbpSubscribable": false,
"name": "ShadowsocksR"
},
{
"downloadUrl": "https://apps.apple.com/app/shadowrocket/id932747118",
"homeUrl": "https://liguangming.com/Shadowrocket",
"id": 37,
"isAbpSubscribable": false,
"name": "Shadowrocket"
},
{
"downloadUrl": "https://github.com/fengyc/URLRedirector/#urlredirector",
"homeUrl": "https://github.com/fengyc/URLRedirector/",
"id": 38,
"isAbpSubscribable": false,
"name": "URLRedirector"
},
{
"homeUrl": "https://www.reddit.com/r/pfBlockerNG/",
"id": 39,
"isAbpSubscribable": false,
"name": "pfBlockerNG"
},
{
"downloadUrl": "https://www.opera.com/",
"homeUrl": "https://www.opera.com/",
"id": 40,
"isAbpSubscribable": true,
"name": "Opera's built-in adblocker"
},
{
"downloadUrl": "https://nssurge.com/mac/v3/Surge-latest.zip",
"homeUrl": "https://nssurge.com/",
"id": 41,
"isAbpSubscribable": false,
"name": "Surge"
},
{
"downloadUrl": "https://github.com/DNSCrypt/dnscrypt-proxy/releases/latest",
"homeUrl": "https://github.com/DNSCrypt/dnscrypt-proxy",
"id": 42,
"isAbpSubscribable": false,
"name": "dnscrypt-proxy"
},
{
"downloadUrl": "https://github.com/pymumu/smartdns/releases",
"homeUrl": "https://github.com/pymumu/smartdns",
"id": 43,
"isAbpSubscribable": false,
"name": "SmartDNS"
}
]

950
data/SoftwareSyntax.json
View file

@ -1,950 +0,0 @@
[
{
"softwareId": 1,
"syntaxId": 1
},
{
"softwareId": 4,
"syntaxId": 1
},
{
"softwareId": 5,
"syntaxId": 1
},
{
"softwareId": 7,
"syntaxId": 1
},
{
"softwareId": 11,
"syntaxId": 1
},
{
"softwareId": 12,
"syntaxId": 1
},
{
"softwareId": 14,
"syntaxId": 1
},
{
"softwareId": 22,
"syntaxId": 1
},
{
"softwareId": 25,
"syntaxId": 1
},
{
"softwareId": 26,
"syntaxId": 1
},
{
"softwareId": 27,
"syntaxId": 1
},
{
"softwareId": 28,
"syntaxId": 1
},
{
"softwareId": 31,
"syntaxId": 1
},
{
"softwareId": 32,
"syntaxId": 1
},
{
"softwareId": 39,
"syntaxId": 1
},
{
"softwareId": 1,
"syntaxId": 2
},
{
"softwareId": 3,
"syntaxId": 2
},
{
"softwareId": 4,
"syntaxId": 2
},
{
"softwareId": 5,
"syntaxId": 2
},
{
"softwareId": 8,
"syntaxId": 2
},
{
"softwareId": 14,
"syntaxId": 2
},
{
"softwareId": 17,
"syntaxId": 2
},
{
"softwareId": 19,
"syntaxId": 2
},
{
"softwareId": 22,
"syntaxId": 2
},
{
"softwareId": 25,
"syntaxId": 2
},
{
"softwareId": 26,
"syntaxId": 2
},
{
"softwareId": 27,
"syntaxId": 2
},
{
"softwareId": 28,
"syntaxId": 2
},
{
"softwareId": 31,
"syntaxId": 2
},
{
"softwareId": 32,
"syntaxId": 2
},
{
"softwareId": 39,
"syntaxId": 2
},
{
"softwareId": 42,
"syntaxId": 2
},
{
"softwareId": 1,
"syntaxId": 3
},
{
"softwareId": 2,
"syntaxId": 3
},
{
"softwareId": 3,
"syntaxId": 3
},
{
"softwareId": 5,
"syntaxId": 3
},
{
"softwareId": 6,
"syntaxId": 3
},
{
"softwareId": 24,
"syntaxId": 3
},
{
"softwareId": 32,
"syntaxId": 3
},
{
"softwareId": 39,
"syntaxId": 3
},
{
"softwareId": 40,
"syntaxId": 3
},
{
"softwareId": 1,
"syntaxId": 4
},
{
"softwareId": 3,
"syntaxId": 4
},
{
"softwareId": 5,
"syntaxId": 4
},
{
"softwareId": 32,
"syntaxId": 4
},
{
"softwareId": 1,
"syntaxId": 6
},
{
"softwareId": 3,
"syntaxId": 6
},
{
"softwareId": 5,
"syntaxId": 6
},
{
"softwareId": 32,
"syntaxId": 6
},
{
"softwareId": 1,
"syntaxId": 7
},
{
"softwareId": 5,
"syntaxId": 7
},
{
"softwareId": 25,
"syntaxId": 7
},
{
"softwareId": 32,
"syntaxId": 7
},
{
"softwareId": 1,
"syntaxId": 8
},
{
"softwareId": 3,
"syntaxId": 8
},
{
"softwareId": 5,
"syntaxId": 8
},
{
"softwareId": 32,
"syntaxId": 8
},
{
"softwareId": 1,
"syntaxId": 9
},
{
"softwareId": 5,
"syntaxId": 9
},
{
"softwareId": 18,
"syntaxId": 9
},
{
"softwareId": 31,
"syntaxId": 9
},
{
"softwareId": 32,
"syntaxId": 9
},
{
"softwareId": 39,
"syntaxId": 9
},
{
"softwareId": 16,
"syntaxId": 10
},
{
"softwareId": 10,
"syntaxId": 11
},
{
"softwareId": 5,
"syntaxId": 12
},
{
"softwareId": 1,
"syntaxId": 13
},
{
"softwareId": 5,
"syntaxId": 13
},
{
"softwareId": 13,
"syntaxId": 13
},
{
"softwareId": 32,
"syntaxId": 13
},
{
"softwareId": 7,
"syntaxId": 14
},
{
"softwareId": 11,
"syntaxId": 14
},
{
"softwareId": 12,
"syntaxId": 14
},
{
"softwareId": 14,
"syntaxId": 14
},
{
"softwareId": 31,
"syntaxId": 14
},
{
"softwareId": 18,
"syntaxId": 15
},
{
"softwareId": 1,
"syntaxId": 16
},
{
"softwareId": 3,
"syntaxId": 16
},
{
"softwareId": 5,
"syntaxId": 16
},
{
"softwareId": 19,
"syntaxId": 16
},
{
"softwareId": 26,
"syntaxId": 16
},
{
"softwareId": 31,
"syntaxId": 16
},
{
"softwareId": 32,
"syntaxId": 16
},
{
"softwareId": 42,
"syntaxId": 16
},
{
"softwareId": 1,
"syntaxId": 17
},
{
"softwareId": 5,
"syntaxId": 17
},
{
"softwareId": 32,
"syntaxId": 17
},
{
"softwareId": 20,
"syntaxId": 18
},
{
"softwareId": 21,
"syntaxId": 19
},
{
"softwareId": 23,
"syntaxId": 20
},
{
"softwareId": 1,
"syntaxId": 21
},
{
"softwareId": 5,
"syntaxId": 21
},
{
"softwareId": 32,
"syntaxId": 21
},
{
"softwareId": 3,
"syntaxId": 22
},
{
"softwareId": 4,
"syntaxId": 22
},
{
"softwareId": 26,
"syntaxId": 22
},
{
"softwareId": 28,
"syntaxId": 22
},
{
"softwareId": 31,
"syntaxId": 22
},
{
"softwareId": 33,
"syntaxId": 23
},
{
"softwareId": 29,
"syntaxId": 24
},
{
"softwareId": 30,
"syntaxId": 25
},
{
"softwareId": 30,
"syntaxId": 26
},
{
"softwareId": 34,
"syntaxId": 27
},
{
"softwareId": 1,
"syntaxId": 28
},
{
"softwareId": 2,
"syntaxId": 28
},
{
"softwareId": 3,
"syntaxId": 28
},
{
"softwareId": 5,
"syntaxId": 28
},
{
"softwareId": 6,
"syntaxId": 28
},
{
"softwareId": 24,
"syntaxId": 28
},
{
"softwareId": 31,
"syntaxId": 28
},
{
"softwareId": 32,
"syntaxId": 28
},
{
"softwareId": 39,
"syntaxId": 28
},
{
"softwareId": 40,
"syntaxId": 28
},
{
"softwareId": 35,
"syntaxId": 29
},
{
"softwareId": 36,
"syntaxId": 29
},
{
"softwareId": 37,
"syntaxId": 29
},
{
"softwareId": 41,
"syntaxId": 29
},
{
"softwareId": 14,
"syntaxId": 30
},
{
"softwareId": 38,
"syntaxId": 31
},
{
"softwareId": 1,
"syntaxId": 32
},
{
"softwareId": 5,
"syntaxId": 32
},
{
"softwareId": 18,
"syntaxId": 32
},
{
"softwareId": 31,
"syntaxId": 32
},
{
"softwareId": 32,
"syntaxId": 32
},
{
"softwareId": 39,
"syntaxId": 32
},
{
"softwareId": 1,
"syntaxId": 33
},
{
"softwareId": 5,
"syntaxId": 33
},
{
"softwareId": 18,
"syntaxId": 33
},
{
"softwareId": 31,
"syntaxId": 33
},
{
"softwareId": 32,
"syntaxId": 33
},
{
"softwareId": 39,
"syntaxId": 33
},
{
"softwareId": 18,
"syntaxId": 34
},
{
"softwareId": 31,
"syntaxId": 34
},
{
"softwareId": 39,
"syntaxId": 34
},
{
"softwareId": 1,
"syntaxId": 35
},
{
"softwareId": 4,
"syntaxId": 35
},
{
"softwareId": 5,
"syntaxId": 35
},
{
"softwareId": 7,
"syntaxId": 35
},
{
"softwareId": 11,
"syntaxId": 35
},
{
"softwareId": 12,
"syntaxId": 35
},
{
"softwareId": 14,
"syntaxId": 35
},
{
"softwareId": 22,
"syntaxId": 35
},
{
"softwareId": 25,
"syntaxId": 35
},
{
"softwareId": 26,
"syntaxId": 35
},
{
"softwareId": 27,
"syntaxId": 35
},
{
"softwareId": 28,
"syntaxId": 35
},
{
"softwareId": 31,
"syntaxId": 35
},
{
"softwareId": 32,
"syntaxId": 35
},
{
"softwareId": 39,
"syntaxId": 35
},
{
"softwareId": 1,
"syntaxId": 36
},
{
"softwareId": 5,
"syntaxId": 36
},
{
"softwareId": 7,
"syntaxId": 36
},
{
"softwareId": 11,
"syntaxId": 36
},
{
"softwareId": 12,
"syntaxId": 36
},
{
"softwareId": 14,
"syntaxId": 36
},
{
"softwareId": 22,
"syntaxId": 36
},
{
"softwareId": 27,
"syntaxId": 36
},
{
"softwareId": 31,
"syntaxId": 36
},
{
"softwareId": 7,
"syntaxId": 37
},
{
"softwareId": 11,
"syntaxId": 37
},
{
"softwareId": 12,
"syntaxId": 37
},
{
"softwareId": 31,
"syntaxId": 37
},
{
"softwareId": 2,
"syntaxId": 38
},
{
"softwareId": 6,
"syntaxId": 38
},
{
"softwareId": 1,
"syntaxId": 39
},
{
"softwareId": 5,
"syntaxId": 39
},
{
"softwareId": 18,
"syntaxId": 39
},
{
"softwareId": 31,
"syntaxId": 39
},
{
"softwareId": 32,
"syntaxId": 39
},
{
"softwareId": 39,
"syntaxId": 39
},
{
"softwareId": 1,
"syntaxId": 40
},
{
"softwareId": 5,
"syntaxId": 40
},
{
"softwareId": 18,
"syntaxId": 40
},
{
"softwareId": 31,
"syntaxId": 40
},
{
"softwareId": 32,
"syntaxId": 40
},
{
"softwareId": 39,
"syntaxId": 40
},
{
"softwareId": 18,
"syntaxId": 41
},
{
"softwareId": 31,
"syntaxId": 41
},
{
"softwareId": 39,
"syntaxId": 41
},
{
"softwareId": 1,
"syntaxId": 42
},
{
"softwareId": 5,
"syntaxId": 42
},
{
"softwareId": 18,
"syntaxId": 42
},
{
"softwareId": 31,
"syntaxId": 42
},
{
"softwareId": 32,
"syntaxId": 42
},
{
"softwareId": 39,
"syntaxId": 42
},
{
"softwareId": 18,
"syntaxId": 43
},
{
"softwareId": 31,
"syntaxId": 43
},
{
"softwareId": 39,
"syntaxId": 43
},
{
"softwareId": 34,
"syntaxId": 44
},
{
"softwareId": 1,
"syntaxId": 45
},
{
"softwareId": 5,
"syntaxId": 45
},
{
"softwareId": 18,
"syntaxId": 45
},
{
"softwareId": 31,
"syntaxId": 45
},
{
"softwareId": 32,
"syntaxId": 45
},
{
"softwareId": 39,
"syntaxId": 45
},
{
"softwareId": 1,
"syntaxId": 46
},
{
"softwareId": 3,
"syntaxId": 46
},
{
"softwareId": 5,
"syntaxId": 46
},
{
"softwareId": 31,
"syntaxId": 46
},
{
"softwareId": 32,
"syntaxId": 46
},
{
"softwareId": 1,
"syntaxId": 47
},
{
"softwareId": 3,
"syntaxId": 47
},
{
"softwareId": 5,
"syntaxId": 47
},
{
"softwareId": 24,
"syntaxId": 47
},
{
"softwareId": 31,
"syntaxId": 47
},
{
"softwareId": 32,
"syntaxId": 47
},
{
"softwareId": 39,
"syntaxId": 47
},
{
"softwareId": 1,
"syntaxId": 48
},
{
"softwareId": 2,
"syntaxId": 48
},
{
"softwareId": 3,
"syntaxId": 48
},
{
"softwareId": 4,
"syntaxId": 48
},
{
"softwareId": 5,
"syntaxId": 48
},
{
"softwareId": 6,
"syntaxId": 48
},
{
"softwareId": 8,
"syntaxId": 48
},
{
"softwareId": 14,
"syntaxId": 48
},
{
"softwareId": 17,
"syntaxId": 48
},
{
"softwareId": 19,
"syntaxId": 48
},
{
"softwareId": 22,
"syntaxId": 48
},
{
"softwareId": 25,
"syntaxId": 48
},
{
"softwareId": 26,
"syntaxId": 48
},
{
"softwareId": 27,
"syntaxId": 48
},
{
"softwareId": 28,
"syntaxId": 48
},
{
"softwareId": 31,
"syntaxId": 48
},
{
"softwareId": 32,
"syntaxId": 48
},
{
"softwareId": 39,
"syntaxId": 48
},
{
"softwareId": 40,
"syntaxId": 48
},
{
"softwareId": 42,
"syntaxId": 48
},
{
"softwareId": 43,
"syntaxId": 49
},
{
"softwareId": 4,
"syntaxId": 50
},
{
"softwareId": 14,
"syntaxId": 50
},
{
"softwareId": 26,
"syntaxId": 50
},
{
"softwareId": 31,
"syntaxId": 50
},
{
"softwareId": 25,
"syntaxId": 51
},
{
"softwareId": 28,
"syntaxId": 52
},
{
"softwareId": 1,
"syntaxId": 53
},
{
"softwareId": 5,
"syntaxId": 53
},
{
"softwareId": 32,
"syntaxId": 53
}
]

230
data/Syntax.json
View file

@ -1,230 +0,0 @@
[
{
"definitionUrl": "https://en.wikipedia.org/wiki/Hosts_(file)",
"id": 1,
"name": "Hosts (localhost IPv4)"
},
{
"id": 2,
"name": "Domains"
},
{
"definitionUrl": "https://adblockplus.org/filters",
"id": 3,
"name": "Adblock Plus"
},
{
"definitionUrl": "https://github.com/gorhill/uBlock/wiki/Static-filter-syntax",
"id": 4,
"name": "uBlock Origin Static"
},
{
"definitionUrl": "https://kb.adguard.com/en/general/how-to-create-your-own-ad-filters",
"id": 6,
"name": "AdGuard"
},
{
"definitionUrl": "https://github.com/gorhill/uBlock/wiki/Dynamic-filtering:-quick-guide",
"id": 7,
"name": "uMatrix/uBO lined dynamic rules"
},
{
"id": 8,
"name": "URLs"
},
{
"id": 9,
"name": "IPs (IPv4; no CIDR)"
},
{
"definitionUrl": "https://blogs.msdn.microsoft.com/ie/2010/12/07/ie9-and-privacy-introducing-tracking-protection/",
"id": 10,
"name": "Tracking Protection List (IE)"
},
{
"id": 11,
"name": "Redirector"
},
{
"definitionUrl": "https://github.com/NanoAdblocker/NanoCore/tree/master/notes",
"id": 12,
"name": "Nano Adblocker Static"
},
{
"definitionUrl": "https://github.com/xd4rker/MinerBlock",
"id": 13,
"name": "MinerBlock"
},
{
"definitionUrl": "https://en.wikipedia.org/wiki/Hosts_(file)",
"id": 14,
"name": "Non-localhost hosts (IPv4)"
},
{
"id": 15,
"name": "IPs (Start-end-range only)"
},
{
"id": 16,
"name": "Domains with wildcards"
},
{
"definitionUrl": "https://github.com/gorhill/uBlock/wiki/Static-filter-syntax#scriptlet-injection",
"id": 17,
"name": "uBlock Origin scriptlet injection"
},
{
"definitionUrl": "https://help.obdev.at/littlesnitch/lsc-rule-group-subscriptions",
"id": 18,
"name": "Little Snitch subscription rules"
},
{
"definitionUrl": "https://www.privoxy.org/user-manual/actions-file.html",
"id": 19,
"name": "Privoxy action file"
},
{
"id": 20,
"name": "dnsmasq domains list"
},
{
"definitionUrl": "https://github.com/gorhill/uBlock/wiki/Static-filter-syntax#include-file-name",
"id": 21,
"name": "!#include compilation"
},
{
"id": 22,
"name": "DNS servers"
},
{
"id": 23,
"name": "Unix-format hosts.deny file"
},
{
"id": 24,
"name": "Unbound"
},
{
"id": 25,
"name": "Response Policy Zones (RPZ)"
},
{
"id": 26,
"name": "BIND"
},
{
"id": 27,
"name": "Windows command line script"
},
{
"definitionUrl": "https://help.eyeo.com/en/adblockplus/how-to-write-filters#basic",
"id": 28,
"name": "Adblocker-syntax domains"
},
{
"id": 29,
"name": "Socks5"
},
{
"definitionUrl": "https://docs.pi-hole.net/ftldns/regex/tutorial/",
"id": 30,
"name": "Pi-hole RegEx"
},
{
"id": 31,
"name": "URLRedirector"
},
{
"id": 32,
"name": "IPs (IPv4; some CIDR)"
},
{
"id": 33,
"name": "IPs (IPv4 + IPv6; no CIDR)"
},
{
"id": 34,
"name": "IPs (IPv4; CIDR only)"
},
{
"definitionUrl": "https://en.wikipedia.org/wiki/Hosts_(file)",
"id": 35,
"name": "Hosts (localhost IPv4 + IPv6)"
},
{
"definitionUrl": "https://en.wikipedia.org/wiki/Hosts_(file)",
"id": 36,
"name": "Hosts (localhost IPv6)"
},
{
"definitionUrl": "https://en.wikipedia.org/wiki/Hosts_(file)",
"id": 37,
"name": "Non-localhost hosts (IPv6)"
},
{
"id": 38,
"name": "Adblock Plus Advanced"
},
{
"id": 39,
"name": "IPs (IPv6 only; no CIDR)"
},
{
"id": 40,
"name": "IPs (IPv6 only; some CIDR)"
},
{
"id": 41,
"name": "IPs (IPv6 only; CIDR only)"
},
{
"id": 42,
"name": "IPs (IPv4 + IPv6; some CIDR)"
},
{
"id": 43,
"name": "IPs (IPv4 + IPv6; CIDR only)"
},
{
"id": 44,
"name": "PowerShell PKG file"
},
{
"id": 45,
"name": "IPs (IPv4; some start-end-range)"
},
{
"id": 46,
"name": "''$important''/''$empty'' only"
},
{
"id": 47,
"name": "Adblocker-syntax domains w/o ABP tag"
},
{
"id": 48,
"name": "Domains with ABP tag"
},
{
"id": 49,
"name": "SmartDNS"
},
{
"id": 50,
"name": "Domains for whitelisting"
},
{
"definitionUrl": "https://github.com/gorhill/uMatrix/wiki/Ruleset-recipes",
"id": 51,
"name": "uMatrix ruleset recipe"
},
{
"id": 52,
"name": "personalDNSfilter whitelisting"
},
{
"definitionUrl": "https://github.com/gorhill/uBlock/wiki/Dynamic-filtering:-quick-guide",
"id": 53,
"name": "uBO lined dynamic rules w/ noop"
}
]

95
data/TPLSubscriptionAssistant.html
View file

@ -1,95 +0,0 @@
<!DOCTYPE html>
<html>
<head>
<title>FilterLists.com for Internet Explorer</title>
</head>
<body>
Because FilterLists.com cannot be loaded in Internet Explorer, here's a workaround to assist in subscribing to TPL lists.<br><br>
In some circumstances, the link will be blocked the first time around. If so, click on "Allow Blocked Content" in the resulting prompt below, and then click the link again.<br><br>
-------------------------------------------------------<br><br>
* International lists:<br><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylist.tpl', 'EasyList')">EasyList</a> - <a href="https://easylist-msie.adblockplus.org/easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylist_noelemhide.tpl', 'EasyList Without Element Hiding Rules')">EasyList Without Element Hiding Rules</a> - <a href="https://easylist-msie.adblockplus.org/easylist_noelemhide.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylist_noadult.tpl', 'EasyList Without Rules for Adult Sites')">EasyList Without Rules for Adult Sites</a> - <a href="https://easylist-msie.adblockplus.org/easylist_noadult.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easyprivacy.tpl', 'EasyPrivacy')">EasyPrivacy</a> - <a href="https://easylist-msie.adblockplus.org/easyprivacy.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easyprivacy_nointernational.tpl', 'EasyPrivacy Without International Rules')">EasyPrivacy Without International Rules</a> - <a href="https://easylist-msie.adblockplus.org/easyprivacy_nointernational.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easyprivacy+easylist.tpl', 'EasyList + EasyPrivacy')">EasyList + EasyPrivacy</a> - <a href="https://easylist-msie.adblockplus.org/easyprivacy+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/adwarefilters.tpl', 'Adware Filters')">Adware Filters</a> - <a href="https://easylist-downloads.adblockplus.org/adwarefilters.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/easytest.tpl', 'EasyTest')">EasyTest</a> - <a href="https://easylist-downloads.adblockplus.org/easytest.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/exceptionrules.tpl', 'Acceptable Ads')">Acceptable Ads</a> - <a href="https://easylist-downloads.adblockplus.org/exceptionrules.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/antiadblockfilters.tpl', 'Adblock Warning Removal List')">Adblock Warning Removal List</a> - <a href="https://easylist-msie.adblockplus.org/antiadblockfilters.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/abp-filters-anti-cv.tpl', 'ABP Anti-Circumvention Filter List')">ABP Anti-Circumvention Filter List</a> - <a href="https://easylist-downloads.adblockplus.org/abp-filters-anti-cv.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/fanboy-annoyance.tpl', 'Fanboy\'s Annoyance List')">Fanboy's Annoyance List</a> - <a href="https://easylist-msie.adblockplus.org/fanboy-annoyance.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylist-cookie.tpl', 'EasyList Cookie List')">EasyList Cookie List</a> - <a href="https://easylist-msie.adblockplus.org/easylist-cookie.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/fanboy-social.tpl', 'Fanboy\'s Social Blocking List')">Fanboy's Social Blocking List</a> - <a href="https://easylist-msie.adblockplus.org/fanboy-social.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/fanboy-notifications.tpl', 'Fanboy\'s Notifications Blocking List')">Fanboy's Notifications Blocking List</a> - <a href="https://easylist-msie.adblockplus.org/fanboy-notifications.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/malwaredomains_full.tpl', 'DNS-BH Malware Domains')">DNS-BH Malware Domains</a> - <a href="https://easylist-msie.adblockplus.org/malwaredomains_full.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://pgl.yoyo.org/adservers/serverlist.tpl', 'Peter Lowe\'s List')">Peter Lowe's List</a> - <a href="https://pgl.yoyo.org/adservers/serverlist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareTPL.tpl', 'Dandelion Sprout\'s Anti-Malware List')">Dandelion Sprout's Anti-Malware List</a> - <a href="https://raw.githubusercontent.com/DandelionSprout/adfilt/master/Alternate%20versions%20Anti-Malware%20List/AntiMalwareTPL.tpl">View ruleset</a><br><br>
* Language-specific lists:<br><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/abpindo+easylist.tpl', 'ABPindo + EasyList')">ABPindo + EasyList</a> - <a href="https://easylist-downloads.adblockplus.org/abpindo+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/abpvn+easylist.tpl', 'ABPVN + EasyList')">ABPVN + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/abpvn+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/bulgarian_list+easylist.tpl', 'Bulgarian List + EasyList')">Bulgarian List + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/bulgarian_list+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/indianlist+easylist.tpl', 'IndianList + EasyList')">IndianList + EasyList</a> - <a href="https://easylist-downloads.adblockplus.org/indianlist+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/indianlist.tpl', 'IndianList')">IndianList</a> - <a href="https://easylist-downloads.adblockplus.org/indianlist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/koreanlist+easylist.tpl', 'KoreanList + EasyList')">KoreanList + EasyList</a> - <a href="https://easylist-downloads.adblockplus.org/koreanlist+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/koreanlist.tpl', 'KoreanList')">KoreanList</a> - <a href="https://easylist-downloads.adblockplus.org/koreanlist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylistchina+easylist.tpl', 'EasyList China + EasyList')">EasyList China + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/easylistchina+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylistchina.tpl', 'EasyList China')">EasyList China</a> - <a href="https://easylist-msie.adblockplus.org/easylistchina.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylistczechslovak+easylist.tpl', 'EasyList Czech and Slovak + EasyList')">EasyList Czech and Slovak + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/easylistczechslovak+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylistdutch+easylist.tpl', 'EasyList Dutch + EasyList')">EasyList Dutch + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/easylistdutch+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylistdutch.tpl', 'EasyList Dutch')">EasyList Dutch</a> - <a href="https://easylist-msie.adblockplus.org/easylistdutch.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylistgermany+easylist.tpl', 'EasyList Germany + EasyList')">EasyList Germany + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/easylistgermany+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylistgermany.tpl', 'EasyList Germany')">EasyList Germany</a> - <a href="https://easylist-msie.adblockplus.org/easylistgermany.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylistitaly+easylist.tpl', 'EasyList Italy + EasyList')">EasyList Italy + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/easylistitaly+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylistitaly.tpl', 'EasyList Italy')">EasyList Italy</a> - <a href="https://easylist-msie.adblockplus.org/easylistitaly.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylistlithuania+easylist.tpl', 'EasyList Lithuania + EasyList')">EasyList Lithuania + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/easylistlithuania+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylistspanish+easylist.tpl', 'EasyList Spanish + EasyList')">EasyList Spanish + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/easylistspanish+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/easylistspanish.tpl', 'EasyList Spanish')">EasyList Spanish</a> - <a href="https://easylist-msie.adblockplus.org/easylistspanish.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/easylistpolish+easylist.tpl', 'EasyList Polish + EasyList')">EasyList Polish + EasyList</a> - <a href="https://easylist-downloads.adblockplus.org/easylistpolish+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/easylistpolish.tpl', 'EasyList Polish')">EasyList Polish</a> - <a href="https://easylist-downloads.adblockplus.org/easylistpolish.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/easylistportuguese+easylist.tpl', 'EasyList Portuguese + EasyList')">EasyList Portuguese + EasyList</a> - <a href="https://easylist-downloads.adblockplus.org/easylistportuguese+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/easylistportuguese.tpl', 'EasyList Portuguese')">EasyList Portuguese</a> - <a href="https://easylist-downloads.adblockplus.org/easylistportuguese.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/israellist+easylist.tpl', 'EasyList Hebrew + EasyList')">EasyList Hebrew</a> - <a href="https://easylist-msie.adblockplus.org/israellist+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/latvianlist+easylist.tpl', 'Latvian List + EasyList')">Latvian List + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/latvianlist+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/liste_ar+liste_fr+easylist.tpl', 'Liste AR + Liste FR + EasyList')">Liste AR + Liste FR + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/liste_ar+liste_fr+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/Liste_AR.tpl', 'Liste AR')">Liste AR</a> - <a href="https://easylist-msie.adblockplus.org/Liste_AR.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/liste_fr+easylist.tpl', 'Liste FR + EasyList')">Liste FR + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/liste_fr+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/liste_fr.tpl', 'Liste FR')">Liste FR</a> - <a href="https://easylist-msie.adblockplus.org/liste_fr.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/rolist+easylist.tpl', 'ROList + EasyList')">ROList + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/rolist+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/ruadlist+easylist.tpl', 'RU AdList + EasyList')">RU AdList + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/ruadlist+easylist.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/advblock.tpl', 'RU AdList')">RU AdList</a> - <a href="https://easylist-msie.adblockplus.org/advblock.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/bitblockext.tpl', 'RU AdList: BitBlock + Fanboy's Annoyance List')">RU AdList: BitBlock + Fanboy's Annoyance List</a> - <a href="https://easylist-downloads.adblockplus.org/bitblockext.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/bitblock.tpl', 'RU AdList: Bitblock')">RU AdList: BitBlock</a> - <a href="https://easylist-msie.adblockplus.org/bitblock.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/cntblock.tpl', 'RU AdList: Counters')">RU AdList: Counters</a> - <a href="https://easylist-msie.adblockplus.org/cntblock.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-downloads.adblockplus.org/antinuha.tpl', 'RU AntiPron List')">RU AntiPron List</a> - <a href="https://easylist-downloads.adblockplus.org/antinuha.tpl">View ruleset</a><br>
<a href="javascript:window.external.msAddTrackingProtectionList('https://raw.githubusercontent.com/DandelionSprout/adfilt/master/NorwegianExperimentalList%20alternate%20versions/DandelionSproutsNorskeFiltre.tpl', 'Dandelion Sprouts nordiske filtre')">Dandelion Sprouts nordiske filtre</a> - <a href="https://raw.githubusercontent.com/DandelionSprout/adfilt/master/NorwegianExperimentalList%20alternate%20versions/DandelionSproutsNorskeFiltre.tpl">View ruleset</a><br><br>
-------------------------------------------------------<br><br>
* Outdated international lists (Last updated more than 24 months ago), that have not been succeeded by more recently updated TPL lists<br><br>
(2016) <a href="javascript:window.external.msAddTrackingProtectionList('https://jansal.net/tpl/fanboy/enhancedstats-addon.tpl', 'Fanboy\'s Enhanced Tracking List')">Fanboy's Enhanced Tracking List</a> - <a href="https://jansal.net/tpl/fanboy/enhancedstats-addon.tpl">View ruleset</a><br>
(2016) <a href="javascript:window.external.msAddTrackingProtectionList('https://jansal.net/tpl/malpatrol.tpl', 'Malware Patrol')">Malware Patrol</a> - <a href="https://jansal.net/tpl/malpatrol.tpl">View ruleset</a><br>
(2016) <a href="javascript:window.external.msAddTrackingProtectionList('https://jansal.net/tpl/ricklist.tpl', 'Rickroll Blacklist')">Rickroll Blacklist</a> - <a href="https://jansal.net/tpl/ricklist.tpl">View ruleset</a><br>
(2013) <a href="javascript:window.external.msAddTrackingProtectionList('https://jansal.net/tpl/m.tpl', 'M')">M</a> - <a href="https://jansal.net/tpl/m.tpl">View ruleset</a><br>
(2013) <a href="javascript:window.external.msAddTrackingProtectionList('https://jansal.net/tpl/Adversity/Adversity.tpl', 'Adversity')">Adversity</a> - <a href="https://jansal.net/tpl/Adversity/Adversity.tpl">View ruleset</a><br>
(2013) <a href="javascript:window.external.msAddTrackingProtectionList('https://jansal.net/tpl/Adversity/Antisocial.tpl', 'Adversity - AntiSocial')">Adversity - AntiSocial</a> - <a href="https://jansal.net/tpl/Adversity/Antisocial.tpl">View ruleset</a><br>
(2013) <a href="javascript:window.external.msAddTrackingProtectionList('https://www.microsoft.com/cms/api/am/binary/RWilsQ', 'Stop Google Tracking')">Stop Google Tracking</a> - <a href="https://www.microsoft.com/cms/api/am/binary/RWilsQ">View ruleset</a><br>
(2011) <a href="javascript:window.external.msAddTrackingProtectionList('https://jansal.net/tpl/popup-block.tpl', 'Popup Block List for Adblock Plus Popup Addon')">Popup Block List for Adblock Plus Popup Addon</a> - <a href="https://jansal.net/tpl/popup-block.tpl">View ruleset</a><br>
(2011) <a href="javascript:window.external.msAddTrackingProtectionList('https://jansal.net/tpl/facebook-connect-optout-adblockplusfilter.tpl', 'Facebook Connect Opt-Out')">Facebook Connect Opt-Out</a> - <a href="https://jansal.net/tpl/facebook-connect-optout-adblockplusfilter.tpl">View ruleset</a><br>
(Unknown year) <a href="javascript:window.external.msAddTrackingProtectionList('https://easy-tracking-protection.truste.com/easy.tpl', 'TRUSTe Easy Tracking Protection List')">TRUSTe Easy Tracking Protection List</a> - <a href="https://easy-tracking-protection.truste.com/easy.tpl">View ruleset</a><br><br>
* Outdated language-specific lists (Last updated more than 24 months ago), that have not been succeeded by more recently updated TPL lists<br><br>
(2016) <a href="javascript:window.external.msAddTrackingProtectionList('https://jansal.net/tpl/adblock-persian.tpl', 'Adblock Persian List')">Adblock Persian List</a> - <a href="http://jansal.net/tpl/adblock-persian.tpl">View ruleset</a><br>
(2015) <a href="javascript:window.external.msAddTrackingProtectionList('https://raw.githubusercontent.com/wiltteri/subscriptions/master/wiltteri.tpl', 'Wiltteri')">Wiltteri</a> - <a href="https://easylist-msie.adblockplus.org/wiltteri+easylist.tpl">View ruleset</a><br>
(2015) <a href="javascript:window.external.msAddTrackingProtectionList('https://easylist-msie.adblockplus.org/wiltteri+easylist.tpl', 'Wiltteri + EasyList')">Wiltteri + EasyList</a> - <a href="https://easylist-msie.adblockplus.org/wiltteri+easylist.tpl">View ruleset</a><br>
(2015) <a href="javascript:window.external.msAddTrackingProtectionList('https://raw.githubusercontent.com/wiltteri/subscriptions/master/wiltteri-reborn.tpl', 'Wiltteri Reborn')">Wiltteri Reborn</a> - <a href="https://raw.githubusercontent.com/wiltteri/subscriptions/master/wiltteri-reborn.tpl">View ruleset</a>
</body>
</html>

207
data/Tag.json
View file

@ -1,207 +0,0 @@
[
{
"description": "Blocks cryptomining and/or cryptojacking",
"id": 1,
"name": "crypto"
},
{
"description": "Blocks advertisements",
"id": 2,
"name": "ads"
},
{
"description": "Blocks trackers and other privacy-invasive resources",
"id": 3,
"name": "privacy"
},
{
"description": "Blocks social media scripts, trackers, widgets, comment sections, etc.",
"id": 4,
"name": "social"
},
{
"description": "Blocks adblock detection scripts",
"id": 5,
"name": "anti-adblock"
},
{
"description": "Blocks malicious resources",
"id": 6,
"name": "malware"
},
{
"description": "Blocks phishing and/or scam resources",
"id": 7,
"name": "phishing"
},
{
"description": "Blocks cookie notices primarily in response to the EU Cookie Law and GDPR",
"id": 8,
"name": "cookies"
},
{
"description": "Blocks subjectively annoying resources",
"id": 9,
"name": "annoyances"
},
{
"description": "Unblocks categorical resources",
"id": 10,
"name": "whitelist"
},
{
"description": "Blocks adult, NSFW, pornographic, etc. resources",
"id": 11,
"name": "nsfw"
},
{
"description": "Redirects traffic through proxies to get around firewalls or service shutdowns",
"id": 12,
"name": "proxy"
},
{
"description": "Extends or blocks functionality from search engines",
"id": 13,
"name": "search"
},
{
"description": "Intended for research only",
"id": 14,
"name": "research"
},
{
"description": "Blocks specific topics/things",
"id": 15,
"name": "topical"
},
{
"description": "Removes obstructing or annoying overlays",
"id": 16,
"name": "overlay"
},
{
"description": "Blocks gambling resources",
"id": 17,
"name": "gambling"
},
{
"description": "Removes website-embedded fonts",
"id": 18,
"name": "fonts"
},
{
"description": "Blocks resources from certain companies",
"id": 19,
"name": "anti-corp"
},
{
"description": "Lists that are of special interest to IT admins",
"id": 20,
"name": "admin"
},
{
"description": "Lists that remove news stories of subjectively low quality",
"id": 21,
"name": "clickbait"
},
{
"description": "Blocks religious or superstitious content",
"id": 22,
"name": "religious"
},
{
"description": "Blocks pages from link-shortening services",
"id": 23,
"name": "shorteners"
},
{
"description": "Blocks piracy-focusing sites",
"id": 24,
"name": "piracy"
},
{
"description": "Blocks political content",
"id": 25,
"name": "politics"
},
{
"description": "Blocks software updates",
"id": 26,
"name": "updates"
},
{
"description": "Userstyles in adblocker syntax that change the appearance of sites",
"id": 27,
"name": "userstyle"
},
{
"description": "Blocks requests that ask you to subscribe to newsletters",
"id": 28,
"name": "newsletters"
},
{
"description": "Prevents \"How can I help you?\" prompts from popping up",
"id": 29,
"name": "helpprompt"
},
{
"description": "Blocks certain cultural content",
"id": 30,
"name": "cultural"
},
{
"description": "Removes empty boxes that remain after other things were blocked",
"id": 31,
"name": "empty-box"
},
{
"description": "Blocks prompts to subscribe to push notifications",
"id": 32,
"name": "push-notes"
},
{
"description": "Blocks promotions of websites' own mobile apps",
"id": 33,
"name": "app-download"
},
{
"description": "Lists designed for units that are neither PCs or phones, e.g. TVs, game consoles, smart-sticks, etc.",
"id": 34,
"name": "media-unit"
},
{
"description": "Lists that deal with specific sites or site groups only, without blocking them entirely",
"id": 35,
"name": "site-specific"
},
{
"description": "Lists known to be so big as to begin to cause loading/processing issues on weaker units, e.g. old phones",
"id": 36,
"name": "≥250K entries"
},
{
"description": "Blocks Dynamic DNS (DDNS) address providers and hosts",
"id": 37,
"name": "dynamic-domains"
},
{
"description": "Pre-stable lists where maintainers are extra interested in feedback",
"id": 38,
"name": "beta"
},
{
"description": "Replaces file and page sources to make them load faster",
"id": 39,
"name": "cdn-replacer"
},
{
"description": "Bypasses paywalls on websites",
"id": 40,
"name": "paywall"
},
{
"description": "Cleans up anime viewing websites to make them easier to use",
"id": 41,
"name": "anime"
}
]

44
data/lint.sh
View file

@ -1,44 +0,0 @@
#!/bin/bash
# Sorts json files by FilterLists conventions documented in Wiki.
jq -S ".|=sort_by(.dependentFilterListId, .dependencyFilterListId)" Dependent.json > Dependent.tmp
mv Dependent.tmp Dependent.json
jq -S ".|=sort_by(.id)" FilterList.json > FilterList.tmp
mv FilterList.tmp FilterList.json
jq -S ".|=sort_by(.filterListId, .languageId)" FilterListLanguage.json > FilterListLanguage.tmp
mv FilterListLanguage.tmp FilterListLanguage.json
jq -S ".|=sort_by(.maintainerId, .filterListId)" FilterListMaintainer.json > FilterListMaintainer.tmp
mv FilterListMaintainer.tmp FilterListMaintainer.json
jq -S ".|=sort_by(.filterListId, .tagId)" FilterListTag.json > FilterListTag.tmp
mv FilterListTag.tmp FilterListTag.json
jq -S ".|=sort_by(.upstreamFilterListId, .forkFilterListId)" Fork.json > Fork.tmp
mv Fork.tmp Fork.json
jq -S ".|=sort_by(.id)" Language.json > Language.tmp
mv Language.tmp Language.json
jq -S ".|=sort_by(.id)" License.json > License.tmp
mv License.tmp License.json
jq -S ".|=sort_by(.id)" Maintainer.json > Maintainer.tmp
mv Maintainer.tmp Maintainer.json
jq -S ".|=sort_by(.mergeFilterListId, .upstreamFilterListId)" Merge.json > Merge.tmp
mv Merge.tmp Merge.json
jq -S ".|=sort_by(.id)" Software.json > Software.tmp
mv Software.tmp Software.json
jq -S ".|=sort_by(.syntaxId, .softwareId)" SoftwareSyntax.json > SoftwareSyntax.tmp
mv SoftwareSyntax.tmp SoftwareSyntax.json
jq -S ".|=sort_by(.id)" Syntax.json > Syntax.tmp
mv Syntax.tmp Syntax.json
jq -S ".|=sort_by(.id)" Tag.json > Tag.tmp
mv Tag.tmp Tag.json

20
docker-compose.agent.tests.yml
View file

@ -1,20 +0,0 @@
version: "3.7"
services:
agent:
image: collinbarrett2/filterlists.agent:test-agent
build:
context: .
dockerfile: server/src/FilterLists.Agent/Dockerfile
target: test-agent
networks:
- test-agent
volumes:
- test-agent-results:/app/tests/FilterLists.Agent.Tests/TestResults
networks:
test-agent:
volumes:
test-agent-results:
external: true

39
docker-compose.data.tests.yml
View file

@ -1,39 +0,0 @@
version: "3.7"
services:
api:
image: collinbarrett2/filterlists.api:test-data
build:
context: .
dockerfile: server/src/FilterLists.Api/Dockerfile
target: test-data
networks:
- test-data
volumes:
- test-data-results:/app/tests/FilterLists.Data.Tests/TestResults
depends_on:
- mariadb
mariadb:
# on update, also update ServerVersion in FilterLists.Data.Constants.ServerVersion
image: mariadb:10.5.4
networks:
- test-data
environment:
- MYSQL_ROOT_PASSWORD=filterlists
- MYSQL_DATABASE=filterlists
- MYSQL_USER=filterlists
- MYSQL_PASSWORD=filterlists
command:
[
"mysqld",
"--character-set-server=utf8mb4",
"--collation-server=utf8mb4_unicode_520_ci",
]
networks:
test-data:
volumes:
test-data-results:
external: true

16
docker-compose.override.yml
View file

@ -1,16 +0,0 @@
version: "3.7"
services:
nginx:
ports:
- "80:80"
volumes:
- ./ops/nginx/conf.d.dev:/etc/nginx/conf.d:ro
api:
build:
dockerfile: server/src/FilterLists.Api/debug.Dockerfile
agent:
build:
dockerfile: server/src/FilterLists.Agent/debug.Dockerfile

13
docker-compose.prod.yml
View file

@ -1,13 +0,0 @@
version: "3.7"
services:
nginx:
ports:
- "443:443"
volumes:
- ./ops/nginx/conf.d:/etc/nginx/conf.d:ro
- ~/cert:/etc/nginx/cert:ro
web:
build:
context: . # hack for https://github.com/docker/compose/issues/4047

20
docker-compose.services.tests.yml
View file

@ -1,20 +0,0 @@
version: "3.7"
services:
api:
image: collinbarrett2/filterlists.api:test-services
build:
context: .
dockerfile: server/src/FilterLists.Api/Dockerfile
target: test-services
networks:
- test-services
volumes:
- test-services-results:/app/tests/FilterLists.Services.Tests/TestResults
networks:
test-services:
volumes:
test-services-results:
external: true

19
docker-compose.web.dcproj
View file

@ -1,19 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="15.0" Sdk="Microsoft.Docker.Sdk">
<PropertyGroup Label="Globals">
<ProjectVersion>2.1</ProjectVersion>
<DockerTargetOS>Linux</DockerTargetOS>
<ProjectGuid>26883901-339a-44cf-8894-45f6a3a5ebab</ProjectGuid>
<DockerLaunchAction>LaunchBrowser</DockerLaunchAction>
<DockerServiceUrl>{Scheme}://localhost:{ServicePort}/{Scheme}://{ServiceHost}:{ServicePort}</DockerServiceUrl>
<DockerServiceName>filterlists.api</DockerServiceName>
</PropertyGroup>
<ItemGroup>
<None Include=".dockerignore" />
<None Include=".env" />
<None Include="docker-compose.override.yml">
<DependentUpon>docker-compose.yml</DependentUpon>
</None>
<None Include="docker-compose.yml" />
</ItemGroup>
</Project>

87
docker-compose.yml
View file

@ -1,87 +0,0 @@
version: "3.7"
services:
nginx:
image: nginx:alpine
restart: always
networks:
- reverse-proxy
depends_on:
- web
- api
volumes:
- ./ops/nginx/svr-includes:/etc/nginx/svr-includes:ro
web:
image: collinbarrett2/filterlists.web:latest
restart: always
build:
context: ./web
target: final
networks:
- reverse-proxy
api:
image: collinbarrett2/filterlists.api:latest
restart: always
build:
context: .
dockerfile: server/src/FilterLists.Api/Dockerfile
target: final
networks:
- reverse-proxy
- api
depends_on:
- mariadb
environment:
ConnectionStrings__FilterListsConnection: ${API_CONNECTION_STRING}
mariadb:
# on update, also update ServerVersion in FilterLists.Data.Constants.ServerVersion
image: mariadb:10.5.4
restart: always
networks:
- api
volumes:
- data:/var/lib/mysql
environment:
MYSQL_ROOT_PASSWORD: ${API_MYSQL_ROOT_PASSWORD}
MYSQL_DATABASE: ${API_MYSQL_DATABASE}
MYSQL_USER: ${API_MYSQL_USER}
MYSQL_PASSWORD: ${API_MYSQL_PASSWORD}
command:
[
"mysqld",
"--character-set-server=utf8mb4",
"--collation-server=utf8mb4_unicode_520_ci",
]
agent:
image: collinbarrett2/filterlists.agent:latest
build:
context: .
dockerfile: server/src/FilterLists.Agent/Dockerfile
target: final
volumes:
- archives:/app/archives
environment:
ApplicationInsights__InstrumentationKey: ${AGENT_APPLICATION_INSIGHTS_INSTRUMENTATION_KEY}
Archive__RepositoryDirectory: ${AGENT_ARCHIVE_GIT_REPOSITORY_DIRECTORY}
Archive__SignatureName: ${AGENT_ARCHIVE_GIT_SIGNATURE_NAME}
Archive__SignatureEmail: ${AGENT_ARCHIVE_GIT_SIGNATURE_EMAIL}
Archive__CommitMessageSuffix: ${AGENT_ARCHIVE_GIT_COMMIT_MESSAGE_SUFFIX}
ConnectionString__FilterListsConnection: ${API_CONNECTION_STRING}
FilterListsApi__BaseUrl: ${AGENT_FILTERLISTS_API_BASE_URL}
FilterListsApi__ClientUserAgent: ${AGENT_FILTERLISTS_API_CLIENT_USER_AGENT}
GitHub__ProductHeaderValue: ${AGENT_GITHUB_PRODUCT_HEADER_VALUE}
GitHub__PersonalAccessToken: ${AGENT_GITHUB_PERSONAL_ACCESS_TOKEN}
GitHub__RepositoryOwner: ${AGENT_GITHUB_REPOSITORY_OWNER}
GitHub__Repository: ${AGENT_GITHUB_REPOSITORY}
networks:
reverse-proxy:
api:
volumes:
data:
archives:

BIN
imgs/icon_filterlists.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 2.6 KiB

BIN
imgs/icon_filterlists.psd
View file

Binary file not shown.

BIN
imgs/logo_filterlists.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 4.4 KiB

BIN
imgs/logo_filterlists.psd
View file

Binary file not shown.

BIN
imgs/logo_lg_filterlists.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 7.4 KiB

656
ops/FilterLists.postman_collection.json
View file

@ -1,656 +0,0 @@
{
"info": {
"_postman_id": "918a11ab-a25e-3c3c-dfb2-295e487ee4d6",
"name": "FilterLists",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
},
"item": [
{
"name": "dependents",
"item": [
{
"name": "dependents/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/dependents/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"dependents",
"seed"
]
}
},
"response": []
}
]
},
{
"name": "forks",
"item": [
{
"name": "forks/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/forks/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"forks",
"seed"
]
}
},
"response": []
}
]
},
{
"name": "languages",
"item": [
{
"name": "languages",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/languages",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"languages"
]
}
},
"response": []
},
{
"name": "languages/{id}",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/languages/122",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"languages",
"122"
]
}
},
"response": []
},
{
"name": "languages/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/languages/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"languages",
"seed"
]
}
},
"response": []
}
]
},
{
"name": "licenses",
"item": [
{
"name": "licenses",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/licenses",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"licenses"
]
}
},
"response": []
},
{
"name": "licenses/{id}",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/licenses/1",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"licenses",
"1"
]
}
},
"response": []
},
{
"name": "licenses/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/licenses/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"licenses",
"seed"
]
}
},
"response": []
}
]
},
{
"name": "lists",
"item": [
{
"name": "lists",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/lists",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"lists"
]
}
},
"response": []
},
{
"name": "lists/{id}",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/lists/1",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"lists",
"1"
]
}
},
"response": []
},
{
"name": "lists/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/lists/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"lists",
"seed"
]
}
},
"response": []
}
]
},
{
"name": "listslanguages",
"item": [
{
"name": "listslanguages/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/listslanguages/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"listslanguages",
"seed"
]
}
},
"response": []
}
]
},
{
"name": "listsmaintainers",
"item": [
{
"name": "listsmaintainers/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/listsmaintainers/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"listsmaintainers",
"seed"
]
}
},
"response": []
}
]
},
{
"name": "liststags",
"item": [
{
"name": "liststags/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/liststags/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"liststags",
"seed"
]
}
},
"response": []
}
]
},
{
"name": "maintainers",
"item": [
{
"name": "maintainers",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/maintainers",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"maintainers"
]
}
},
"response": []
},
{
"name": "maintainers/{id}",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/maintainers/2",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"maintainers",
"2"
]
}
},
"response": []
},
{
"name": "maintainers/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/maintainers/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"maintainers",
"seed"
]
}
},
"response": []
}
]
},
{
"name": "merges",
"item": [
{
"name": "merges/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/merges/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"merges",
"seed"
]
}
},
"response": []
}
]
},
{
"name": "rules",
"item": [
{
"name": "rules",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/rules",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"rules"
]
}
},
"response": []
}
]
},
{
"name": "software",
"item": [
{
"name": "software",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/software",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"software"
]
}
},
"response": []
},
{
"name": "software/{id}",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/software/1",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"software",
"1"
]
}
},
"response": []
},
{
"name": "software/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/software/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"software",
"seed"
]
}
},
"response": []
}
]
},
{
"name": "softwaresyntaxes",
"item": [
{
"name": "softwaresyntaxes/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/softwaresyntaxes/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"softwaresyntaxes",
"seed"
]
}
},
"response": []
}
]
},
{
"name": "syntaxes",
"item": [
{
"name": "syntaxes",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/syntaxes",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"syntaxes"
]
}
},
"response": []
},
{
"name": "syntaxes/{id}",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/syntaxes/1",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"syntaxes",
"1"
]
}
},
"response": []
},
{
"name": "syntaxes/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/syntaxes/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"syntaxes",
"seed"
]
}
},
"response": []
}
]
},
{
"name": "tags",
"item": [
{
"name": "tags",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/tags",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"tags"
]
}
},
"response": []
},
{
"name": "tags/{id}",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/tags/1",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"tags",
"1"
]
}
},
"response": []
},
{
"name": "tags/seed",
"request": {
"method": "GET",
"header": [],
"body": {},
"url": {
"raw": "{{baseUrl}}/{{version}}/tags/seed",
"host": [
"{{baseUrl}}"
],
"path": [
"{{version}}",
"tags",
"seed"
]
}
},
"response": []
}
]
}
]
}

33
ops/nginx/azure-pipelines.yaml
View file

@ -1,33 +0,0 @@
trigger:
batch: false
branches:
include: [master]
paths:
include: [ops/nginx/*, docker-compose.yml, docker-compose.prod.yml]
pr: none
pool:
vmImage: ubuntu-latest
steps:
- task: CopyFiles@2
displayName: stage files
inputs:
SourceFolder: $(Build.SourcesDirectory)
Contents: |
ops/nginx/**
docker-compose.yml
docker-compose.prod.yml
TargetFolder: $(Build.ArtifactStagingDirectory)
CleanTargetFolder: true
OverWrite: true
condition: eq(variables['Build.SourceBranch'], 'refs/heads/master')
- task: PublishBuildArtifacts@1
displayName: publish files
inputs:
PathtoPublish: $(Build.ArtifactStagingDirectory)
ArtifactName: drop
publishLocation: Container
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))

26
ops/nginx/conf.d.dev/filterlists.conf
View file

@ -1,26 +0,0 @@
server {
listen 80 default_server;
location ^~ /api {
rewrite ^/api/(.*)$ /$1 break;
proxy_pass http://api:80;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location / {
proxy_pass http://web:80;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}

54
ops/nginx/conf.d/filterlists.conf
View file

@ -1,54 +0,0 @@
server {
listen 443 default_server ssl http2;
server_name filterlists.com;
include svr-includes/*;
ssl_certificate /etc/nginx/cert/filterlists_cert.pem;
ssl_certificate_key /etc/nginx/cert/filterlists_key.pem;
add_header Content-Security-Policy "default-src 'none'; script-src 'self' 'unsafe-inline'; connect-src 'self'; img-src 'self' data:; style-src 'self' 'unsafe-inline'; manifest-src 'self';" always;
location ^~ /api {
rewrite ^/api/(.*)$ /$1 break;
proxy_pass http://api:80;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
location / {
proxy_pass http://web:80;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
# Redirect http to https
server {
listen 80 default_server;
server_name filterlists.com;
return 301 https://filterlists.com$request_uri;
}
# Redirect www to non-www
server {
listen 80;
listen 443 ssl http2;
server_name www.filterlists.com;
ssl_certificate /etc/nginx/cert/filterlists_cert.pem;
ssl_certificate_key /etc/nginx/cert/filterlists_key.pem;
return 301 https://filterlists.com$request_uri;
}

77
ops/nginx/conf.d/static-files.conf
View file

@ -1,77 +0,0 @@
# https://raw.githubusercontent.com/h5bp/server-configs-nginx/master/h5bp/web_performance/cache_expiration.conf
# ----------------------------------------------------------------------
# | Cache expiration |
# ----------------------------------------------------------------------
# Serve resources with far-future expiration date.
#
# (!) If you don't control versioning with filename-based cache busting, you
# should consider lowering the cache times to something like one week.
#
# https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control
# https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Expires
# https://nginx.org/en/docs/http/ngx_http_headers_module.html#expires
map $sent_http_content_type $expires {
default 1M;
# No content
"" off;
# CSS
~*text/css 1y;
# Data interchange
~*application/atom\+xml 1h;
~*application/rdf\+xml 1h;
~*application/rss\+xml 1h;
~*application/json 4h;
~*application/ld\+json 0;
~*application/schema\+json 0;
~*application/geo\+json 0;
~*application/xml 0;
~*text/calendar 0;
~*text/xml 0;
# Favicon (cannot be renamed!) and cursor images
~*image/vnd.microsoft.icon 1w;
~*image/x-icon 1w;
# HTML
~*text/html 0;
# JavaScript
~*application/javascript 1y;
~*application/x-javascript 1y;
~*text/javascript 1y;
# Manifest files
~*application/manifest\+json 1w;
~*application/x-web-app-manifest\+json 0;
~*text/cache-manifest 0;
# Markdown
~*text/markdown 0;
# Media files
~*audio/ 1M;
~*image/ 1M;
~*video/ 1M;
# WebAssembly
~*application/wasm 1y;
# Web fonts
~*font/ 1M;
~*application/vnd.ms-fontobject 1M;
~*application/x-font-ttf 1M;
~*application/x-font-woff 1M;
~*application/font-woff 1M;
~*application/font-woff2 1M;
# Other
~*text/x-cross-domain-policy 1w;
}
expires $expires;

6
ops/nginx/svr-includes/security.conf
View file

@ -1,6 +0,0 @@
server_tokens off;
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "SAMEORIGIN" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Referrer-Policy "no-referrer" always;

18
ops/nginx/svr-includes/ssl.conf
View file

@ -1,18 +0,0 @@
# https://ssl-config.mozilla.org/#server=nginx&config=intermediate&ocsp=false
ssl_session_timeout 1d;
ssl_session_cache shared:MozSSL:10m; # about 40000 sessions
ssl_session_tickets off;
ssl_dhparam /etc/nginx/cert/dhparam.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
ssl_prefer_server_ciphers off;
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
# https://support.cloudflare.com/hc/en-us/articles/204899617-Authenticated-Origin-Pulls
ssl_client_certificate /etc/nginx/cert/cloudflare.crt;
ssl_verify_client on;

2
ops/scripts/cron
View file

@ -1,2 +0,0 @@
# FilterLists Capture Snapshots
*/1 * * * * nice -n 19 /usr/bin/flock -n /tmp/filterlistsagent.lockfile /var/www/filterlists/FilterLists.Agent > /dev/null 2>&1

4
ops/scripts/sql/createDb.sql
View file

@ -1,4 +0,0 @@
CREATE USER 'filterlists'@'localhost' IDENTIFIED BY 'password';
CREATE DATABASE filterlistsdata CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_520_ci;
GRANT ALL PRIVILEGES ON filterlistsdata.* TO 'filterlists'@'localhost';
FLUSH PRIVILEGES;

4
ops/scripts/sql/duplicateRules.sql
View file

@ -1,4 +0,0 @@
SELECT Raw, COUNT(*) 'Count'
FROM rules
GROUP BY Raw COLLATE utf8mb4_bin
HAVING COUNT > 1;

5
ops/scripts/sql/orphanedSnapshotsRules.sql
View file

@ -1,5 +0,0 @@
SELECT SnapshotId, COUNT(*)
FROM snapshots_rules
JOIN snapshots ON snapshots_rules.SnapshotId = snapshots.Id
WHERE snapshots.WasSuccessful = FALSE
GROUP BY snapshots.Id;

7
ops/scripts/sql/resetSnapshotsRules.sql
View file

@ -1,7 +0,0 @@
SET FOREIGN_KEY_CHECKS = 0;
TRUNCATE snapshots_rules;
TRUNCATE snapshots;
TRUNCATE rules;
SET FOREIGN_KEY_CHECKS = 1;
ALTER TABLE snapshots AUTO_INCREMENT = 1;
ALTER TABLE rules AUTO_INCREMENT = 1;

6
ops/scripts/sql/snapshotMetrics.sql
View file

@ -1,6 +0,0 @@
SELECT snapshots.CreatedDateUtc, FilterlistId, HttpStatusCode, CAST(WasSuccessful AS UNSIGNED), WaybackTimestamp, COUNT(snapshots_rules.RuleId) AS 'Count'
FROM snapshots
JOIN snapshots_rules ON snapshots.Id = snapshots_rules.SnapshotId
GROUP BY snapshots.Id
ORDER BY snapshots.CreatedDateUtc DESC
LIMIT 5;

9
ops/scripts/sql/snapshotTimeAnalysis.sql
View file

@ -1,9 +0,0 @@
SELECT
FilterListId,
ModifiedDateUtc - CreatedDateUtc AS ProcessingTimeSecs,
CreatedDateUtc,
ModifiedDateUtc,
WasSuccessful
FROM snapshots
WHERE ModifiedDateUtc - CreatedDateUtc > 30 OR WasSuccessful = 0
ORDER BY ProcessingTimeSecs DESC;

73
server/FilterLists.Api.sln
View file

@ -1,73 +0,0 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 16
VisualStudioVersion = 16.0.29326.143
MinimumVisualStudioVersion = 10.0.40219.1
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "src", "src", "{00CBD1AF-455E-4DF5-AB24-A5C2C18EEB6F}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tests", "tests", "{EA11453B-FB14-4F0E-A19B-B8089498F677}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FilterLists.Data", "src\FilterLists.Data\FilterLists.Data.csproj", "{E604D351-D471-41FD-BA82-BAE12B342B4D}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FilterLists.Data.Migrations", "src\FilterLists.Data.Migrations\FilterLists.Data.Migrations.csproj", "{312A8819-5CAB-4F43-B182-50CAD5D8FEF9}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FilterLists.Services", "src\FilterLists.Services\FilterLists.Services.csproj", "{51186C58-1627-4743-B86D-4B355D8A152B}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FilterLists.Api", "src\FilterLists.Api\FilterLists.Api.csproj", "{1B8105E2-0721-4DC0-8243-02932A05D98B}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FilterLists.Data.Tests", "tests\FilterLists.Data.Tests\FilterLists.Data.Tests.csproj", "{EB65E67D-3C29-4555-8D9B-C90801880F51}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "FilterLists.Services.Tests", "tests\FilterLists.Services.Tests\FilterLists.Services.Tests.csproj", "{0F419F45-EC3F-409B-95A2-F6AEE21E4041}"
EndProject
Project("{E53339B2-1760-4266-BCC7-CA923CBCF16C}") = "docker-compose.web", "..\docker-compose.web.dcproj", "{26883901-339A-44CF-8894-45F6A3A5EBAB}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Release|Any CPU = Release|Any CPU
EndGlobalSection
GlobalSection(ProjectConfigurationPlatforms) = postSolution
{E604D351-D471-41FD-BA82-BAE12B342B4D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{E604D351-D471-41FD-BA82-BAE12B342B4D}.Debug|Any CPU.Build.0 = Debug|Any CPU
{E604D351-D471-41FD-BA82-BAE12B342B4D}.Release|Any CPU.ActiveCfg = Release|Any CPU
{E604D351-D471-41FD-BA82-BAE12B342B4D}.Release|Any CPU.Build.0 = Release|Any CPU
{312A8819-5CAB-4F43-B182-50CAD5D8FEF9}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{312A8819-5CAB-4F43-B182-50CAD5D8FEF9}.Debug|Any CPU.Build.0 = Debug|Any CPU
{312A8819-5CAB-4F43-B182-50CAD5D8FEF9}.Release|Any CPU.ActiveCfg = Release|Any CPU
{312A8819-5CAB-4F43-B182-50CAD5D8FEF9}.Release|Any CPU.Build.0 = Release|Any CPU
{51186C58-1627-4743-B86D-4B355D8A152B}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{51186C58-1627-4743-B86D-4B355D8A152B}.Debug|Any CPU.Build.0 = Debug|Any CPU
{51186C58-1627-4743-B86D-4B355D8A152B}.Release|Any CPU.ActiveCfg = Release|Any CPU
{51186C58-1627-4743-B86D-4B355D8A152B}.Release|Any CPU.Build.0 = Release|Any CPU
{1B8105E2-0721-4DC0-8243-02932A05D98B}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{1B8105E2-0721-4DC0-8243-02932A05D98B}.Debug|Any CPU.Build.0 = Debug|Any CPU
{1B8105E2-0721-4DC0-8243-02932A05D98B}.Release|Any CPU.ActiveCfg = Release|Any CPU
{1B8105E2-0721-4DC0-8243-02932A05D98B}.Release|Any CPU.Build.0 = Release|Any CPU
{EB65E67D-3C29-4555-8D9B-C90801880F51}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{EB65E67D-3C29-4555-8D9B-C90801880F51}.Debug|Any CPU.Build.0 = Debug|Any CPU
{EB65E67D-3C29-4555-8D9B-C90801880F51}.Release|Any CPU.ActiveCfg = Release|Any CPU
{EB65E67D-3C29-4555-8D9B-C90801880F51}.Release|Any CPU.Build.0 = Release|Any CPU
{0F419F45-EC3F-409B-95A2-F6AEE21E4041}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{0F419F45-EC3F-409B-95A2-F6AEE21E4041}.Debug|Any CPU.Build.0 = Debug|Any CPU
{0F419F45-EC3F-409B-95A2-F6AEE21E4041}.Release|Any CPU.ActiveCfg = Release|Any CPU
{0F419F45-EC3F-409B-95A2-F6AEE21E4041}.Release|Any CPU.Build.0 = Release|Any CPU
{26883901-339A-44CF-8894-45F6A3A5EBAB}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{26883901-339A-44CF-8894-45F6A3A5EBAB}.Debug|Any CPU.Build.0 = Debug|Any CPU
{26883901-339A-44CF-8894-45F6A3A5EBAB}.Release|Any CPU.ActiveCfg = Release|Any CPU
{26883901-339A-44CF-8894-45F6A3A5EBAB}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
EndGlobalSection
GlobalSection(NestedProjects) = preSolution
{E604D351-D471-41FD-BA82-BAE12B342B4D} = {00CBD1AF-455E-4DF5-AB24-A5C2C18EEB6F}
{312A8819-5CAB-4F43-B182-50CAD5D8FEF9} = {00CBD1AF-455E-4DF5-AB24-A5C2C18EEB6F}
{51186C58-1627-4743-B86D-4B355D8A152B} = {00CBD1AF-455E-4DF5-AB24-A5C2C18EEB6F}
{1B8105E2-0721-4DC0-8243-02932A05D98B} = {00CBD1AF-455E-4DF5-AB24-A5C2C18EEB6F}
{EB65E67D-3C29-4555-8D9B-C90801880F51} = {EA11453B-FB14-4F0E-A19B-B8089498F677}
{0F419F45-EC3F-409B-95A2-F6AEE21E4041} = {EA11453B-FB14-4F0E-A19B-B8089498F677}
EndGlobalSection
GlobalSection(ExtensibilityGlobals) = postSolution
SolutionGuid = {369BF43A-097E-44B5-A755-FD4D97CBA241}
EndGlobalSection
EndGlobal

118
server/azure-pipelines.agent.yaml
View file

@ -1,118 +0,0 @@
trigger:
batch: false
branches:
include: ["master"]
paths:
include:
[
"server/azure-pipelines.agent.yaml",
"server/src/FilterLists.Agent/*",
"server/tests/FilterLists.Agent.Tests/*",
"server/**/.editorconfig",
".dockerignore",
"docker-compose.yml",
"docker-compose.prod.yml",
"docker-compose.agent.tests.yml",
]
pr:
autoCancel: false
branches:
include: ["master"]
paths:
include:
[
"server/azure-pipelines.agent.yaml",
"server/src/FilterLists.Agent/*",
"server/tests/FilterLists.Agent.Tests/*",
"server/**/.editorconfig",
".dockerignore",
"docker-compose.yml",
"docker-compose.prod.yml",
"docker-compose.agent.tests.yml",
]
pool:
vmImage: "ubuntu-latest"
steps:
- task: Docker@2
displayName: "create volume test-agent-results"
inputs:
command: "volume"
arguments: "create test-agent-results"
- task: DockerCompose@0
displayName: "run agent"
inputs:
dockerComposeFile: "docker-compose.agent.tests.yml"
dockerComposeCommand: "run agent"
- task: Docker@2
displayName: "create container test-agent-results"
inputs:
command: "container"
arguments: "create --name test-agent-results -v test-agent-results:/results hello-world"
condition: "succeededOrFailed()"
- task: Docker@2
displayName: "copy out test-agent results"
inputs:
command: "cp"
arguments: "test-agent-results:/results $(System.DefaultWorkingDirectory)"
condition: "succeededOrFailed()"
- task: PublishTestResults@2
displayName: "publish test results"
inputs:
testResultsFormat: "VSTest"
testResultsFiles: "**/*.trx"
condition: "succeededOrFailed()"
- task: DockerCompose@0
displayName: "build final"
inputs:
dockerComposeFile: "docker-compose.yml"
additionalDockerComposeFiles: "docker-compose.prod.yml"
dockerComposeCommand: "build agent"
condition: "and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))"
- task: Docker@2
displayName: "tag build number"
inputs:
containerRegistry: "collinbarrett2-dockerhub"
repository: "collinbarrett2/filterlists.agent"
command: "tag"
arguments: "collinbarrett2/filterlists.agent:latest collinbarrett2/filterlists.agent:$(Build.BuildId)"
condition: "and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))"
- task: Docker@2
displayName: "push"
inputs:
containerRegistry: "collinbarrett2-dockerhub"
repository: "collinbarrett2/filterlists.agent"
command: "push"
tags: |
latest
$(Build.BuildId)
condition: "and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))"
- task: CopyFiles@2
displayName: "stage docker-compose files"
inputs:
SourceFolder: "$(Build.SourcesDirectory)"
Contents: |
docker-compose.yml
docker-compose.prod.yml
TargetFolder: "$(Build.ArtifactStagingDirectory)"
CleanTargetFolder: true
OverWrite: true
condition: "and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))"
- task: PublishBuildArtifacts@1
displayName: "publish docker-compose files"
inputs:
PathtoPublish: "$(Build.ArtifactStagingDirectory)"
ArtifactName: "drop"
publishLocation: "Container"
condition: "and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))"

172
server/azure-pipelines.api.yaml
View file

@ -1,172 +0,0 @@
trigger:
batch: false
branches:
include: [master]
paths:
include:
[
server/azure-pipelines.api.yaml,
server/src/FilterLists.Api/*,
server/src/FilterLists.Services/*,
server/src/FilterLists.Data/*,
server/src/FilterLists.Data.Migrations/*,
server/tests/FilterLists.Services.Tests/*,
server/tests/FilterLists.Data.Tests/*,
server/**/.editorconfig,
.dockerignore,
docker-compose.yml,
docker-compose.prod.yml,
docker-compose.data.tests.yml,
docker-compose.services.tests.yml,
]
pr:
autoCancel: false
branches:
include: [master]
paths:
include:
[
server/azure-pipelines.api.yaml,
server/src/FilterLists.Api/*,
server/src/FilterLists.Services/*,
server/src/FilterLists.Data/*,
server/src/FilterLists.Data.Migrations/*,
server/tests/FilterLists.Services.Tests/*,
server/tests/FilterLists.Data.Tests/*,
server/**/.editorconfig,
.dockerignore,
docker-compose.yml,
docker-compose.prod.yml,
docker-compose.data.tests.yml,
docker-compose.services.tests.yml,
]
pool:
vmImage: ubuntu-latest
steps:
- task: DockerCompose@0
displayName: build test-data
inputs:
dockerComposeFile: docker-compose.data.tests.yml
dockerComposeCommand: build api
- task: Docker@2
displayName: create volume test-data-results
inputs:
command: volume
arguments: create test-data-results
- task: DockerCompose@0
displayName: up test-data db
inputs:
dockerComposeFile: docker-compose.data.tests.yml
dockerComposeCommand: up -d mariadb
- task: DockerCompose@0
displayName: run test-data
inputs:
dockerComposeFile: docker-compose.data.tests.yml
dockerComposeCommand: run api
- task: DockerCompose@0
displayName: build test-services
inputs:
dockerComposeFile: docker-compose.services.tests.yml
dockerComposeCommand: build api
- task: Docker@2
displayName: create volume test-services-results
inputs:
command: volume
arguments: create test-services-results
- task: DockerCompose@0
displayName: run test-services
inputs:
dockerComposeFile: docker-compose.services.tests.yml
dockerComposeCommand: run api
- task: Docker@2
displayName: create container test-data-results
inputs:
command: container
arguments: create --name test-data-results -v test-data-results:/results hello-world
condition: succeededOrFailed()
- task: Docker@2
displayName: copy out test-data results
inputs:
command: cp
arguments: test-data-results:/results $(System.DefaultWorkingDirectory)
condition: succeededOrFailed()
- task: Docker@2
displayName: create container test-services-results
inputs:
command: container
arguments: create --name test-services-results -v test-services-results:/results hello-world
condition: succeededOrFailed()
- task: Docker@2
displayName: copy out test-services results
inputs:
command: cp
arguments: test-services-results:/results $(System.DefaultWorkingDirectory)
condition: succeededOrFailed()
- task: PublishTestResults@2
displayName: publish test results
inputs:
testResultsFormat: VSTest
testResultsFiles: "**/*.trx"
condition: succeededOrFailed()
- task: DockerCompose@0
displayName: build final
inputs:
dockerComposeFile: docker-compose.yml
additionalDockerComposeFiles: docker-compose.prod.yml
dockerComposeCommand: build api
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
- task: Docker@2
displayName: tag build number
inputs:
containerRegistry: collinbarrett2-dockerhub
repository: collinbarrett2/filterlists.api
command: tag
arguments: collinbarrett2/filterlists.api:latest collinbarrett2/filterlists.api:$(Build.BuildId)
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
- task: Docker@2
displayName: push
inputs:
containerRegistry: collinbarrett2-dockerhub
repository: collinbarrett2/filterlists.api
command: push
tags: |
latest
$(Build.BuildId)
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
- task: CopyFiles@2
displayName: stage docker-compose files
inputs:
SourceFolder: $(Build.SourcesDirectory)
Contents: |
docker-compose.yml
docker-compose.prod.yml
TargetFolder: $(Build.ArtifactStagingDirectory)
CleanTargetFolder: true
OverWrite: true
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))
- task: PublishBuildArtifacts@1
displayName: publish docker-compose files
inputs:
PathtoPublish: $(Build.ArtifactStagingDirectory)
ArtifactName: drop
publishLocation: Container
condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/master'))

205
server/azure-pipelines.migrate.yaml
View file

@ -1,205 +0,0 @@
# Adds an Entity Framework Core migration to a pull request that changes data in data/*.json
# TODO: if data/* or server/src/FilterLists.Data.Migrations/* change in origin/master, re-trigger this Pipeline
trigger: none
pr:
autoCancel: true
branches:
include: [master]
paths:
include: [data/*, server/src/FilterLists.Data.Migrations/Migrations/*]
pool:
vmImage: ubuntu-latest
steps:
- checkout: none
# TODO: switch to using Azure Pipelines Repository Resource to use GitHub service connection rather than PAT
- task: Bash@3
displayName: checkout source branch
inputs:
targetType: inline
script: |
FORKURI=$(curl -X GET "https://api.github.com/repos/$BUILD_REPOSITORY_NAME/pulls/$SYSTEM_PULLREQUEST_PULLREQUESTNUMBER" | jq -r '.head.repo.clone_url')
git clone --single-branch -b "$SYSTEM_PULLREQUEST_SOURCEBRANCH" "$FORKURI" .
- task: Bash@3
displayName: abort if just migrated
name: abortJustMigrated
env:
GITHUBNAME: $(GITHUBNAME)
inputs:
targetType: inline
script: |
LASTAUTHOR=$(git log -n 1 --pretty=format:"%an")
LASTMSG=$(git log -n 1 --pretty=format:"%s")
if [[ $LASTAUTHOR == "$GITHUBNAME" ]] && [[ $LASTMSG == "feat(data): migrate PR #$SYSTEM_PULLREQUEST_PULLREQUESTNUMBER" ]] ; then
echo "Just migrated. Aborting..."
else
echo "##vso[task.setvariable variable=aborted;isOutput=true]false"
fi
- task: ShellScript@2
displayName: lint json
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), succeeded())
inputs:
scriptPath: data/lint.sh
- task: Bash@3
displayName: git config
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), succeeded())
env:
GITHUBNAME: $(GITHUBNAME)
GITHUBEMAIL: $(GITHUBEMAIL)
inputs:
targetType: inline
script: |
git config --global user.name "$GITHUBNAME"
git config --global user.email "$GITHUBEMAIL"
- task: Bash@3
displayName: commit lint results
name: commitLint
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), succeeded())
inputs:
targetType: inline
workingDirectory: data
script: |
DIFF=$(git status -s -- . | wc -l)
echo "$DIFF file(s) required linting"
if (( $DIFF > 0 )) ; then
git add .
git commit -m "refactor(data): lint PR #$SYSTEM_PULLREQUEST_PULLREQUESTNUMBER"
fi
- task: UseDotNet@2
displayName: use latest dotnet sdk
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), succeeded())
inputs:
version: 3.x
- task: CmdLine@2
displayName: install ef dotnet tool
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), succeeded())
inputs:
script: dotnet tool install -g dotnet-ef
- task: Bash@3
displayName: revert existing PR migration
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), succeeded())
env:
GITHUBNAME: $(GITHUBNAME)
inputs:
targetType: inline
workingDirectory: server/src
script: |
MIGLIST=$(dotnet ef migrations list -p FilterLists.Data.Migrations -s FilterLists.Api)
echo "$MIGLIST"
if [[ $MIGLIST == *$SYSTEM_PULLREQUEST_PULLREQUESTNUMBER ]] ; then
echo "A migration already exists for PR #$SYSTEM_PULLREQUEST_PULLREQUESTNUMBER. Reverting..."
REVERTHASH=$(git log -n 1 --author="$GITHUBNAME" --grep="feat(data): migrate PR #$SYSTEM_PULLREQUEST_PULLREQUESTNUMBER" --pretty=format:"%H")
git revert --no-edit "$REVERTHASH"
fi
- task: Bash@3
displayName: sync with upstream
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), succeeded())
inputs:
targetType: inline
script: |
git remote add upstream https://github.com/collinbarrett/FilterLists.git
git fetch upstream
git merge upstream/master
CONFLICTS=$(git ls-files -u | wc -l)
if [ "$CONFLICTS" -gt 0 ] ; then
echo "Conflicts with upstream repository. Aborting..."
exit 1
fi
- task: CmdLine@2
displayName: add migration
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), succeeded())
inputs:
workingDirectory: server/src
script: dotnet ef migrations add $(System.PullRequest.PullRequestNumber) -p FilterLists.Data.Migrations -s FilterLists.Api
- task: Bash@3
displayName: commit or abandon no-op migration
name: commitOrAbandon
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), succeeded())
inputs:
targetType: inline
workingDirectory: server/src/FilterLists.Data.Migrations
script: |
DIFF=$(git status -s -- . | wc -l)
echo "$DIFF file(s) changed"
if (( $DIFF != 3 )) ; then
echo "No-op migration. Effective EF migrations add/change 3 files (.Designer.cs, .cs, and *ModelSnapshot.cs). Abandoning..."
else
git add .
git commit -m "feat(data): migrate PR #$SYSTEM_PULLREQUEST_PULLREQUESTNUMBER"
echo "##vso[task.setvariable variable=committed;isOutput=true]true"
fi
- task: DockerCompose@0
displayName: build test-data
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), eq(variables['commitOrAbandon.committed'], 'true'), succeeded())
inputs:
dockerComposeFile: docker-compose.data.tests.yml
dockerComposeCommand: build api
- task: Docker@2
displayName: create volume test-data-results
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), eq(variables['commitOrAbandon.committed'], 'true'), succeeded())
inputs:
command: volume
arguments: create test-data-results
- task: DockerCompose@0
displayName: up test-data db
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), eq(variables['commitOrAbandon.committed'], 'true'), succeeded())
inputs:
dockerComposeFile: docker-compose.data.tests.yml
dockerComposeCommand: up -d mariadb
- task: DockerCompose@0
displayName: run test-data
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), eq(variables['commitOrAbandon.committed'], 'true'), succeeded())
inputs:
dockerComposeFile: docker-compose.data.tests.yml
dockerComposeCommand: run api
- task: Docker@2
displayName: create container test-data-results
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), eq(variables['commitOrAbandon.committed'], 'true'), succeededOrFailed())
inputs:
command: container
arguments: create --name test-data-results -v test-data-results:/results hello-world
- task: Docker@2
displayName: copy out test-data results
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), eq(variables['commitOrAbandon.committed'], 'true'), succeededOrFailed())
inputs:
command: cp
arguments: test-data-results:/results $(System.DefaultWorkingDirectory)
- task: PublishTestResults@2
displayName: publish test results
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), eq(variables['commitOrAbandon.committed'], 'true'), succeededOrFailed())
inputs:
testResultsFormat: VSTest
testResultsFiles: "**/*.trx"
- task: Bash@3
displayName: push
condition: and(eq(variables['abortJustMigrated.aborted'], 'false'), succeeded())
env:
GITHUBPAT: $(GITHUBPAT)
inputs:
targetType: inline
script: |
git config --global credential.helper store
echo "https://$GITHUBPAT:x-oauth-basic@github.com" >> ~/.git-credentials
git push origin "$SYSTEM_PULLREQUEST_SOURCEBRANCH"

7
server/src/FilterLists.Agent/AppSettings/ApplicationInsightsSettings.cs
View file

@ -1,7 +0,0 @@
namespace FilterLists.Agent.AppSettings
{
public class ApplicationInsightsSettings
{
public string InstrumentationKey { get; set; }
}
}

10
server/src/FilterLists.Agent/AppSettings/ArchiveSettings.cs
View file

@ -1,10 +0,0 @@
namespace FilterLists.Agent.AppSettings
{
public class ArchiveSettings
{
public string CommitMessageSuffix { get; set; }
public string RepositoryDirectory { get; set; }
public string SignatureName { get; set; }
public string SignatureEmail { get; set; }
}
}

7
server/src/FilterLists.Agent/AppSettings/ConnectionStringSettings.cs
View file

@ -1,7 +0,0 @@
namespace FilterLists.Agent.AppSettings
{
public class ConnectionStringSettings
{
public string FilterListsConnection { get; set; }
}
}

10
server/src/FilterLists.Agent/AppSettings/FilterListsApiSettings.cs
View file

@ -1,10 +0,0 @@
using System;
namespace FilterLists.Agent.AppSettings
{
public class FilterListsApiSettings
{
public Uri BaseUrl { get; set; }
public string ClientUserAgent { get; set; }
}
}

10
server/src/FilterLists.Agent/AppSettings/GitHubSettings.cs
View file

@ -1,10 +0,0 @@
namespace FilterLists.Agent.AppSettings
{
public class GitHubSettings
{
public string ProductHeaderValue { get; set; }
public string PersonalAccessToken { get; set; }
public string RepositoryOwner { get; set; }
public string Repository { get; set; }
}
}

24
server/src/FilterLists.Agent/CommandLineOptions.cs
View file

@ -1,24 +0,0 @@
using CommandLine;
namespace FilterLists.Agent
{
public class CommandLineOptions
{
public CommandLineOptions(bool archiveLists, bool validateUrls)
{
ArchiveLists = archiveLists;
ValidateUrls = validateUrls;
}
[Option('a', "archive", Required = false, HelpText = "Archive copies of all lists in a git repository.")]
public bool ArchiveLists { get; }
[Option('v', "validate", Required = false, HelpText = "Validate all URLs in the FilterLists database.")]
public bool ValidateUrls { get; }
public bool Any()
{
return ArchiveLists || ValidateUrls;
}
}
}

15
server/src/FilterLists.Agent/Core/GitHub/IIssuesRepository.cs
View file

@ -1,15 +0,0 @@
using System.Collections.Generic;
using System.Threading.Tasks;
using Octokit;
namespace FilterLists.Agent.Core.GitHub
{
public interface IIssuesRepository
{
Task<IEnumerable<Issue>> GetAllIssuesAsync(RepositoryIssueRequest repositoryIssueRequest);
Task<Issue> CreateIssueAsync(NewIssue newIssue);
Task<Issue> UpdateIssueAsync(int issueNumber, IssueUpdate issueUpdate);
}
}

12
server/src/FilterLists.Agent/Core/Lists/IListRepository.cs
View file

@ -1,12 +0,0 @@
using System;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
namespace FilterLists.Agent.Core.Lists
{
public interface IListRepository
{
Task<Stream> GetAsStreamAsync(Uri url, CancellationToken cancellationToken);
}
}

11
server/src/FilterLists.Agent/Core/Lists/IListViewUrlRepository.cs
View file

@ -1,11 +0,0 @@
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
namespace FilterLists.Agent.Core.Lists
{
public interface IListViewUrlRepository
{
Task<IEnumerable<ListViewUrl>> GetAllAsync(CancellationToken cancellationToken);
}
}

10
server/src/FilterLists.Agent/Core/Lists/ListViewUrl.cs
View file

@ -1,10 +0,0 @@
using System;
namespace FilterLists.Agent.Core.Lists
{
public class ListViewUrl
{
public int Id { get; private set; }
public Uri ViewUrl { get; private set; }
}
}

11
server/src/FilterLists.Agent/Core/Urls/Entity.cs
View file

@ -1,11 +0,0 @@
namespace FilterLists.Agent.Core.Urls
{
public enum Entity
{
License,
FilterList,
Maintainer,
Software,
Syntax
}
}

45
server/src/FilterLists.Agent/Core/Urls/EntityUrl.cs
View file

@ -1,45 +0,0 @@
using System;
using System.Collections.Generic;
using System.Linq;
namespace FilterLists.Agent.Core.Urls
{
public class EntityUrl
{
public EntityUrl(Entity entity, int id, Uri url)
{
Entity = entity;
Id = id;
Url = url;
}
public Entity Entity { get; }
public int Id { get; }
public Uri Url { get; }
public List<string> ValidationMessages { get; } = new List<string>();
public bool IsValid()
{
return !ValidationMessages.Any();
}
public void SetBroken()
{
ValidationMessages.Add("This URL might be broken and should maybe be fixed, replaced, or removed.");
}
public void SetSupportsHttps()
{
ValidationMessages.Add(
"This URL uses the HTTP protocol but appears to be able to support and should probably be changed to HTTPS.");
}
public void SetRedirectsTo(Uri uri)
{
ValidationMessages.Add($"This URL appears to redirect and should maybe be changed to \"{uri}\".");
}
}
}

11
server/src/FilterLists.Agent/Core/Urls/IEntityUrlRepository.cs
View file

@ -1,11 +0,0 @@
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
namespace FilterLists.Agent.Core.Urls
{
public interface IEntityUrlRepository
{
Task<IEnumerable<EntityUrl>> GetAllAsync(CancellationToken cancellationToken);
}
}

10
server/src/FilterLists.Agent/Core/Urls/IEntityUrlValidator.cs
View file

@ -1,10 +0,0 @@
using System.Threading;
using System.Threading.Tasks;
namespace FilterLists.Agent.Core.Urls
{
public interface IEntityUrlValidator
{
Task<EntityUrl> ValidateAsync(EntityUrl entityUrl, CancellationToken cancellationToken);
}
}

40
server/src/FilterLists.Agent/Dockerfile
View file

@ -1,40 +0,0 @@
# Context: .
# Command: docker build -f server/src/FilterLists.Agent/Dockerfile .
# init base
FROM mcr.microsoft.com/dotnet/core/runtime-deps:3.1.7-alpine as base
ENTRYPOINT ["./FilterLists.Agent"]
WORKDIR /app
# init build
FROM mcr.microsoft.com/dotnet/core/sdk:3.1-alpine AS build
# restore
WORKDIR /app/src/FilterLists.Agent
COPY server/src/FilterLists.Agent/FilterLists.Agent.csproj .
RUN dotnet restore
# build
COPY server/src/FilterLists.Agent/. .
RUN dotnet build -c Release -r linux-musl-x64 --no-restore
# init Agent.Tests
FROM build AS test-agent
ENTRYPOINT ["dotnet", "test", "--logger:trx"]
# restore Agent.Tests
WORKDIR /app/tests/FilterLists.Agent.Tests
COPY server/tests/FilterLists.Agent.Tests/FilterLists.Agent.Tests.csproj .
RUN dotnet restore
# build Agent.Tests
COPY server/tests/FilterLists.Agent.Tests/. .
RUN dotnet build -c Release --no-restore
# publish
FROM build AS publish
RUN dotnet publish -c Release -r linux-musl-x64 -o out --no-restore --no-build
# final
FROM base as final
COPY --from=publish /app/src/FilterLists.Agent/out .

20
server/src/FilterLists.Agent/Extensions/ConfigurationExtensions.cs
View file

@ -1,20 +0,0 @@
using System;
using Microsoft.Extensions.Configuration;
namespace FilterLists.Agent.Extensions
{
public static class ConfigurationExtensions
{
public static TSettings GetSettings<TSettings>(this IConfiguration configuration) where TSettings : new()
{
var settings = new TSettings();
configuration.GetSection<TSettings>().Bind(settings);
return settings;
}
public static IConfigurationSection GetSection<TSettings>(this IConfiguration configuration)
{
return configuration.GetSection(typeof(TSettings).Name.Replace("Settings", "", StringComparison.Ordinal));
}
}
}

13
server/src/FilterLists.Agent/Extensions/UriExtensions.cs
View file

@ -1,13 +0,0 @@
using System;
using System.IO;
namespace FilterLists.Agent.Extensions
{
public static class UriExtensions
{
public static string GetExtension(this Uri uri)
{
return Path.GetExtension(uri.AbsolutePath);
}
}
}

33
server/src/FilterLists.Agent/Features/Lists/ArchiveLists.cs
View file

@ -1,33 +0,0 @@
using System.Threading;
using System.Threading.Tasks;
using FilterLists.Agent.Core.Lists;
using MediatR;
namespace FilterLists.Agent.Features.Lists
{
public static class ArchiveLists
{
public class Command : IRequest
{
}
public class Handler : AsyncRequestHandler<Command>
{
private readonly IMediator _mediator;
private readonly IListViewUrlRepository _repo;
public Handler(IMediator mediator, IListViewUrlRepository listViewUrlRepository)
{
_mediator = mediator;
_repo = listViewUrlRepository;
}
protected override async Task Handle(Command request, CancellationToken cancellationToken)
{
var lists = await _repo.GetAllAsync(cancellationToken);
await _mediator.Send(new DownloadLists.Command(lists), cancellationToken);
await _mediator.Send(new CommitLists.Command(), cancellationToken);
}
}
}
}

36
server/src/FilterLists.Agent/Features/Lists/CommitLists.cs
View file

@ -1,36 +0,0 @@
using System;
using FilterLists.Agent.AppSettings;
using LibGit2Sharp;
using MediatR;
using Microsoft.Extensions.Options;
namespace FilterLists.Agent.Features.Lists
{
public static class CommitLists
{
public class Command : IRequest
{
}
public class Handler : RequestHandler<Command>
{
private readonly ArchiveSettings _archiveSettings;
private readonly IRepository _repository;
public Handler(IOptions<ArchiveSettings> archiveOptions, IRepository repository)
{
_archiveSettings = archiveOptions.Value;
_repository = repository;
}
protected override void Handle(Command request)
{
Commands.Stage(_repository, "*");
var utcNow = DateTime.UtcNow;
var signature = new Signature(_archiveSettings.SignatureName, _archiveSettings.SignatureEmail, utcNow);
_repository.Commit(utcNow.ToShortDateString() + _archiveSettings.CommitMessageSuffix, signature,
signature);
}
}
}
}

112
server/src/FilterLists.Agent/Features/Lists/DownloadLists.cs
View file

@ -1,112 +0,0 @@
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;
using FilterLists.Agent.Core.Lists;
using FilterLists.Agent.Extensions;
using MediatR;
using Microsoft.Extensions.Logging;
namespace FilterLists.Agent.Features.Lists
{
public static class DownloadLists
{
public class Command : IRequest
{
public Command(IEnumerable<ListViewUrl> listInfo)
{
ListInfo = listInfo;
}
public IEnumerable<ListViewUrl> ListInfo { get; }
}
public class Handler : AsyncRequestHandler<Command>
{
private const int MaxDegreeOfParallelism = 5;
private static readonly Dictionary<string, Func<ListViewUrl, IRequest>> CommandsByExtension
= new Dictionary<string, Func<ListViewUrl, IRequest>>
{
{"", l => new DownloadRawText.Command(l)},
{".7z", l => new DownloadSevenZip.Command(l)},
{".acl", l => new DownloadRawText.Command(l)},
{".action", l => new DownloadRawText.Command(l)},
{".all", l => new DownloadRawText.Command(l)},
{".aspx", l => new DownloadRawText.Command(l)},
{".bat", l => new DownloadRawText.Command(l)},
{".blacklist", l => new DownloadRawText.Command(l)},
{".conf", l => new DownloadRawText.Command(l)},
{".csv", l => new DownloadRawText.Command(l)},
{".dat", l => new DownloadRawText.Command(l)},
{".deny", l => new DownloadRawText.Command(l)},
{".host", l => new DownloadRawText.Command(l)},
{".hosts", l => new DownloadRawText.Command(l)},
{".ips", l => new DownloadRawText.Command(l)},
{".ipset", l => new DownloadRawText.Command(l)},
{".json", l => new DownloadRawText.Command(l)},
{".list", l => new DownloadRawText.Command(l)},
{".lsrules", l => new DownloadRawText.Command(l)},
{".netset", l => new DownloadRawText.Command(l)},
{".p2p", l => new DownloadRawText.Command(l)},
{".php", l => new DownloadRawText.Command(l)},
{".tpl", l => new DownloadRawText.Command(l)},
{".txt", l => new DownloadRawText.Command(l)},
{".zip", l => new DownloadZip.Command(l)}
};
private readonly ILogger<Handler> _logger;
private readonly IMediator _mediator;
public Handler(ILogger<Handler> logger, IMediator mediator)
{
_logger = logger;
_mediator = mediator;
}
protected override async Task Handle(Command request, CancellationToken cancellationToken)
{
var downloader = BuildDownloader(cancellationToken);
var orderedListInfo = request.ListInfo;
foreach (var listInfo in orderedListInfo)
await downloader.SendAsync(listInfo, cancellationToken);
downloader.Complete();
await downloader.Completion;
}
private ActionBlock<ListViewUrl> BuildDownloader(CancellationToken cancellationToken)
{
return new ActionBlock<ListViewUrl>(
async l =>
{
var errorMessage = $"Error downloading list {l.Id} from {l.ViewUrl}.";
var extension = l.ViewUrl.GetExtension();
if (CommandsByExtension.ContainsKey(extension))
{
var command = CommandsByExtension[extension].Invoke(l);
try
{
await _mediator.Send(command, cancellationToken);
}
catch (HttpRequestException ex)
{
_logger.LogError(ex, errorMessage);
}
catch (TaskCanceledException ex)
{
_logger.LogError(ex, errorMessage);
}
}
else
{
_logger.LogError($"The file extension of list {l.Id} from {l.ViewUrl} is not supported.");
}
},
new ExecutionDataflowBlockOptions {MaxDegreeOfParallelism = MaxDegreeOfParallelism}
);
}
}
}
}

51
server/src/FilterLists.Agent/Features/Lists/DownloadRawText.cs
View file

@ -1,51 +0,0 @@
using System.IO;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using FilterLists.Agent.Core.Lists;
using FilterLists.Agent.Extensions;
using MediatR;
namespace FilterLists.Agent.Features.Lists
{
public static class DownloadRawText
{
public class Command : IRequest
{
public Command(ListViewUrl listViewUrl)
{
ListViewUrl = listViewUrl;
}
public ListViewUrl ListViewUrl { get; }
}
public class Handler : AsyncRequestHandler<Command>
{
private const string RepoDirectory = "archives";
private readonly string[] _extensionsToRewrite = {"", ".aspx", ".p2p", ".php"};
private readonly IListRepository _repo;
public Handler(IListRepository listRepository)
{
_repo = listRepository;
}
protected override async Task Handle(Command request, CancellationToken cancellationToken)
{
var destinationPath = BuildDestinationPath(request);
using var input = await _repo.GetAsStreamAsync(request.ListViewUrl.ViewUrl, cancellationToken);
using var output = File.OpenWrite(destinationPath);
await input.CopyToAsync(output, cancellationToken);
}
private string BuildDestinationPath(Command request)
{
var sourceExtension = request.ListViewUrl.ViewUrl.GetExtension();
var destinationExtension = _extensionsToRewrite.Contains(sourceExtension) ? ".txt" : sourceExtension;
var destinationPath = Path.Combine(RepoDirectory, $"{request.ListViewUrl.Id}{destinationExtension}");
return destinationPath;
}
}
}
}

46
server/src/FilterLists.Agent/Features/Lists/DownloadSevenZip.cs
View file

@ -1,46 +0,0 @@
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using FilterLists.Agent.Core.Lists;
using MediatR;
using SharpCompress.Archives;
using SharpCompress.Archives.SevenZip;
using SharpCompress.Common;
namespace FilterLists.Agent.Features.Lists
{
public static class DownloadSevenZip
{
public class Command : IRequest
{
public Command(ListViewUrl listViewUrl)
{
ListViewUrl = listViewUrl;
}
public ListViewUrl ListViewUrl { get; }
}
public class Handler : AsyncRequestHandler<Command>
{
private const string RepoDirectory = "archives";
private readonly IListRepository _repo;
public Handler(IListRepository listRepository)
{
_repo = listRepository;
}
protected override async Task Handle(Command request, CancellationToken cancellationToken)
{
var destinationDirectoryPath = Path.Combine(RepoDirectory, $"{request.ListViewUrl.Id}");
using var input = await _repo.GetAsStreamAsync(request.ListViewUrl.ViewUrl, cancellationToken);
using var archive = SevenZipArchive.Open(input);
foreach (var entry in archive.Entries)
if (!entry.IsDirectory)
entry.WriteToDirectory(destinationDirectoryPath,
new ExtractionOptions {ExtractFullPath = true, Overwrite = true});
}
}
}
}

45
server/src/FilterLists.Agent/Features/Lists/DownloadZip.cs
View file

@ -1,45 +0,0 @@
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using FilterLists.Agent.Core.Lists;
using MediatR;
using SharpCompress.Common;
using SharpCompress.Readers;
namespace FilterLists.Agent.Features.Lists
{
public static class DownloadZip
{
public class Command : IRequest
{
public Command(ListViewUrl listViewUrl)
{
ListViewUrl = listViewUrl;
}
public ListViewUrl ListViewUrl { get; }
}
public class Handler : AsyncRequestHandler<Command>
{
private const string RepoDirectory = "archives";
private readonly IListRepository _repo;
public Handler(IListRepository listRepository)
{
_repo = listRepository;
}
protected override async Task Handle(Command request, CancellationToken cancellationToken)
{
var destinationDirectoryPath = Path.Combine(RepoDirectory, $"{request.ListViewUrl.Id}");
using var input = await _repo.GetAsStreamAsync(request.ListViewUrl.ViewUrl, cancellationToken);
using var reader = ReaderFactory.Open(input);
while (reader.MoveToNextEntry())
if (!reader.Entry.IsDirectory)
reader.WriteEntryToDirectory(destinationDirectoryPath,
new ExtractionOptions {ExtractFullPath = true, Overwrite = true});
}
}
}
}

54
server/src/FilterLists.Agent/Features/Urls/BuildGitHubIssueBody.cs
View file

@ -1,54 +0,0 @@
using System.Collections.Generic;
using System.Linq;
using System.Text;
using FilterLists.Agent.Core.Urls;
using MediatR;
namespace FilterLists.Agent.Features.Urls
{
public static class BuildGitHubIssueBody
{
public class Command : IRequest<string>
{
public Command(IEnumerable<EntityUrl> invalidEntityUrls)
{
InvalidEntityUrls = invalidEntityUrls;
}
public IEnumerable<EntityUrl> InvalidEntityUrls { get; }
}
public class Handler : RequestHandler<Command, string>
{
private const string IssueHeader =
"<p>This issue is auto-generated by the <a href=\"https://github.com/collinbarrett/FilterLists/tree/master/src/FilterLists.Agent\">FilterLists.Agent</a>. It is updated at about 9am UTC daily.</p><p>We rely on the help of the community to ensure that the FilterLists site data remains up-to-date. The URLs listed below have been automatically flagged and may <a href=\"https://github.com/collinbarrett/FilterLists/tree/master/data\">need to be updated</a>. Please consider submitting a PR against this issue updating some or all of the URLs accordingly.</p><p>Thanks for your contributions!</p>";
protected override string Handle(Command request)
{
var body = new StringBuilder();
body.Append(IssueHeader);
var entityInvalidUrlGroups = request.InvalidEntityUrls.GroupBy(i => i.Entity);
foreach (var entityInvalidUrls in entityInvalidUrlGroups)
{
body.Append(
$"<h1><a href=\"https://github.com/collinbarrett/FilterLists/blob/master/data/{entityInvalidUrls.Key}.json\">{entityInvalidUrls.Key}.json</a></h1>");
body.Append("<ul>");
foreach (var invalidUrl in entityInvalidUrls)
{
body.Append(
$"<li><a href=\"{invalidUrl.Url.OriginalString}\">{invalidUrl.Url.OriginalString}</a>");
body.Append("<ul>");
foreach (var message in invalidUrl.ValidationMessages)
body.Append($"<li>{message}</li>");
body.Append("</ul>");
body.Append("</li>");
}
body.Append("</ul>");
}
return body.ToString();
}
}
}
}

50
server/src/FilterLists.Agent/Features/Urls/CloseUrlValidationIssue.cs
View file

@ -1,50 +0,0 @@
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using FilterLists.Agent.Core.GitHub;
using MediatR;
using Octokit;
namespace FilterLists.Agent.Features.Urls
{
public static class CloseUrlValidationIssue
{
public class Command : IRequest
{
}
public class Handler : AsyncRequestHandler<Command>
{
private const string AgentBotLabel = "agent bot";
private readonly IIssuesRepository _repo;
public Handler(IIssuesRepository issuesRepository)
{
_repo = issuesRepository;
}
protected override async Task Handle(Command request, CancellationToken cancellationToken)
{
var issue = await GetOpenIssueOrNull();
if (issue is null)
return;
var updateIssue = issue.ToUpdate();
updateIssue.State = ItemState.Closed;
updateIssue.Body = issue.Body +
"<h1>Update:</h1>This issue is being closed since all URLs seem to now be valid.";
await _repo.UpdateIssueAsync(issue.Number, updateIssue);
}
private async Task<Issue> GetOpenIssueOrNull()
{
var issueRequest = new RepositoryIssueRequest
{
Filter = IssueFilter.All,
State = ItemStateFilter.Open,
Labels = {AgentBotLabel}
};
return (await _repo.GetAllIssuesAsync(issueRequest)).FirstOrDefault();
}
}
}
}

75
server/src/FilterLists.Agent/Features/Urls/CreateOrUpdateUrlValidationIssue.cs
View file

@ -1,75 +0,0 @@
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using FilterLists.Agent.Core.GitHub;
using FilterLists.Agent.Core.Urls;
using MediatR;
using Octokit;
namespace FilterLists.Agent.Features.Urls
{
public static class CreateOrUpdateUrlValidationIssue
{
public class Command : IRequest
{
public Command(IEnumerable<EntityUrl> invalidEntityUrls)
{
InvalidEntityUrls = invalidEntityUrls;
}
public IEnumerable<EntityUrl> InvalidEntityUrls { get; }
}
public class Handler : AsyncRequestHandler<Command>
{
private const string AgentBotLabel = "agent bot";
private const string HelpWantedLabel = "help wanted";
private const string DataLabel = "data";
private const string IssueTitle = "BOT: url validation errors";
private readonly IMediator _mediator;
private readonly IIssuesRepository _repo;
public Handler(IIssuesRepository issuesRepository, IMediator mediator)
{
_repo = issuesRepository;
_mediator = mediator;
}
protected override async Task Handle(Command request, CancellationToken cancellationToken)
{
var issue = await GetOpenIssueOrNull() ?? await CreateBaseIssue();
await UpdateIssue(issue, request.InvalidEntityUrls, cancellationToken);
}
private async Task<Issue> GetOpenIssueOrNull()
{
var issueRequest = new RepositoryIssueRequest
{
Filter = IssueFilter.All,
State = ItemStateFilter.Open,
Labels = {AgentBotLabel}
};
return (await _repo.GetAllIssuesAsync(issueRequest)).FirstOrDefault();
}
private async Task<Issue> CreateBaseIssue()
{
var newIssue = new NewIssue(IssueTitle);
return await _repo.CreateIssueAsync(newIssue);
}
private async Task UpdateIssue(Issue issue, IEnumerable<EntityUrl> invalidEntityUrls,
CancellationToken cancellationToken)
{
var updateIssue = issue.ToUpdate();
updateIssue.AddLabel(HelpWantedLabel);
updateIssue.AddLabel(AgentBotLabel);
updateIssue.AddLabel(DataLabel);
updateIssue.Body = await _mediator.Send(new BuildGitHubIssueBody.Command(invalidEntityUrls),
cancellationToken);
await _repo.UpdateIssueAsync(issue.Number, updateIssue);
}
}
}
}

40
server/src/FilterLists.Agent/Features/Urls/ValidateAllUrls.cs
View file

@ -1,40 +0,0 @@
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using FilterLists.Agent.Core.Urls;
using MediatR;
namespace FilterLists.Agent.Features.Urls
{
public static class ValidateAllUrls
{
public class Command : IRequest
{
}
public class Handler : AsyncRequestHandler<Command>
{
private readonly IMediator _mediator;
private readonly IEntityUrlRepository _repo;
public Handler(IMediator mediator, IEntityUrlRepository entityUrlRepository)
{
_mediator = mediator;
_repo = entityUrlRepository;
}
protected override async Task Handle(Command request, CancellationToken cancellationToken)
{
var entityUrls = await _repo.GetAllAsync(cancellationToken);
var validatedEntityUrls = await _mediator.Send(new ValidateUrls.Command(entityUrls), cancellationToken);
var invalidEntityUrls = validatedEntityUrls.Where(e => !e.IsValid()).ToList();
if (invalidEntityUrls.Any())
await _mediator.Send(new CreateOrUpdateUrlValidationIssue.Command(invalidEntityUrls),
cancellationToken);
else
await _mediator.Send(new CloseUrlValidationIssue.Command(), cancellationToken);
}
}
}
}

55
server/src/FilterLists.Agent/Features/Urls/ValidateUrls.cs
View file

@ -1,55 +0,0 @@
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks.Dataflow;
using FilterLists.Agent.Core.Urls;
using MediatR;
namespace FilterLists.Agent.Features.Urls
{
public static class ValidateUrls
{
public class Command : IRequest<IEnumerable<EntityUrl>>
{
public Command(IEnumerable<EntityUrl> entityUrls)
{
EntityUrls = entityUrls;
}
public IEnumerable<EntityUrl> EntityUrls { get; }
}
public class Handler : IRequestHandler<Command, IEnumerable<EntityUrl>>
{
private const int MaxDegreeOfParallelism = 5;
private readonly IEntityUrlValidator _entityUrlValidator;
public Handler(IEntityUrlValidator entityUrlValidator)
{
_entityUrlValidator = entityUrlValidator;
}
public async Task<IEnumerable<EntityUrl>> Handle(Command request, CancellationToken cancellationToken)
{
var validator = BuildValidator(cancellationToken);
foreach (var entityUrl in request.EntityUrls)
await validator.SendAsync(entityUrl, cancellationToken);
validator.Complete();
var validatedEntityUrls = new List<EntityUrl>();
while (await validator.OutputAvailableAsync(cancellationToken))
validatedEntityUrls.Add(await validator.ReceiveAsync(cancellationToken));
await validator.Completion;
return validatedEntityUrls;
}
private TransformBlock<EntityUrl, EntityUrl> BuildValidator(CancellationToken cancellationToken)
{
return new TransformBlock<EntityUrl, EntityUrl>(
e => _entityUrlValidator.ValidateAsync(e, cancellationToken),
new ExecutionDataflowBlockOptions {MaxDegreeOfParallelism = MaxDegreeOfParallelism}
);
}
}
}
}

58
server/src/FilterLists.Agent/FilterLists.Agent.csproj
View file

@ -1,58 +0,0 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>netcoreapp3.1</TargetFramework>
<Nullable>enable</Nullable>
<PublishTrimmed>true</PublishTrimmed>
<PublishReadyToRun>true</PublishReadyToRun>
<OutputType>Exe</OutputType>
<RuntimeIdentifiers>win10-x64;linux-musl-x64</RuntimeIdentifiers>
<DockerDefaultTargetOS>Linux</DockerDefaultTargetOS>
<DockerComposeProjectPath>..\..\docker-compose.dcproj</DockerComposeProjectPath>
</PropertyGroup>
<PropertyGroup>
<Authors>Collin M. Barrett</Authors>
<Company>Collin M. Barrett</Company>
<Product>FilterLists</Product>
<Description>The independent, comprehensive directory of filter and host lists for advertisements, trackers, malware, and annoyances.</Description>
<Copyright>Copyright (c) 2020 Collin M. Barrett</Copyright>
<RepositoryUrl>https://github.com/collinbarrett/FilterLists</RepositoryUrl>
<RepositoryType>git</RepositoryType>
</PropertyGroup>
<ItemGroup>
<RuntimeHostConfigurationOption Include="System.Globalization.Invariant" Value="true" />
</ItemGroup>
<ItemGroup>
<PackageReference Include="CommandLineParser" Version="2.8.0" />
<PackageReference Include="LibGit2Sharp" Version="0.26.2" />
<PackageReference Include="MediatR" Version="8.1.0" />
<PackageReference Include="MediatR.Extensions.Microsoft.DependencyInjection" Version="8.1.0" />
<PackageReference Include="Microsoft.ApplicationInsights" Version="2.14.0" />
<PackageReference Include="Microsoft.ApplicationInsights.PerfCounterCollector" Version="2.14.0" />
<PackageReference Include="Microsoft.CodeAnalysis.FxCopAnalyzers" Version="3.3.0">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
<PackageReference Include="Microsoft.DotNet.Analyzers.Compatibility" Version="0.2.12-alpha" />
<PackageReference Include="Microsoft.Extensions.Configuration.EnvironmentVariables" Version="3.1.7" />
<PackageReference Include="Microsoft.Extensions.Configuration.Json" Version="3.1.7" />
<PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="3.1.7" />
<PackageReference Include="Microsoft.Extensions.Http" Version="3.1.7" />
<PackageReference Include="Microsoft.Extensions.Http.Polly" Version="3.1.7" />
<PackageReference Include="Microsoft.Extensions.Localization" Version="3.1.7" />
<PackageReference Include="Microsoft.Extensions.Logging" Version="3.1.7" />
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="3.1.7" />
<PackageReference Include="Microsoft.VisualStudio.Azure.Containers.Tools.Targets" Version="1.10.8" />
<PackageReference Include="Octokit" Version="0.48.0" />
<PackageReference Include="Polly" Version="7.2.1" />
<PackageReference Include="RestSharp" Version="106.11.4" />
<PackageReference Include="Serilog.Extensions.Logging" Version="3.0.1" />
<PackageReference Include="Serilog.Sinks.ApplicationInsights" Version="3.1.0" />
<PackageReference Include="Serilog.Sinks.Console" Version="3.1.1" />
<PackageReference Include="SharpCompress" Version="0.26.0" />
</ItemGroup>
</Project>

30
server/src/FilterLists.Agent/Infrastructure/ApplicationInsights/AgentTelemetryClient.cs
View file

@ -1,30 +0,0 @@
using System;
using System.Threading;
using Microsoft.ApplicationInsights;
namespace FilterLists.Agent.Infrastructure.ApplicationInsights
{
public class AgentTelemetryClient : IDisposable
{
public AgentTelemetryClient(string instrumentationKey)
{
TelemetryClient = new TelemetryClient {InstrumentationKey = instrumentationKey};
}
public TelemetryClient TelemetryClient { get; }
public void Dispose()
{
Dispose(true);
GC.SuppressFinalize(this);
}
protected virtual void Dispose(bool disposing)
{
if (!disposing)
return;
TelemetryClient.Flush();
Thread.Sleep(5000);
}
}
}

34
server/src/FilterLists.Agent/Infrastructure/ApplicationInsights/QuickPulseTelemetryModuleBuilder.cs
View file

@ -1,34 +0,0 @@
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse;
namespace FilterLists.Agent.Infrastructure.ApplicationInsights
{
public class QuickPulseTelemetryModuleBuilder
{
private readonly TelemetryConfiguration _telemetryConfiguration;
public QuickPulseTelemetryModuleBuilder(TelemetryConfiguration telemetryConfiguration)
{
_telemetryConfiguration = telemetryConfiguration;
}
public QuickPulseTelemetryModule Build()
{
QuickPulseTelemetryProcessor processor = null;
_telemetryConfiguration.TelemetryProcessorChainBuilder
.Use(next =>
{
processor = new QuickPulseTelemetryProcessor(next);
return processor;
})
.Build();
var quickPulseTelemetryModule = new QuickPulseTelemetryModule();
quickPulseTelemetryModule.Initialize(_telemetryConfiguration);
quickPulseTelemetryModule.RegisterTelemetryProcessor(processor);
foreach (var telemetryProcessor in _telemetryConfiguration.TelemetryProcessors)
if (telemetryProcessor is ITelemetryModule telemetryModule)
telemetryModule.Initialize(_telemetryConfiguration);
return quickPulseTelemetryModule;
}
}
}

23
server/src/FilterLists.Agent/Infrastructure/ApplicationInsights/ServiceCollectionExtensions.cs
View file

@ -1,23 +0,0 @@
using FilterLists.Agent.AppSettings;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse;
using Microsoft.Extensions.DependencyInjection;
namespace FilterLists.Agent.Infrastructure.ApplicationInsights
{
public static class ServiceCollectionExtensions
{
public static void AddApplicationInsights(this IServiceCollection services,
ApplicationInsightsSettings applicationInsightsSettings)
{
services.AddSingleton(b =>
{
var telemetryConfiguration = new TelemetryConfiguration(applicationInsightsSettings.InstrumentationKey);
return new QuickPulseTelemetryModuleBuilder(telemetryConfiguration).Build();
});
// init never resolved singleton
services.BuildServiceProvider().GetService<QuickPulseTelemetryModule>();
}
}
}

74
server/src/FilterLists.Agent/Infrastructure/DependencyInjection/ServiceCollectionExtensions.cs
View file

@ -1,74 +0,0 @@
using System;
using CommandLine;
using FilterLists.Agent.AppSettings;
using FilterLists.Agent.Extensions;
using FilterLists.Agent.Infrastructure.ApplicationInsights;
using FilterLists.Agent.Infrastructure.Disk;
using FilterLists.Agent.Infrastructure.FilterListsApi;
using FilterLists.Agent.Infrastructure.GitHub;
using FilterLists.Agent.Infrastructure.Polly;
using FilterLists.Agent.Infrastructure.Web;
using MediatR;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Serilog;
namespace FilterLists.Agent.Infrastructure.DependencyInjection
{
public static class ServiceCollectionExtensions
{
private static IConfigurationRoot Configuration { get; } = new ConfigurationBuilder()
.AddEnvironmentVariables()
.AddJsonFile("appsettings.json", true, true)
#if DEBUG
.AddJsonFile("appsettings.Development.json", true, true)
#endif
.Build();
private static ApplicationInsightsSettings ApplicationInsightsSettings { get; } =
Configuration.GetSettings<ApplicationInsightsSettings>();
public static void ConfigureServices(this IServiceCollection services)
{
services.ConfigureSettings();
services.AddLogging();
services.AddApplicationInsights(ApplicationInsightsSettings);
services.AddLocalization();
services.AddTransient<Parser>();
services.AddMediatR(AppDomain.CurrentDomain.GetAssemblies());
services.AddPollyPolicyRegistry();
services.AddDiskServices(Configuration.GetSettings<ArchiveSettings>());
services.AddFilterListsApiServices();
services.AddGitHubServices(Configuration.GetSettings<GitHubSettings>());
services.AddWebServices();
}
private static void ConfigureSettings(this IServiceCollection services)
{
services.Configure<ApplicationInsightsSettings>();
services.Configure<ArchiveSettings>();
services.Configure<ConnectionStringSettings>();
services.Configure<FilterListsApiSettings>();
services.Configure<GitHubSettings>();
}
private static void Configure<TSettings>(this IServiceCollection services) where TSettings : class
{
services.Configure<TSettings>(Configuration.GetSection<TSettings>());
}
private static void AddLogging(this IServiceCollection services)
{
services.AddLogging(b =>
{
var instrumentationKey = ApplicationInsightsSettings.InstrumentationKey;
var telemetryClient = new AgentTelemetryClient(instrumentationKey).TelemetryClient;
var logger = new LoggerConfiguration()
.WriteTo.Console()
.WriteTo.ApplicationInsights(telemetryClient, TelemetryConverter.Traces)
.CreateLogger();
b.AddSerilog(logger);
});
}
}
}

15
server/src/FilterLists.Agent/Infrastructure/DependencyInjection/ServiceProviderBuilder.cs
View file

@ -1,15 +0,0 @@
using System;
using Microsoft.Extensions.DependencyInjection;
namespace FilterLists.Agent.Infrastructure.DependencyInjection
{
public static class ServiceProviderBuilder
{
public static IServiceProvider Build()
{
var serviceCollection = new ServiceCollection();
serviceCollection.ConfigureServices();
return serviceCollection.BuildServiceProvider();
}
}
}

19
server/src/FilterLists.Agent/Infrastructure/Disk/ServiceCollectionExtensions.cs
View file

@ -1,19 +0,0 @@
using FilterLists.Agent.AppSettings;
using LibGit2Sharp;
using Microsoft.Extensions.DependencyInjection;
namespace FilterLists.Agent.Infrastructure.Disk
{
public static class ServiceCollectionExtensions
{
public static void AddDiskServices(this IServiceCollection services, ArchiveSettings archiveSettings)
{
services.AddTransient<IRepository, Repository>(s =>
{
if (!Repository.IsValid(archiveSettings.RepositoryDirectory))
Repository.Init(archiveSettings.RepositoryDirectory);
return new Repository(archiveSettings.RepositoryDirectory);
});
}
}
}

55
server/src/FilterLists.Agent/Infrastructure/FilterListsApi/EntityUrlRepository.cs
View file

@ -1,55 +0,0 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using FilterLists.Agent.Core.Urls;
using FilterLists.Agent.Infrastructure.FilterListsApi.EntityUrls;
using RestSharp;
namespace FilterLists.Agent.Infrastructure.FilterListsApi
{
public class EntityUrlRepository : IEntityUrlRepository
{
private static readonly Dictionary<string, (string, Entity)> EndpointAndEntityByDto =
new Dictionary<string, (string, Entity)>
{
{nameof(LicenseUrls), ("licenses", Entity.License)},
{nameof(ListUrls), ("lists", Entity.FilterList)},
{nameof(MaintainerUrls), ("maintainers", Entity.Maintainer)},
{nameof(SoftwareUrls), ("software", Entity.Software)},
{nameof(SyntaxUrls), ("syntaxes", Entity.Syntax)}
};
private readonly IFilterListsApiClient _apiClient;
public EntityUrlRepository(IFilterListsApiClient apiClient)
{
_apiClient = apiClient;
}
public async Task<IEnumerable<EntityUrl>> GetAllAsync(CancellationToken cancellationToken)
{
return (await Task.WhenAll(
Task.Run(() => GetDistinctForEntity<LicenseUrls>(cancellationToken), cancellationToken),
Task.Run(() => GetDistinctForEntity<ListUrls>(cancellationToken), cancellationToken),
Task.Run(() => GetDistinctForEntity<MaintainerUrls>(cancellationToken), cancellationToken),
Task.Run(() => GetDistinctForEntity<SoftwareUrls>(cancellationToken), cancellationToken),
Task.Run(() => GetDistinctForEntity<SyntaxUrls>(cancellationToken), cancellationToken)))
.SelectMany(r => r);
}
private async Task<IEnumerable<EntityUrl>> GetDistinctForEntity<TModel>(CancellationToken cancellationToken)
{
var request = new RestRequest($"{EndpointAndEntityByDto[typeof(TModel).Name].Item1}/seed");
var response = await _apiClient.ExecuteAsync<IEnumerable<TModel>>(request, cancellationToken);
return response.SelectMany(e =>
{
var propertyInfos = e.GetType().GetProperties();
var id = (int)propertyInfos.First(p => p.Name == nameof(LicenseUrls.Id)).GetValue(e);
return propertyInfos.Where(p => p.GetValue(e) != null && p.PropertyType == typeof(Uri)).Select(p =>
new EntityUrl(EndpointAndEntityByDto[typeof(TModel).Name].Item2, id, (Uri)p.GetValue(e)));
}).GroupBy(e => e.Url).Select(e => e.First());
}
}
}

10
server/src/FilterLists.Agent/Infrastructure/FilterListsApi/EntityUrls/LicenseUrls.cs
View file

@ -1,10 +0,0 @@
using System;
namespace FilterLists.Agent.Infrastructure.FilterListsApi.EntityUrls
{
public class LicenseUrls
{
public int Id { get; private set; }
public Uri DescriptionUrl { get; private set; }
}
}

20
server/src/FilterLists.Agent/Infrastructure/FilterListsApi/EntityUrls/ListUrls.cs
View file

@ -1,20 +0,0 @@
using System;
namespace FilterLists.Agent.Infrastructure.FilterListsApi.EntityUrls
{
public class ListUrls
{
public int Id { get; private set; }
public Uri ChatUrl { get; private set; }
public Uri DescriptionSourceUrl { get; private set; }
public Uri DonateUrl { get; private set; }
public Uri ForumUrl { get; private set; }
public Uri HomeUrl { get; private set; }
public Uri IssuesUrl { get; private set; }
public Uri PolicyUrl { get; private set; }
public Uri SubmissionUrl { get; private set; }
public Uri ViewUrl { get; private set; }
public Uri ViewUrlMirror1 { get; private set; }
public Uri ViewUrlMirror2 { get; private set; }
}
}

10
server/src/FilterLists.Agent/Infrastructure/FilterListsApi/EntityUrls/MaintainerUrls.cs
View file

@ -1,10 +0,0 @@
using System;
namespace FilterLists.Agent.Infrastructure.FilterListsApi.EntityUrls
{
public class MaintainerUrls
{
public int Id { get; private set; }
public Uri HomeUrl { get; private set; }
}
}

11
server/src/FilterLists.Agent/Infrastructure/FilterListsApi/EntityUrls/SoftwareUrls.cs
View file

@ -1,11 +0,0 @@
using System;
namespace FilterLists.Agent.Infrastructure.FilterListsApi.EntityUrls
{
public class SoftwareUrls
{
public int Id { get; private set; }
public Uri DownloadUrl { get; private set; }
public Uri HomeUrl { get; private set; }
}
}

10
server/src/FilterLists.Agent/Infrastructure/FilterListsApi/EntityUrls/SyntaxUrls.cs
View file

@ -1,10 +0,0 @@
using System;
namespace FilterLists.Agent.Infrastructure.FilterListsApi.EntityUrls
{
public class SyntaxUrls
{
public int Id { get; private set; }
public Uri DefinitionUrl { get; private set; }
}
}

45
server/src/FilterLists.Agent/Infrastructure/FilterListsApi/FilterListsApiClient.cs
View file

@ -1,45 +0,0 @@
using System.Threading;
using System.Threading.Tasks;
using FilterLists.Agent.AppSettings;
using Microsoft.Extensions.Localization;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using RestSharp;
namespace FilterLists.Agent.Infrastructure.FilterListsApi
{
public interface IFilterListsApiClient
{
Task<TResponse> ExecuteAsync<TResponse>(IRestRequest restRequest, CancellationToken cancellationToken);
}
public class FilterListsApiClient : IFilterListsApiClient
{
private readonly IStringLocalizer<FilterListsApiClient> _localizer;
private readonly ILogger<FilterListsApiClient> _logger;
private readonly IRestClient _restClient;
public FilterListsApiClient(IStringLocalizer<FilterListsApiClient> stringLocalizer,
ILogger<FilterListsApiClient> logger, IOptions<FilterListsApiSettings> filterListsApiOptions)
{
_localizer = stringLocalizer;
_logger = logger;
var filterListsApiSettings = filterListsApiOptions.Value;
_restClient = new RestClient(filterListsApiSettings.BaseUrl)
{
UserAgent = filterListsApiSettings.ClientUserAgent
};
}
public async Task<TResponse> ExecuteAsync<TResponse>(IRestRequest restRequest,
CancellationToken cancellationToken)
{
var response = await _restClient.ExecuteTaskAsync<TResponse>(restRequest, cancellationToken);
if (response.ErrorException != null)
_logger.LogError(response.ErrorException,
_localizer["Error retrieving response from the FilterLists API."]);
return response.Data;
}
}
}

24
server/src/FilterLists.Agent/Infrastructure/FilterListsApi/ListViewUrlRepository.cs
View file

@ -1,24 +0,0 @@
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using FilterLists.Agent.Core.Lists;
using RestSharp;
namespace FilterLists.Agent.Infrastructure.FilterListsApi
{
public class ListViewUrlRepository : IListViewUrlRepository
{
private readonly IFilterListsApiClient _apiClient;
public ListViewUrlRepository(IFilterListsApiClient filterListsApiClient)
{
_apiClient = filterListsApiClient;
}
public async Task<IEnumerable<ListViewUrl>> GetAllAsync(CancellationToken cancellationToken)
{
var listsRequest = new RestRequest("lists");
return await _apiClient.ExecuteAsync<IEnumerable<ListViewUrl>>(listsRequest, cancellationToken);
}
}
}

16
server/src/FilterLists.Agent/Infrastructure/FilterListsApi/ServiceCollectionExtensions.cs
View file

@ -1,16 +0,0 @@
using FilterLists.Agent.Core.Lists;
using FilterLists.Agent.Core.Urls;
using Microsoft.Extensions.DependencyInjection;
namespace FilterLists.Agent.Infrastructure.FilterListsApi
{
public static class ServiceCollectionExtensions
{
public static void AddFilterListsApiServices(this IServiceCollection services)
{
services.AddSingleton<IFilterListsApiClient, FilterListsApiClient>();
services.AddTransient<IListViewUrlRepository, ListViewUrlRepository>();
services.AddTransient<IEntityUrlRepository, EntityUrlRepository>();
}
}
}

70
server/src/FilterLists.Agent/Infrastructure/GitHub/IssuesRepository.cs
View file

@ -1,70 +0,0 @@
using System.Collections.Generic;
using System.Threading.Tasks;
using FilterLists.Agent.AppSettings;
using FilterLists.Agent.Core.GitHub;
using Microsoft.Extensions.Localization;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Octokit;
namespace FilterLists.Agent.Infrastructure.GitHub
{
public class IssuesRepository : IIssuesRepository
{
private readonly IGitHubClient _gitHubClient;
private readonly GitHubSettings _gitHubSettings;
private readonly IStringLocalizer<IssuesRepository> _localizer;
private readonly ILogger<IssuesRepository> _logger;
public IssuesRepository(IGitHubClient gitHubClient, IOptions<GitHubSettings> gitHubOptions,
IStringLocalizer<IssuesRepository> stringLocalizer, ILogger<IssuesRepository> logger)
{
_gitHubClient = gitHubClient;
_gitHubSettings = gitHubOptions.Value;
_localizer = stringLocalizer;
_logger = logger;
}
public async Task<IEnumerable<Issue>> GetAllIssuesAsync(RepositoryIssueRequest repositoryIssueRequest)
{
try
{
return await _gitHubClient.Issue.GetAllForRepository(_gitHubSettings.RepositoryOwner,
_gitHubSettings.Repository, repositoryIssueRequest);
}
catch (ApiException ex)
{
_logger.LogError(ex, _localizer["Failed getting all Issues from the GitHub API."]);
throw;
}
}
public async Task<Issue> CreateIssueAsync(NewIssue newIssue)
{
try
{
return await _gitHubClient.Issue.Create(_gitHubSettings.RepositoryOwner, _gitHubSettings.Repository,
newIssue);
}
catch (ApiException ex)
{
_logger.LogError(ex, _localizer["Failed creating Issue with the GitHub API."]);
throw;
}
}
public async Task<Issue> UpdateIssueAsync(int issueNumber, IssueUpdate issueUpdate)
{
try
{
return await _gitHubClient.Issue.Update(_gitHubSettings.RepositoryOwner, _gitHubSettings.Repository,
issueNumber, issueUpdate);
}
catch (ApiException ex)
{
_logger.LogError(ex, _localizer["Failed updating Issue with the GitHub API."]);
throw;
}
}
}
}

18
server/src/FilterLists.Agent/Infrastructure/GitHub/ServiceCollectionExtensions.cs
View file

@ -1,18 +0,0 @@
using FilterLists.Agent.AppSettings;
using FilterLists.Agent.Core.GitHub;
using Microsoft.Extensions.DependencyInjection;
using Octokit;
namespace FilterLists.Agent.Infrastructure.GitHub
{
public static class ServiceCollectionExtensions
{
public static void AddGitHubServices(this IServiceCollection services, GitHubSettings gitHubSettings)
{
services.AddSingleton<IGitHubClient, GitHubClient>(s =>
new GitHubClient(new ProductHeaderValue(gitHubSettings.ProductHeaderValue))
{Credentials = new Credentials(gitHubSettings.PersonalAccessToken)});
services.AddTransient<IIssuesRepository, IssuesRepository>();
}
}
}

25
server/src/FilterLists.Agent/Infrastructure/Polly/ServiceCollectionExtensions.cs
View file

@ -1,25 +0,0 @@
using System;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.Extensions.DependencyInjection;
using Polly;
namespace FilterLists.Agent.Infrastructure.Polly
{
public static class ServiceCollectionExtensions
{
public static void AddPollyPolicyRegistry(this IServiceCollection services)
{
var registry = services.AddPolicyRegistry();
IAsyncPolicy<HttpResponseMessage> waitAndRetryTooManyRequests = Policy
.HandleResult<HttpResponseMessage>(r => r.StatusCode == HttpStatusCode.TooManyRequests)
.WaitAndRetryAsync(3,
(retryCount, response, context) =>
response.Result?.Headers.RetryAfter.Delta ?? TimeSpan.FromMilliseconds(120),
async (response, timespan, retryCount, context) => { await Task.CompletedTask; });
registry.Add(nameof(waitAndRetryTooManyRequests), waitAndRetryTooManyRequests);
}
}
}

74
server/src/FilterLists.Agent/Infrastructure/Web/EntityUrlValidator.cs
View file

@ -1,74 +0,0 @@
using System;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading;
using System.Threading.Tasks;
using FilterLists.Agent.AppSettings;
using FilterLists.Agent.Core.Urls;
using Microsoft.Extensions.Options;
namespace FilterLists.Agent.Infrastructure.Web
{
public class EntityUrlValidator : IEntityUrlValidator
{
private readonly HttpClient _httpClient;
public EntityUrlValidator(HttpClient httpClient, IOptions<FilterListsApiSettings> filterListsApiOptions)
{
httpClient.Timeout = TimeSpan.FromSeconds(90);
var header = new ProductHeaderValue(filterListsApiOptions.Value.ClientUserAgent);
var userAgent = new ProductInfoHeaderValue(header);
httpClient.DefaultRequestHeaders.UserAgent.Add(userAgent);
_httpClient = httpClient;
}
public async Task<EntityUrl> ValidateAsync(EntityUrl entityUrl, CancellationToken cancellationToken)
{
var url = entityUrl.Url;
if (!Uri.IsWellFormedUriString(url.OriginalString, UriKind.Absolute))
{
entityUrl.SetBroken();
return entityUrl;
}
if (url.Scheme == Uri.UriSchemeHttp)
if (await IsHttpsSupported(url, cancellationToken))
entityUrl.SetSupportsHttps();
try
{
using var response = await GetAsync(url, cancellationToken);
if ((int)response.StatusCode >= 300 && (int)response.StatusCode < 400)
entityUrl.SetRedirectsTo(response.Headers.Location);
else if (!response.IsSuccessStatusCode)
entityUrl.SetBroken();
}
catch (Exception ex) when (ex is HttpRequestException ||
ex is TaskCanceledException)
{
entityUrl.SetBroken();
}
return entityUrl;
}
private async Task<bool> IsHttpsSupported(Uri url, CancellationToken cancellationToken)
{
var httpsUrl = new UriBuilder(url.OriginalString) {Scheme = Uri.UriSchemeHttps, Port = -1}.Uri;
try
{
using var response = await GetAsync(httpsUrl, cancellationToken);
return response.IsSuccessStatusCode;
}
catch (Exception ex) when (ex is HttpRequestException ||
ex is TaskCanceledException)
{
return false;
}
}
private async Task<HttpResponseMessage> GetAsync(Uri url, CancellationToken cancellationToken)
{
return await _httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead, cancellationToken);
}
}
}

42
server/src/FilterLists.Agent/Infrastructure/Web/ListRepository.cs
View file

@ -1,42 +0,0 @@
using System;
using System.Diagnostics.CodeAnalysis;
using System.IO;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Threading;
using System.Threading.Tasks;
using FilterLists.Agent.AppSettings;
using FilterLists.Agent.Core.Lists;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
namespace FilterLists.Agent.Infrastructure.Web
{
public class ListRepository : IListRepository
{
private readonly HttpClient _httpClient;
private readonly ILogger<ListRepository> _logger;
public ListRepository(HttpClient httpClient, ILogger<ListRepository> logger,
IOptions<FilterListsApiSettings> filterListsApiOptions)
{
httpClient.Timeout = TimeSpan.FromSeconds(90);
var header = new ProductHeaderValue(filterListsApiOptions.Value.ClientUserAgent);
var userAgent = new ProductInfoHeaderValue(header);
httpClient.DefaultRequestHeaders.UserAgent.Add(userAgent);
_httpClient = httpClient;
_logger = logger;
}
[SuppressMessage("Microsoft.Reliability", "CA2000:DisposeObjectsBeforeLosingScope")]
public async Task<Stream> GetAsStreamAsync(Uri url, CancellationToken cancellationToken)
{
var response = await _httpClient.GetAsync(url, cancellationToken);
if (response.IsSuccessStatusCode)
return await response.Content.ReadAsStreamAsync();
_logger.LogError($"Error downloading list from {url}. {response.StatusCode}");
return Stream.Null;
}
}
}

21
server/src/FilterLists.Agent/Infrastructure/Web/ServiceCollectionExtensions.cs
View file

@ -1,21 +0,0 @@
using System.Net.Http;
using FilterLists.Agent.Core.Lists;
using FilterLists.Agent.Core.Urls;
using Microsoft.Extensions.DependencyInjection;
namespace FilterLists.Agent.Infrastructure.Web
{
public static class ServiceCollectionExtensions
{
public static void AddWebServices(this IServiceCollection services)
{
services.AddHttpClient<IListRepository, ListRepository>()
.AddPolicyHandlerFromRegistry("waitAndRetryTooManyRequests");
services.AddHttpClient<IEntityUrlValidator, EntityUrlValidator>().ConfigureHttpMessageHandlerBuilder(b =>
{
b.PrimaryHandler = new HttpClientHandler {AllowAutoRedirect = false};
b.Build();
}).AddPolicyHandlerFromRegistry("waitAndRetryTooManyRequests");
}
}
}

35
server/src/FilterLists.Agent/Program.cs
View file

@ -1,35 +0,0 @@
using System;
using System.Threading.Tasks;
using CommandLine;
using FilterLists.Agent.Features.Lists;
using FilterLists.Agent.Features.Urls;
using FilterLists.Agent.Infrastructure.DependencyInjection;
using MediatR;
using Microsoft.Extensions.DependencyInjection;
namespace FilterLists.Agent
{
public static class Program
{
private static IServiceProvider _serviceProvider;
public static async Task Main(string[] args)
{
_serviceProvider = ServiceProviderBuilder.Build();
var parser = _serviceProvider.GetService<Parser>();
await parser.ParseArguments<CommandLineOptions>(args).MapResult(async o =>
{
if (o.Any())
{
var mediator = _serviceProvider.GetService<IMediator>();
if (o.ArchiveLists)
await mediator.Send(new ArchiveLists.Command());
if (o.ValidateUrls)
await mediator.Send(new ValidateAllUrls.Command());
}
},
e => Task.FromResult(0)
);
}
}
}

Some files were not shown because too many files have changed in this diff Show more