ekultek
|
a302562893
|
moved the encoding declaration to the main file, it will now be started at the beginning of the program, also removed the call to dict the proxy (issue #287)
|
2017-12-09 22:20:48 -06:00 |
|
ekultek
|
b16a9c184a
|
getting rid of the '...' at the end of the output string, got sick of typing it os now it won't be there anymore
|
2017-12-07 07:27:20 -06:00 |
|
ekultek
|
30ffde95c3
|
created a get_page function that does the requests for me, this way I'm not calling requests.get over and over again
|
2017-12-02 11:06:16 -06:00 |
|
ekultek
|
97187c07f0
|
fixes issue #173 crawler will ignore SSL certificates, fixes #174 caches the found firewall into memory, incase we run across it again we don't waste our time trying to discover it, fixes #175 and #176 if there are unicode chars in the value it will not be saved
|
2017-11-28 08:14:47 -06:00 |
|
ekultek
|
5029005829
|
patch for an issue where a failed status code would try to create an issue, will now output the status code, the reason, and shutdown
|
2017-11-27 09:55:41 -06:00 |
|
ekultek
|
6f62049eb8
|
optimization for the time it takes to load the program, should be quicker now, created a search for public PGP keys will take your sites domain and search for any public PGP keys it can find, renamed some files to make more sense
|
2017-11-22 15:07:24 -06:00 |
|
ekultek
|
52af437fe4
|
moved shutdown, startup, and prompt to lib.core.common due to how much they are used
|
2017-11-20 14:27:08 -06:00 |
|
ekultek
|
9eca1950cb
|
created constants of all the log filenames, adjusted files accordingly
|
2017-11-20 14:04:11 -06:00 |
|
ekultek
|
90ba39f3ed
|
patch for an issue where if the site refuses connection it would keep trying to crawl it (issue #134)
|
2017-11-03 09:49:58 -05:00 |
|
ekultek
|
51db83833b
|
patch for a reported issue where it would fail with from a website with Unicode
|
2017-10-31 13:18:40 -05:00 |
|
ekultek
|
182e588774
|
patched a privately reported issue that would not allow you to run the blackwidow crawler behind a proxy, forgot a space in the README file
|
2017-10-28 16:53:13 -05:00 |
|
ekultek
|
f75cabb876
|
added a new requirement BeautifulSoup, fixed the crawler will pull all links with an 'a' tag and descriptor of 'href', bumped version number
|
2017-10-28 16:28:18 -05:00 |
|
ekultek
|
80d9565585
|
files edited to reflect the move of the settings, errors, and tamper scripts folders
|
2017-10-10 17:01:13 -05:00 |
|
ekultek
|
fa43d499c5
|
edited for the new write_to_file function
|
2017-10-03 14:53:45 -05:00 |
|
ekultek
|
bf922eeb7c
|
fixed the blackwidow log file issue #14
|
2017-09-28 11:16:21 -05:00 |
|
cclauss
|
b7fb8350d8
|
Simplify with 'or'. 'Else pass' is unnecessary
|
2017-09-25 16:28:50 +02:00 |
|
ekultek
|
03498f3d9f
|
created a website spider, will pull all useable URL's from a given webpage
|
2017-09-11 11:43:41 -05:00 |
|