9 commits

Author	SHA1	Message	Date
ekultek	90ba39f3ed	patch for an issue where if the site refuses connection it would keep trying to crawl it (issue #134)	2017-11-03 09:49:58 -05:00
ekultek	51db83833b	patch for a reported issue where it would fail with from a website with Unicode	2017-10-31 13:18:40 -05:00
ekultek	182e588774	patched a privately reported issue that would not allow you to run the blackwidow crawler behind a proxy, forgot a space in the README file	2017-10-28 16:53:13 -05:00
ekultek	f75cabb876	added a new requirement BeautifulSoup, fixed the crawler will pull all links with an 'a' tag and descriptor of 'href', bumped version number	2017-10-28 16:28:18 -05:00
ekultek	80d9565585	files edited to reflect the move of the settings, errors, and tamper scripts folders	2017-10-10 17:01:13 -05:00
ekultek	fa43d499c5	edited for the new write_to_file function	2017-10-03 14:53:45 -05:00
ekultek	bf922eeb7c	fixed the blackwidow log file issue #14	2017-09-28 11:16:21 -05:00
cclauss	b7fb8350d8	Simplify with 'or'. 'Else pass' is unnecessary	2017-09-25 16:28:50 +02:00
ekultek	03498f3d9f	created a website spider, will pull all useable URL's from a given webpage	2017-09-11 11:43:41 -05:00