WebCrawler

A Web Crawler which crawls the webpage in BFS order and returns the depth from origin ,most frequent word and number of valid external links on the page

Instructions:

1. Install Beautiful Soup using pip install beautifulsoup4
2. Run BFS_Crawler.py using python BFS_Crawler.py
3. Enter a valid url starting with http:// or https://
4. Enter an integer for specifying max limit for number of pages to be crawled
5. Enter an integer for specifying timeout (in seconds)

Result will pe printed on the console and a log.txt file will be generated