Extensible / customizable web crawling engines / frameworks / libraries?
I have a relatively simple case. I basically want to store link data between different websites and don't want to restrict domains. I know I can write my own crawler using some http client library, but I feel like I will be doing unnecessary work - make sure pages are not checked more than once while working out how to read and use the robots.txt file, maybe even trying to make it parallel and distributed and I'm sure there is a lot more that I haven't thought of yet.
So, I need a web crawling framework that takes care of these things, allowing me to dictate what to do with the responses (in my case, just fetch links and store them). Most searchers seem to assume that you are indexing web pages for search, and that is no good, I need something customizable.
I want to store link information in MongoDB database , so I need to be able to define how links are stored in the framework. And while I posed the question as a language agnostic, it also means that I have to restrict my choice of structure to one of the MongoDB supported languages (Python, Ruby, Perl, PHP, Java, and C ++), which is a very wide web. I prefer dynamic languages, but I am open to any suggestions.
I was able to find Scrapy (which looks neat) and JSpider (which seems good, but maybe too "heavy" based on the 121-page user manual), but I wanted to see if there are other good options where I'm missing.
source to share
I assume you have already searched Stack Overflow yourself as there are quite a few similar questions in the web-crawler tag . Without applying any of the following circumstances, I will not refrain from developing and just listed a few that seem to me worth considering for this task:
- Python
- Scrapy : I mention it again to make it look promising, especially because it is based on Twisted , a modern and highly regarded web engine.
- Mechanize : Based on WWW-Mechanize , see Perl below.
- A similar question related to Python , some more details on Mechanism, Scrapy and others.
- Ruby (never used them at all)
- Perl
- WWW-Mechanize : Very mature project, well documented.
- Java
- Nutch : Fairly mature project, well-documented, dedicated extensibility based on Apache Lucene , which is very mature and has a strong community. there are still problems with advanced integration scenarios, see this question .
- Heritrix : very mature project, well documented, dedicated extensibility, web archive foundation ; seems to be more efficient for more complex integration scenarios for some, see this question again .
Okay, good luck with the review;)
source to share
StormCrawler was not around when this question was asked, but it would fit the bill quite well. It is in Java, is highly modular and scalable, and can be configured to do what was described above.
source to share