Clever Geek Handbook

.Based on web crawler sample

I am using VSTS 2008 + C # + .Net 3.5. I want to find a tool (open source) that crawls all web pages for a website and for any other domain pages that are linked by that website, I want to skip crawling them (I only need a page for that specific domain only) ... For crawling a webpage, I want to store them in a local file directory.

Any samples or ready-to-use open source tools?

+2


source to share


2 answers


Arachnode.net might be what you are looking for.



+4


source


I am the author of AN.

AN indexes all languages ​​by default. There is nothing to customize.



  • Mike
+3


source






More articles:


All Articles