2 projects tagged "web crawler"
Updated 18 Nov 2012
SpiderBot
| Pop | 21.24 |
|---|---|
| Vit | 13.59 |
SpiderBot crawls the Web, retrieves content, and performs actions on the content. It is an effort to design and develop a truly pipelined distributed Web crawler.
Updated 25 Jun 2009
Methanol
| Pop | 34.51 |
|---|---|
| Vit | 1.02 |
Methanol is a modular, customizable Web crawling system with crawlers optimized for speed. It is designed to allow the administrator to set up any kind of filetype handling, parsing, and indexing rules.
