2 projects tagged "web crawler"
Updated 25 Jun 2009
Methanol
| Pop | 34.42 |
|---|---|
| Vit | 1.02 |
Methanol is a modular, customizable Web crawling system with crawlers optimized for speed. It is designed to allow the administrator to set up any kind of filetype handling, parsing, and indexing rules.
Updated 18 Nov 2012
SpiderBot
| Pop | 21.19 |
|---|---|
| Vit | 13.63 |
SpiderBot crawls the Web, retrieves content, and performs actions on the content. It is an effort to design and develop a truly pipelined distributed Web crawler.
