3 projects tagged "web crawler"
Updated 18 Nov 2012
SpiderBot
| Pop | 21.17 |
|---|---|
| Vit | 13.66 |
SpiderBot crawls the Web, retrieves content, and performs actions on the content. It is an effort to design and develop a truly pipelined distributed Web crawler.
Updated 02 May 2013
OpenSearchServer
| Pop | 923.65 |
|---|---|
| Vit | 52.42 |
OpenSearchServer is a stable, high-performance search engine and a suite of high-powered full text search algorithms. Documents can be indexed in sixteen languages. Multi-lingual analyzers slice sentences into words, then run lemmatisation algorithms on words based on the document's language. Numerous document formats are supported, such as XML, HTML/XHTML, PDF, Word, PowerPoint, RTF, OpenOffice, plain text, MP3/4, Ogg, FLAC, etc. The Web interface, built around the Zkoss framework, provides an easy way to manage OSS. The integration is fast using the PHP client or the API (XML over HTTP). The crawlers of OpenSearchServer go through Web sites, file systems, and databases to rapidly and easily build your index.
Updated 25 Jun 2009
Methanol
| Pop | 33.90 |
|---|---|
| Vit | 1.02 |
Methanol is a modular, customizable Web crawling system with crawlers optimized for speed. It is designed to allow the administrator to set up any kind of filetype handling, parsing, and indexing rules.
