Spider Legs is a prototype, medium-duty, parallel, and highly programmable framework for writing Web robots with Perl. Spider Legs is not a search engine or content indexing system, but this software may be a useful part of such a project. If you need to download content from multiple Web sites on a periodic basis, Spider Legs can help you. It provides a logging system, allows multiple Web sites to be crawled at the same time, and uses the standard LWP at its core.
|Tags||Internet Web Software Development Libraries Application Frameworks|