ItSucks is a Web spider with the ability to
download (and resume) files. It is also highly
customizable with regular expressions and download
templates. All backend functionality is also available
in a separate library.
OpenSearchServer is a powerful, enterprise-class, search engine program. Using its Web user interface, crawlers (Web, file, database, etc.), and REST/RESTFul API, you can integrate advanced full-text search capabilities into your application.