Nutch is highly scalable Web searching software which builds on top of Apache Hadoop and Lucene Java. Key features include a Web crawler, indexer, crawl management tools, parsers for HTML, PDF, DOC, and several other document formats, and an expandable architecture that allows you to plug in additional functionality such as document parsers, custom scoring algorithms, custom content parsers, protocols, and more.
Catacomb is a WebDAV repository module for use with the Apache WebDAV module, mod_dav. Apache mod_dav parses WebDAV and DeltaV protocol requests into operations on a repository providing persistent storage of resources and their properties. The default repository for mod_dav is provided by a separate module, mod_dav_fs, which stores resource bodies as files in the filesystem, and stores properties in a (G)DBM database. It could be used for server side searching and versioning of files over the HTTP protocol.
Compass is a Java framework which makes it simple to map your Java object model into a search engine. It is built on top of the Lucene search engine. Compass features declarative mapping technology OSEM (similar to O/R database mapping), transaction management, Google-like query syntax, externalization of common metadata, and much much more.
Ojax provides an OAI-PMH harvester to harvest metadata from multiple repositories and builds a single searchable Lucene index of this data. OJAX provides a dynamic user experience enabled by Ajax and full OpenSearch integration with the IE 7 and FireFox 2 search fields. It supports OpenSearch Discovery, OpenSearch Atom results, and OpenSearch Search Suggestions. Loosely coupled Web Services and OpenSearch Atom standard allow integration with a range of VLEs, CMSs, institutional repositories, and portals.
Python Web Graph Generator is a threaded Web graph (Power law random graph) generator. It can generate a synthetic Web graph of about one million nodes in a few minutes on a desktop machine. It supports both directed and undirected graphs. It implements a threaded variant of the RMAT algorithm. A little tweak can produce graphs representing social networks or community networks. It can also output connected components in a graph.
Redland is a set of C libraries providing a high-level API for the Resource Description Framework (RDF), allowing it to be stored, parsed, serialized, queried, and manipulated. It has an object-based, modular design and comes with detailed reference documentation and examples. Redland supports all RDF vocabularies such as FOAF, RSS 1.0, Dublin Core, DOAP, and OWL, the query languages SPARQL and RDQL, and all RDF syntaxes including Turtle, RDF/XML, RDF/JSON, RSS, Atom, RDFa, and GRDDL.