Nutch is highly scalable Web searching software which builds on top of Apache Hadoop and Lucene Java. Key features include a Web crawler, indexer, crawl management tools, parsers for HTML, PDF, DOC, and several other document formats, and an expandable architecture that allows you to plug in additional functionality such as document parsers, custom scoring algorithms, custom content parsers, protocols, and more.
"TCP Input Text" extracts TCP ports and fully qualified domain names (FQDN) from search results into a .csv file and individual shell scripts for nmap and nc (a.k.a. netcat) to provide assurance of a listening TCP service since the time that has passed of the last crawl performed by the search engine.
ratproxy is a semi-automated, largely passive Web application security audit tool optimized for accurate and sensitive detection, and automatic annotation, of potential problems and security-relevant design patterns based on the observation of existing, user-initiated traffic in complex Web 2.0 environments.
skipfish is a high-performance, easy, and sophisticated Web application security testing tool. It features a single-threaded multiplexing HTTP stack, heuristic detection of obscure Web frameworks, and advanced, differential security checks capable of detecting blind injection vulnerabilities, stored XSS, and so forth.