This is a tool to collect information from web servers and to spider the web sites. This was written for the Open Source Security Testing Methodology (OSSTM) located on http://www.ideahamster.org/osstmm- description.htm. The spider is a multi-threaded resusable module that can be used in other projects.
SitemapGen4j is a Java library to generate XML sitemaps. It supports gzipped output, sitemap validation, and sitemap index generation. It can also generate Google-specific sitemaps, such as Mobile sitemaps, Geo sitemaps, Code Search sitemaps, Google News sitemaps, and Video sitemaps.