SWEC is a program that automates testing of dynamic Web sites. It parses each HTML file it finds for links, and if those links are within the site specified, it will check that page as well. In addition to parsing and locating links, it will also parse the pages looking for known errors and report those. It will report if a page cannot be read (by either returning a 404, 500, or similar).
phoneutria is a Web crawler that is multi-threaded, scalable, high performance, extensible, and polite. It can be used to crawl, index, load-test, or even download any Web or enterprise domain and is configurable through a XML configuration file. Phoneutria can be used for either checking the links of a Web site or for load-testing purposes (i.e. the level of politeness can be configured). It provides a plug-in mechanism for further extensions.