Site Checker can be used to find broken links in Web pages. First, it retrieves the list of all pages of the Web site, either static pages or dynamic pages generated from data stored in a database. Then it checks the links in each of the pages to verify if they are on the list. Links to external sites can also be verified by performing HTTP requests to the remote Web servers to check whether the pages still exist.
phoneutria is a Web crawler that is multi-threaded, scalable, high performance, extensible, and polite. It can be used to crawl, index, load-test, or even download any Web or enterprise domain and is configurable through a XML configuration file. Phoneutria can be used for either checking the links of a Web site or for load-testing purposes (i.e. the level of politeness can be configured). It provides a plug-in mechanism for further extensions.