With LinkChecker, you can check HTML documents and Web sites for broken links. It features recursion, robots.txt exclusion protocol support, HTTP proxy support, i18n support, multithreading, regular expression filtering rules for links, and user/password checking for authorized pages. Output can be colored or normal text, HTML, SQL, CSV, or a sitemap graph in DOT, GML, or XML format. Supported link types are HTTP/1.1 and 1.0, HTTPS, FTP, mailto:, news:, nntp:, Telnet, and local files.
GNU Wget is a utility for noninteractive download of files from the Web. It supports HTTP and FTP protocols, as well as retrieval through HTTP proxies. It can follow HTML links, download many pages, and convert the links for local viewing. It can also mirror FTP hierarchies or only those files that have changed. Wget has been designed for robustness over slow network connections; if a download fails due to a network problem, it will keep retrying until the whole file has been retrieved.
Site Checker can be used to find broken links in Web pages. First, it retrieves the list of all pages of the Web site, either static pages or dynamic pages generated from data stored in a database. Then it checks the links in each of the pages to verify if they are on the list. Links to external sites can also be verified by performing HTTP requests to the remote Web servers to check whether the pages still exist.
HTTrack is an easy-to-use offline browser utility. It allows you to download a Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure. Simply open a page of the mirrored Web site in your browser, and you can browse the site from link to link, as if you were viewing it online. HTTrack can also update an existing mirrored site, and resume interrupted downloads. WebHTTrack is a Web-based GUI for HTTrack.
urlwatch is a script intended to help you watch URLs and get notified (via email) of any changes. The change notification will include the URL that has changed and a unified diff of what has changed. The script works out of a single directory, so there is no need to install anything. State files are kept in the same folder. The script supports stripping parts of a page that are always changing through the use of a filter hook function. It is typically run as a cronjob.
webcheck is a Web site checking tool for Web masters. It crawls a given Web site and generates a number of reports. The whole system is pluggable, allowing extra reports and checks to be added easily. It supports retrieving Web sites over HTTP, file, and FTP protocols and produces reports on site structure, broken links, old Web pages, overviews of external links, and more. The links that webcheck considers external are configurable through regular expressions, and webcheck honors robots.txt.
Web-Analiser is a script to gather and analyze statistics of Website traffic. It provides the whole gamut of every possible statistics and reports, which analyze visitors, pages, and Website traffic. It is designed to operate both one and several domains (Websites) at once. The statistics are maintained, in which each Website dealt with separately and together. It includes a great number of statistical reports, making it possible to have a complete idea of your audience.
Websitary is a script that monitors Web pages, RSS feeds, and podcasts and reports what's new. For many tasks, it reuses other programs (such as w3m, diff, and webdiff) to do the actual work. By default, it works on an ASCII basis, i.e. with the output of text-based Web browsers. With the help of some friends, it can also work with HTML.