GNU Wget is a utility for noninteractive download of files from the Web. It supports HTTP and FTP protocols, as well as retrieval through HTTP proxies. It can follow HTML links, download many pages, and convert the links for local viewing. It can also mirror FTP hierarchies or only those files that have changed. Wget has been designed for robustness over slow network connections; if a download fails due to a network problem, it will keep retrying until the whole file has been retrieved.
Site Checker can be used to find broken links in Web pages. First, it retrieves the list of all pages of the Web site, either static pages or dynamic pages generated from data stored in a database. Then it checks the links in each of the pages to verify if they are on the list. Links to external sites can also be verified by performing HTTP requests to the remote Web servers to check whether the pages still exist.
HTTrack is an easy-to-use offline browser utility. It allows you to download a Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure. Simply open a page of the mirrored Web site in your browser, and you can browse the site from link to link, as if you were viewing it online. HTTrack can also update an existing mirrored site, and resume interrupted downloads. WebHTTrack is a Web-based GUI for HTTrack.
SEO SpyGlass Professional is a feature-rich backlink analysis software for webmasters and SEOs who need to outrank their competition in all major search engines. SEO SpyGlass unearths your competitors' link-building strategies and lets you capitalize on their best optimization techniques. On the basis of competition research, the software develops a sure-winning step-by-step optimization plan that will lead your website to the top of Google and other search engines. This software is also available within SEO PowerSuite, which includes WebSite Auditor, LinkAssistant, and Rank Tracker as well.
SEO SpyGlass Enterprise is a feature-rich backlink analysis software for professional SEOs and webmasters. The tools provide the deepest possible insight into your competitors' link building strategies. It lets you milk your competitors of their most SEO-productive backlink sources and use these sources in your own SEO campaign. You'll check all backlinks for Google PageRank and Alexa Rank, scrutinize the used anchor texts and URLs, discover how many backlinks come from forums and blogs, from homepages, from DMOZ-listed sites and much more. The software generates 5 types of reports that can be further printed out, sent to your clients by e-mail or made available on a Web site. You can brand your reports with a company logo and set their color schemes and data layout. This software is also available within SEO PowerSuite, which includes WebSite Auditor, LinkAssistant, and Rank Tracker as well.
webcheck is a Web site checking tool for Web masters. It crawls a given Web site and generates a number of reports. The whole system is pluggable, allowing extra reports and checks to be added easily. It supports retrieving Web sites over HTTP, file, and FTP protocols and produces reports on site structure, broken links, old Web pages, overviews of external links, and more. The links that webcheck considers external are configurable through regular expressions, and webcheck honors robots.txt.
Web-Analiser is a script to gather and analyze statistics of Website traffic. It provides the whole gamut of every possible statistics and reports, which analyze visitors, pages, and Website traffic. It is designed to operate both one and several domains (Websites) at once. The statistics are maintained, in which each Website dealt with separately and together. It includes a great number of statistical reports, making it possible to have a complete idea of your audience.
phoneutria is a Web crawler that is multi-threaded, scalable, high performance, extensible, and polite. It can be used to crawl, index, load-test, or even download any Web or enterprise domain and is configurable through a XML configuration file. Phoneutria can be used for either checking the links of a Web site or for load-testing purposes (i.e. the level of politeness can be configured). It provides a plug-in mechanism for further extensions.
SWEC is a program that automates testing of dynamic Web sites. It parses each HTML file it finds for links, and if those links are within the site specified, it will check that page as well. In addition to parsing and locating links, it will also parse the pages looking for known errors and report those. It will report if a page cannot be read (by either returning a 404, 500, or similar).