curl and libcurl is a tool for transferring files using URL syntax. It supports HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, DICT, TELNET, LDAP, POP3, IMAP, SMTP, RTSP, RTMP, and FILE, as well as HTTP-post, HTTP-put, cookies, FTP upload, resumed transfers, passwords, port numbers, SSL certificates, Kerberos, and proxies. It is powered by libcurl, the client-side URL transfer library. There are bindings to libcurl for about 40 languages and environments.
DansGuardian is a Web content filtering proxy that uses Squid to do all the fetching. It filters using multiple methods including, but not limited to, phrase matching, file extension matching, MIME type matching, PICS filtering, and URL/domain blocking. It has the ability to switch off filtering by certain criteria including username, domain name, source IP, etc. The configurable logging produces a log in an easy to read format. It has the option to only log text-based pages, thus significantly reducing redundant information (such as every image on a page).
DeleGate is a multi-purpose application level gateway or proxy server that mediates communication of various protocols, applying cache and conversion for mediated data, controlling access from clients, and routing toward servers. It translates protocols between clients and servers, converting between IPv4 and IPv6, applying SSL (TLS) to arbitrary protocols, merging several servers into a single server view with aliasing and filtering. It can be used as a simple origin server for some protocols (HTTP, FTP, and NNTP).
Harvest is a system to collect information and make it searchable using a Web interface. It can collect information using HTTP, FTP, NNTP, and local files. Supported formats include HTML, DVI, PS, fulltext, mail, man pages, news, troff, WordPerfect, C sources, and many more. Adding support for new formats is easy due to Harvest's modular design.
httpMonitor is a Python program which lets you monitor the HTTP traffic between a browser and a Web server. You configure your browser to use httpMonitor as its proxy to ensure that all HTTP traffic between browser and server goes via httpMonitor. In a XML-configuration file you can specify a Python-function as processor for the HTTP-messages you are interested in (requests or responses, which match certain criteria). You may have as many processors as you want, without modifying the actual program. The only thing you need to do is configure the httpMonitor and write your processors.
HTTrack is an easy-to-use offline browser utility. It allows you to download a Web site from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer. HTTrack arranges the original site's relative link-structure. Simply open a page of the mirrored Web site in your browser, and you can browse the site from link to link, as if you were viewing it online. HTTrack can also update an existing mirrored site, and resume interrupted downloads. WebHTTrack is a Web-based GUI for HTTrack.
Kannel is a WAP gateway. It attempts to provide this essential part of the WAP infrastructure freely to everyone so the market potential for WAP services, both from wireless operators and specialized service providers, will be realized as efficiently as possible. It also works as an SMS gateway for GSM networks. Almost all GSM phones can use it to send and receive SMS messages, so this is a way to serve many more clients than just those using a WAP phone. Kannel was among the first WAP gateways to be certified as WAP 1.1 compliant.