The Ex-Crawler Project is divided into three subprojects. The main part is the Ex-Crawler daemon server, a highly configurable and flexible Web crawler written in Java. It comes with its own socket server, with which you can manage the server, users, distributed grid/volunteer computing, and much more. Crawled information is stored in a database (Currently MySQL, PostgreSQL, and MSSQL are supported). The second part is a graphical (Java Swing) distributed grid/volunteer computing client, including user computer state detection, based on JADIF Project. The Web search engine is written in PHP. It comes with a Content Management System, user language detection and multi-language support, and templates using Smarty, including an application framework that is partly forked from Joomla 1.5, so that Joomla components can be adapted quickly.
osclass is Web-based software for creating classified ads. The ads can be general or specialized by category (such as cars, jobs, or real estate). It is easy to install on private/dedicated or shared hosting services. It has lots of options for customization. It is integrated with the TinyMCE WYSIWYG editor. It can be extended through modules (for vertical ads) or extensions (plugins). It supports multiple languages. The look and feel are can be changed with themes. It is SEO friendly. It has an integrated search engine with full text support. An unlimited number of categories and sub-categories can be defined. It has anti-spam facilities that are enabled out of the box (with CAPTCHAs and Askimet).
Large Database Backup is a class to back up large MySQL databases into multiple files. It can list the tables of a given MySQL database, and generate SQL statements to create and insert records with the current values of the table rows, as if it were creating a new database. The class stores the generated SQL statements in files. A full backup process can be split into multiple iterations so as to not exceed the PHP default script execution time limit. Each iteration dumps a limited number of table rows. An iteration can be executed by a script that redirects the page request to itself at the end to proceed to the next iteration.
Change Web Site Mode is a PHP class that can be used to switch the access to sites between test and production versions. Each Web site must have a copy of all files in the production server and in a test server. This class allows the user to switch between the two copies of the site without having to change the URL. It provides a GUI interface for switching between the test mode and production site. The test version may be run in the local machine or in another machine.
Site Checker can be used to find broken links in Web pages. First, it retrieves the list of all pages of the Web site, either static pages or dynamic pages generated from data stored in a database. Then it checks the links in each of the pages to verify if they are on the list. Links to external sites can also be verified by performing HTTP requests to the remote Web servers to check whether the pages still exist.
Id3 Tag Correction and Downloading is a PHP class that can be used to correct tags of MP3 tags of music with records stored in a MySQL database. It scans a MySQL database table with information about music albums to see which albums do not yet have the associated ID3 tags. The class accesses the Web services APIs of sites like audioscrobbler.com, last.fm, lyricwiki.org, and lyricsfly.com to correct the pending music tags and get other details about the stored music.