Webtools 4 larbin

Larbin is a Web crawler intended to fetch a large number of Web pages to fill the database of a search engine. With a network fast enough, it should be able to fetch more than 100 millions pages on a standard PC. This set of PHP and Perl scripts, called webtools4larbin, can handle the output of Larbin.