Projects / crawl

crawl

Crawl starts a depth-first traversal of the Web at the specified URLs. It stores all JPEG images that match the configured constraints. Crawl is fairly fast and allows for graceful termination. After terminating crawl, it is possible to restart it at exactly the same spot where it was terminated. It also keeps a persistent database that allows multiple crawls without revisiting sites.

Tags
Licenses
Operating Systems

Recent releases

  •  18 May 2003 02:42

    Release Notes: Crawling is more reliable, and crawl performance has improved for large crawls.

    •  29 Jan 2002 03:47

      Release Notes: A complete rewrite for higher performance, including support for all media types, asynchronous DNS lookups, and optional wait time between hosts. The configuration file specifies permittable size of media objects depending on media-type, etc.

      •  13 Dec 2001 02:36

        Release Notes: This release has portability fixes, and supports downloading of media types other than just images.

        •  03 Jul 2001 20:18

          Release Notes: This release fixes a bug where crawl would stop early when encountering errors on subsequent connections. The verbosity level and number of concurrent connections are tunable now

          •  18 Jun 2001 20:09

            No changes have been submitted for this release.

            Screenshot

            Project Spotlight

            OpenStack4j

            A Fluent OpenStack client API for Java.

            Screenshot

            Project Spotlight

            TurnKey TWiki Appliance

            A TWiki appliance that is easy to use and lightweight.