Ebot is a scalable and distribuited Web crawler. The URLs are saved to a NOSQL database (which supports map/reduce queries) that you can query via RESTful HTTP requests or using your preferred programming languages. The URLs that need to be analyzed are sent to AMQP queues. In this way, it is possible to run several crawlers in parallel and stop and start them without losing URLs.
Zato is an Enterprise Service Bus (ESB) and app server written by pragmatists for pragmatists. It provides an HA load-balancer, hot-deployment, and hot-reconfiguration almost everywhere (forget server restarts). It has a browser-based GUI, CLI, and API (forget XML configuration). It supports many protocols, industry standards, and data formats, including HTTP, JSON, SOAP, REST, AMQP, JMS WebSphere MQ, ZeroMQ, Redis, SQL, and FTP.