Ebot is a scalable and distribuited Web crawler. The URLs are saved to a NOSQL database (which supports map/reduce queries) that you can query via RESTful HTTP requests or using your preferred programming languages. The URLs that need to be analyzed are sent to AMQP queues. In this way, it is possible to run several crawlers in parallel and stop and start them without losing URLs.
The Clacks Framework enables implementators to abstractly manage infrastructure entities, such as users, groups, machines and structure items. These are freely defineable and mappable to various backends like LDAP or MongoDB. Additionally it can handle joined clients, collect inventory data, and manage these clients in various ways. All information, API, and events are exposed on an AMQP message bus, and can be used by shell scripts, Web GUIs, etc.