22 projects tagged "storage"
TomP2P is a P2P-based high performance key-value pair storage library. Each peer has a table (either disk-based or memory-based) to store its values. A single value can be queried or updated with a secondary key. The underlying communication framework uses Java NIO to handle many concurrent connections.
Memventi is a Venti daemon. It speaks the same Venti protocol as the real Venti in Plan 9 from Bell Labs. It is a storage server that stores data blocks up to 56KB using its SHA-1 hash (called its score) to address it. It keeps a mapping of score to disk location in memory (in a memory-efficient manner). Blocks written cannot be removed, and blocks are only written once. Memventi writes new blocks to an append-only file, thereby making file corruption due to bugs practically impossible.
chunkd is a simple storage service based on the GET, PUT, and DELETE operations. It is meant to be used in building larger, replicated, distributed storage systems. Clients connect via TCP and remotely manage storage. Typical applications include distributed filesystems and other distributed storage applications.
xNBD is yet another NBD (Network Block Device) server program, which is fully compatible with the NBD client driver in the Linux kernel. In comparison with the original NBD server, it provides possibly better I/O performance by using mmap(), concurrent access from multiple clients, distributed Copy-on-Write disk, and live storage migration for virtual machines.
KVZone is an evaluation tool for comparing performance of key-value stores. It provides various configuration parameters to match the expected application workload. It provides adapters for existing key value stores such as TokyoCabinet and simple databases such as sqlite3 and BerkeleyDB that can be used as key-value stores. New adapters can easily be written to evaluate other key-value stores.
S3QL is a file system that stores all its data online. It supports Amazon S3, Google Storage, and OpenStack and effectively provides you with a hard disk of dynamic, infinite capacity that can be accessed from any computer with Internet access. S3QL provides a standard, full featured Unix file system that is conceptually indistinguishable from any local file system. Additional features include compression, encryption, data de-duplication, immutable trees, and snapshotting, which make it especially suitable for online backup and archiving. The design favors simplicity and elegance over performance and feature-creep. Care has been taken to make the source code as readable and serviceable as possible. Solid error detection, error handling, and extensive automated test cases are provided.
Opendedup is a project that develops a deduplication based filesystem for Windows and Linux called SDFS. SDFS is designed to support the unique needs of virtual environments and supports enhanced functionality for VMWare, Xen, and KVM. It can deduplicate a petabyte or more of data. It supports over 3TB per gigabyte of memory at a 128k chunk size. It can perform deduplication/reduplication at a line speed of 250 MB/s or more. VMware support: it works with vms, and can deduplicate at 4k block sizes. This is required to deduplicate Virtual Machines effectively Deduplicated data can be stored locally, on the network across multiple nodes, or in the cloud. The filesystem can deduplicate inline or periodically based on needs. This can be changed on the fly. There is support for file or folder level snapshots.
piler is an email archiving application with all the features required for an average company. The most prominent features are full text searching, deduplication, compression, encryption, digital fingerprinting, policy rules, access control, a built-in SMTP server, and Active Directory/openldap support.