5 projects tagged "Deduplication"

Pop	323.89
Vit	21.30

Pcompress is a utility to do compression/decompression and deduplication in parallel by splitting input data into chunks. It has a modular structure and includes support for multiple algorithms like LZMA, Bzip2, PPMD, LZ4, etc., with KECCAK/BLAKE2/SHA-256/512 chunk checksums. SSE optimizations for the bundled LZMA are included. It also implements chunk-level Content-Aware Deduplication and Delta Compression features based on a Rabin Fingerprinting scheme. It has low metadata overhead and overlaps I/O and compression to achieve maximum parallelism. It has AES encryption capability and uses Scrypt from Tarsnap to generate per-session unique keys from passwords. It can work in pipe mode, reading from stdin and writing to stdout. It also provides some adaptive compression modes in which a suitable algorithm is chosen per chunk based on heuristics.

Pop	83.07
Vit	1.44

BlackHole is an data de-duplicating network block device that also supports mirroring, snapshots, and support for multiple LUNs using the same data store. It is filesystem agnostic and has been tested with ext2/3/4, NTFS, ReiserFS, and the Oracle Cluster File System (OCFS2). It supports encryption, compression, and multiple storage backends. The hashing scheme used is user configurable. The program exports an NBD device which can be mounted in Linux and GNU/Hurd.

Pop	131.33
Vit	8.06

Duke is a fast and flexible record linkage engine. It does not use the traditional blocking (sort by key) approach, but instead relies on Lucene. This makes it high-performance (able to process 1,000,000 records in ~10 minutes). Duke can be run from the command line, but also has an API allowing incremental linking applications to be built easily. It supports reading data from CSV, JDBC, SPARQL, and NTriples, and also supports a number of string comparators and string normalizers.

Pop	177.68
Vit	4.95

Lessfs is a high performance inline data deduplicating file system for Linux. Lessfs complies to the POSIX standard and is very useful for backup purposes as well as providing storage for virtual machine images. Although lessfs is a file system that is implemented in user space with FUSE, it offers decent performance. Lessfs is capable of handling data rates up to 350MB/sec. It supports filesystem encryption.

Pop	433.43
Vit	35.04

Arkeia Network Backup is designed for organizations that require fast, easy-to-use, and affordable data protection. It backs up critical data to disk, tape, and cloud storage. Arkeia protects all major virtual platforms including VMware, Hyper-V, XenServer, and more than 200 physical platforms including Windows, Mac OS X, Linux, Netware, most UNIX flavors, and BSDs. The company’s source-side Progressive Deduplication technology helps users realize better performance at a lower cost by reducing data volumes. Arkeia’s deduplication is crucial to accelerating replication of on-premise backups to private or public clouds.