S3QL is a file system that stores all its data online. It supports Amazon S3, Google Storage, and OpenStack and effectively provides you with a hard disk of dynamic, infinite capacity that can be accessed from any computer with Internet access. S3QL provides a standard, full featured Unix file system that is conceptually indistinguishable from any local file system. Additional features include compression, encryption, data de-duplication, immutable trees, and snapshotting, which make it especially suitable for online backup and archiving. The design favors simplicity and elegance over performance and feature-creep. Care has been taken to make the source code as readable and serviceable as possible. Solid error detection, error handling, and extensive automated test cases are provided.
Weed-FS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files, and to serve the files fast! Instead of supporting full POSIX file system semantics, it implements only a key-file mapping. Instead of managing all file metadata in a central master, it manages file volumes in the central master and lets volume servers manage files and the metadata. This relieves concurrency pressure from the central master and spreads file metadata into volume servers' memories, allowing faster file access with just one disk read operation. It is modelled on Facebook's Haystack design paper. Only 40 bytes of disk storage are required for each file's metadata, and disk reads are O(1).