column

column is a high-performance, columnar, in-memory storage engine written in Go, designed to support fast querying, updates, and iteration with zero allocations and bitmap indexing. It is optimized for cache-friendly data layout, zero heap allocation during querying, and efficient batch updates and deletes. The engine supports SIMD-enabled aggregate functions (e.g., sum, avg, min, max) and filtering operations using bitmap indexing, allowing for fast retrieval and columnar projection.

It offers transaction isolation, allowing for commit and rollback operations, and supports concurrent updates with sharded latches. The system also includes features for row expiration based on time-to-live or expiration columns, atomic merging of values, primary keys for use cases where offsets cannot be used, and a change data stream for consistent commit streaming. Additionally, it supports concurrent snapshotting, enabling the storage of the entire collection into a file.

column leverages a columnar storage approach, organizing data in structures of arrays (SoA), which enhances iteration and filtering efficiency. It also adds bitmap indexing to facilitate complex filter queries using binary operations. The library supports sorted indexes for transient, consistently sorted data, and allows for the storage of complex binary records.

The tool is designed for high performance, as evidenced by benchmarks showing minimal allocation and fast operation times even with large datasets. It also supports streaming changes for real-time data synchronization and offers snapshot and restore functionality for data persistence. The library is open-source and maintained by Roman Atachiants, licensed under the MIT License.