Project category
Data & Storage Infra tech stacks
Databases, object storage, data pipelines, and query engines. 15 production examples.
Production Data & Storage Infra projects
- Headless CSV Ingestion Engine Classic elekcsv Keeping parsing, compiled-validator generation, fuzzy column mapping and bitmap error tracking in one dependency-free core package lets every UI framework reuse identical import logic instead of six separate implementations.
- S3-Compatible Object Storage Classic RustFS RustFS splits a MinIO-style object store into dozens of single-purpose Rust crates behind one binary, keeps extra protocols behind cargo features, and moves inter-node traffic to gRPC, so storage, healing and IAM evolve separately.
- Distributed Blob & File Storage Classic SeaweedFS Decoupling directory metadata from physical volume storage eliminates central metadata bottlenecks, enabling massive horizontal scaling for small and large files alike.
- Columnar Real-Time Analytical Database (OLAP) Classic ClickHouse A single CMake super-project that vendors ~150 native dependencies (Arrow, gRPC, AWS/Azure/GCS SDKs, NuRaft, Delta Lake kernel) under contrib/ lets ClickHouse ship a largely self-contained, dependency-free binary and Docker image.
- Instant GraphQL Engine & Unified Data Access Classic Hasura Hasura runs two independently evolving query engines from one repository — a Haskell V2 server and a Rust V3 engine — and extends database support through an out-of-process Data Connector protocol, so a new database backend plugs in as a separate agent instead of new code inside either engine.
- High-Performance S3 Object Storage Classic MinIO Implementing erasure coding, IAM, and multi-protocol event notifications directly in the Go server, instead of delegating to RAID or an external message broker, keeps a deployment to a single binary while still integrating with existing enterprise identity and messaging systems through native client libraries.
- Automated GraphQL API Engine Classic PostGraphile Synthesizing GraphQL APIs directly from PostgreSQL catalogs eliminates manual resolver boilerplate while Grafast planning solves N+1 query overhead at the database layer.
- Geo-Distributed Object Storage Engine Classic Garage Garage splits RPC, replication, storage and each API protocol into separate Rust crates around a pluggable embedded metadata engine (LMDB, SQLite or Fjall), so a geo-distributed cluster can tolerate node and network failures without depending on an external coordination service or database.
- Time-Series SQL Database Classic QuestDB Combining a zero-GC Java engine with C++/Rust vector acceleration gives QuestDB ultra-low latency ingestion and microsecond SQL queries over billions of metrics.
- Multi-Threaded In-Memory Key-Value Store Classic Dragonfly A fiber-based, shared-nothing execution model (one thread and one keyspace shard per core) lets Dragonfly scale vertically across cores, which single-threaded Redis cannot do without external sharding, while staying protocol-compatible so existing clients need no changes.
- Multi-Model Real-Time Database Classic SurrealDB SurrealDB puts document, graph, relational and vector models behind one SurrealQL engine in a single Rust binary that runs embedded, in the browser via WebAssembly, or as a server over pluggable storage (RocksDB, SurrealKV, TiKV), with WebSocket live queries and row-level permissions built into the core.
- Local-First Sync Engine Classic ElectricSQL Postgres logical replication and shape-based HTTP sync live in a dedicated Elixir OTP service so any TypeScript, React, or Elixir client can subscribe to partial replicas without each client re-implementing replication.
- Developer Data Platform Classic Crowd.dev Decomposing ingestion, enrichment, and merging into many independent Temporal-orchestrated worker services under services/apps lets crowd.dev scale each pipeline stage separately, while shared @crowd/* libraries keep database, queue, search, and cache access consistent across all of them.
- Data Pipeline Engine Hybrid Mage AI A Tornado Python server handles orchestration over a SQLAlchemy and Alembic-modeled Postgres database, while the pipeline editor ships as an independent Next.js frontend, and Spark or dbt support is layered in through opt-in Docker images and bundled adapters.
- Search Engine Classic Meilisearch Keeping the milli search engine as a dependency-free library separate from the actix-web server crate lets ranking and indexing logic be tested and reused independently of HTTP, while per-index LMDB environments make each index independently portable and snapshot-able.
AGENTS.md rules and agent skills for Data & Storage Infra
nextjs-clickhouse-analytics/AGENTS.mdNext.js + ClickHouse + Prisma (Analytics & Observability)Architecture guidelines for high-volume event ingestion and analytical queries using ClickHouse alongside a PostgreSQL/Prisma control plane. advancedrust-axum-postgres/AGENTS.mdRust + Axum + PostgreSQL API (sqlx & Tokio)Rust API and backend rules: Axum 0.8 on Tokio, compile-time SQL checks with sqlx 0.9, Tower middleware, typed errors and a cargo check loop for AI agents. advanced
Questions about Data & Storage Infra stacks
How many Data & Storage Infra tech stacks are in the directory?
15 verified production Data & Storage Infra projects are currently indexed, each with a full architecture breakdown and copy-paste AI scaffolding prompt.
What AGENTS.md rules or agent skills apply to Data & Storage Infra projects?
Next.js + ClickHouse + Prisma (Analytics & Observability), Rust + Axum + PostgreSQL API (sqlx & Tokio) are battle-tested architecture rules for this category, each with a full AGENTS.md, CLAUDE.md, Cursor .mdc file and a Claude Code SKILL.md.
How do I get a custom stack recommendation for a Data & Storage Infra project?
Take the STACK IT FAST architecture survey and select "Data & Storage Infra" as your project type to get a ranked rule recommendation and an instant AI scaffolding prompt tailored to a "Data & Storage Infra" project.