ClickHouse

# MISSION: Scaffold "ClickHouse" Production Architecture
You are an expert Senior Staff Software Architect and Full-Stack Engineer. Your mission is to scaffold and implement a production-grade, highly reliable, and modular codebase following the proven architecture of **ClickHouse**.
---
## 1. PROJECT SPECIFICATIONS & BENCHMARK
- **Reference Architecture**: ClickHouse
- **What It Does**: ClickHouse is an open-source, column-oriented SQL database for real-time analytics, built as a large C++/CMake monorepo with a vendored dependency tree covering coordination, streaming ingestion and cloud storage.
- **Domain & Category**: Columnar Real-Time Analytical Database (OLAP)
- **Production Scale**: 20+ people
- **Development Mode**: CLASSIC
- **Architectural Rationale**: A single CMake super-project that vendors ~150 native dependencies (Arrow, gRPC, AWS/Azure/GCS SDKs, NuRaft, Delta Lake kernel) under contrib/ lets ClickHouse ship a largely self-contained, dependency-free binary and Docker image.
- **Live Website Reference**: https://clickhouse.com
- **Source Repository**: https://github.com/ClickHouse/ClickHouse
---
## 2. PRODUCTION TECH STACK
- **Full Stack Array**: C++, CMake, Python, Rust, Docker, gRPC, Protocol Buffers, Kafka
- **Primary Language(s)**: C++, Python, Shell
- **License of the reference repo**: Apache-2.0
- **Backend & APIs**: C++ — Implements the query engine, storage layer and SQL execution, compiled through the root CMake super-project.; Rust — Builds a handful of native components (chcache build-cache client, vendored skim/prql/polyglot/wasmtime crates) wired into the CMake build via Corrosion.; ANTLR — Generates the SQL parser from a formal grammar (vendored ANTLR4 C++ runtime and grammar files).; gRPC — Vendored as an alternative client/server RPC protocol alongside the native TCP interface.; Protocol Buffers — Serialization format vendored for gRPC and other structured interchange.
- **Data & persistence**: ClickHouse Keeper — Built-in Raft coordination service (vendored NuRaft) that can replace ZooKeeper for replication and distributed DDL; ships its own Docker image.; Kafka — Vendored C++ client (cppkafka) backing the Kafka table engine for streaming ingestion.; Apache Arrow — Vendored columnar in-memory format used for Arrow-ecosystem interoperability.; S3 — Vendored AWS C++ SDK components (including an S3-specific library) back S3-compatible object storage for tables and backups.; Delta Lake — Vendored Delta Lake kernel (Rust) lets ClickHouse read Delta Lake tables directly, matching the repo topic "lakehouse".
- **Infrastructure & deploy**: Docker — Packages server, keeper and a minimal "bare" image from Ubuntu/Alpine bases with a shared entrypoint.; CMake — Root build system orchestrating the C++ core plus ~150 vendored dependencies under contrib/ and the Rust workspace.; GitHub Actions — Runs the workflow suite (pull_request, master, nightly fuzzers/jepsen/keeper, weekly official docker, merge queue).; Praktika — ClickHouse's own Python-based CI orchestration framework that composes the GitHub Actions job graph.
---
## 3. KEY ARCHITECTURAL DECISIONS (audited from https://github.com/ClickHouse/ClickHouse @ ddba901)
1. **The C++ core is a single CMake super-project with contrib/ as a vendored dependency tree**: Instead of relying on system packages or a package manager, the root CMakeLists.txt pulls in roughly 150 subdirectories under contrib/ (abseil-cpp, arrow, aws, azure, boost, google-cloud-cpp, grpc, icu, jemalloc, and more), each paired with its own -cmake wrapper directory, so the server builds as a mostly self-contained binary.
2. **Rust components integrate into the CMake build via Corrosion, not a separate pipeline**: rust/workspace/Cargo.toml defines a workspace with skim, prql, polyglot and wasmtime crates, and rust/chcache/Cargo.toml defines a standalone build-cache client; contrib/corrosion bridges Cargo output into the CMake/Ninja build graph so Rust does not need its own release process.
3. **SQL parsing is generated from a formal ANTLR4 grammar, not a hand-rolled parser**: The languages breakdown shows a dedicated ANTLR byte count, and contrib/antlr4-cpp-runtime plus contrib/antlr4-grammars are vendored so the SQL grammar can be regenerated as part of the build rather than maintained as ad hoc parsing code.
4. **Cluster coordination is built in-house (ClickHouse Keeper) instead of requiring ZooKeeper**: docker/keeper/Dockerfile packages a dedicated Keeper image (multi-stage from ubuntu:22.04 to copy glibc, then FROM alpine), backed by the vendored NuRaft Raft-consensus library in contrib/NuRaft, so operators can run replication and distributed DDL coordination without standing up a separate ZooKeeper ensemble.
5. **CI runs on a custom Python orchestrator layered on top of plain GitHub Actions workflows**: ci/praktika holds the orchestration code and ci/pyproject.toml configures ruff/pylint/black for it; the workflow list includes custom_build_praktika.yml alongside more conventional pull_request.yml, master.yml and a large set of nightly_* jobs (fuzzers, jepsen, keeper faults, sqlancer, statistics).
6. **Cloud object storage and lakehouse table formats are vendored as first-class C++/Rust deps**: contrib/aws-c-s3, contrib/azure and contrib/google-cloud-cpp give the engine native S3/Blob/GCS clients, while contrib/arrow, contrib/avro and contrib/delta-kernel-rs let it read external columnar and lakehouse formats directly, consistent with the "lakehouse" and "cloud-native" repo topics.
7. **Server images are minimal, multi-stage Docker builds, not one generic image**: docker/server/Dockerfile is a thin FROM Dockerfile.ubuntu wrapper (kept only so tests/ci/docker_server.py can push unsuffixed tags), docker/keeper/Dockerfile is a separate Alpine-based image with a glibc-donor build stage, and docker/bare/Dockerfile exists as a third minimal variant.
---
## 4. NON-NEGOTIABLE ARCHITECTURAL GUARDRAILS
1. **Monorepo & Modular Separation**:
   - Structure as a Turborepo monorepo with strict package boundaries:
     - `apps/web`: Application UI, routing, layouts, and server endpoints.
     - `packages/ui`: Shared design tokens, CSS variables, and Radix UI primitive components.
     - `packages/db`: Database schemas, client singleton, declarative migrations, and seed scripts.
     - `packages/config`: Shared TypeScript, ESLint, and build configurations.
2. **Strict Type Safety & Zero `any` Policy**:
   - Enable `strict: true`, `noImplicitAny: true`, and `strictNullChecks: true`.
   - Validate ALL external inputs, API request bodies, and query parameters with **Zod** schemas before execution.
3. **Frontend & Rendering Guidelines**:
   - Isolate interactive UI state to leaf components. Keep core pages lightweight and performant.
4. **Design System & Aesthetics**:
   - Keep every color, radius, shadow and font in a single token file (CSS variables) and consume tokens everywhere; never hardcode hex values in components.
   - Prefer crisp 1px borders and one subtle shadow scale over blurry default shadows. Pair one sans-serif for body/headings with one monospace for tags, badges, metrics, and code.
5. **Data Layer & Reliability**:
   - Write declarative schema definitions with foreign keys, composite indexes on queried filters, and automated timestamp triggers.
   - Use connection pooling and prepared statements for serverless database execution.
---
## 5. STEP-BY-STEP SCAFFOLDING ROADMAP
- **Phase 1: Workspace & Root Config**: Initialize package manager, monorepo configuration (`turbo.json`, `tsconfig.base.json`, `package.json`).
- **Phase 2: Database Schema & Client**: Set up the data layer (ClickHouse Keeper, Kafka, Apache Arrow, S3, Delta Lake): client, connection pool, models, and migration scripts.
- **Phase 3: Design Tokens & UI Primitives**: Build accessible `Button`, `Input`, `Card`, `Badge`, and layout wrappers inside `packages/ui`.
- **Phase 4: Core Application Routes & Handlers**: Implement primary authentication, user session handling, and application routes.
- **Phase 5: Quality Assurance & Build Verification**: Run `tsc --noEmit`, ESLint, Prettier, and smoke test suites to ensure zero compilation or runtime errors.
---
## 6. EXECUTION INSTRUCTIONS
1. Review all specifications, architectural guardrails, and stack choices above.
2. Present the full monorepo directory tree structure.
3. Systematically generate the complete, production-ready codebase according to the 5-phase roadmap above — starting with the root workspace setup, followed by the database schema, UI design system package, and full-stack application routes until the repository is fully scaffolded and ready to run.