OpenObserve

# MISSION: Scaffold "OpenObserve" Production Architecture
You are an expert Senior Staff Software Architect and Full-Stack Engineer. Your mission is to scaffold and implement a production-grade, highly reliable, and modular codebase following the proven architecture of **OpenObserve**.
---
## 1. PROJECT SPECIFICATIONS & BENCHMARK
- **Reference Architecture**: OpenObserve
- **What It Does**: OpenObserve is an open-source, AGPL-3.0 Rust observability platform that stores logs, metrics, traces and RUM data as Parquet files on S3-compatible storage, queried with SQL and PromQL from a single binary.
- **Domain & Category**: Cloud-Native Observability Engine
- **Production Scale**: 6-20 People
- **Development Mode**: CLASSIC
- **Architectural Rationale**: Storing telemetry as Parquet on object storage instead of a traditional database, paired with DataFusion for SQL and dedicated PromQL crates, is what the project credits for its lower storage cost, while Cargo feature flags keep enterprise-only code optional inside the same open-core binary.
- **Live Website Reference**: https://openobserve.ai
- **Source Repository**: https://github.com/openobserve/openobserve
---
## 2. PRODUCTION TECH STACK
- **Full Stack Array**: Rust, TypeScript, Vue, Vite, Axum, Tokio, gRPC, OpenTelemetry, Prometheus, Docker, Kubernetes, DataFusion, Parquet, Arrow, S3
- **Primary Language(s)**: TypeScript, Rust, Vue
- **License of the reference repo**: AGPL-3.0
- **Frontend**: Vue — Vue 3 single-page app for logs, traces, metrics, dashboards, alerts and pipelines.; TypeScript — Primary language for the web UI, type-checked with vue-tsc across several tsconfig targets.; Vite — Builds and serves the UI, mounted under the /web/ base path (npm run dev runs "vite --base /web/").; Monaco Editor — Powers SQL/VRL/PromQL editing surfaces in the query builder and pipeline editor.
- **Backend & APIs**: Rust — The entire server is one Rust binary; the workspace root Cargo.toml lists 40+ internal crates under src/.; Axum — HTTP framework for the ingest, search and management APIs (axum, axum-extra, axum-server workspace deps).; Tokio — Async runtime underlying the server and all workspace crates.; gRPC — tonic/tonic-prost provide inter-node RPC for the clustered/HA deployment mode.; OpenTelemetry — Native OTLP ingestion (opentelemetry, opentelemetry-otlp, opentelemetry-proto workspace deps) as the primary ingest protocol.
- **Data & persistence**: Parquet — Logs, metrics and traces are stored as Parquet files; the parquet crate is a direct workspace dependency.; Arrow — Columnar in-memory format used across ingestion, search and compaction (arrow, arrow-flight, arrow-json workspace deps).; DataFusion — SQL query engine executed over the Parquet/Arrow data for logs and trace search.; Tantivy — Full-text search index used to accelerate log search (tantivy, tantivy-fst workspace deps).
- **Infrastructure & deploy**: Docker — deploy/build/Dockerfile is a multi-stage build: Node builds the web UI, then Rust compiles the release binary into a Debian runtime image.; Kubernetes — deploy/k8s holds Kubernetes manifests for clustered/HA deployment.; GitHub Actions — db-testing.yml, api-testing.yml, cargo-deny.yml and dozens more workflows gate every PR before merge.
- **Tooling, testing & ops**: Model Context Protocol — src/mcp is a workspace crate; the main binary depends on openobserve-mcp to serve an MCP endpoint for AI agents.
---
## 3. KEY ARCHITECTURAL DECISIONS (audited from https://github.com/openobserve/openobserve @ b85030e)
1. **Enterprise features are optional Cargo feature flags, not a separate repo**: Cargo.toml declares an "enterprise" feature that pulls in an optional o2_enterprise dependency across most crates (audit, common, core, db, jobs, search_service), plus "cloud" and "vectorscan" features layered on top, so the same open-core binary can be built with or without proprietary code.
2. **Telemetry data lands as Parquet on S3-compatible storage, not a database**: The root Cargo.toml depends directly on parquet, arrow and object_store, and the README attributes the "140x lower storage cost" claim to this Parquet-on-S3 design rather than a row-oriented database, with DataFusion executing SQL queries over the resulting files.
3. **Logs and traces query with SQL; metrics query with SQL or PromQL**: Dedicated promql and promql-service workspace crates sit alongside the DataFusion-based search and search_service crates in Cargo.toml, matching the README’s claim of supporting both query languages instead of a proprietary one.
4. **Full-text search runs on a Tantivy index layered over Parquet**: tantivy and tantivy-fst are direct workspace dependencies used by the tantivy_utils crate (a workspace member in Cargo.toml) alongside the DataFusion/Parquet search path, giving log search an inverted index without a separate search cluster.
5. **The web UI is one Vue SPA served under a fixed /web/ base path**: web/package.json’s dev and build scripts pass --base /web/ to Vite, and postbuild copies the compiled assets into dist/src/assets, so the UI is designed to be served by the same Rust binary rather than a separate static host.
6. **An MCP server ships as a first-party workspace crate**: src/mcp appears as a workspace member in Cargo.toml, and the top-level openobserve binary depends on openobserve-mcp.workspace = true, making Model Context Protocol access to OpenObserve data part of the shipped binary rather than an external add-on.
---
## 4. NON-NEGOTIABLE ARCHITECTURAL GUARDRAILS
1. **Monorepo & Modular Separation**:
   - Structure as a Turborepo monorepo with strict package boundaries:
     - `apps/web`: Application UI, routing, layouts, and server endpoints.
     - `packages/ui`: Shared design tokens, CSS variables, and Radix UI primitive components.
     - `packages/db`: Database schemas, client singleton, declarative migrations, and seed scripts.
     - `packages/config`: Shared TypeScript, ESLint, and build configurations.
2. **Strict Type Safety & Zero `any` Policy**:
   - Enable `strict: true`, `noImplicitAny: true`, and `strictNullChecks: true`.
   - Validate ALL external inputs, API request bodies, and query parameters with **Zod** schemas before execution.
3. **Frontend & Rendering Guidelines**:
   - Utilize Vite 6 with TanStack Router for fully type-safe routing. Manage server state and caching via TanStack Query v5 with optimistic UI updates.
4. **Design System & Aesthetics**:
   - Keep every color, radius, shadow and font in a single token file (CSS variables) and consume tokens everywhere; never hardcode hex values in components.
   - Prefer crisp 1px borders and one subtle shadow scale over blurry default shadows. Pair one sans-serif for body/headings with one monospace for tags, badges, metrics, and code.
5. **Data Layer & Reliability**:
   - Write declarative schema definitions with foreign keys, composite indexes on queried filters, and automated timestamp triggers.
   - Use connection pooling and prepared statements for serverless database execution.
---
## 5. STEP-BY-STEP SCAFFOLDING ROADMAP
- **Phase 1: Workspace & Root Config**: Initialize package manager, monorepo configuration (`turbo.json`, `tsconfig.base.json`, `package.json`).
- **Phase 2: Database Schema & Client**: Set up the data layer (Parquet, Arrow, DataFusion, Tantivy): client, connection pool, models, and migration scripts.
- **Phase 3: Design Tokens & UI Primitives**: Build accessible `Button`, `Input`, `Card`, `Badge`, and layout wrappers inside `packages/ui`.
- **Phase 4: Core Application Routes & Handlers**: Implement primary authentication, user session handling, and application routes.
- **Phase 5: Quality Assurance & Build Verification**: Run `tsc --noEmit`, ESLint, Prettier, and smoke test suites to ensure zero compilation or runtime errors.
---
## 6. EXECUTION INSTRUCTIONS
1. Review all specifications, architectural guardrails, and stack choices above.
2. Present the full monorepo directory tree structure.
3. Systematically generate the complete, production-ready codebase according to the 5-phase roadmap above — starting with the root workspace setup, followed by the database schema, UI design system package, and full-stack application routes until the repository is fully scaffolded and ready to run.