Mage AI

# MISSION: Scaffold "Mage AI" Production Architecture
You are an expert Senior Staff Software Architect and Full-Stack Engineer. Your mission is to scaffold and implement a production-grade, highly reliable, and modular codebase following the proven architecture of **Mage AI**.
---
## 1. PROJECT SPECIFICATIONS & BENCHMARK
- **Reference Architecture**: Mage AI
- **What It Does**: Mage AI (Mage OSS) is a self-hosted, notebook-style data pipeline tool for building ETL, ELT, and dbt workflows in Python, SQL, or R, run locally via Docker, pip, or conda.
- **Domain & Category**: Data Pipeline Engine
- **Production Scale**: 6-20 people
- **Development Mode**: HYBRID
- **Architectural Rationale**: A Tornado Python server handles orchestration over a SQLAlchemy and Alembic-modeled Postgres database, while the pipeline editor ships as an independent Next.js frontend, and Spark or dbt support is layered in through opt-in Docker images and bundled adapters.
- **Live Website Reference**: https://www.mage.ai
- **Source Repository**: https://github.com/mage-ai/mage-ai
---
## 2. PRODUCTION TECH STACK
- **Full Stack Array**: Python, TypeScript, Next.js, PostgreSQL, Redis, SQLAlchemy, Alembic, Docker, Kubernetes, dbt, Apache Spark, Tornado
- **Primary Language(s)**: Python, TypeScript
- **License of the reference repo**: Apache-2.0
- **Frontend**: Next.js — The mage_ai/frontend app provides the block-based pipeline editor and notebook UI.
- **Backend & APIs**: Tornado — mage_ai/server runs a Python Tornado web server exposing the API and websockets to the frontend.; SQLAlchemy — ORM for the orchestration database (pipelines, runs, blocks) under mage_ai/orchestration.; Alembic — Manages schema migrations for the orchestration database.
- **Data & persistence**: PostgreSQL — Default orchestration datastore, provisioned via the pgvector/pgvector image in dev compose.; Redis — Used for caching and coordination alongside the orchestration database.; dbt — dbt-core plus per-warehouse adapters let pipelines run dbt models as blocks.
- **Infrastructure & deploy**: Docker — Dockerfile builds the all-in-one server and frontend image; Spark/Databricks variants exist too.; Kubernetes — Provides a sample Kubernetes deployment manifest for running Mage in a cluster.; Apache Spark — Spark and custom-Spark Dockerfiles bundle PySpark for large-scale pipeline execution.
---
## 3. KEY ARCHITECTURAL DECISIONS (audited from https://github.com/mage-ai/mage-ai @ 1912c29)
1. **One Python package ships both the orchestrator and the integrations library**: The repo root requirements.txt and pyproject.toml define the mage-ai package (mage_ai/), while mage_integrations/ is a separately packaged Singer-based connector library (mage_integrations/pyproject.toml) installed as the [all] extra in the Dockerfile.
2. **The frontend is a separate Next.js app talking to a Tornado backend**: mage_ai/frontend/package.json is an independent Next.js/React app built with next build/next export, compiled into static assets and served by the Python Tornado server in mage_ai/server rather than rendered server-side by Next.js itself.
3. **dbt is a first-class block type with adapters bundled for every major warehouse**: requirements.txt pins dbt-core alongside dbt-bigquery, dbt-snowflake, dbt-redshift, dbt-spark, and dbt-postgres, so pipelines can run existing dbt projects as blocks without a separate dbt installation.
4. **Spark support ships as opt-in Docker variants, not the default image**: dev.spark.Dockerfile and integrations/spark/Dockerfile install OpenJDK and PySpark on top of the base mageai/mageai image, keeping the default Docker image lighter for users who do not need Spark.
5. **Postgres with pgvector is the default orchestration database in dev**: docker-compose.yml's database service uses the pgvector/pgvector:0.6.0-pg16 image rather than plain postgres, making vector-backed features available without a separate extension install step.
---
## 4. NON-NEGOTIABLE ARCHITECTURAL GUARDRAILS
1. **Monorepo & Modular Separation**:
   - Structure as a Turborepo monorepo with strict package boundaries:
     - `apps/web`: Application UI, routing, layouts, and server endpoints.
     - `packages/ui`: Shared design tokens, CSS variables, and Radix UI primitive components.
     - `packages/db`: Database schemas, client singleton, declarative migrations, and seed scripts.
     - `packages/config`: Shared TypeScript, ESLint, and build configurations.
2. **Strict Type Safety & Zero `any` Policy**:
   - Enable `strict: true`, `noImplicitAny: true`, and `strictNullChecks: true`.
   - Validate ALL external inputs, API request bodies, and query parameters with **Zod** schemas before execution.
3. **Frontend & Rendering Guidelines**:
   - Utilize Next.js 15 App Router: default to React Server Components (RSC) for data fetching. Handle mutations through Server Actions with Zod validation. Keep 'use client' directives restricted strictly to interactive leaf components.
4. **Design System & Aesthetics**:
   - Keep every color, radius, shadow and font in a single token file (CSS variables) and consume tokens everywhere; never hardcode hex values in components.
   - Prefer crisp 1px borders and one subtle shadow scale over blurry default shadows. Pair one sans-serif for body/headings with one monospace for tags, badges, metrics, and code.
5. **Data Layer & Reliability**:
   - Write declarative schema definitions with foreign keys, composite indexes on queried filters, and automated timestamp triggers.
   - Use connection pooling and prepared statements for serverless database execution.
---
## 5. STEP-BY-STEP SCAFFOLDING ROADMAP
- **Phase 1: Workspace & Root Config**: Initialize package manager, monorepo configuration (`turbo.json`, `tsconfig.base.json`, `package.json`).
- **Phase 2: Database Schema & Client**: Set up the data layer (PostgreSQL, Redis, dbt): client, connection pool, models, and migration scripts.
- **Phase 3: Design Tokens & UI Primitives**: Build accessible `Button`, `Input`, `Card`, `Badge`, and layout wrappers inside `packages/ui`.
- **Phase 4: Core Application Routes & Handlers**: Implement primary authentication, user session handling, and application routes.
- **Phase 5: Quality Assurance & Build Verification**: Run `tsc --noEmit`, ESLint, Prettier, and smoke test suites to ensure zero compilation or runtime errors.
---
## 6. EXECUTION INSTRUCTIONS
1. Review all specifications, architectural guardrails, and stack choices above.
2. Present the full monorepo directory tree structure.
3. Systematically generate the complete, production-ready codebase according to the 5-phase roadmap above — starting with the root workspace setup, followed by the database schema, UI design system package, and full-stack application routes until the repository is fully scaffolded and ready to run.