Crowd.dev

# MISSION: Scaffold "Crowd.dev" Production Architecture
You are an expert Senior Staff Software Architect and Full-Stack Engineer. Your mission is to scaffold and implement a production-grade, highly reliable, and modular codebase following the proven architecture of **Crowd.dev**.
---
## 1. PROJECT SPECIFICATIONS & BENCHMARK
- **Reference Architecture**: Crowd.dev
- **What It Does**: crowd.dev (LFX Community Data Platform) is an open-source platform for aggregating and analyzing community and developer activity data across many integrations, built as a TypeScript monorepo of Temporal-orchestrated worker services.
- **Domain & Category**: Developer Data Platform
- **Production Scale**: 6-20 people
- **Development Mode**: CLASSIC
- **Architectural Rationale**: Decomposing ingestion, enrichment, and merging into many independent Temporal-orchestrated worker services under services/apps lets crowd.dev scale each pipeline stage separately, while shared @crowd/* libraries keep database, queue, search, and cache access consistent across all of them.
- **Live Website Reference**: https://crowd.dev
- **Source Repository**: https://github.com/CrowdDotDev/crowd.dev
---
## 2. PRODUCTION TECH STACK
- **Full Stack Array**: TypeScript, Vue, Vite, Pinia, Element Plus, Node.js, PostgreSQL, Sequelize, Redis, Temporal, OpenSearch, Kafka, Python, Docker
- **Primary Language(s)**: TypeScript, Vue, Python
- **License of the reference repo**: Apache-2.0
- **Frontend**: Vue — Component framework for the community data platform dashboard in frontend/.; Vite — Dev server and build tool for the frontend, with mode-specific builds for localhost, staging, and prod.; Pinia — State management store for the Vue dashboard.; Element Plus — Vue UI component library used for tables, forms, and layout across the dashboard.; Socket.IO — Client-side realtime connection to backend services, via socket.io-client.
- **Backend & APIs**: Node.js — Runtime for the API and the many worker services under services/apps/*, run via tsx.; Sequelize — ORM and CLI-driven migrations for the backend PostgreSQL schema.; Temporal — Orchestrates long-running and multi-step workflows such as data sink processing, via @crowd/temporal.; Python — Implements standalone integration services, including the git integration worker.
- **Data & persistence**: PostgreSQL — Primary relational database for members, organizations, and activities.; OpenSearch — Search and indexing backend used by the data-sink-worker via the @crowd/opensearch package.; Redis — Caching and shared state across services via the @crowd/redis workspace package.; Kafka — Event streaming backbone for integration and worker services, provisioned via a local scaffold.
- **Infrastructure & deploy**: Docker — Scaffold Dockerfiles provision local Kafka Connect, Redis, Nginx, Temporal, and packages-db services.; pnpm — Workspace and package manager for the monorepo, pinned via packageManager in package.json.
---
## 3. KEY ARCHITECTURAL DECISIONS (audited from https://github.com/CrowdDotDev/crowd.dev @ 1c661ca)
1. **The backend is decomposed into many single-purpose workers, not one API monolith**: services/apps/ contains a package.json per worker (data_sink_worker, integration_run_worker, integration_stream_worker, entity_merging_worker, categorization_worker, members_enrichment_worker, organizations_enrichment_worker, automatic_onboarding_worker, and more), each independently startable and each depending on shared @crowd/* workspace libraries.
2. **Temporal orchestrates cross-service workflows instead of ad-hoc job chaining**: services/apps/data_sink_worker/package.json depends on @crowd/temporal, and scripts/scaffold/temporal/Dockerfile provisions a local Temporal server, giving multi-step data pipelines (ingest, enrich, merge) durable, resumable workflow state.
3. **Sequelize with a CLI-driven migration path manages the PostgreSQL schema**: backend/package.json defines sequelize-cli:source and sequelize-cli:build scripts that run the sequelize CLI against src/database/migrations, compiled through the same build step (tsc) used for the backend API and job-generator services.
4. **One Python service handles git integration, alongside an otherwise all-TypeScript backend**: services/apps/git_integration/pyproject.toml and services/apps/mailing_list_integration/pyproject.toml are the only Python manifests in the services/apps/ tree, isolating integrations that benefit from Python tooling (for example git log parsing) from the rest of the Node.js worker fleet.
5. **Local infra (Kafka, Redis, Temporal, Nginx) is provisioned via scaffold Dockerfiles**: scripts/scaffold/ contains dedicated Dockerfiles for kafka-connect, redis, nginx, and temporal, plus scripts/packages-db/Dockerfile, giving contributors a reproducible local dependency stack without a single monolithic compose file.
6. **OpenSearch is a first-class dependency of the data pipeline, not an optional add-on**: The @crowd/opensearch workspace package is a direct dependency of services/apps/data_sink_worker/package.json, meaning ingested community activity data is indexed into OpenSearch as part of the core data-sink workflow, not a separate opt-in step.
7. **Frontend builds are environment-specific rather than a single production build**: frontend/package.json defines build:localhost, build:staging, and build:production as separate Vite mode builds, and includes component documentation tooling (docs:storybook, docs:tailwind via tailwind-config-viewer) alongside the app build.
---
## 4. NON-NEGOTIABLE ARCHITECTURAL GUARDRAILS
1. **Monorepo & Modular Separation**:
   - Structure as a Turborepo monorepo with strict package boundaries:
     - `apps/web`: Application UI, routing, layouts, and server endpoints.
     - `packages/ui`: Shared design tokens, CSS variables, and Radix UI primitive components.
     - `packages/db`: Database schemas, client singleton, declarative migrations, and seed scripts.
     - `packages/config`: Shared TypeScript, ESLint, and build configurations.
2. **Strict Type Safety & Zero `any` Policy**:
   - Enable `strict: true`, `noImplicitAny: true`, and `strictNullChecks: true`.
   - Validate ALL external inputs, API request bodies, and query parameters with **Zod** schemas before execution.
3. **Frontend & Rendering Guidelines**:
   - Utilize Vite 6 with TanStack Router for fully type-safe routing. Manage server state and caching via TanStack Query v5 with optimistic UI updates.
4. **Design System & Aesthetics**:
   - Keep every color, radius, shadow and font in a single token file (CSS variables) and consume tokens everywhere; never hardcode hex values in components.
   - Prefer crisp 1px borders and one subtle shadow scale over blurry default shadows. Pair one sans-serif for body/headings with one monospace for tags, badges, metrics, and code.
5. **Data Layer & Reliability**:
   - Write declarative schema definitions with foreign keys, composite indexes on queried filters, and automated timestamp triggers.
   - Use connection pooling and prepared statements for serverless database execution.
---
## 5. STEP-BY-STEP SCAFFOLDING ROADMAP
- **Phase 1: Workspace & Root Config**: Initialize package manager, monorepo configuration (`turbo.json`, `tsconfig.base.json`, `package.json`).
- **Phase 2: Database Schema & Client**: Set up the data layer (PostgreSQL, OpenSearch, Redis, Kafka): client, connection pool, models, and migration scripts.
- **Phase 3: Design Tokens & UI Primitives**: Build accessible `Button`, `Input`, `Card`, `Badge`, and layout wrappers inside `packages/ui`.
- **Phase 4: Core Application Routes & Handlers**: Implement primary authentication, user session handling, and application routes.
- **Phase 5: Quality Assurance & Build Verification**: Run `tsc --noEmit`, ESLint, Prettier, and smoke test suites to ensure zero compilation or runtime errors.
---
## 6. EXECUTION INSTRUCTIONS
1. Review all specifications, architectural guardrails, and stack choices above.
2. Present the full monorepo directory tree structure.
3. Systematically generate the complete, production-ready codebase according to the 5-phase roadmap above — starting with the root workspace setup, followed by the database schema, UI design system package, and full-stack application routes until the repository is fully scaffolded and ready to run.