# Django + PostgreSQL + Redis + Celery — AI Agent Guidelines & Architecture Rules

> Production guidelines for Django 5 enterprise applications, PostgreSQL connection pooling, Redis caching, and Celery async workers.
> Technologies: Django, Python, PostgreSQL, Redis, Celery, React

---

## AGENTS.md
```markdown
# Project Architecture & Guidelines (Django + PostgreSQL + Redis + Celery)

## 1. System Architecture
- **Backend Framework**: Django 5.x with WSGI/ASGI (Gunicorn or Uvicorn).
- **Database**: PostgreSQL with connection pooling (PgBouncer or Django 5.1 native pool).
- **Caching & Broker**: Redis for Django cache backend, session store, and Celery task broker.
- **Async Worker Queue**: Celery for asynchronous background job execution.

## 2. PostgreSQL Connection Management (Critical)
- Configure Django 5.1+ database connection pool in `settings.py`:
  ```python
  DATABASES = {
      'default': {
          'ENGINE': 'django.db.backends.postgresql',
          'NAME': env('DB_NAME'),
          'USER': env('DB_USER'),
          'PASSWORD': env('DB_PASSWORD'),
          'HOST': env('DB_HOST'),
          'PORT': env('DB_PORT', default='5432'),
          'OPTIONS': {
              'pool': {
                  'min_size': 2,
                  'max_size': 10,
                  'timeout': 10,
              },
          },
          'CONN_MAX_AGE': 0, # Must be 0 when using native pooling or PgBouncer
      }
  }
  ```
- If deploying behind PgBouncer in `transaction` mode:
  - Disable server-side cursors: `'DISABLE_SERVER_SIDE_CURSORS': True`.
  - Set `CONN_MAX_AGE = 0` to prevent persistent connections from conflicting with the external pooler.

## 3. Redis & Celery Best Practices
- Configure thread safety and broker connection limits:
  ```python
  CELERY_BROKER_URL = env('REDIS_URL')
  CELERY_RESULT_BACKEND = env('REDIS_URL')
  CELERY_RESULT_BACKEND_THREAD_SAFE = True
  CELERY_TASK_ACKS_LATE = True
  CELERY_WORKER_PREFETCH_MULTIPLIER = 1
  ```
- Use `django-redis` with `BlockingConnectionPool` to prevent unbounded Redis socket creation under heavy load:
  ```python
  CACHES = {
      'default': {
          'BACKEND': 'django_redis.cache.RedisCache',
          'LOCATION': env('REDIS_URL'),
          'OPTIONS': {
              'CLIENT_CLASS': 'django_redis.client.DefaultClient',
              'CONNECTION_POOL_CLASS': 'redis.BlockingConnectionPool',
              'CONNECTION_POOL_CLASS_KWARGS': {'max_connections': 50, 'timeout': 20},
          },
      }
  }
  ```

## 4. Background Job & Task Rules
- All Celery tasks MUST be idempotent. Network glitches can cause worker retries.
- Separate CPU-heavy queues from fast I/O queues (e.g., `high-priority`, `default`, `analytics`).
- Pass record IDs (primary keys) to Celery tasks instead of serialized model instances to prevent stale data race conditions.

## 5. Common Pitfalls to Avoid
- ❌ Calculating connection pool without worker count: Total connections = `(Gunicorn Workers × DB Pool) + (Celery Workers × Concurrency)`. Ensure this is within PostgreSQL `max_connections`.
- ❌ Unindexed Foreign Keys: Always ensure database models define `db_index=True` on filtered columns.
- ❌ Blocking the HTTP Request Loop: Offload any third-party API calls, email dispatches, or heavy reporting to Celery.

## 6. Testing Conventions
- Use `pytest` + `pytest-django` with `--reuse-db` for fast local iteration; drop `--reuse-db` in CI to catch migration drift.
- Use `factory_boy` for model factories instead of fixtures — fixtures rot as schemas evolve, factories don't.
- Test Celery tasks synchronously with `CELERY_TASK_ALWAYS_EAGER = True` in the test settings module, and assert idempotency by calling the task twice.
- Cover connection-pool-sensitive code paths (long transactions, `CONN_MAX_AGE` interactions) with integration tests against a real Postgres instance, not SQLite.

## 7. Git Workflow & PR Conventions
- Conventional Commits (`feat:`, `fix:`, `refactor:`) with the Django app name in scope, e.g. `fix(billing): correct Stripe webhook idempotency key`.
- Every migration file ships in the same PR as the model change that generated it — never a follow-up PR.
- Run `python manage.py makemigrations --check --dry-run` in CI to block unmigrated model changes from merging.
- Require `pytest` and `ruff check .` green before merge; squash-merge to keep `main` bisectable.
```

---

## CLAUDE.md
```markdown
# CLAUDE.md — Django + PostgreSQL + Redis + Celery

Refer to @AGENTS.md for complete connection pooling and task queue rules.

## Common Commands
- `python manage.py runserver` - Start local development server
- `python manage.py migrate` - Apply database migrations
- `python manage.py makemigrations` - Create new database migrations
- `celery -A project worker --loglevel=info` - Start Celery worker process
- `pytest` - Run test suite

## Claude Specific Directives
- When writing database queries, use `select_related()` for foreign keys and `prefetch_related()` for M2M to prevent N+1 queries.
- Pass only model IDs (strings/integers) to Celery tasks.
```

---

## .cursor/rules/stack.mdc
```markdown
---
description: Django + PostgreSQL + Redis + Celery architecture rules
globs: ["**/*.py"]
alwaysApply: true
---

# Django + PostgreSQL + Redis + Celery

- Django 5.x, PostgreSQL with native connection pooling or PgBouncer, Redis for cache + Celery broker.
- `CONN_MAX_AGE = 0` when using native pooling or PgBouncer transaction mode; disable server-side cursors behind PgBouncer.
- All Celery tasks MUST be idempotent — pass primary keys, never serialized model instances, to avoid stale-data races.
- `django-redis` with `BlockingConnectionPool`; set Redis `maxmemory-policy` per use case (never evict keys backing active queues).
- Separate Celery queues by workload class (`high-priority`, `default`, `analytics`) — never mix CPU-bound and I/O-bound tasks in one queue.
- `select_related()` for FKs, `prefetch_related()` for M2M — no N+1 queries.
- `db_index=True` on every filtered/foreign-key column.
- Never block the HTTP request-response cycle with third-party API calls, email sends, or heavy reports — offload to Celery.
```

---

## Architecture Overview & Best Practices
## Architecture Overview

Production conventions for high-throughput, enterprise-scale Python backends using **Django**, **PostgreSQL**, **Redis**, and **Celery**.

### Verified Real-World Adoption

This architecture powers heavy, analytics-intensive platforms including **PostHog**, **Sentry**, **Plane**, **Baserow**, and **Authentik**.

### Key Architectural Nuances

- **Calculated Connection Sizing**: Balances Gunicorn web workers and Celery background processes against PgBouncer pool limits.
- **Idempotent Queue Execution**: Enforces `CELERY_TASK_ACKS_LATE` with atomic database transactions.