# Codebase architecture

Markdown version of <https://sam-dumont.github.io/immich-memories/docs/contribute/architecture/>.

## Runtime boundaries

The web UI and terminal use the same CLI workflow: **choose a cut → render it → optionally upload it**. A render consumes the chosen cut; it does not choose footage again. The store keeps reusable facts, settings and history; per-attempt records explain the decisions in one cut.

## Data and service boundaries

Pick a service in the diagram to see the data it holds and what it runs. These are service boundaries: they don't say where the GPU work runs.

**Deployment: One app beside Immich**

Prepare a cut:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (Your app host)
  - **App + CLI**, Choose the cut: Reads Immich metadata and previews, saves reusable facts, then chooses the cut. On GPU and Full tiers, Laya runs in the app, even with a remote preparation service.
  - Storage: Store · cache · output files
- **Your browser** (Desktop or phone)
  - **Web UI**, App URL · authenticated if exposed: Connects to the app, which serves cached media or fetches it from Immich. Configure authentication before opening access to other machines.

1. **Metadata + previews**: Preparation starts with your library; originals remain in Immich.
2. **Browser → app**: UI requests and media go through the app, behind the same login.

The default film needs no model server or GPU. The app keeps the store and outputs; Immich keeps your originals.

Render a film:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (Your app host)
  - **App + CLI**, Assemble the film: Downloads the selected originals and renders locally with FFmpeg.
  - Storage: Store · cache · output files
- **Your browser** (Desktop or phone)
  - **Web UI**, App URL · authenticated if exposed: Connects to the app, which serves cached media or fetches it from Immich. Configure authentication before opening access to other machines.

1. **Selected originals**: The app downloads media for the chosen cut.
2. **Browser → app**: UI requests and media go through the app, behind the same login.

The default film needs no model server or GPU. The app keeps the store and outputs; Immich keeps your originals.

Open the app:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (Your app host)
  - **App + CLI**, App listener · port 8080: Serves the UI and proxies previews on the app listener. For Compose, open the published host port in your browser; the container still listens on 8080. The browser does not receive your Immich API key.
  - Storage: Store · cache · output files
- **Your browser** (Desktop or phone)
  - **Web UI**, App URL · authenticated if exposed: Connects to the app, which serves cached media or fetches it from Immich. Configure authentication before opening access to other machines.

1. **Preview requests**: The app keeps the Immich key on the server.
2. **Browser → app**: UI requests and media go through the app, behind the same login.

The default film needs no model server or GPU. The app keeps the store and outputs; Immich keeps your originals.

**Deployment: App + one GPU service**

Prepare a cut:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (Your app host)
  - **App + CLI**, Choose the cut: Reads Immich metadata and previews, saves reusable facts, then chooses the cut. On GPU and Full tiers, Laya runs in the app, even with a remote preparation service.
  - Storage: Store · cache · output files
- **GPU service** (NVIDIA host · one container)
  - **Facts + captions**, Port 8092 · /facts and /v1: Receives previews for classification and captioning. These routes have no built-in authentication; keep them private.
  - Storage: Model cache · render scratch

1. **Metadata + previews**: Preparation starts with your library; originals remain in Immich.
2. **Preview tiles + frames**: Images go to the caption service; the text reader is a separate role.

One GPU container shares the GPU across phases and releases owned models between phases. Reader and generated-music services are separate choices, not included here.

Render a film:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (Your app host)
  - **App + CLI**, Assemble the film: Hands the chosen cut to the render worker. The UI and run history stay here.
  - Storage: Store · cache · output files
- **GPU service** (NVIDIA host · one container)
  - **Render worker**, Port 8092 · /render: Receives the cut and Immich URL/API key, fetches originals and returns the film. /render requires a bearer token.
  - Storage: Model cache · render scratch

1. **Cut + Immich credentials**: The authenticated render request includes the Immich URL and API key. Use HTTPS across an untrusted network.
2. **Worker → originals**: The worker needs its own network route to Immich. It returns the finished film to the app.
3. **Optional delivery**: After rendering, the app can upload the finished film if you enable delivery.

One GPU container shares the GPU across phases and releases owned models between phases. Reader and generated-music services are separate choices, not included here.

Open the app:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (Your app host)
  - **App + CLI**, App listener · port 8080: Serves the UI and proxies previews on the app listener. For Compose, open the published host port in your browser; the container still listens on 8080. The browser does not receive your Immich API key.
  - Storage: Store · cache · output files
- **Your browser** (Desktop or phone)
  - **Web UI**, App URL · authenticated if exposed: Connects to the app, which serves cached media or fetches it from Immich. Configure authentication before opening access to other machines.

1. **Preview requests**: The app keeps the Immich key on the server.
2. **Browser → app**: UI requests and media go through the app, behind the same login.

One GPU container shares the GPU across phases and releases owned models between phases. Reader and generated-music services are separate choices, not included here.

**Deployment: Distributed Kubernetes services**

Prepare a cut:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (App pod · one replica)
  - **App + CLI**, Choose the cut: Reads Immich metadata and previews, saves reusable facts, then chooses the cut. On GPU and Full tiers, Laya runs in the app, even with a remote preparation service.
  - Storage: Data/cache · output · model PVCs
- **Preparation services** (Pods + supplied reader endpoint)
  - **Inference**, GPU reservation · 1 allocation: Receives sampled frames for classification. The overlay reserves one advertised NVIDIA GPU allocation for this pod.
  - **Caption server**, GPU allocation must be configured: Receives picture tiles and frame strips. The shipped CUDA caption deployment has no GPU resource reservation. Configure allocation or advertised sharing before scheduling it.
  - **Text reader**, Existing API · not deployed here: The overlay expects an existing text-reader endpoint and a served model name. This service receives annotation text, not pictures. Configure it separately; it can run on your network or another permitted endpoint.
  - Storage: Service model caches

1. **Metadata + previews**: Preparation starts with your library; originals remain in Immich.
2. **Preview tiles + frames**: Classification and captions go to separate services. The text reader receives annotations through its separately configured endpoint.

Service boundaries, not node placement. Reserve GPUs explicitly; a namespace is not an authentication boundary. Reader and music endpoints are configured separately.

Render a film:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (App pod · one replica)
  - **App + CLI**, Assemble the film: Hands the chosen cut to the render worker. The UI and run history stay here.
  - Storage: Data/cache · output · model PVCs
- **Render sidecar** (Same pod as the app)
  - **Render worker**, App pod GPU reservation · 1 allocation: The app and render sidecar share their pod. The app calls the authenticated worker on 127.0.0.1:8093. The sidecar receives the Immich key, fetches originals and returns the film to the app. The diagram does not promise that all workloads fit on two GPU nodes.
  - Storage: Worker scratch/cache · ephemeral

1. **Cut + Immich credentials**: The authenticated render request includes the Immich URL and API key. Use HTTPS across an untrusted network.
2. **Worker → originals**: The worker needs its own network route to Immich. It returns the finished film to the app.
3. **Optional delivery**: After rendering, the app can upload the finished film if you enable delivery.

Service boundaries, not node placement. Reserve GPUs explicitly; a namespace is not an authentication boundary. Reader and music endpoints are configured separately.

Open the app:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (App pod · one replica)
  - **App + CLI**, App listener · port 8080: Serves the UI and proxies previews on the app listener. For Compose, open the published host port in your browser; the container still listens on 8080. The browser does not receive your Immich API key.
  - Storage: Data/cache · output · model PVCs
- **Your browser** (Desktop or phone)
  - **Web UI**, App URL · authenticated if exposed: Connects to the app, which serves cached media or fetches it from Immich. Configure authentication before opening access to other machines.
  - **HTTPS proxy + OIDC**, Operator-provided services: The example enables OIDC but does not deploy an ingress or identity provider. Supply the HTTPS proxy and identity registration; the app handles the login flow. This card groups the access dependencies, not one container.

1. **Preview requests**: The app keeps the Immich key on the server.
2. **Browser → app**: UI requests and media go through the app, behind the same login.

Service boundaries, not node placement. Reserve GPUs explicitly; a namespace is not an authentication boundary. Reader and music endpoints are configured separately.

**Deployment: Local services on Apple Silicon**

Prepare a cut:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (Your Mac · native app)
  - **App + CLI**, Choose the cut: Reads Immich metadata and previews, saves reusable facts, then chooses the cut. On GPU and Full tiers, Laya runs in the app, even with a remote preparation service.
  - **Owned local reader**, llama.cpp · when enabled: A separate app-owned process reads annotation text. A blank reader URL selects this runtime when the reader is enabled. Its model is released before local rendering and music.
  - Storage: Store · cache · output files
- **Caption server** (Same Mac · separate runtime)
  - **MLX captions**, localhost:8092/v1: Start the caption server separately. It receives picture tiles; keep its bind address on localhost when only this Mac uses it.

1. **Metadata + previews**: Preparation starts with your library; originals remain in Immich.
2. **Preview tiles + frames**: Images go to the caption service; the text reader is a separate role.

The app and caption runtime share one Mac. Captions and the reader are enabled in this recipe; generated music is optional.

Render a film:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (Your Mac · native app)
  - **App + CLI**, Assemble the film: Downloads the selected originals and renders locally with FFmpeg.
  - Storage: Store · cache · output files
- **Local rendering** (Same Mac · separate runtime)
  - **Native FFmpeg rendering**, Runs inside the app workflow: Rendering uses VideoToolbox encoding and Metal title effects where available, with CPU fallback. This is not a remote worker or another machine. Docker on macOS cannot use Metal directly.

1. **Selected originals**: The app downloads media for the chosen cut.
2. **Local render work**: No remote handoff: processing and output stay on this Mac.

The app and caption runtime share one Mac. Captions and the reader are enabled in this recipe; generated music is optional.

Open the app:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (Your Mac · native app)
  - **App + CLI**, Native UI · port 8080: Serves the UI on the native app port and proxies previews. The browser does not receive your Immich API key.
  - Storage: Store · cache · output files
- **Your browser** (Desktop or phone)
  - **Web UI**, App URL · authenticated if exposed: Connects to the app, which serves cached media or fetches it from Immich. Configure authentication before opening access to other machines.

1. **Preview requests**: The app keeps the Immich key on the server.
2. **Browser → app**: UI requests and media go through the app, behind the same login.

The app and caption runtime share one Mac. Captions and the reader are enabled in this recipe; generated music is optional.

**Diagram: Every way in ends in the same film run: the web UI, the daily timer or an OS timer.** [(image)](https://sam-dumont.github.io/immich-memories/diagrams/architecture-detailed.svg)

```text
┌──────────────────────┐  ┌───────────────────────────┐  ┌───────────────────────────┐  ┌────────────────────────┐  ┌────────────────────┐
│                      │  │                           │  │                           │  │                        │  │       FFmpeg       │
│                      │  │                           │  │                           │  │                        │  │       Store        │
│       Browser        ├─►│          FastAPI          ├─►│         Job runner        ├─►│        Film run        ├─►│SQLite or PostgreSQL│
│                      │  │/app, /api/v1, /api/trigger│  │     one job at a time     │ ││generate, or runs render│  │   Films, caches,   │
│                      │  │                           │  │                           │ ││                        │  │        logs        │
└──────────────────────┘  └─────────────┬─────────────┘  └───────────────────────────┘ │└────────────────────────┘  └────────────────────┘
                                        │                                              │             ▲
                                        │                                              │             │
                                        ├───────────────┬──────────────────────────────┘             │
                                        │               │                                            │
                                        │               │                                            │
┌──────────────────────┐  ┌─────────────┴─────────────┐ │┌───────────────────────────┐               │
│                      │  │                           │ ││                           │               │
│   In-process timer   ├─┐│          auto run         │ ►│     Automation runner     ├───────────────┘
│      once a day      │ ││ the same runner, as a CLI │ ││lease, cooldown, what's due│
└──────────────────────┘ │└───────────────────────────┘ │└───────────────────────────┘
                         │              ▲               │
                         │              │               │
                         └──────────────┼───────────────┘
                                        │
                                        │
┌──────────────────────┐                │
│launchd, systemd, cron├────────────────┘
└──────────────────────┘
```

Groups:

- Your app host
  - Web server process · immich-memories ui, port 8080
    - FastAPI
    - Job runner
    - Automation runner
    - In-process timer
  - Or an OS timer · auto install
    - launchd, systemd, cron
    - auto run
  - Film run
  - FFmpeg Store; Films, caches, logs

Arrows:

- Browser → FastAPI
- FastAPI → Job runner
- FastAPI → Automation runner
- In-process timer → Automation runner
- Job runner → Film run
- Automation runner → Film run
- launchd, systemd, cron → auto run
- auto run → Film run
- Film run → FFmpeg Store; Films, caches, logs

**Diagram: Each helper is one setting away, and none of them is on until you set it.** [(image)](https://sam-dumont.github.io/immich-memories/diagrams/architecture-detailed-2.svg)

```text
┌────────┐  ┌───────────────────────────┐
│        │  │                           │
│        │  │    llama-server reader    │
│Film run├─►│llm.enabled, blank base_url│
│generate│  │     ACE-Step, lib mode    │
│        │  │     ace_step.mode: lib    │
└────┬───┘  └───────────────────────────┘
     │
     │
     │
     │
     │
     │      ┌───────────────────────────┐
     │      │         Immich API        │
     ├─────►│      API key: reads;      │
     │      │ writes only with upload on│
     │      └───────────────────────────┘
     │
     │
     │
     │
     │
     │      ┌───────────────────────────┐
     │      │                           │
     │      │      Inference :8092      │
     │      │  inference.facts_base_url │
     ├─────►│     Captions :8092/v1     │
     │      │      caption_base_url     │
     │      │       Render worker       │
     │      │   render.worker_base_url  │
     │      └───────────────────────────┘
     │
     │
     │
     │
     │
     │      ┌───────────────────────────┐
     │      │                           │
     │      │        Reader model       │
     └─────►│        llm.base_url       │
            │        ACE-Step API       │
            │      ace_step.api_url     │
            └───────────────────────────┘
```

Groups:

- Your network
  - Your app host
    - Film run
    - llama-server reader; ACE-Step, lib mode
  - GPU box or service pods · optional
    - Inference :8092; Captions :8092/v1; Render worker
  - Other services you run · optional
    - Reader model; ACE-Step API
  - Immich API

Arrows:

- Film run → llama-server reader; ACE-Step, lib mode
- Film run → Immich API
- Film run → Inference :8092; Captions :8092/v1; Render worker
- Film run → Reader model; ACE-Step API

The text reader can be an app-owned local process or an API server. Captioning is a separate role; explicitly enabling LLM captions sends pictures to that configured model. A render worker additionally receives the chosen cut and Immich key, then fetches originals. Keep these services inside the network boundaries described in [Privacy](https://sam-dumont.github.io/immich-memories/docs/run/privacy.md).

## Composition over inheritance

The four main orchestrators compose smaller service objects through constructor injection. The
lifecycle a run reports is the `OperationalPhase` enum in `operations/phases.py` (discovery →
download → analysis → selection → render → music → delivery → complete), and it spans two entry
points. Selection runs first: `generate` (the web UI's Cut runs the same command) →
`build_smart_pipeline(editorial_context)` in `analysis/runtime/composition.py` →
`SmartPipeline.run_editorial_source()` → `RuntimeEditorialPlanner.plan_source()`, which runs
preparation, the two readings, the structure and story planners, and certifies the timing.
`generate_memory()` in `generate.py` then does extract → assemble → music → upload. Hand it no
clips and it raises rather than going to find some.

`analysis/` is 18 packages, one per stage of that selection. Top to bottom: `runtime` (the
composition the CLI drives), `structure`, `polish`, `selection`, `stories`, `film`, `audience`,
`special_days`, `reading`, `preparation`, `captions`, `clips`, `facts`, `sources`, `corpus`,
`places`, `judgments` and `llm` (the model transport). A stage imports only the stages below it,
and only the modules a package lists in its `__all__`. The order lives in the
`analysis-stages-are-layered` contract in `pyproject.toml`: `make arch-check` fails a change that
breaks it, and `make private-imports` fails one that imports another module's `_private` names.
Config is its own package, `config/`, with the section models in `config/models/`. The full map is
in `ARCHITECTURE.md`.

| Orchestrator | Services | What it does |
| - | - | - |
| **VideoAssembler** | FFmpegProber, ClipEncoder, AssemblyEngine, AudioMixerService, TitleInserter | Assembles clips into final video |
| **SmartPipeline** | RuntimeEditorialPlanner (from `build_smart_pipeline`) | Runs the story-first selection and projects its plan into a `PipelineResult` |
| **ImmichClient** | SearchService, AllAssetsService, AssetService, PersonService, AlbumService | Talks to the Immich API |
| **TitleScreenGenerator** | RenderingService, EndingService, TripService | Creates title/ending screens |

The editorial route has Protocol-typed ports rather than services: the providers and the people
loader, the structure planner, the judges it calls out to, and `EditorialAttempt` in `operations/`
for the durable attempt tree and its OS lease. On disk each attempt is
`<cache>/editorial-runs/<key>/attempts/<id>/`, and the banked facts and
answers live in the store (the `db/` package, tables in `db/tables/annotations.py` and
`db/tables/model_answers.py`).
[ARCHITECTURE.md](https://github.com/sam-dumont/immich-memories/blob/main/ARCHITECTURE.md)
names every port and the file it lives in, with the full module map.

## Verification

[CI and quality gates](https://sam-dumont.github.io/immich-memories/docs/contribute/ci.md) explain which checks run for each kind of diff. [Testing](https://sam-dumont.github.io/immich-memories/docs/contribute/testing.md) covers local suites.

## How to add a feature

### A new processing capability

1. Create a service class in the relevant package (for example `processing/my_service.py`)
2. Inject it into the orchestrator's `__init__` in `video_assembler.py`
3. Add tests in `tests/test_my_service.py`

### A new API endpoint

1. Add the method to the relevant service in `api/` (for example `search_service.py`)
2. Add a delegating method on `ImmichClient` in `api/immich.py`
3. Add the model to `api/models.py` if needed, and test against a mock HTTP client

### A new memory type

1. Add the value to the `MemoryType` enum in `memory_types/registry.py`
2. Write a factory function in `memory_types/factory.py` and decorate it with `@register_preset`: the decorator *is* the registration, there is no second list to edit there
3. Add date builder logic if the type needs its own, in `memory_types/date_builders.py`
4. Add it to `OFFERED_MEMORY_TYPES` in `memory_types/registry.py` for `--memory-type`. Add its fields to `FIELDS` in `web/src/routes/create/+page.svelte` too: the Memory page has its own map and does not discover presets from the Python tuple
5. Document it in [`docs-site/docs/make/memory-types.mdx`](https://sam-dumont.github.io/immich-memories/docs/make/memory-types.md)
6. Give it a row in `tests/features/inventory.yaml` with its feature test: `make feature-check` fails while a memory type has none ([Feature tests](https://sam-dumont.github.io/immich-memories/docs/contribute/testing.md#feature-tests))

### A new CLI command

1. Create a new file in `cli/` (for example `cli/my_cmd.py`) with a `register_my_commands(main)` function, like `cli/hardware_cmd.py`
2. Import it and call it with the others at the bottom of `cli/__init__.py`
3. Run `make docs-cli` to regenerate the [CLI reference](https://sam-dumont.github.io/immich-memories/docs/reference/cli-reference.md): `make docs-cli-check` fails CI until you do
4. Add the docs page under `docs-site/docs/make/cli/`, add its ID to `docs-site/sidebars.ts`, and run `make docs-build`
5. Give the command and each flag a row in `tests/features/inventory.yaml`: `make feature-check` fails while one has none

## File naming conventions

- `_prefixed.py`: private helpers for their own package. Keep imports inside the owning package; public package entry points expose shared behavior
- `*_service.py`: composed service classes
- `*_models.py`: data models (Pydantic or dataclass)
- `*_helpers.py`: standalone helper functions
- `*.py` (no prefix): public modules and standalone classes. Re-export shims belong in
  `__init__.py` and nowhere else
