# One GPU service

Markdown version of <https://sam-dumont.github.io/immich-memories/docs/run/reference-setup/>.

Keep the app on the NAS and move classifiers, captions, Demucs stems and rendering to one NVIDIA
container. The text reader and ACE-Step music generation are separate choices. Laya runs in the app.

**Diagram: Basic is one container. GPU adds inference and captions; Full connects your reader.** [(image)](https://sam-dumont.github.io/immich-memories/diagrams/deploy-compose.svg)

```text
┌───────────────────────────┐
│                           │
│      immich-memories      │
│host port → container :8080│
│       config + store      │
│    named config volume    │
│           Films           │
│          ./output         │
└───────────────────────────┘
              │
              │
              ├─────────────────────────┬───────────────────────┐
              │                         │                       │
              ▼                         ▼                       ▼
┌───────────────────────────┐  ┌─────────────────┐  ┌───────────────────────┐
│                           │  │                 │  │                       │
│         inference         │  │                 │  │                       │
│   container + host :8092  │  │                 │  │                       │
│         captioner         │  │   postgres:16   │  │     GPU box :8092     │
│container :8092, host :8094│  │no published port│  │facts, captions, render│
│       caption models      │  │                 │  │                       │
│     one-shot download     │  │                 │  │                       │
└─────────────┬─────────────┘  └─────────────────┘  └───────────────────────┘
              │
              │
              │
              │
              ▼
┌───────────────────────────┐
│                           │
│      Existing reader      │
│     URL + served model    │
└───────────────────────────┘
```

Groups:

- Basic: the whole app · docker-compose.yml · CPU, rules, up to 1080p
  - immich-memories; config + store; Films
- - GPU tier: captions and checks · docker-compose.gpu.yml · docker-compose.cuda.yml · NVIDIA driver and Container Toolkit
  * inference; captioner; caption models
- - Full tier · docker-compose.full.yml · connect and enable your reader
  * Existing reader
- - PostgreSQL, any tier · docker-compose.postgres.yml · the store moves off SQLite
  * postgres:16
- Or: GPU work on another private machine · docker-compose.gpu-worker.yml · set GPU\_BOX on the app; rendering takes its own URL
  - GPU box :8092

Arrows:

- immich-memories; config + store; Films → inference; captioner; caption models
- immich-memories; config + store; Films → postgres:16
- immich-memories; config + store; Films → GPU box :8092
- inference; captioner; caption models → Existing reader

## One GPU service

Use the CUDA inference image matching the app version, the NVIDIA Container Toolkit and a
private address reachable from the app. From a checkout of that release:

```bash
export GPU_WORKER_IMAGE=ghcr.io/sam-dumont/immich-memories/inference:YOUR_APP_TAG-cuda
export IMMICH_URL=https://photos.example.com
export RENDER_WORKER_TOKEN=$(openssl rand -hex 32)
export GPU_WORKER_BIND_ADDRESS=192.168.1.50
docker compose -f services/inference/compose.gpu-worker.yaml up -d
```

Save the token in your secret manager and use the same value in the app. The recipe defaults to
loopback binding if you omit `GPU_WORKER_BIND_ADDRESS`. The model-cache volume holds reusable
weights and runtime caches; render-scratch holds the worker's downloaded originals and temporary
render files. It is separate from the app's finished-output volume. The service reserves one
NVIDIA GPU and does not set a RAM limit.

Configure the app in `config.yaml` or the environment. The worker URL and token can't be set
from Settings, since the worker receives your Immich key:

```yaml
advanced:
  inference:
    facts_base_url: http://192.168.1.50:8092
    fallback_to_local: false
  editorial:
    preparation:
      caption_base_url: http://192.168.1.50:8092/v1
render:
  worker_base_url: http://192.168.1.50:8092/render
  worker_token: ${RENDER_WORKER_TOKEN}
  allow_insecure_http: true
  fallback_to_local: false
```

`allow_insecure_http` is an explicit opt-in for this trusted LAN: render requests include the
Immich API key. Use HTTPS through a proxy otherwise, and omit that opt-in. Only `/render` checks
the bearer token; `/facts`, `/audio/stems` and `/v1` have no built-in authentication. Do not expose
this listener to the internet.

**Deployment: App + one GPU service**

Prepare a cut:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (Your app host)
  - **App + CLI**, Choose the cut: Reads Immich metadata and previews, saves reusable facts, then chooses the cut. On GPU and Full tiers, Laya runs in the app, even with a remote preparation service.
  - Storage: Store · cache · output files
- **GPU service** (NVIDIA host · one container)
  - **Facts + captions**, Port 8092 · /facts and /v1: Receives previews for classification and captioning. These routes have no built-in authentication; keep them private.
  - Storage: Model cache · render scratch

1. **Metadata + previews**: Preparation starts with your library; originals remain in Immich.
2. **Preview tiles + frames**: Images go to the caption service; the text reader is a separate role.

One GPU container shares the GPU across phases and releases owned models between phases. Reader and generated-music services are separate choices, not included here.

Render a film:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (Your app host)
  - **App + CLI**, Assemble the film: Hands the chosen cut to the render worker. The UI and run history stay here.
  - Storage: Store · cache · output files
- **GPU service** (NVIDIA host · one container)
  - **Render worker**, Port 8092 · /render: Receives the cut and Immich URL/API key, fetches originals and returns the film. /render requires a bearer token.
  - Storage: Model cache · render scratch

1. **Cut + Immich credentials**: The authenticated render request includes the Immich URL and API key. Use HTTPS across an untrusted network.
2. **Worker → originals**: The worker needs its own network route to Immich. It returns the finished film to the app.
3. **Optional delivery**: After rendering, the app can upload the finished film if you enable delivery.

One GPU container shares the GPU across phases and releases owned models between phases. Reader and generated-music services are separate choices, not included here.

Open the app:

- **Your Immich** (Existing Immich server)
  - **Immich API**, Your configured URL: Provides metadata, previews and original media. Upload is optional; the app does not change the source originals.
  - Storage: Your photo and video library
- **Immich Memories** (Your app host)
  - **App + CLI**, App listener · port 8080: Serves the UI and proxies previews on the app listener. For Compose, open the published host port in your browser; the container still listens on 8080. The browser does not receive your Immich API key.
  - Storage: Store · cache · output files
- **Your browser** (Desktop or phone)
  - **Web UI**, App URL · authenticated if exposed: Connects to the app, which serves cached media or fetches it from Immich. Configure authentication before opening access to other machines.

1. **Preview requests**: The app keeps the Immich key on the server.
2. **Browser → app**: UI requests and media go through the app, behind the same login.

One GPU container shares the GPU across phases and releases owned models between phases. Reader and generated-music services are separate choices, not included here.

Check from the app's environment:

```bash
immich-memories preflight -v
# When the app runs in Docker Compose:
docker compose exec immich-memories immich-memories preflight -v
```

The Compose health check verifies the inference listener and authenticated render health. It
does not certify caption responses or every model's provider. Preflight checks reachability,
model advertisement and reported capabilities; preparation validates synthetic caption controls
before sending library pictures.

## Optional generated music

The CUDA worker separates Demucs stems but does not generate ACE-Step music. Connect your own
[ACE-Step API server](https://sam-dumont.github.io/immich-memories/docs/better/music.md#api-server), or use local library mode on a native Linux
or Mac checkout. The external music server owns its model lifetime. Configure its model and
planner on that server; the app's local `model_variant`, `use_lm` and `cpu_offload` settings do not
change the remote service. The app generates only when the server's `/health` reports an XL model,
and plays a bundled track otherwise. When the music server runs on another host than the worker, the
music is made while the worker renders the film; on the same host, the film renders first. Sharing
one card still requires room for every resident service.
The [tested ACE-Step build and single-GPU limits](https://sam-dumont.github.io/immich-memories/docs/better/music.md#sharing-one-gpu-with-the-worker)
record the service used for the measured cold runs. Kubernetes time-slicing does not unload models.

## Memory and scheduling

Work changes phase between classifiers, captions, audio and rendering. The worker unloads
classifier weights and stops its owned caption subprocess when switching away from them.
Demucs releases after separation. Captions restart on demand.

Rendering waits up to 60 seconds for active model work to finish. Conflicting requests return
`503` with `Retry-After: 1`; classifier queue overflow separately returns `429`. Native work keeps
its GPU ownership even when its client cancels. Requests for the same model phase wait up to
60 seconds for cleanup, then enter that service's existing queue. `/queue` reports classifiers,
not a combined queue for all phases.

One container saves duplicate services and image layers. It still needs enough host RAM and
VRAM for the largest active phase, scratch space for originals and output, and headroom for other
GPU users. The bundled caption command sets an 8192-token context and a 128 MiB host prompt cache
(`LLAMA_ARG_CACHE_RAM`, override it in the worker's environment); it keeps llama.cpp's automatic
slot count. Separate-reader or music services can still compete for the card. Measure your workload before raising concurrency.

## Scratch space for 4K

For 4K rendering, [the Kubernetes storage guidance](https://sam-dumont.github.io/immich-memories/docs/run/reference/kubernetes.md) recommends
**8 GiB of disk-backed `/tmp`**; size it for your workload. Keep model files and finished output
on their own volumes. A model-only service's small temporary volume can fill during
source preparation after rendering is added.

In Kubernetes, leave the scratch `emptyDir` disk-backed and set its `sizeLimit` for that work.
The node also needs the corresponding free ephemeral storage. A RAM-backed `emptyDir` charges
the container's memory budget; increasing a RAM limit does not increase disk scratch space.

## Other deployments

The shipped Kubernetes inference and caption overlays and render sidecar deploy **separate**
services; there is no unified-worker overlay. Use the [Kubernetes reference](https://sam-dumont.github.io/immich-memories/docs/run/reference/kubernetes.md)
for those manifests. The [multi-service cluster example](https://sam-dumont.github.io/immich-memories/docs/run/reference/cluster-example.md) is for
operators deliberately distributing work across nodes.

For Apple Silicon, use the [Mac example](https://sam-dumont.github.io/immich-memories/docs/run/reference/mac-example.md). The CUDA worker cannot use
Metal. Service API details and offline model provisioning are in the
[inference reference](https://sam-dumont.github.io/immich-memories/docs/reference/inference-service.md).
