# Photos, Live Photos and HDR processing

Markdown version of <https://sam-dumont.github.io/immich-memories/docs/reference/media-processing/>.

Photos compete in the same pool as videos, on every tier. There is no separate photo pipeline: a
still that wins a slot is animated at render time and holds the screen for the seconds the editor
gave it. Every image in range comes back from Immich, Live Photo stills included.

`--no-photos` takes stills out for one run; `photos.enabled` is the lasting switch.

A photo edited in Immich's own editor (crop, rotate, mirror; `isEdited` on the asset) is read as
Immich renders the edit: the previews preparation and selection read ask for `edited=true`, and so
does the download the renderer animates. Its size check is skipped, since the edited render's size
has nothing to do with the original's. A picture whose edit state changed since it was last prepared
gets its preview, pixel facts, heads, caption and face reading again on the next run. An edited HDR
photo plays as SDR, because Immich's edited render carries no gain map. Videos and Live Photo clips
always come from the original: the editor does not touch them. A server older than Immich 2.5 sends
no `isEdited`, and every picture reads as unedited.

## Ken Burns, and where the pan lands

One renderer for every photo. A move on a picture with faces has the whole picture at one end
(zoom 1.0) and its closest frame at the other. Every choice is seeded from the asset ID, so a
re-render moves the same way:

| Film | Closest frame | Direction |
| - | - | - |
| About one person | That person, 25-35 % in; other faces only if they still fit | In |
| About several people, or several faces in any film | The tightest crop holding every face plus half a face of air on each side, 15-50 % in; a 20 % pan across the group when that crop would zoom less than 15 % | In for a film about several people; in or out per photo otherwise |
| Anything else | The lone face, 25-35 % in | In or out, per photo |
| Nobody in the picture, any film | Three times in four, a pan from one end of the long side to the other at 12-15 %, with at most a 3 % zoom drift; otherwise a soft zoom from the whole picture to the centre at 12-15 % | Pan either way; a soft zoom goes in for a film about people, in or out otherwise |

The move is one continuous glide, size and position together. Each still draws one of four easings
(in, out, in-out or linear) from the same seed, blended with linear so no move stalls at either end.
It animates at the film's own frame rate, so a 24 fps film doesn't drop frames out of every pan.

A film is 30 fps. It goes to 60 (or 50) only when more than half of its video seconds are 60 (or
50\) fps footage; one fast clip among slower ones doesn't count. Slow motion, recorded at 100 fps or
more and played slowed, counts as 30. A film made only of 24 or 25 fps footage keeps that rate.
Photos follow the film.

The ceiling is 50 % because the photo is prepared at 1.5 times the canvas; closer would upscale it.
The face boxes come from Immich's faces endpoint, one request per still; framing runs no face
detector of its own, and a still whose boxes can't be read moves toward its centre.

A photo move is encoded with the hardware encoder the app has verified (NVIDIA, VAAPI and the
others), or as fast near-lossless x264 in software; the film's own encode does the real
compression. In an HDR film, a still is rendered in the film's transfer. On Basic, a device with
hardware H.264 but no hardware HEVC prepares photos as SDR H.264, tone-mapping HDR photos first.
With hardware encoding off, preparation runs in software.

Video clips are never cropped. A landscape clip in a portrait film keeps its whole frame and
[`scale_mode`](https://sam-dumont.github.io/immich-memories/docs/reference/config-reference.md) fills the rest: `blur` (default) puts a blurred, zoomed
copy behind the sharp one, `fit` uses black bars. A photo that does not fill the canvas gets the
same bands. Face-aware video cropping is not offered: a moving subject needs per-frame tracking,
not one face position.

## Live Photos

A merge, its short final-frame hold and the certified cut are encoded losslessly: NVENC's
lossless mode on an NVIDIA encoder, x264 at its fastest preset everywhere else. The film's own
encode is the only lossy step. The hold keeps the source cadence and records its encoder beside
the verified frame count.

A selected motion interval keeps its verified frame rate through encoding. When frame rounding
leaves a permitted shortfall, the renderer holds the final frame and checks the encoded duration
before assembly. This applies to software and hardware encoders.

Some MOV probes report a 4-tick sample duration even when successive pictures arrive every
20 ticks. The renderer checks those actual presentation timestamps before allowing one final
frame hold. A rounded container endpoint can use that measured interval; a larger missing tail
still stops the render.

### Original-source admission

Before sealing a cut, the app downloads only the Live Photo clips the cut keeps and checks
that each one plays in full. The check is tied to the source bytes, the video stream, the decoder
and the presentation rules, and a matching check is reused. Rejected candidates are never
downloaded for it.

A Live Photo whose clip fails plays as its still in the same slot; no other picture takes its
place, and the timing is worked out again before the cut is certified.

### Presentation and audio boundaries

The renderer checks both the compressed samples and the decoded frames, then cuts on the MOV
track's verified sample table and edit list. Reference samples count toward completeness but are
never shown. Each joined segment keeps its own video and audio boundaries, and audio keeps its
own clock.

A rounding gap within the allowance holds the last video frame. An audible audio tail is never
replaced with silence to fit, and a larger missing video tail is never invented. The encoded
segment is checked against the certified interval before assembly.

### Bursts become footage

An iPhone records about 3 seconds of video with every photo. Most libraries hold thousands of them,
and a rapid burst is several seconds of continuous footage nobody meant to shoot.

Three photos of an Italian hilltop, fired off in a row, each about 3 seconds and overlapping:

**Video: Hilltop Live Photo 1 of 3 (3 s)** [source\_1.mp4](https://sam-dumont.github.io/immich-memories/demos/live-photos/italian_hilltop/source_1.mp4)\
![Poster frame: Hilltop Live Photo 1 of 3](https://sam-dumont.github.io/immich-memories/img/live-photo-preview.jpg)

**Video: Hilltop Live Photo 2 of 3 (3 s)** [source\_2.mp4](https://sam-dumont.github.io/immich-memories/demos/live-photos/italian_hilltop/source_2.mp4)

**Video: Hilltop Live Photo 3 of 3 (2 s)** [source\_3.mp4](https://sam-dumont.github.io/immich-memories/demos/live-photos/italian_hilltop/source_3.mp4)

Merged, 4.5 seconds of continuous footage:

**Video: The three hilltop Live Photos, merged (5 s)** [merged.mp4](https://sam-dumont.github.io/immich-memories/demos/live-photos/italian_hilltop/merged.mp4)

A bike race, 6 Live Photos merged into 8.1 seconds:

**Video: Six Live Photos at a bike race, merged into one shot (8 s)** [merged.mp4](https://sam-dumont.github.io/immich-memories/demos/live-photos/bike_race/merged.mp4)

These clips are the project author's own footage, published with permission.

**A Live Photo is a photograph.** Its still competes with the photographs and wins or loses as one.
Whether it plays as motion is a rendering question, asked afterwards, about a picture that has
already won its place. The answer is the same on every tier, model or not:

Diagram outline:

- Live Photo kept in the cut → Other Live Photos within 10 s?
- Other Live Photos within 10 s? → Kept picture's own clip at least 2.5 s? (no, lone)
- Other Live Photos within 10 s? → Stitch measured and at least 2.5 s? (yes, a burst)
- Stitch measured and at least 2.5 s? → Kept picture's own clip at least 2.5 s? (no)
- Kept picture's own clip at least 2.5 s? → Plays as a still (no)
- Kept picture's own clip at least 2.5 s? → Motion residual at least 1.5? (yes)
- Stitch measured and at least 2.5 s? → Motion residual at least 1.5? (yes)
- Motion residual at least 1.5? → Plays as a still (no)
- Motion residual at least 1.5? → Clip shows the subject? (yes)
- Clip shows the subject? → Plays as a still (often missing)
- Clip shows the subject? → Original clip plays correctly? (yes)
- Original clip plays correctly? → Plays as a still (broken frames or timing)
- Original clip plays correctly? → Plays as motion, up to 6 s (yes)

```mermaid
flowchart TD
  L[Live Photo kept in the cut] --> B{Other Live Photos<br/>within 10 s?}
  B -- no, lone --> O
  B -- yes, a burst --> J{Stitch measured and<br/>at least 2.5 s?}
  J -- no --> O{Kept picture's own clip<br/>at least 2.5 s?}
  O -- no --> S
  O -- yes --> R
  J -- yes --> R{Motion residual at least 1.5?}
  R -- no --> S[Plays as a still]
  R -- yes --> F{Clip shows the subject?}
  F -- often missing --> S
  F -- yes --> P{Original clip plays correctly?}
  P -- broken frames or timing --> S
  P -- yes --> M[Plays as motion, up to 6 s]
```

1. **The video half is not footage.** It leaves the video pool: it belongs to a photograph.
2. **Bursts.** Photos within 10 s of each other form a burst. A
   burst collapses to one unit before the editor chooses: the favourite carries it, otherwise the
   sharpest, best-exposed frame, and the siblings are not separately selectable.
3. **Length.** The clip that would play must last 2.5 s, whether it is a stitch or one Live Photo's own clip. A lone clip
   runs 1.2 to 3.3 s; a 1.2 s clip is half a sentence, a 2.9 s one reads as a moment. When a
   burst's joins cannot be measured, or its stitch comes out short (shots a fraction of a second
   apart overlap so much that their stitch can be barely longer than one clip), the kept
   picture's own clip gets the same test. Under the floor, the picture is its still.
4. **Motion.** The residual is the optical flow left once the camera's own movement is removed,
   over 12 frames. At 1.5 or above something happened in every burst we measured; below it the
   answer is a coin flip, and a still always works where a dead clip does not.
5. **The subject.** Where the clip's frames were read and the subject is often out of frame (the
   phone already on its way to the pocket), the picture plays as its still.
6. **The original.** The selected clip must decode with valid frame timing. A damaged or
   truncated clip, or one that fails to download, uses the still for the same slot. If the still
   fails too, the picture leaves the film and the film is that much shorter: one picture never
   costs the whole film.

A clip only ever costs a picture its motion, never its place, favourite or not. A Live Photo that
plays is offered beside the videos, and a true video always plays whatever its residual.

The one setting that decides anything is `include_live_photos` (true); the 2.5 s floor is fixed.
`--include-live-photos` cannot turn the feature on when the config says false: both must
agree. With it off, a Live Photo is used as its still, never dropped.

A shared album can hold a downscaled copy of a Live Photo pointing at the same video. The video
belongs to both copies, so whichever the cut keeps plays it.

On Basic tier, Live Photo merges fit within 1920×1080 in landscape or 1080×1920 in portrait.
If the hardware encodes H.264 but cannot encode HEVC, HDR companions are tone-mapped to SDR
during preparation. Each companion's HLG or PQ transfer determines its color conversion.
Changing the tier or hardware settings rebuilds certified merges under the new policy.

> **Person-filtered memories:** The person tag is the boundary. An untagged Live Photo does not enter a person memory because it was
> shot beside a tagged one. A tagged frame can then lack neighbours to render as motion; it stays a
> photograph rather than widening the memory to pictures without that person.

### Merging on audio, not timestamps

A clip's video does not start at exactly `shutter_time - 1.5 s`, and tens of milliseconds of drift
on a rapid burst means audible clicks and gaps. So the joins are measured, and only for the bursts
the film keeps: the draft plans every burst from its metadata, then a kept burst's clips are
downloaded once and lined up on their sound. The answer is banked per pair, so the next film over the
same pictures downloads nothing.

1. Extract 48 kHz mono audio from each clip
2. Compute an STFT spectrogram (1024-sample window, 256 hop, one fingerprint every 5 ms)
3. For each pair, correlate the first 20 frames of clip B (about 107 ms) against clip A
4. Cut at the midpoint between consecutive shutters; extend a clip to cover any hole
5. Normalize exposure, fade the audio 30 ms at each join, concatenate

Three photos at t = 0, 0.5 s and 2 s, each clip about 3 s:

| Clip | Plays from | Plays to | Duration |
| - | - | - | - |
| Photo 1 | -1.5 s | midpoint(0, 0.5) = 0.25 s | \~1.75 s |
| Photo 2 | 0.25 s | midpoint(0.5, 2.0) = 1.25 s | \~1.0 s |
| Photo 3 | 1.25 s | 3.5 s | \~2.25 s |

These times share the first shutter as zero. The span is 5.0 s in total.

Clips that do not overlap stay separate. With no shared content to line up on, nothing is stitched:
the kept picture plays its own clip if it lasts 2.5 s and moves, and its still otherwise. The merge
is lossless (see [Live Photos](#live-photos)), and an HDR burst keeps its 10-bit HLG or PQ
transfer unless Basic hardware needs SDR H.264; the final render decides the rest.

### Devices

Immich normalizes Live Photos and Motion Photos through `livePhotoVideoId`. The one device branch is
Google: a Pixel gets a 1.5 s assumed clip length instead of 3.0 s. A Pixel Motion Photo is 0.7 to
1.3 s with no audio track and does not overlap the next shot, so four rapid Pixel photos stay four
clips. Everything else, Samsung included, takes the Apple path.

## HDR, end to end

The default `output.codec: h264` produces SDR: HDR sources are tone-mapped for the final film.
To preserve HDR, choose `output.codec: h265` with `output.hdr_mode: auto` or `hdr`. The final
encoding plan decides whether HDR video (HLG or PQ) stays HDR. An HDR photograph can be rebuilt
from its gain map before that output conversion.

**Apple, iPhone 12 and later.** A HEIC carries an 8-bit Display P3 image and a grayscale gain map;
the headroom comes from two MakerNote tags read together, `0x0021` and `0x0030`. The renderer applies
`HDR = SDR x (1 + (headroom - 1) x gain)` and ships 10-bit PQ, BT.2020, which is Apple's own shape
and never darkens a pixel. When the MakerNote does not parse it falls back to `exiftool` if
installed; a photograph whose headroom cannot be read renders at the brightness of its base image.

**Android Ultra HDR.** A JPEG with an MPF gain map and `hdrgm` XMP metadata, rebuilt with its
per-channel gamma and offsets.

`pi-heif` (libheif, decode only) reads HEIC, because FFmpeg only reads a HEIC's thumbnail tiles. Title text
over HDR is drawn at HLG graphics white, so a caption does not glare above the picture.

`output.hdr_mode` is `auto` (HDR when H.265 is selected and any selected source is HDR), `hdr` or `sdr`.
`auto` with H.264 still produces SDR. HDR output is H.265 only: `hdr` with H.264 or ProRes is refused, never silently flattened. Which encoders take 10-bit is on
[Hardware encoding](https://sam-dumont.github.io/immich-memories/docs/run/hardware.md).
