Skip to main content

Photos, Live Photos and HDR processing

Photos compete in the same pool as videos, on every tier. There is no separate photo pipeline: a still that wins a slot is animated at render time and holds the screen for the seconds the editor gave it. Every image in range comes back from Immich, Live Photo stills included.

--no-photos takes stills out for one run; photos.enabled is the lasting switch.

A photo edited in Immich's own editor (crop, rotate, mirror; isEdited on the asset) is read as Immich renders the edit: the previews preparation and selection read ask for edited=true, and so does the download the renderer animates. Its size check is skipped, since the edited render's size has nothing to do with the original's. A picture whose edit state changed since it was last prepared gets its preview, pixel facts, heads, caption and face reading again on the next run. An edited HDR photo plays as SDR, because Immich's edited render carries no gain map. Videos and Live Photo clips always come from the original: the editor does not touch them. A server older than Immich 2.5 sends no isEdited, and every picture reads as unedited.

Ken Burns, and where the pan lands​

One renderer for every photo. A move on a picture with faces has the whole picture at one end (zoom 1.0) and its closest frame at the other. Every choice is seeded from the asset ID, so a re-render moves the same way:

FilmClosest frameDirection
About one personThat person, 25-35 % in; other faces only if they still fitIn
About several people, or several faces in any filmThe tightest crop holding every face plus half a face of air on each side, 15-50 % in; a 20 % pan across the group when that crop would zoom less than 15 %In for a film about several people; in or out per photo otherwise
Anything elseThe lone face, 25-35 % inIn or out, per photo
Nobody in the picture, any filmThree times in four, a pan from one end of the long side to the other at 12-15 %, with at most a 3 % zoom drift; otherwise a soft zoom from the whole picture to the centre at 12-15 %Pan either way; a soft zoom goes in for a film about people, in or out otherwise

The move is one continuous glide, size and position together. Each still draws one of four easings (in, out, in-out or linear) from the same seed, blended with linear so no move stalls at either end. It animates at the film's own frame rate, so a 24 fps film doesn't drop frames out of every pan.

A film is 30 fps. It goes to 60 (or 50) only when more than half of its video seconds are 60 (or 50) fps footage; one fast clip among slower ones doesn't count. Slow motion, recorded at 100 fps or more and played slowed, counts as 30. A film made only of 24 or 25 fps footage keeps that rate. Photos follow the film.

The ceiling is 50 % because the photo is prepared at 1.5 times the canvas; closer would upscale it. The face boxes come from Immich's faces endpoint, one request per still; framing runs no face detector of its own, and a still whose boxes can't be read moves toward its centre.

A photo move is encoded with the hardware encoder the app has verified (NVIDIA, VAAPI and the others), or as fast near-lossless x264 in software; the film's own encode does the real compression. In an HDR film, a still is rendered in the film's transfer. On Basic, a device with hardware H.264 but no hardware HEVC prepares photos as SDR H.264, tone-mapping HDR photos first. With hardware encoding off, preparation runs in software.

Video clips are never cropped. A landscape clip in a portrait film keeps its whole frame and scale_mode fills the rest: blur (default) puts a blurred, zoomed copy behind the sharp one, fit uses black bars. A photo that does not fill the canvas gets the same bands. Face-aware video cropping is not offered: a moving subject needs per-frame tracking, not one face position.

Live Photos​

A merge, its short final-frame hold and the certified cut are encoded losslessly: NVENC's lossless mode on an NVIDIA encoder, x264 at its fastest preset everywhere else. The film's own encode is the only lossy step. The hold keeps the source cadence and records its encoder beside the verified frame count.

A selected motion interval keeps its verified frame rate through encoding. When frame rounding leaves a permitted shortfall, the renderer holds the final frame and checks the encoded duration before assembly. This applies to software and hardware encoders.

Some MOV probes report a 4-tick sample duration even when successive pictures arrive every 20 ticks. The renderer checks those actual presentation timestamps before allowing one final frame hold. A rounded container endpoint can use that measured interval; a larger missing tail still stops the render.

Original-source admission​

Before sealing a cut, the app downloads only the Live Photo clips the cut keeps and checks that each one plays in full. The check is tied to the source bytes, the video stream, the decoder and the presentation rules, and a matching check is reused. Rejected candidates are never downloaded for it.

A Live Photo whose clip fails plays as its still in the same slot; no other picture takes its place, and the timing is worked out again before the cut is certified.

Presentation and audio boundaries​

The renderer checks both the compressed samples and the decoded frames, then cuts on the MOV track's verified sample table and edit list. Reference samples count toward completeness but are never shown. Each joined segment keeps its own video and audio boundaries, and audio keeps its own clock.

A rounding gap within the allowance holds the last video frame. An audible audio tail is never replaced with silence to fit, and a larger missing video tail is never invented. The encoded segment is checked against the certified interval before assembly.

Bursts become footage​

An iPhone records about 3 seconds of video with every photo. Most libraries hold thousands of them, and a rapid burst is several seconds of continuous footage nobody meant to shoot.

Three photos of an Italian hilltop, fired off in a row, each about 3 seconds and overlapping:

Merged, 4.5 seconds of continuous footage:

A bike race, 6 Live Photos merged into 8.1 seconds:

These clips are the project author's own footage, published with permission.

A Live Photo is a photograph. Its still competes with the photographs and wins or loses as one. Whether it plays as motion is a rendering question, asked afterwards, about a picture that has already won its place. The answer is the same on every tier, model or not:

100%
  1. The video half is not footage. It leaves the video pool: it belongs to a photograph.
  2. Bursts. Photos within 10 s of each other form a burst. A burst collapses to one unit before the editor chooses: the favourite carries it, otherwise the sharpest, best-exposed frame, and the siblings are not separately selectable.
  3. Length. The clip that would play must last 2.5 s, whether it is a stitch or one Live Photo's own clip. A lone clip runs 1.2 to 3.3 s; a 1.2 s clip is half a sentence, a 2.9 s one reads as a moment. When a burst's joins cannot be measured, or its stitch comes out short (shots a fraction of a second apart overlap so much that their stitch can be barely longer than one clip), the kept picture's own clip gets the same test. Under the floor, the picture is its still.
  4. Motion. The residual is the optical flow left once the camera's own movement is removed, over 12 frames. At 1.5 or above something happened in every burst we measured; below it the answer is a coin flip, and a still always works where a dead clip does not.
  5. The subject. Where the clip's frames were read and the subject is often out of frame (the phone already on its way to the pocket), the picture plays as its still.
  6. The original. The selected clip must decode with valid frame timing. A damaged or truncated clip, or one that fails to download, uses the still for the same slot. If the still fails too, the picture leaves the film and the film is that much shorter: one picture never costs the whole film.

A clip only ever costs a picture its motion, never its place, favourite or not. A Live Photo that plays is offered beside the videos, and a true video always plays whatever its residual.

The one setting that decides anything is include_live_photos (true); the 2.5 s floor is fixed. --include-live-photos cannot turn the feature on when the config says false: both must agree. With it off, a Live Photo is used as its still, never dropped.

A shared album can hold a downscaled copy of a Live Photo pointing at the same video. The video belongs to both copies, so whichever the cut keeps plays it.

On Basic tier, Live Photo merges fit within 1920×1080 in landscape or 1080×1920 in portrait. If the hardware encodes H.264 but cannot encode HEVC, HDR companions are tone-mapped to SDR during preparation. Each companion's HLG or PQ transfer determines its color conversion. Changing the tier or hardware settings rebuilds certified merges under the new policy.

Person-filtered memories

The person tag is the boundary. An untagged Live Photo does not enter a person memory because it was shot beside a tagged one. A tagged frame can then lack neighbours to render as motion; it stays a photograph rather than widening the memory to pictures without that person.

Merging on audio, not timestamps​

A clip's video does not start at exactly shutter_time - 1.5 s, and tens of milliseconds of drift on a rapid burst means audible clicks and gaps. So the joins are measured, and only for the bursts the film keeps: the draft plans every burst from its metadata, then a kept burst's clips are downloaded once and lined up on their sound. The answer is banked per pair, so the next film over the same pictures downloads nothing.

  1. Extract 48 kHz mono audio from each clip
  2. Compute an STFT spectrogram (1024-sample window, 256 hop, one fingerprint every 5 ms)
  3. For each pair, correlate the first 20 frames of clip B (about 107 ms) against clip A
  4. Cut at the midpoint between consecutive shutters; extend a clip to cover any hole
  5. Normalize exposure, fade the audio 30 ms at each join, concatenate

Three photos at t = 0, 0.5 s and 2 s, each clip about 3 s:

ClipPlays fromPlays toDuration
Photo 1-1.5 smidpoint(0, 0.5) = 0.25 s~1.75 s
Photo 20.25 smidpoint(0.5, 2.0) = 1.25 s~1.0 s
Photo 31.25 s3.5 s~2.25 s

These times share the first shutter as zero. The span is 5.0 s in total.

Clips that do not overlap stay separate. With no shared content to line up on, nothing is stitched: the kept picture plays its own clip if it lasts 2.5 s and moves, and its still otherwise. The merge is lossless (see Live Photos), and an HDR burst keeps its 10-bit HLG or PQ transfer unless Basic hardware needs SDR H.264; the final render decides the rest.

Devices​

Immich normalizes Live Photos and Motion Photos through livePhotoVideoId. The one device branch is Google: a Pixel gets a 1.5 s assumed clip length instead of 3.0 s. A Pixel Motion Photo is 0.7 to 1.3 s with no audio track and does not overlap the next shot, so four rapid Pixel photos stay four clips. Everything else, Samsung included, takes the Apple path.

HDR, end to end​

The default output.codec: h264 produces SDR: HDR sources are tone-mapped for the final film. To preserve HDR, choose output.codec: h265 with output.hdr_mode: auto or hdr. The final encoding plan decides whether HDR video (HLG or PQ) stays HDR. An HDR photograph can be rebuilt from its gain map before that output conversion.

Apple, iPhone 12 and later. A HEIC carries an 8-bit Display P3 image and a grayscale gain map; the headroom comes from two MakerNote tags read together, 0x0021 and 0x0030. The renderer applies HDR = SDR x (1 + (headroom - 1) x gain) and ships 10-bit PQ, BT.2020, which is Apple's own shape and never darkens a pixel. When the MakerNote does not parse it falls back to exiftool if installed; a photograph whose headroom cannot be read renders at the brightness of its base image.

Android Ultra HDR. A JPEG with an MPF gain map and hdrgm XMP metadata, rebuilt with its per-channel gamma and offsets.

pi-heif (libheif, decode only) reads HEIC, because FFmpeg only reads a HEIC's thumbnail tiles. Title text over HDR is drawn at HLG graphics white, so a caption does not glare above the picture.

output.hdr_mode is auto (HDR when H.265 is selected and any selected source is HDR), hdr or sdr. auto with H.264 still produces SDR. HDR output is H.265 only: hdr with H.264 or ProRes is refused, never silently flattened. Which encoders take 10-bit is on Hardware encoding.