Optional optical-flow interpolation (Practical-RIFE–style IFNet) between consecutive SDXL stills so the full-frame background warps from keyframe to keyframe instead of relying only on smoothstep crossfades.
- Background mode must be
sdxl-stills. RIFE is not used for static Ken Burns uploads or for AnimateDiff (AnimateDiff’s init stills are generated with RIFE disabled so seed PNGs stay raw). - Defaults (UI): Ken Burns on SDXL stills and Morph keyframes (RIFE) are on unless the user turns them off. Programmatic defaults match via
OrchestratorInputs.sdxl_ken_burns/sdxl_rife_morph.
| Control | Effect |
|---|---|
| Morph keyframes (RIFE) | After SDXL keyframes exist, runs RIFE between each adjacent pair and replaces the sampling timeline with a denser control set. |
| RIFE subdivisions (2^N steps between keyframes) | Integer N in [2, 8]; 2^N centered IFNet samples per segment (see “Centered sampling” below). Higher = smoother morphs and longer RIFE bake. Default 4 → 16 IFNet samples per segment; N=8 → 256 (heavy). |
Ken Burns (if enabled) is applied after the background sample at time t, unchanged from pre-RIFE behaviour.
Sampling: background_frame(t) uses linear blending between neighbouring RIFE samples (dense optical-flow timeline). Sparse SDXL-only keyframes still use smoothstep crossfades—double-easing dense RIFE was slowing motion toward every sample and stretching the blend from the last approximate IFNet frame to the exact endpoint still.
Plain centered sampling (j + 0.5)/n (the v2 / v3 timeline) avoided emitting exact SDXL stills at internal keyframe boundaries, but it still placed the boundary IFNet samples of every segment at s ≈ 0.5/n and s ≈ (n-0.5)/n (≈ 0.031 / 0.969 at the default N = 4). At those near-endpoint timesteps IFNet's flow displacement collapses toward zero, so the predicted frames are visually near-identical to the SDXL keyframe pixel-wise. The compositor then renders an extended ~T_seg/n window of "near-keyframe" pixels around every internal keyframe time, which the viewer perceives as the morph pausing on the original still, even though no exact still is in the timeline.
The dense morph timeline therefore samples IFNet at inset-warped centered timesteps. For each segment [kf_i, kf_{i+1}] of duration T_seg:
- Sample IFNet at
s_j = inset + (1 - 2*inset) * (j + 0.5)/nforj ∈ [0, n)wheren = 2**N. Theinsetis v7.2 capped by a target boundary-velocity ratio (default 5× body velocity, override viaGLITCHFRAME_RIFE_BOUNDARY_VELOCITY_RATIO) — see the next section for the derivation. Atrife_exp ≤ 4(n ≤ 16) the cap is a no-op andinsetstays at the v7.1 default of 0.12; at higherrife_expthe cap shrinks it (e.g.0.0362atN = 6,0.0096atN = 8). An explicitGLITCHFRAME_RIFE_IFNET_INSETvalue bypasses the cap and is honoured byte-exactly (clamped to[0, 0.45]) so byte-exact reproduction against older bakes / manual experimentation continues to work. The closest-to-boundary samples sit ats ≈ insetands ≈ 1 - insetinstead of the legacy≈ 0.5/n/≈ 1 - 0.5/n, so every dense sample carries visible flow displacement and is visually distinct from both bracketing keyframes. - Wall-clock placement of body samples is decoupled from the IFNet timestep and stays uniform centered:
t_j = t_kf_i + T_seg * (j + 0.5)/n. Apparent motion velocity is therefore constant — only the visual content of each frame is shifted away from the keyframe.
v4 / v5 left a subtler perceptual issue at every internal keyframe time even after the inset moved boundary samples away from s ≈ 0 / s ≈ 1: the last body sample of segment i (IFNet(kf_i, kf_{i+1}, s ≈ 1 - inset)) and the first body sample of segment i + 1 (IFNet(kf_{i+1}, kf_{i+2}, s ≈ inset)) come from different optical-flow pairs but both look visually similar to the same shared keyframe kf_{i+1}. The flow fields are pair-specific: the dominant content (e.g. the cat in a duck→cat→tiger sequence) sits at slightly different pixel coordinates in each — the last body of segment i is kf_i warped almost all the way to kf_{i+1}, the first body of segment i + 1 is kf_{i+1} warped slightly toward kf_{i+2}. The compositor's pixel-space linear blend across that pair-mismatched seam therefore translates the dominant object across the boundary window, perceived by the user as a brief "skip" of the object's position at every keyframe.
v6 tried a cross-pair bridge and made it worse. v6 added one IFNet inference at every internal t_{i+1} on the cross pair (kf_i, kf_{i+2}) at s = 0.5. The intent was that this bridge would carry a different flow direction (kf_i → kf_{i+2} instead of into/out of kf_{i+1}), breaking the assumed static plateau. In practice it introduced a third pair-mismatched warp at the seam: the bridge's spatial layout of the dominant content was different from both flanking body samples, so the compositor's linear blend across three pair-mismatched warps over the same wall-clock window made the per-frame translation of the dominant object bigger, not smaller — perceived as "skipping entire frames" / "clear move of the object" on every transition.
v7 anchored on the exact SDXL keyframe and introduced a different artifact. v7 dropped the cross-pair bridge and placed the byte-exact SDXL still keyframes[i + 1] at every internal t = t_{i+1}, on the (stated) assumption that the still is byte-identical to IFNet(kf_i, kf_{i+1}, s = 1.0) modulo identity behaviour at the endpoints. That assumption is wrong: Practical-RIFE's IFNet (see pipeline/rife_vendor/ifnet_hdv3.py) has no identity branch at s = 0 or s = 1 — it unconditionally runs all five flow-refinement blocks with timestep baked into the feature concatenation, so its render at s = 1.0 is visually close to kf_{i+1} but carries the network's characteristic warp + merge-mask texture, which differs from the SDXL VAE output (the VAE is sharper / more saturated; IFNet is slightly smoother). v7 therefore had the compositor blend IFNet(seg_i body, s≈0.86) → raw VAE still → IFNet(seg_{i+1} body, s≈0.14) — a 1–2-video-frame texture discontinuity on every internal keyframe, visible regardless of how clean the underlying SDXL stills were. Users reported this as a "blip" / "skip" at every boundary.
v7.1 (schema v8): IFNet-rendered anchors. For each internal boundary at t = t_{i+1} (between segment i and segment i + 1):
- Render
IFNet(kf_i, kf_{i+1}, s = 1.0)on the prior pair and place it att = t_{i+1}(one frame, one extra IFNet inference per internal boundary).
The anchor is now produced by the same network path that produces every body sample of seg_i, so it shares IFNet's texture signature with its left neighbour. The compositor's pixel-space linear blend stays within a single texture and a single optical-flow pair on the pre-anchor side, and crosses pairs on the post-anchor side only between two IFNet renders both dominated by kf_{i+1} content (anchor: full warp toward kf_{i+1}; first body of seg_{i+1}: only inset along the next flow), so the cross-pair flow residual at the boundary is at its minimum visible scale.
- just before
t_{i+1}: blendlast_body(seg_i)→anchorwithin pair(kf_i, kf_{i+1})— same flow field, same texture signature, no spatial or stylistic mismatch. - exactly at
t_{i+1}:IFNet(kf_i, kf_{i+1}, s = 1.0)— IFNet's own render ofkf_{i+1}, visually dominated bykf_{i+1}content. - just after
t_{i+1}: blendanchor→first_body(seg_{i+1})across pair(kf_i, kf_{i+1})→ pair(kf_{i+1}, kf_{i+2}). Both samples are IFNet renders (no VAE↔IFNet texture hand-off — the v7 regression is gone), and both arekf_{i+1}-dominant (anchor ats = 1.0of prior; first body ats ≈ insetof next), so the cross-pair flow residual is small.
The residual IFNet-timestep velocity cost on each side of every internal keyframe is inset / (T_seg/(2n)) = 2 · n · inset / T_seg per second over the T_seg/(2n) boundary cell. The ratio to the body's own IFNet-timestep velocity (1 - 2 · inset) / T_seg simplifies to
ratio = 2 · n · inset / (1 - 2 · inset)
(T_seg cancels). At inset = 0.12, n = 16 that's ~5 × the body velocity — the v7 / v7.1 design target, perceived as a brief speed-up into every keyframe rather than a discrete jump. But the ratio scales linearly in n: with the same constant inset = 0.12 it becomes ~80 × the body velocity at rife_exp = 8 (n = 256), which translates to ~25 % IFNet-timestep change per video frame at 30 fps — a clear pre-anchor "skip" and (symmetrically, because the post-anchor first body of the next segment sits inset = 0.12 along the next flow) a visible post-anchor cross-pair residual on every keyframe boundary at heavy rife_exp. (This was the dominant artifact reported under v7.1 at rife_exp = 8 — the texture blip was gone, but the velocity spike scales with n and the v7.1 reference inset was calibrated only at n = 16.)
v7.2 (schema v9): boundary-velocity cap. v7.2 caps the inset so the ratio stays bounded for every rife_exp:
inset_cap = ratio_target / (2 · n + 2 · ratio_target)
inset_effective = min(DEFAULT_IFNET_TIMESTEP_INSET, inset_cap)
with ratio_target = 5 by default (override via GLITCHFRAME_RIFE_BOUNDARY_VELOCITY_RATIO, clamped to [0, 100]). At rife_exp ≤ 4 (n ≤ 16) the cap evaluates ≥ 0.119 — at or just below the v7.1 default 0.12 — so the min returns either 0.12 (low rife_exp) or 0.119 (at n = 16 exactly), within the user-perceptible motion threshold of the v7.1 reference. At higher rife_exp the cap kicks in (e.g. 0.0362 at n = 64, 0.0096 at n = 256), which has two beneficial effects: (a) the pre-anchor IFNet-timestep velocity spike stays at ratio_target × body velocity regardless of n, eliminating the visible "skip" right before every keyframe; and (b) the boundary body samples are tightened to s ≈ cap / s ≈ 1 − cap (e.g. s ≈ 0.011 / s ≈ 0.989 at n = 256), which also minimises the post-anchor cross-pair residual because the first body of the next segment now sits at a flow displacement of only ~1 % toward kf_{i+2} instead of the v7.1 ~12 %. An explicit GLITCHFRAME_RIFE_IFNET_INSET value bypasses the cap; GLITCHFRAME_RIFE_IFNET_INSET = 0 still collapses the spike to ~0 (perfect velocity continuity) at the cost of letting the per-pair body samples near every keyframe become visually near-identical to the keyframe itself (the original "stops on the still" symptom at low rife_exp; at high rife_exp the near-identical wall-clock window shrinks to T_seg/n and is too brief to perceive).
Cost: +1 IFNet inference per internal keyframe vs. v7 (total_segs − 1 extra forward passes per bake; sub-second on a 3090 for typical 20–40-keyframe songs). Still well below v6's N − 2 cross-pair bridge calls per boundary. v7.2 does not change the cost relative to v7.1 — only the IFNet timesteps at which the body / bookend samples are evaluated change with rife_exp.
The v4 sampling eliminated the internal keyframe pause but introduced a velocity discontinuity at the song's bookends: keeping the exact SDXL still at t = 0 (s = 0) and t = duration (s = 1) leaves an IFNet jump of inset + span/(2n) between the bookend and the first body sample over only T_seg/(2n) wall-clock seconds. At the legacy inset = 0.12 and N = 4 (n = 16) that was a 0.144 IFNet-timestep jump in 0.25 s — roughly 6× the body's pace span/T_seg. The compositor renders ~7–8 video frames flying through 14 % of the morph in that window, which the viewer reads as a brief "skip" right before the song's closing still (and a symmetric one at the open).
v5 fixes this by sampling the bookends at IFNet timesteps that match the body velocity:
- Start bookend at
t = keyframe_times[0]: IFNet ats = inseton the first keyframe pair(kf_0, kf_1)— IFNet jump to the first body sample becomesspan/(2n)(half a body step), which divided byT_seg/(2n)wall-clock equalsspan/T_seg(the body velocity). - End bookend at
t = keyframe_times[-1]: symmetrically, IFNet ats = 1 - inseton the last keyframe pair(kf_{N-1}, kf_N).
The visual cost is small: IFNet at s = inset is within ~inset flow displacement of kf_0 (and symmetrically for kf_N), so the song still visibly opens and closes on the generated background image — just with a hint of motion instead of a freeze that snaps. The eye reads the resulting uniform velocity as a smooth open/close instead of the previous skip. Setting GLITCHFRAME_RIFE_IFNET_INSET=0 collapses the bookends to s = 0 / s = 1 (≈ exact stills under IFNet), which is also the legacy uniform-velocity case (no skip at inset = 0 either); cross-pair bridges still apply at inset = 0 (they're independent of the body inset).
Spacing: within a segment the wall-clock spacing of body samples is uniform T_seg / n (e.g. with T_seg = 8 s and N = 4 that's 0.5 s). Across every internal keyframe boundary the spacing is halved to T_seg / (2n) per side because the v7.1 IFNet-rendered anchor sits exactly on t_kf. The only other spacing irregularity is the short T_seg/(2n) gap between the velocity-matched bookend and its neighbouring body sample, where IFNet velocity now matches the body — visually smooth. Spacing is independent of the v7.2 boundary-velocity cap — only the IFNet timesteps at which the boundary body samples / the bookends / the (unchanged) anchor are evaluated shrink with rife_exp; the wall-clock grid is unaffected.
Frame count formula: frame_count = total_segs * n_steps + (total_segs - 1) + 2 — body samples + internal-keyframe anchors + two velocity-matched bookends. For N = 31 keyframes (30 segments) at rife_exp = 4 (n_steps = 16): 30 * 16 + 29 + 2 = 511 frames (identical count to v6 / v7 / v7.1 / v7.2; only the content of the total_segs − 1 boundary slots and the IFNet timesteps at which body / bookend samples are evaluated change between schema versions — cross-pair bridge in v6, raw SDXL still in v7, IFNet endpoint render in v7.1, and v7.2 additionally caps the body / bookend inset by the boundary-velocity ratio at high rife_exp).
Cache invalidation: RIFE_MANIFEST_SCHEMA_VERSION is 9 (= v7.2). v1 → centered (v2) introduced no exact internal stills; v2 → v3 added keyframes_content_hash (SHA-256 over keyframe RGB payloads in order) so replacing or editing keyframe PNGs invalidates the RIFE cache even when text prompts and layout are unchanged; v3 → v4 introduced inset-warped centered sampling; v4 → v5 replaced the exact-still bookends with velocity-matched IFNet bookends (eliminating the skip at song open / close); v5 → v6 raised the body inset default 0.12 → 0.25 and inserted a cross-pair IFNet bridge at every internal keyframe; v6 → v7 reverted the inset to 0.12 and replaced the cross-pair bridge with the exact SDXL keyframe still at every internal boundary (eliminating the cross-pair pixel-blend region that caused the v6 "skipping entire frames" / "clear move of the object" artifact); v7 → v8 (v7.1) replaced the raw SDXL still anchor with IFNet(kf_i, kf_{i+1}, s = 1.0) after we discovered Practical-RIFE's IFNet has no identity branch at the endpoints — the raw VAE still and the IFNet render differ by a per-network texture signature, and inserting the VAE still as an anchor introduced a 1–2-video-frame texture "blip" at every internal boundary; v8 → v9 (v7.2) caps the body / bookend inset by a boundary-velocity ratio target (default 5×) so the pre-anchor IFNet-timestep velocity spike stays bounded at high rife_exp (was ~80× body velocity at rife_exp = 8, now ~5× regardless of rife_exp) and the post-anchor cross-pair residual collapses to its minimum visible scale (both flanking samples now sit at IFNet timesteps very close to the shared keyframe). At rife_exp ≤ 4 v9's behaviour is byte-identical to v8 up to the 0.12 → 0.119 cap rounding at n = 16; at rife_exp ≥ 5 body / bookend bytes change. Older manifests are silently re-baked the next time RIFE runs.
- Vendor code:
pipeline/rife_vendor/— warper +IFNet(MIT, derived from Practical-RIFE). - Runtime:
pipeline/rife_runtime.py— downloads weights viahuggingface_hub, builds the dense timeline withrife_build_morph_timeline. - Integration:
pipeline/background_stills.py— after SDXLensure,_apply_rife_morph_if_neededeither loads cache or runs RIFE;background_frame(t)usesRifeMorphManifest.timeswhen RIFE is active (linear interpolation within segments), elseBackgroundManifest.keyframe_timeswith smoothstep. - Factory / orchestrator:
create_background_source(..., sdxl_rife_morph=..., rife_exp=...);OrchestratorInputscarries the same flags.
- Checkpoint: Hugging Face
MonsterMMORPG/RIFE_4_26(train_log/flownet.pkl, ~24 MB). First use downloads intoMODEL_CACHE_DIRunderrife_practical/…(same Windows symlink-safe hub behaviour as WhisperX / SDXL caches). - Code follows upstream Practical-RIFE MIT terms; verify checkpoint license/host terms for your redistribution scenario.
cache/<hash>/background/
manifest.json # SDXL stills (unchanged)
keyframe_*.png
manifest_rife.json # RIFE cache key + per-frame times
rife_timeline/
rife_000000.png
...
Invalidates when the SDXL manifest key changes (prompt / model / resolution / keyframe count), when RIFE settings change (rife_exp, output width/height, repo id), when the manifest schema version changes (e.g. v1 → v2 centered sampling, v2 → v3 pixel fingerprint), when keyframe image bytes change (v3 keyframes_content_hash), when timeline PNGs are missing, or on a fresh bake (old rife_timeline/*.png and manifest_rife.json are removed before writing new frames).
- CUDA for the RIFE bake (same machine class as SDXL). If RIFE is on and CUDA is unavailable, the pipeline raises a clear error.
- Extra GPU time after SDXL: one forward pass per generated intermediate between pairs (scales with subdivision N and number of keyframe intervals).
rife_runtime.py mirrors what upstream Practical-RIFE does to reach its quoted 3090-class throughput, none of which were enabled previously:
torch.inference_mode()around every IFNet forward pass. Disables autograd graph construction (we never call.backward()); cuts inference latency ~15–20 % and roughly halves transient VRAM versus.eval()alone.torch.backends.cudnn.benchmark = Truewhile baking. IFNet runs the same input shape thousands of times per timeline, so cudnn's algorithm-search cache pays off after the first call. The flag is saved/restored around the bake so other CUDA users (SDXL, AnimateDiff) keep their previous setting.- FP16 (auto-on for CUDA).
load_rife_modelcallsnet.half()on CUDA and_prepare_pair_tensorsmatches the model's dtype. The 3090's tensor cores roughly halve forward-pass time and also halve IFNet's VRAM footprint. Outputs are quantised to uint8 immediately, so any FP16 round-off is lost in the cast — visually byte-equivalent in spot checks.pipeline/rife_vendor/warplayer.pykeys its_backwarp_gridcache ondtypeso the sameF.grid_samplecall works in either precision (PyTorch ≥ 1.10 requiresinputandgriddtypes to match). Disable withGLITCHFRAME_RIFE_FP16=0if you ever need byte-exact reproducibility against an FP32 baseline.
Combined, the GPU phase at N=8 on a 3090-class card drops from ~30+ minutes to roughly 5–10 minutes per typical SDXL keyframe set — small enough that the save phase (covered above) becomes the next thing to watch.
The dense morph timeline is persisted as one PNG per frame under cache/<hash>/background/rife_timeline/. PIL's PNG encoder is single-threaded and CPU-bound (zlib), so on high-rife_exp runs (e.g. N=8 → 256 frames per segment, often 5 000+ frames at 1080p) the save phase used to dominate wall-clock far more than the actual GPU bake. NVMe disk bandwidth was never the bottleneck — the encoder was, and the in-RAM accumulation behind it.
_apply_rife_morph_if_needed now uses a streaming bake plus a lazy on-disk source for both fresh bakes and cache hits:
- Streaming write during the bake.
rife_build_morph_timelineaccepts anon_frame(idx, rgb, t)callback (keep_frames=Falseskips the in-RAM list). The bake fans frames into a smallThreadPoolExecutor(max(2, min(8, os.cpu_count()))) of PNG writers running withcompress_level=1(lossless; ~3–5× faster encode at the cost of ~30–50 % larger cache files; user-visible keyframes still use the default6). A small in-flight queue caps RAM at roughlyworkers × frame_size. The PNG manifest is written last, after every writer drains, so a crashed bake leaves an inconsistent state that fails_rife_timeline_pngs_completeon retry rather than presenting a half-written manifest. - Lazy in-RAM source for sampling.
_RifeFrameSourceis a thread-safe disk-backedSequence[np.ndarray]with a small LRU cache (default 16 frames)._interpolate_frameonly ever reads two adjacent frames per call, and the compositor walks time monotonically, so this size is enough to keep every read warm. Both the cache-hit path and the post-bake handoff install this source instead of decoding all PNGs into alist[np.ndarray].
Combined, peak RAM for the RIFE timeline drops from frame_count × frame_size (≈ 36 GB at 1080p × 5 889 frames) to roughly workers × frame_size during the bake and cache_size × frame_size during render — typically well under 200 MB. The save phase wall-clock collapses from hours to minutes on typical hardware.
tests/test_rife_manifest.py—RifeMorphManifestJSON round-trip andmatches_key.
docs/technical/background-stills.md— SDXL keyframe planning and crossfade baseline.docs/technical/background-modes.md—BackgroundSourcefactory and modes table.