Scene Orchestrator#

A scene here is a cinematic clip baked to a PNG flipbook — an mp4 imported once (ffmpeg extracts it frame-by-frame at import time) and played back as a flat sequence of images, exactly like a character animation. It is not the same thing as a cutscene: a cutscene (core/cutscene.js, authored by the Cutscene Editor) is an ordered list of scripted engine steps — NPC walks, SAY lines, waits — scoped to one room. A cutscene's PLAYSCENE step plays a Scene Orchestrator scene as one of its steps and blocks until it ends; the two tools edit two different data shapes and are not interchangeable. There is no video codec anywhere at runtime — by design (see the closing callout) — a scene is just PNG frames advancing on a clock, so it plays identically on every platform the engine targets, including ports that can't rely on <video>.

You reach it from the Hub (dev server running). The toolbar has a Scene dropdown, + New, Save (Ctrl+S, writes the descriptor and registers it), a Clips manager, a Replace Color tool (eyedropper + tolerance/blend, bakes a recolored PNG for a clip), Undo/Redo, and GC (sweeps unused scene assets to a reversible trash folder). The left column is a preview (with pan/zoom and transport controls) over a timeline of tracks; the right panel shows properties for whatever track, keyframe, or cue is selected.

Limits: how much video a scene can hold#

A scene plays PNG frames, and the player decodes every frame of every clip in the scene before it starts — that is what makes playback identical on every platform, and it is also the ceiling. The budget that matters is not the mp4's size but frames × width × height × 4 bytes once decoded. The importer shows that figure next to the disk estimate and gates on it: amber above 1 GB (it loads, but slowly, and a modest machine may not manage it), red above 3 GB (the import is refused — lower the fps or the scale, or trim). The same limit is enforced during the import itself, so there is no way to slip a clip past it.

What that means in seconds, at the frame sizes and rates you will use most:

Scenes are for short cinematic pieces — a logo, a title card, a few seconds of motion. Anything longer belongs in a video editor, cut down to the pieces you actually need.

Recording for import (OBS or any screen recorder):

• Resolution = your project's resolution (1920 × 1080 for a 1080p project), never larger. Bigger frames are scaled down at draw time anyway; they only cost disk, RAM and import time. If you do hand it a 4K recording, the importer drops the scale to fit the project width on its own.
• 24–30 fps. 60 fps doubles the frame count for no visible gain in a flipbook. The importer resamples to the fps you pick, so a 60 fps source works — it just decodes twice the frames.
• Bitrate barely matters. Every frame is fully decoded on import, so a high-quality master costs nothing extra; what weighs is the number and size of the PNGs it turns into.
• Keep clips to seconds, not minutes, and trim in the importer (from + count) when the recording has more than the shot you want.

Anatomy of a scene#

A scene descriptor ({ id, duration, bg?, endMode?, keepMusic?, onEnd?, tracks: [...] }) is a pure data object — serialized as a small ES module and saved to projects/<id>/scenes/<sceneId>.js. Saving also idempotently adds an entry to core/scenes/index.js, but that file is just a tiny id→descriptor map; in practice the runtime never needs the save-time registration because shell/main.js's _ensureScene() dynamic-imports the descriptor lazily on first PLAYSCENE/REWINDSCENE and registers it then — core stays fully decoupled from project paths.

A scene has five kinds of tracks, all sharing one timeline:

• Sprite tracks (the default, no kind) — a clip (a named Frame[] from the project's clip registry) plus a keyframe list ({ t, x, y, scale, rot, alpha, ease }) driving an independent transform curve on top of the clip's own frame-by-frame playback. in/out gate when the track is visible in scene time; z orders layers; anchor and an optional fx preset (a per-clip visual filter, resolved to an actual filter only in the shell) round it out. Two clocks run side by side here: the master timeline (keyframes, seekable, eased) and the clip-internal clock (frame cycling, reusing the exact same tickAnim() primitive character animations use). The registry itself (scenes/clips.js) is one flat Frame[] per clip ({ sprite, ms } per frame). An entry in any other shape is listed as skipped in the Clips manager, with its id, and written back untouched when you save — never dropped.
• Audio tracks (kind: 'audio') — a positioned, trimmable audio clip (src, at, trimStart/trimEnd, vol, fadeIn/fadeOut, muted). When the master clock crosses at, it fires a single edge-triggered SCENEAUDIO:<src>|<trimStart>|<dur>|<vol>|<fadeIn>|<fadeOut> token; the shell plays it as a one-shot on the music bus (GameAudio.playMusic), stopped automatically (with the clip's own fade-out) when the scene ends — unless the scene sets keepMusic (see How a scene ends).
• Cue tracks (kind: 'cue') — a list of { t, token } pairs: arbitrary effect tokens fired the instant the master clock reaches t. This is how a scene fires anything mid-playback — a flag, a sound, a room warp — without waiting for the scene to finish.
• FX tracks (kind: 'fx') — a full-screen filter preset (glitch/blur/etc.) with its own in/out window and opacity, drawn over the whole frame rather than as a positioned sprite.
• Text tracks (kind: 'text') — an on-screen caption or title. Like a sprite track it carries a keyframe list (position, scale, rotation, alpha over time, eased per segment) and a z, but instead of a clip it draws a line of text with a full style batch: font and size, color, alignment, bold / italic / underline / strike, an outline (outlineWidth), a drop shadow, and lineHeight/letterSpacing. The text itself is authored monolingually and backed by a lid, so the Translation Editor fills the other languages later — the same content-i18n rule as spoken lines and item names. It renders identically in the editor preview, the MP4 export, and in-game (core carries the shape; the shell resolves the lid and font).

Editing on the timeline#

Two pointer tools sit on the toolbar. ➤ Select (V) is the default — click a track or clip to select it, drag to move it along its lane. ✂ Cut (C) splits a clip at the playhead marker: click any clip and it becomes two back-to-back clips. Each half restarts its own frame playback from frame 0, so a cut is a genuine split into two independent clips, not a trim of one.

Tracks reorder vertically: grab a track's header and drag it up or down into a new slot, and the editor renumbers z by the resulting top-to-bottom order — what you see stacked in the timeline is the paint order on screen. A sprite track's own clip can be dragged vertically out of its lane to the same effect.

When two audio clips overlap in time, a ⤬ crossfade helper appears in the selected clip's properties: one click sets the earlier clip's fadeOut and the later clip's fadeIn to the overlap length, turning the overlap into a DAW-style crossfade built from the plain fade fields the runtime already plays — no new track type, no special data.

The effect-token pipeline (one grammar, one dispatch)#

Both cue tokens and the scene's onEnd list are ordinary effect tokens from the same shared manifest every other system uses (rules, dialogs, cutscenes) — authored through the same typed picker widget, never free text for an ID-bearing argument. The verb list shown here is curated for scene context: it hides tokens that only make sense inside live room logic (inventory scroll, hotspot restore, save/load, item pickup) and promotes room-warp / audio / flag / dialog / cutscene / scene verbs to the top; cue pickers additionally promote the audio quartet (SOUND/MUSIC/STOPMUSIC/PLAYFX) first, since a mid-scene cue is almost always an audio beat.

onEnd holds a list, and its rows carry a ⠿ grip — drag one to change the order the tokens fire in when the scene ends. A cue has no grip on purpose: it holds a single token slot, so there is nothing to reorder; you move a cue in time by dragging it along its track instead.

WAIT and WAITFOR are left out for a different reason: a scene is a timeline, so you space its beats by moving them along the track, not by pausing between tokens. Where a delay belongs to live play instead — a spoken line that needs a moment before the room changes — that's a reaction chain, and both waits are available there.

At runtime, core/scenes/timeline.js's pure tickScene() returns a fired array of due tokens each frame; shell/main.js runs it through the same two-tier applyEffects() dispatch used everywhere else in the engine — Tier-1 tokens (SETFLAG, GIVEITEM, WARPTOROOM, etc.) mutate state directly, and whatever's left over (SCENEAUDIO, SAY, MUSIC) falls through to the generic action dispatcher. One rule store, one grammar, no separate scene-only interpreter.

How a scene ends#

Every track's in/out window is inclusive at out: a track with no explicit out inherits scene.duration, and it stays on screen through the very last millisecond. Cutting to black before the end is the author's call — lower that track's own out. (Before this, the frame that landed exactly on duration culled every track, so the closing frame of every scene was solid black.)

Two scene-level fields decide what happens when the clock reaches duration:

• endMode — 'end' (default) finishes the scene and hands the game back. 'hold' parks it there instead: the master clock keeps running, so transformAt holds the last keyframe while looping clips keep cycling — the closing shot stays alive rather than freezing into a photo. A held scene ends only when the player skips it (the same hold-to-skip gesture, or Escape), and onEnd is deferred to that moment — firing it on crossing duration would run the very WARPTOROOM / structure-advance that yanks the player out of the shot you asked to hold. This is the classic end-credits scene: a title flying, music underneath, until the player decides to leave.
• keepMusic — by default the scene's music (an audio track or a MUSIC: cue — both ride the music bus) is stopped on the way out. Set this and it survives into whatever comes next: the menu, the next structure block, the room behind it.

Both are edited in the scene panel and only serialized when they differ from the default, so a scene that ends the ordinary way stays byte-identical to before.

Playback: skip and rewind#

The player can click to skip a step and hold to skip a whole scene: holding fills a ~1.1s progress bar, and on completion the scene's clock is warped straight to duration so the next tick crosses the end naturally and fires onEnd — the runtime never nulls out the active scene directly, so onEnd can't be stranded by a skip. In a scene with endMode: 'hold' the completed skip ends the scene outright (warping the clock would only re-enter the hold).

The hint for all this stays out of the shot until the player has tried something: it appears on their first key or click during the scene, not over its opening frame. A held scene also surfaces it on its own after a few seconds parked at the end, so nobody sits through the credits without knowing how to leave.

REWINDSCENE:<sceneId>[|speed=N] (default speed 2) plays a scene backward with a VHS skin, driven by the pure seekScene() — a side-effect-free render at an absolute, decreasing time. This is visual-only: seekScene fires no cue, audio, or onEnd tokens while rewinding, by construction (it never touches sceneState or the fired array). There's no authoring UI for it in the Scene Orchestrator itself — it's a runtime-level token, reachable from a cutscene's raw effect slot (the Cutscene Editor's escape hatch) or anywhere else a token can be dispatched, not a distinct track or step type here.

Audio while a scene plays#

A scene owns the soundscape: starting one immediately ducks room ambient (before the async clip load even finishes, so nothing bleeds under a scene during the load gap), and ambient is resumed the moment the scene ends — including the abnormal exits (rewind finishing, or a load failure). A scene's own audio tracks ride the shared music bus via SCENEAUDIO and are unaffected by the ducking, since they're intentionally part of the scene, not the room.

Importing a clip from video#

Clips → NEW CLIP FROM VIDEO picks an mp4 and extracts it in a scene mode that skips the character-only steps (chroma-key cleanup, character-canvas anchoring) and writes full frames to assets/scenes/<clip>/. Knobs: fps (resample — a stylized cutscene often lives at 12–15 fps), scale (1× / 0.75× / 0.5× — pre-set to fit your project's width when the source is wider), and an in/out trim (from + count). Three presets (Light 12fps·0.5×, Balanced 15fps·0.75×, Quality 24fps·1×) set the first two at once.

Preview frame 0 does more than show a thumbnail: it samples a handful of frames spread across the clip to measure how big the PNGs really get (a dark first frame alone can be off by a quarter), and reads the clip's size, fps and length. From then on the line under the buttons shows both costs live as you move the knobs — disk (≈ MB, frames × bytes per frame) and RAM when the scene plays (the decoded figure the limits above are about) — and turns amber or red with the gate.

The import runs next to its destination: frames are extracted straight into a working folder beside assets/scenes/<clip>/ on the project's own drive (never the system temp folder), the progress bar follows ffmpeg frame by frame, and the finished folder is swapped into place at the end. Before starting, the importer checks the free space on that drive against the estimate and refuses up front rather than failing halfway. Opaque sources are written as RGB; alpha is kept only when the video actually carries one. On import, the new clip is added to the project's clip registry (persisted the next time you save from the Clips manager). The ffmpeg used is the one you pointed Hub Config at, or the one on your PATH.

Workflow#

1. Pick an existing Scene or + New to start one blank.
2. Clips → NEW CLIP FROM VIDEO to bring in cinematic footage (or reuse an existing clip), then +🎬 / +🔊 / +✨ on the timeline to add a sprite, audio, or full-screen FX track — plus Add text / Add trigger from the add-track panel for a caption or a cue track.
3. For a sprite track, place keyframes on the timeline to move/scale/rotate/fade it over time, with easing per segment; drag its in/out handles to gate visibility.
4. Add cue tokens at specific times for mid-scene beats (sound, flags, warps); set the scene's onEnd list for what happens once it finishes.
5. Preview with the transport controls (play/pause, full rewind/forward, stop) and the zoomable timeline; use Replace Color if you need a recolored variant of an imported PNG.
6. Save — writes projects/<id>/scenes/<sceneId>.js and registers the id.
7. Wire it in from a PLAYSCENE:<id> (or REWINDSCENE:<id>) token anywhere a token can fire — most commonly a Cutscene Editor PLAYSCENE step, a hotspot reaction, or a Game Structure Designer scene block.

A scene's cue and keyframe times are baked against the imported clip's frame count and fps at import time. Every t on a cue, a keyframe, or an in/out window is an absolute millisecond offset into the scene's master clock — there's no relationship kept back to the source mp4 once frames are extracted. Re-importing a trimmed or re-encoded version of the same footage (different fps, different trim points) produces a new frame sequence whose timing no longer lines up with cues and keyframes authored against the old one — they'll fire early, late, or into blank frames. Treat an imported clip as final before spending time placing cues and keyframes against it; if you must re-import, expect to re-check every cue and keyframe on tracks using that clip.