Voice Editor#

The Voice Editor bakes local text-to-speech voice-acting for the game's spoken lines and generates the lip-sync mouth-shape cues that drive a talking character's animation while a line plays. It reads/writes a per-project voice manifest (voice/voice_manifest.json) that declares a voice per character and a set of voice lines (which line, spoken by which character, with which per-line parameter overrides). The manifest does not store the spoken text: a voice line's id is its lid, so the words are a strings.json record and the manifest only points at it; generation itself runs through a local model (VoxCPM2) via an offline Python pipeline (tools/voicegen/), invoked from the editor through dev-server job endpoints. Nothing here ships in the game runtime — only the resulting WAVs and lip-sync JSON files do.

You reach it from the Hub (dev server running). The left VOICE column holds the selected character's voice configuration; the right VOICE LINES column lists that character's lines, either as a compact paginated list or, once you expand one, a single focused line with its text, takes, and lip-sync controls. The text is shown read-only — see below.

What gets voiced#

The editor doesn't invent a separate script — it imports the game's own spoken text via 📥 Import from game, so a voice line's text IS the line the game actually displays, with no double authoring:

• strings.json (the content-i18n string table, core/strings.js) is the primary source — but only its spoken lines, not every row in the table: strings.json also carries hotspot and item names, room descriptions, GUI text, verb labels, engine system messages, and achievement text, and none of those are something a character would ever say out loud. parseNameLid(), the same address-parser the string table uses everywhere else, classifies each lid by what it's an overlay for; only its 'line' kind — a SAY/NPCSAY — becomes a candidate, keyed by the lid itself as the line id. This is the same table the Dialog Editor and Cutscene Editor write to when you author a SAY:#lid / NPCSAY:#lid token — see their docs for how a lid is minted and resolved. Because the line id is the lid, the baked WAV path (.../<char>/<lid>/<lang>/baked.wav) is derivable from the lid alone; core/strings.js's voicePathForLid() is the single pure builder both the editor and the runtime call.
• Dialog trees contribute each node's NPC line too. The id comes from the line's address rather than its text — dialogVoiceLid(dialogId, nodeId), shape dlg_<dialog>_<node> — so re-wording the node later doesn't orphan the baked clip. It's also the same lid the string table stores that node's translations under (the literal in the dialog file stays the default-locale source): one address keys the on-screen text, its translations, and its voice clip. Player choice lines are not imported (the speaker would be whichever playable character is active, which the importer can't resolve).
• Legacy literal SAY:/NPCSAY: tokens still living directly in an unmigrated room file are picked up as a fallback, deduplicated by character+text and given a slugified id.

The import modal shows every candidate line with its speaking character and text, greys out ones you already have (matched by lid or by character+text), and lets you select-all/select-none before adding. Added lines are otherwise ordinary rows you can retitle, delete, or add manually (+ Line) — importing is a convenience, not the only path in.

The text is read-only here#

The per-language text boxes on a focused line display the line's words; they don't author them. Because a voice line's id is its lid, the text is a record of the project's strings.json, and it is edited where content is authored: in the game (the SAY: token itself) and in the Translation Editor for the other locales.

This used to be a text field that wrote into the manifest, which meant the same words lived in two files with nothing keeping them in step — and the two feed different consumers: the tools/voicegen/ pipeline bakes the WAV from one while the engine renders the subtitle from the other. One edit on either side and the player hears one line and reads another. A line that still carries its own text is a legacy or not-yet-lidified line; the box says so, and the pipeline keeps reading it through a back-compat branch.

Per-character voice configuration#

Every game character (from characters.json) appears automatically as a voice card; you can also add extra voices (+ Extra voice) for NPCs that aren't playable game characters, with a freely editable id. Each voice has:

• Source — Voice Design (a text prompt describing the desired voice, no audio needed — VoxCPM2 synthesizes a speaker from the description) or Human (cloned from a reference WAV).
• For Human, a reference WAV — either recorded live through the browser's mic (with a listen-before-save preview) or imported from a file — plus an optional style prompt that nudges emotion/pace at synthesis time without changing the cloned timbre.
• Shared voice params — rotary knobs for cfg_value and inference_timesteps, plus a seed (blank = random each bake, or pin one for reproducibility with a 🎲 reroll). The knob set itself isn't hardcoded in the editor: both it and generate.py read the same tools/voicegen/param_spec.json, so adding a knob is a spec change, not an editor rewrite.
• A voice preview box (venv-gated) synthesizes an ad-hoc phrase with the character's current config and plays it back immediately — audition a tweak before committing it to a real bake.

Per-line overrides (in the focused line view) let a specific line override any of those same knobs; a dimmed knob means it's currently inheriting the character default, and double-clicking a line-override knob deletes the override rather than just resetting its value.

Generation: the local TTS pipeline#

Baking calls into tools/voicegen/generate.py, which must run under a dedicated Python venv (imports torch/voxcpm — never loaded by core/, shell/, or dev_server.py itself). The editor never shells out directly to a venv path; it POSTs to /api/voice/generate, which starts a background job, and the editor polls /api/voice/generate-status once a second until it reports done. Two entry points:

• ⟳ Regenerate (per focused line/language) — forces a fresh take.
• Bake character — batch-bakes every line belonging to the selected character, skipping any line whose existing take already matches the current text + params + reference hash (so re-running costs nothing once everything's current).

The first time a bake is attempted, window.ensureModelConsent('voxcpm2', …) (tools/_ai-consent.js) gates it: the model is a multi-GB Hugging Face download, so the user is warned once before it's pulled — subsequent calls check localStorage and a server-side /api/model-status cache probe, and only re-prompt if neither says yes. The model id/label/size come from the shared tools/ai-models.json manifest, not hardcoded strings — the Audio Editor's MusicGen gate uses the identical mechanism.

Takes are non-destructive#

Every (re)generation appends a numbered take (take_01.wav, take_02.wav, …) rather than overwriting; a per-line takes.json records each take's params/text/hash and which one is baked (canonical — the file lip-sync and the runtime actually play). The focused line view lists every take with a play button and a radio-style set as baked control, so you can audition alternates and pick a favorite without losing the others. A take can also be routed through 🎚 Voice FX, which opens the full Audio Editor pedalboard in a modal against that one take and posts back a new take (never mutates the original) when you save there.

Status badges#

Each line/language shows one of: missing (no takes yet), takes (takes exist, none canonical), baked (canonical take's hash matches the current text/params — up to date), or stale (canonical take exists but its hash no longer matches — the manifest changed since that take was generated). The badge is computed from the SAME hash generate.py uses to decide whether a bake is needed, so the editor's status and the CLI's "nothing to bake" decision can never disagree.

Lip-sync cue baking#

👄 Bake lip-sync (per line, or Bake lip-sync (all) for the whole character) is a separate action from voice generation, offered independently of the VoxCPM2 venv — all it needs is an already-baked baked.wav and the Rhubarb Lip Sync speech model. That model (about 70 MB) isn't in the installer: the first bake asks whether to fetch it, shows the download percentage while it does, and from then on it's cached and every later bake runs offline. Decline and the bake simply doesn't happen — nothing is downloaded behind your back. If you'd rather supply your own copy of Rhubarb, point Hub Config at it and that path wins instead. It runs synchronously through /api/voice/lipsync (Rhubarb is fast enough that the HTTP response IS the result, no polling), writing lipsync.json next to the WAV: { dur, rec, shapes, cues:[{t, shape}, …] }, one cue per Preston-Blair mouth shape (A–H, X = rest), timed in seconds against baked.wav. English lines use Rhubarb's text-assisted pocketSphinx recognizer (fed the line's own text as a dialog hint for accuracy); every other language falls back to the phonetic, language-agnostic recognizer. The focused line view renders the baked cues as a read-only, proportional cue-timeline strip (one colored segment per cue, width ∝ duration) so you can sanity-check a bake without opening the runtime.

At runtime, shell/voice.js is what actually consumes this: whenever a voiced SAY/NPCSAY line begins playback, it lazily fetches that line's lipsync.json and points a single global lip-sync target (VOICE.lipsyncActive) at the speaking actor. Each frame, visemeFrame(shape, frameCount) maps the active cue's shape to an openness value (0 for closed/rest, up to 1 for wide-open) and picks the corresponding frame out of however many frames the actor's _talk animation happens to have — 2 frames (closed/open) and up to 9 (one per shape) both work. No cues, voices muted, or clip already ended all fall back to the ordinary talk-flap animation, so an un-lip-synced line still animates, just without mouth-shape precision.

Staleness and regeneration#

Both bakes key off a hash of text + effective params + source/design/ref + reference-file mtime. Edit a line's text (or a character's design prompt, style prompt, params, or reference clip) and its badge flips from baked to stale — but the existing baked.wav and lipsync.json are left untouched and still play in-game exactly as before. Nothing forces a re-bake; stale is a warning, not a gate. Regenerating the voice (⟳ Regenerate / Bake character) produces a new take against the new hash and, if you set it baked, a new baked.wav — but lip-sync is not re-baked automatically. Re-run 👄 Bake lip-sync for that line afterward, or the cue timeline stays wired to whatever the previous baked.wav sounded like.

Workflow#

1. 📥 Import from game to pull in candidate lines from strings.json (SAY/NPCSAY lids) and dialog trees; select the ones you want and add them.
2. Pick a character in the toolbar, configure its Source (Design prompt or Human reference + optional style), and tune the shared voice params.
3. Use Voice preview to audition the config on a throwaway phrase before spending a real bake.
4. Bake character (or ⟳ Regenerate a single focused line) — wait for the consent gate the first run, then the job poll.
5. Review takes per line, listen, and pick the one to set as baked if the first isn't the best.
6. 👄 Bake lip-sync (per line or for the whole character) once the voice take you want is canonical.
7. 💾 Save the manifest. Everything above (regen, set-baked, lip-sync bake) reads the manifest from disk, so an unsaved edit blocks those actions until you save first. If someone else (or a CLI run) wrote the manifest since you loaded it, Save shows a conflict prompt — keep your local edits or overwrite theirs, never resolved automatically — and a server that can't be reached surfaces an error instead of hanging.

Stale voice and stale lip-sync can silently drift apart from the text. Editing a SAY line's text after it's been voiced does not invalidate the audio or its mouth cues — it only flips a badge. If you skip regeneration, the character will keep speaking the old line (correctly lip-synced to itself) while the on-screen subtitle shows the new, edited text. And if you regenerate the voice but forget to re-bake lip-sync, you get a fresh, correct-sounding clip driving mouth cues that were timed against the previous recording — usually still watchable, but no longer precisely matched. Treat stale badges as a to-do list, not a display quirk.