Technical Docs

How SideWave turns speech into pasted text, and where each part lives in the source. The repository README covers building and testing.

Overview

SideWave is a Swift app built with SwiftUI, with AppKit for the global shortcut, the floating preview, and Accessibility-based delivery. project.yml generates the Xcode project with XcodeGen, and Swift Package Manager supplies the dependencies.

AppState coordinates the UI and the session. Services under Sources/SideWave/Services/ are grouped by area:

• Audio/: input devices, channel selection, microphone test, and standby.
• Transcription/: TranscriptionService, which owns capture, the speech models, segments, and the staged transcript.
• Refine/: DictationFormatterService, which runs Apple Intelligence and Gemma.
• Paste/: TypingService and window targeting.
• Vocabulary/, History/, Documents/, Meetings/, and Models/.

SideWave2 and SideWaveKit are a smaller rebuild around one explicit session state machine. Apart from the Refine output checks in Sources/Shared, they're separate from the main app and serve as a reference for moving its engine one tested boundary at a time.

Dictation pipeline

Capture

Audio comes from AVAudioEngine on the selected input. Auto channel selection picks the strongest active channel of a multichannel interface; you can also pick a channel or a mix. The tap converts audio to 16 kHz mono for the speech models. The shortcut starts capture on key down, and an optional standby setting keeps the input device awake between sessions to shorten startup.

Draft, confirmation, and segments

Three models run during a session in the default Complete mode:

• A streaming Parakeet end-of-utterance model reads 160 ms chunks and produces the gray live draft.
• Silero VAD runs on the CPU, so it doesn't compete with the speech models for the Neural Engine, and tracks where speech starts and stops.
• Parakeet TDT v3 (or Ultra, or v2) confirms the audio in segments. Its encoder takes at most 15 seconds of audio, so a segment ends at 14.8 seconds, or earlier at a pause when you choose a faster cadence.

When a segment finalizes, the CTC 110M model rescores it against your enabled glossary terms. Number formatting, filler and profanity filters, and glossary replacements follow.

Guards against invented words

Speech models can produce short phrases from background noise. While VAD is running, SideWave commits only text with detected speech behind it: a segment recorded before VAD has heard any speech is dropped, and words timed after the end of detected speech are trimmed. Without VAD the guards stand aside rather than drop words. A ghost gate also holds back draft words that keep growing while the microphone is silent.

Refine during and after the session

Each finalized segment goes to Refine. The request can still edit the last two sentences of the previous segment (up to 64 words), so a sentence that continues across a boundary gets repaired, and it includes up to 400 earlier words as read-only context. Template instructions, glossary hints, the speech profile, and the output language are part of the prompt.

When you stop, SideWave refines whatever live Refine hasn't finished, in requests of up to 160 words, each able to revise the two sentences before it. A segment that failed earlier gets another try. The final pass has a budget of 10 to 25 seconds, scaled to the unfinished words. A part the model declines is kept as captured and left out of later requests; when the budget runs out, the rest is kept as captured. Either way SideWave pastes and shows a notice. When translating, it keeps the text on Home and in History instead, so untranslated words aren't pasted.

Apple Intelligence on macOS 27 applies its default guardrails to structured output even with permissive content transformations, which apply only to plain text. When it declines a Standard, Concise, or Professional refine, SideWave asks again for plain text before keeping the part as captured.

Document templates (Email, Instructions, AI Prompt, Custom) clean each segment as ordinary prose while you talk and apply the document format once after you stop, which avoids formatting a half-finished thought.

Models for Refine

Apple Intelligence uses the Foundation Models framework. On macOS 27 it keeps a bounded session across turns; on macOS 26, ordinary dictation uses plain-text output because the structured path there kept false sentence breaks. Gemma 4 runs through MLX on the GPU and reuses its verified token prefix within a session. Changing the template, instructions, vocabulary, profile, or language resets that cache. Every result passes length and ratio checks before it replaces the recognized text.

Delivery

SideWave records the target app (and, when it can, the text field) when you start. Before pasting it checks that the target is still in front. TypingService tries Unicode key events, Accessibility insertion, and clipboard paste with the previous clipboard restored, in an order that depends on the app and the text length. It retries only when it has evidence the text didn't arrive, so an unconfirmed paste is never repeated. Undelivered text stays on Home and in History.

Shortcut

A CGEvent tap watches key and modifier events. Modifier-only shortcuts such as Right Option come through as flagsChanged events, and a press that's combined with another key is ignored. A hold past 500 ms (adjustable) is push-to-talk, and two taps within 300 ms paste the last dictation again. macOS can disable event taps, so a background check re-enables or recreates the tap.

Storage and recovery

SideWave's data lives in ~/Library/Application Support/SideWave/: history.json, vocabulary.json, document drafts, meeting sessions, and recovery journals. Meeting audio and drafts use owner-only permissions. A dictation's journal is removed once its text is saved to History. After a crash, the next launch turns readable journal lines into a Recovered entry. Unreadable data files are kept rather than overwritten.

Meetings

Meeting capture writes mono audio at the device's native rate to numbered files, and every 10 seconds it flushes the file and updates a manifest, so an interruption loses little. After recording, Parakeet transcribes the saved audio and NVIDIA Nemotron 3 assigns up to eight speakers. Long transcripts are summarized in sections sized with the model's tokenizer, then combined into one set of notes.

Models and downloads

Speech models download from Hugging Face into ~/Library/Application Support/FluidAudio/Models/; Gemma uses the shared Hugging Face cache. Before loading, SideWave checks tokenizer and template files, shard indexes, and weight headers, and it repairs missing small files without replacing weights. Core ML compiles each model on first load, which takes longer than later launches.

App-managed requests go through an observer that records host, path, size, and time for the Data Transparency window. It rejects request bodies and uploads, and it never logs transcript text.

Releases

scripts/release.sh builds the Release configuration, signs the nested code and the app with Developer ID and a secure timestamp, notarizes and staples the app, and packages SideWave.dmg, which is notarized too. It signs the DMG for Sparkle and writes the appcast entry. scripts/validate_release.py checks signatures, entitlements, and the published feed. Sparkle reads https://sidewave.dev/appcast.xml and verifies each update's EdDSA signature.

Dependencies

Model licenses are listed in the acknowledgments.