Architecture¶

Select the diagram to open it at full size and zoom in.

Edit the D2 source. After editing, regenerate the SVG using the command in the repository README; Zensical serves the SVG as a static asset rather than rendering D2 code blocks.

Scope¶

One operator: a shared password entered in the UI; no accounts, roles, or multi-user support.
Exactly two tabs: Start a meeting (Together API key, expected number of people, named voice enrollment, start/stop capture) and Meeting (live transcript and session status).
One active meeting at a time on one server instance. Session and enrollments are ephemeral. No meeting history, file upload, summary, or transcript export.
One configured language. Set it on the server and provide the corresponding Together language hint; validate it against the chosen Parakeet model.
Physical setup: several people in one room speak into one shared microphone connected to the operator's computer. One physical loudspeaker/output device may exist but is not an additional audio input or a person. The one microphone sends a mixed audio feed; there are no per-person channels. No conferencing bot or capture from another application is included.

Responsibilities¶

SvelteKit serves the two-tab UI, verifies the shared password, accepts enrollment recordings, and enforces one active meeting. After login, it issues an HTTP-only, SameSite session cookie (Secure under HTTPS). It issues a short-lived, meeting-scoped ticket for the live audio connection. The operator enters the Together key in Start a meeting; keep it in backend memory for the session, pass it to the speech service over a protected internal channel, and clear it on stop. Never return it to the browser or put it in a URL, cookie, local storage, transcript, or log. Require HTTPS/WSS when accessed off localhost.

The speech service is a persistent backend process. It accepts enrollment audio from SvelteKit, loads a speaker-verification model, computes reference embeddings, and holds the named profiles in memory. For the meeting, it authenticates the audio ticket, timestamps and buffers PCM chunks, runs streaming diarization (locally or through a hosted inference endpoint), streams the same audio to Together, compares clean live speaker samples with enrolled embeddings, and merges speaker labels with transcribed text. It sends revisions back to the UI. It must preserve Nemotron's per-meeting streaming state; a fresh stateless inference call for every chunk would lose speaker continuity.

Together transcribes with nvidia/parakeet-tdt-0.6b-v3 using its real-time API, provided that the exact model is confirmed on a live connection. Nemotron-3-Diarization reports anonymous, arrival-ordered speaker activity for up to eight people; it does not consume an enrolled reference embedding. The embedding model and matcher are separate backend components that map Nemotron's session-local speaker labels to names. Nemotron's internal speaker cache is not a verification-embedding store and cannot be initialized with the profiles from enrollment.

Meeting flow¶

In Start a meeting, enter the expected number of people (1–8) and the Together key. Each person takes a turn speaking alone into the same microphone to record one named, clean voice sample. Enrollment audio is sent to the backend; the backend validates it and computes an embedding with a dedicated speaker-verification model such as SpeechBrain ECAPA-TDNN. Show the status of each slot and require all requested slots to succeed before starting. The entered count is a validation hint, not a guarantee that Nemotron will find exactly that many speakers.
The operator starts the meeting. The browser captures a single 16 kHz mono PCM feed from the shared microphone and sends chunks to the speech service using the meeting-scoped ticket. The backend assigns sample-based audio offsets and feeds the same stream to Nemotron and Together. It does not rely on each person having their own microphone.
Nemotron emits timed speaker-activity probabilities/segments, including possible overlap, labeled anonymously in arrival order. These labels are stable only within that inference session, not across meetings or reconnects.
For a sufficiently long, clean, non-overlapping segment attributed to an anonymous speaker, the backend extracts a new embedding with the same verification model used for enrollment. It compares that vector with the named reference vectors. Only when a calibrated threshold and margin over competing names are met does it map that session-local label to a name; otherwise show Speaker 1, Speaker 2, etc., or Unknown if speaker attribution itself is unclear. Neither reference vectors nor live verification vectors are injected into Nemotron.
Together supplies replaceable interim text and final utterance text. The backend joins available timing and Nemotron's speaker activity with those utterances and pushes a revisable live transcript to Meeting. Do not promise exact word-level named attribution until Together's actual live timestamp behavior is verified.
On stop, flush remaining transcription, close the live inference streams, and clear the Together key and enrollment profiles. The final text may remain on the currently open browser page but is not persisted.

Inference-server choices¶

Hosted: Baseten's Nemotron 3 streaming deployment exposes a live WebSocket for 16 kHz mono PCM and returns anonymous speaker turns. It requires a separate Baseten deployment and backend-configured API key. The speech service still performs enrollment, verification, naming, and alignment; its Nemotron inference call points to Baseten instead of a local model. Baseten's separate streaming diarized transcription preset uses its own multitalker Parakeet, not the requested Together model, and does not provide enrolled names.

Self-hosted: Run a persistent Python service on a suitable Linux NVIDIA GPU. Load the Nemotron-3-Diarization checkpoint once with NeMo, preserve its cache for the meeting, and run the verification model there. Implement the audio WebSocket and separate Together connection. NVIDIA's streaming ASR integration guide describes the per-session inference structure; its example driver itself replays a file rather than providing this complete server. Triton can serve stateful models but would add complexity to a one-stream POC.

Neither is a turnkey enrollment server: Both routes need a backend matcher. NeMo-Speech.cpp documents local serving for a Sortformer diarizer but does not establish support for this exact Nemotron-3 checkpoint plus enrolled identification. For the shortest model-serving path, use Baseten if a second hosted provider is acceptable; otherwise self-host NeMo.

Limits to test before implementation¶

Transcription timing: Together lists nvidia/parakeet-tdt-0.6b-v3 among its STT models and documents a real-time WebSocket, but the documented live events show text without word timestamps. Confirm this exact model accepts a live connection and inspect its event payloads. Otherwise text-to-speaker alignment is approximate at the utterance level.
Overlapping voices: Nemotron can flag simultaneous speakers. A single conventional ASR stream on mixed audio can still merge or omit their words, and a clean verification embedding cannot be computed from strongly overlapped speech. NVIDIA's supported overlap-aware Nemotron + multitalker streaming Parakeet pipeline is a different ASR deployment from Together's Parakeet TDT model.
Latency: Nemotron publishes 0.32–1.04-second low-latency input buffers; inference, networking, ASR finalization, and verification add time. Live updates are feasible; instant, guaranteed-final named words are not.
Acoustics: All participants must be audible through the shared microphone. Enroll them through that microphone, individually, under similar conditions, then test speaker matching with the actual room and equipment. One shared mic does not guarantee equal audio quality for every person.