assemblyai

AssemblyAI Speech-to-Text and Voice AI

AssemblyAI provides speech-to-text APIs, audio intelligence models, and an LLM Gateway for applying language models to transcripts. This skill corrects common mistakes that training data gets wrong — deprecated APIs, discontinued SDKs, and non-obvious auth patterns.

Authentication

All endpoints use the same header:

Authorization: YOUR_API_KEY

NOT Authorization: Bearer ... — just the raw API key, no Bearer prefix. This is the #1 mistake.

Base URLs

Streaming EU region: As of March 2026, the EU region moved from AWS eu-west-1 (Ireland) to AWS eu-north-1 (Stockholm). The customer-facing endpoint host (streaming.eu.assemblyai.com) is unchanged.

SDKs

Only Python, JS/TS, and Ruby SDKs are maintained. For Java, Go, or C#, use the REST API directly.

Speech-to-Text Models

Pre-Recorded

Use speech_models as a priority list with fallback: ["universal-3-pro", "universal-2"] (the default when omitted).

Universal-3.5 Pro is also available for pre-recorded/async transcription, not just streaming. Opt in with speech_models: ["universal-3-5-pro"] on POST /v2/transcript. It supports contextual prompt (a plain-language description of the audio — domain/scenario/full detail) and keyterms_prompt (up to 1,000 terms), does native code-switching, and auto-falls back to Universal-2 for languages outside its 18. Note: the formal OpenAPI speech_models enum still lists only universal-3-pro/universal-2, but the async API accepts universal-3-5-pro.

Streaming

For realtime/streaming STT, use speech_model: "universal-3-5-pro" by default. The raw API parameter is optional and defaults to universal-3-5-pro; set it explicitly when pinning behavior or using an SDK that requires the field. The mode connection param (universal-3-5-pro / u3-rt-pro) trades off accuracy vs latency: min_latency, balanced (default), max_accuracy.

Medical Mode (Add-On)

domain: "medical-v1" enables Medical Mode — an add-on that improves accuracy for medical terminology (medications, procedures, conditions, dosages). Works with both pre-recorded and streaming models.

• Pre-recorded: Universal-3 Pro (domain: "medical-v1" in request body), Universal-2
• Streaming: universal-3-5-pro, u3-rt-pro, universal-streaming-english, universal-streaming-multilingual
• Supported languages: English, Spanish, German, French (4 languages only)
• Billed as a separate add-on. If used with an unsupported language, the API ignores domain and returns a warning — transcript still completes and you are NOT charged for Medical Mode.

Prompting (Universal-3.5 Pro)

prompt and keyterms_prompt are complementary — use either, or both together. Neither changes the output format, and both work the same way for streaming and async (POST /v2/transcript). Transcription behavior (verbatim, punctuation, formatting) is built in and managed by AssemblyAI.

• prompt (string, ~1500 chars max): a plain-language description of the audio — its domain, scenario, or full details. It carries context, not instructions — formatting/behavioral commands (punctuation rules, "transcribe verbatim", "don't…") are not supported and are ignored. The model stays grounded in the audio: irrelevant or only-partially-applicable context won't make it insert words that weren't spoken, so you can safely send the same description on every session/segment.
• keyterms_prompt (string[]): an explicit list of names/brands/domain terms to boost. Streaming: up to 100 terms, ≤50 chars each. Async: up to 1,000 terms.

Contextual prompt — three levels of specificity (use the least specific that covers your case):

Best practices:

• Start with no prompt and no keyterms_prompt — the model is optimized out of the box. Add context only for domain vocabulary it gets wrong, starting at the broadest level.
• Write plain, complete sentences that describe the recording; keep it to one short block, not a keyword list (that's what keyterms_prompt is for).
• Keyterms: use exact spelling/capitalization; avoid common words (over-inclusion causes overcorrection/hallucination).
• Specify language via language_code (preferred) or by naming it in the prompt (e.g. "Spanish customer support call…").
• Streaming: update both mid-session via UpdateConfiguration; a new keyterms array replaces the prior set, [] clears it.

Sync STT API (short-form audio, ≤120s)

A separate synchronous endpoint for short clips — send audio in one HTTP request, get the transcript back in the response. No polling, no transcript ID, no upload step. Ideal for voice-message transcription, short call recordings, or voice-agent pipelines that do their own turn detection and submit completed utterances.

• Endpoint: POST https://sync.assemblyai.com/transcribe (global default — routes to nearest region; use sync.us.assemblyai.com / sync.eu.assemblyai.com for data residency)
• Required header: X-AAI-Model — the current quickstart uses universal-3-5-pro; u3-sync-pro (Universal-3 Pro) is also accepted (the value in the formal spec enum)
• Auth: Authorization: YOUR_API_KEY (Bearer prefix optional here, unlike the async REST API; or pass ?token=YOUR_API_KEY)
• Body: multipart/form-data with an audio part (Content-Type: audio/wav or audio/pcm) and an optional config JSON part
• config fields: prompt (≤4096 chars), word_boost (string[], ≤2048 chars total — this is the documented keyterms param for Sync, not keyterms_prompt), conversation_context (string or string[] — prior conversation turns oldest-first, for continuity across a multi-turn conversation; oldest dropped when over the context budget), language_code (ISO 639-1 string or list, default en — steers the default prompt toward the named language(s); ignored when a custom prompt is set), and for audio/pcm also sample_rate + channels (required for raw PCM; WAV reads them from its header)
• Audio limits: 80ms–120s, ≤40MB, 16-bit only, mono/stereo (stereo down-mixed), sample rates 8000/16000/22050/24000/32000/44100/48000 Hz
• Response: { text, words[{text, start, end, confidence}], confidence, audio_duration_ms, session_id } — word timestamps use start/end (integer milliseconds), same field names as the async API; only the clip-level audio_duration_ms carries the _ms suffix
• 30s per-request deadline (504 inference_timeout). For audio >120s use the async REST API; for live mic audio use Streaming.

curl -X POST https://sync.assemblyai.com/transcribe \
  -H 'Authorization: YOUR_API_KEY' \
  -H 'X-AAI-Model: universal-3-5-pro' \
  -F 'audio=@sample.wav;type=audio/wav'

LeMUR is Deprecated

LeMUR is deprecated (sunset March 31, 2026 — already sunset). Use the LLM Gateway instead. The LLM Gateway is an OpenAI-compatible API. Key difference: you pass transcript text directly in messages (no transcript_ids). Transcribe first, then include transcript.text in your prompt.

See references/llm-gateway.md for models, tool calling, structured outputs, and examples.

Key Gotchas

Common Mistakes

Reference Files

Read the relevant reference file based on what the user needs:

API Spec Source of Truth

https://github.com/AssemblyAI/assemblyai-api-spec