Voice (TTS & STT)

Speech to text (STT) turns a recording into words. Text to speech (TTS) turns words into audio. Add either or both to a normal Backboard message.

This guide also covers the live STT → LLM → TTS pipeline. For a single model that listens and speaks directly, use Realtime Audio.

​

Choose a path

For HTTP, stream: false returns one response. stream: true returns events as the work runs. Streaming the response is not the same as streaming microphone input.

Use the SDK guide for Python and Node.js examples. The requests below use the primary POST /threads/messages API directly. Set BACKBOARD_API_KEY in your backend environment. No assistant or thread creation calls are needed. Omit IDs on the first request; the response returns thread_id and assistant_id. Send the thread ID next time to continue, or only the assistant ID to start a new conversation sharing its memory and saved configuration. Omit both to start fresh.

​

Mode 1: STT + LLM

Send question.wav as audio_file. Backboard transcribes it, sends the words to the language model, and returns a text answer.

curl --fail-with-body https://app.backboard.io/api/threads/messages \
  -H "X-API-Key: $BACKBOARD_API_KEY" \
  -F 'voice={"stt":{"provider":"openai","model":"gpt-4o-mini-transcribe","language":"en"}}' \
  -F 'llm_provider=openai' -F 'model_name=gpt-4o-mini' \
  -F 'send_to_llm=true' \
  -F 'audio_file=@question.wav'

The voice field is a JSON string inside multipart form data. The file is a separate field, not a property inside voice.stt.

​

Mode 2: LLM + TTS

Send a text question. Backboard generates an answer and speaks the answer, not the original question. No audio upload is needed.

curl --fail-with-body https://app.backboard.io/api/threads/messages \
  -H "X-API-Key: $BACKBOARD_API_KEY" \
  -H 'Content-Type: application/json' \
  -d "{
    \"content\":\"Explain gravity in one sentence.\",
    \"llm_provider\":\"openai\",\"model_name\":\"gpt-4o-mini\",
    \"send_to_llm\":true,
    \"voice\":{\"tts\":{\"provider\":\"openai\",\"model\":\"tts-1\",\"voice\":\"alloy\",\"output_format\":\"mp3\"}}
  }"

​

Mode 3: STT + LLM + TTS

Send a recording and receive a spoken answer. Combine the same two settings:

curl --fail-with-body https://app.backboard.io/api/threads/messages \
  -H "X-API-Key: $BACKBOARD_API_KEY" \
  -F 'voice={"stt":{"provider":"openai","model":"gpt-4o-mini-transcribe"},"tts":{"provider":"openai","model":"tts-1","voice":"alloy","output_format":"mp3"}}' \
  -F 'llm_provider=openai' -F 'model_name=gpt-4o-mini' \
  -F 'send_to_llm=true' -F 'audio_file=@question.wav'

​

What comes back?

A non-streaming response includes content and voice_records. This is an illustrative subset; IDs, URLs, and wording vary. Only the requested STT/TTS results are populated.

{
  "thread_id": "CREATED_THREAD_ID",
  "assistant_id": "CREATED_ASSISTANT_ID",
  "content": "Gravity pulls objects toward each other.",
  "voice_records": {
    "stt": {"provider":"openai","model":"gpt-4o-mini-transcribe","transcript":"What is gravity?"},
    "tts": {"provider":"openai","model":"tts-1","voice":"alloy","output_format":"mp3","audio_url":"SIGNED_AUDIO_URL"}
  }
}

voice_records.stt can also include audio_url, language, duration_seconds, usage counts, and raw provider_output. voice_records.tts can include character counts, duration, usage, and raw provider output. Treat signed URLs as temporary.

To continue, add "thread_id": "CREATED_THREAD_ID" to your next JSON body, or -F 'thread_id=CREATED_THREAD_ID' to multipart. To share the assistant without sharing conversation history, send assistant_id instead. When both IDs are supplied, the thread takes precedence. system_prompt and tools overrides are per-request; they do not edit the assistant’s saved configuration.

​

Stream the response

Add -F 'stream=true' to a multipart request, or "stream": true to JSON. Use curl -N to see events immediately.

For a full pipeline, expect STT, then the text answer, then TTS. Lifecycle events may appear between these stages:

stt_stream_start → stt_text_delta* → stt_stream_end
→ content_streaming* → tts_stream_start → tts_audio_chunk* → tts_stream_end

* means repeated events. Not every STT model emits partial text.

HTTP events use Server-Sent Events (SSE), for example:

data: {"type":"content_streaming","content":"Gravity pulls"}

data: {"type":"tts_audio_chunk","data":"BASE64_AUDIO_BYTES","chunk_index":0}

Network reads can split an SSE event in the middle. Use the SDK’s stream parser or buffer complete SSE records before parsing JSON. Do not parse each network chunk as if it were a whole event. The SDK example shows how to collect the real bytes into an MP3.

​

Choose a provider

You can mix providers: for example, ElevenLabs transcription with OpenAI speech. Use supported model/voice identifiers for your provider. Examples:

​

STT parameters

For HTTP requests, voice.stt contains:

{
  "stt": {
    "provider": "elevenlabs",
    "model": "scribe_v2",
    "provider_options": {"elevenlabs":{"diarize":true,"timestamps_granularity":"word"}}
  }
}

OpenAI settings can include response_format, prompt, temperature, and timestamp_granularities; support varies by model. For example, use whisper-1 with response_format: "verbose_json" when requesting word timestamps. See OpenAI transcription and ElevenLabs transcription.

​

TTS parameters

{
  "tts": {
    "provider": "openai", "model": "tts-1", "voice": "alloy",
    "output_format": "mp3", "provider_options": {"openai":{"speed":1.1}}
  }
}

ElevenLabs supports voice tuning under provider_options.elevenlabs.voice_settings, including stability and similarity. See OpenAI speech and ElevenLabs speech. Tell users that the spoken voice is AI-generated.

​

Live microphone pipeline

This section replaces the separate “Realtime Voice (WebSocket)” guide. Use it when you want ElevenLabs STT → a text model → optional TTS. For a native audio model, use Realtime Audio instead.

Connect from a trusted backend to:

wss://app.backboard.io/api/threads/messages?api_key=URL_ENCODED_API_KEY

Do not expose this API key in browser code or log the connection URL. A browser app should use your authenticated backend as a relay for this pipeline. The single-use ticket API belongs to native Realtime Audio, not this endpoint.

Send this JSON setup first, then wait for session.begin:

Use the thread_id returned by a previous /threads/messages request. This older pipeline needs an existing thread; native realtime also supports starting with no IDs.

{
  "thread_id": "YOUR_THREAD_ID",
  "voice": {
    "stt": {"model":"scribe_v2_realtime","sample_rate":16000,"commit_strategy":"manual"},
    "tts": {"provider":"openai","model":"tts-1","voice":"alloy","output_format":"mp3"}
  },
  "send_to_llm": true,
  "llm_provider": "openai",
  "llm_model": "gpt-4o-mini",
  "memory": "off"
}

Unlike the HTTP message request, the live pipeline uses llm_model for the text model. Omit voice.tts to receive only transcripts and a text answer.

1. Send raw 16 kHz, mono, signed PCM16 little-endian chunks as binary messages.
2. With manual turns, send {"type":"commit"} after the utterance.
3. Receive the user’s transcript.final, then text in content_streaming.
4. If TTS is enabled, decode tts_audio_chunk.data and play it as the selected format.
5. Wait for input.resumed before the next input turn.

transcript.partial* → transcript.final → input.paused
→ content_streaming* → tts_audio_chunk* → tts_stream_end → input.resumed

This pipeline takes turns: audio sent between input.paused and input.resumed is ignored. Stop forwarding microphone audio during that time. Your local player may still have queued speech after input.resumed; wait for playback to finish before unmuting the mic to avoid transcribing the assistant’s voice.

For hands-free use, set voice.stt.commit_strategy to "vad" (the default) instead. Keep sending microphone audio and silence, and let the provider detect the end of a turn. Do not send manual commits in that mode.

Send {"type":"stop"} when done. Handle error, tts_error, and unexpected disconnects. See the complete SDK pipeline example.

​

Audio formats

• HTTP STT: upload a real audio file, such as WAV or MP3. Supported containers and upload limits vary by provider and model; check the provider references above.
• HTTP/streamed TTS: use output_format. OpenAI supports formats including MP3, WAV, PCM, Opus, AAC, and FLAC. ElevenLabs uses identifiers such as mp3_44100_128 or pcm_24000. Do not request WAV for ElevenLabs streaming TTS.
• Live pipeline input: raw mono PCM16, not a file container. Convert recordings with ffmpeg -i question.wav -f s16le -ac 1 -ar 16000 question-16k.pcm.
• Native realtime: different negotiated audio formats; see Realtime Audio.

​

Language codes

voice.stt.language is an optional ISO 639-1 language code: for example en (English), es (Spanish), fr (French), hi (Hindi), or ja (Japanese). Omit it for automatic detection. Supported languages depend on the provider.

​

Tools and memory

For HTTP voice messages, tools and memory are part of the normal message run. Use tool calling to handle required actions; do not assume that a tool request is already a finished spoken answer. For live audio conversations with hosted and custom tools, use the Realtime Audio tool guide.