Voice (TTS & STT) - Backboard API

Use send_message / sendMessage, the preferred entry point, to add speech to a message without creating an assistant or thread first. Use stream_voice / streamVoice for the live microphone pipeline. For one model that hears and speaks directly, use Realtime Audio.

Set up once

Install pip install 'backboard-sdk[voice]' for Python, or npm install backboard-sdk for Node.js. Set BACKBOARD_API_KEY on your backend. Never put it in a browser bundle.

Create the client, then send your first message. Backboard creates the assistant and thread automatically and returns both IDs. The snippets below run inside the Python main() function or after the Node.js setup.

import asyncio
import os
from backboard import BackboardClient

async def main():
    async with BackboardClient(api_key=os.environ["BACKBOARD_API_KEY"]) as client:
        # Paste a message example here.
        pass

asyncio.run(main())

Each example below starts fresh because it omits IDs. Pass thread_id / threadId to continue a conversation, or only assistant_id / assistantId for a new thread sharing that assistant’s memory and saved configuration. Per-call system_prompt and tools are not saved to the assistant; re-pass overrides when needed.

Mode 1: STT + LLM

Send a recording and receive a text answer. Replace question.wav with your file.

response = await client.send_message(
    audio_file="question.wav",
    voice={"stt": {"provider": "openai", "model": "gpt-4o-mini-transcribe"}},
    llm_provider="openai", model_name="gpt-4o-mini", send_to_llm="true",
)
print(response.content)
print("Conversation IDs:", response.thread_id, response.assistant_id)
for message in response.messages:
    records = message.get("voice_records") or {}
    if records.get("stt"):
        print("You said:", records["stt"].get("transcript"))

Mode 2: LLM + TTS

Send text and get a spoken answer. The result contains the answer and a temporary audio URL. This speaks the model’s reply, not the text of the question itself.

response = await client.send_message(
    content="Explain gravity in one sentence.",
    voice={"tts": {"provider": "openai", "model": "tts-1", "voice": "alloy"}},
    llm_provider="openai", model_name="gpt-4o-mini", send_to_llm="true",
)
print(response.content)
for message in response.messages:
    records = message.get("voice_records") or {}
    if records.get("tts"):
        print("Speech URL:", records["tts"].get("audio_url"))

Continue the conversation

After either non-streaming example, reuse the returned thread ID:

follow_up = await client.send_message(
    "Explain that more simply.", thread_id=response.thread_id,
    voice={"tts": {"provider": "openai", "model": "tts-1", "voice": "alloy"}},
)
# For a fresh conversation sharing the same assistant instead:
# follow_up = await client.send_message("New question", assistant_id=response.assistant_id)

Mode 3: STT + LLM + TTS

To send voice and get voice back, use audio_file / audioFile with both settings. Use this voice value in the Mode 1 example, then read the speech URL as in Mode 2:

{
  "stt": {"provider":"openai","model":"gpt-4o-mini-transcribe"},
  "tts": {"provider":"openai","model":"tts-1","voice":"alloy","output_format":"mp3"}
}

You can mix providers by changing just one half, for example stt: {provider: "elevenlabs", model: "scribe_v2"} with the OpenAI TTS settings above. You do not need a separate request for each stage.

Stream and save a spoken reply

This example uploads question.wav, prints the streamed text, and saves the streamed speech to reply.mp3. It uses the client from setup and starts a new conversation. The SDK parses the SSE stream for you.

import base64

stream = await client.send_message(
    audio_file="question.wav",
    voice={
        "stt": {"provider": "openai", "model": "gpt-4o-mini-transcribe"},
        "tts": {"provider": "openai", "model": "tts-1", "voice": "alloy", "output_format": "mp3"},
    },
    llm_provider="openai", model_name="gpt-4o-mini", send_to_llm="true", stream=True,
)
with open("reply.mp3", "wb") as output:
    async for event in stream:
        if event.get("thread_id"):
            thread_id = event["thread_id"]
        if event.get("assistant_id"):
            assistant_id = event["assistant_id"]
        kind = event.get("type")
        if kind == "stt_stream_end":
            print("You said:", event.get("transcript"))
        elif kind == "content_streaming":
            print(event.get("content", ""), end="", flush=True)
        elif kind == "tts_audio_chunk":
            output.write(base64.b64decode(event["data"]))
        elif kind == "tts_stream_end":
            print("\nSpeech URL:", event.get("audio_url"))
        elif kind in ("error", "tts_error"):
            raise RuntimeError(event)

Keep returned IDs when they appear in events; an early progress event may not contain them yet. Streaming event fields stay snake_case in both SDKs.

To stream a spoken response to text, remove audio_file / audioFile and voice.stt, then add content. To stream text only from audio, remove voice.tts and the audio-writing branch.

For immediate playback, send decoded chunks to an incremental player instead of a file. Use a decoder for the requested TTS format; these MP3 chunks are not PCM. The Realtime Audio guide has a raw-PCM microphone/playback example for native conversations.

STT parameters

voice.stt accepts provider, model, optional language, and provider-specific options. The SDK passes the nested object through without renaming its keys:

{"stt":{"provider":"openai","model":"whisper-1","language":"en","provider_options":{"openai":{"response_format":"verbose_json","timestamp_granularities":["word"]}}}}

Use this object as voice in an audio-file example. Check model support before adding options; not every transcription model supports word timestamps.

TTS parameters

voice.tts accepts provider, model, optional voice, output_format, and provider-specific options:

{"tts":{"provider":"openai","model":"tts-1","voice":"alloy","output_format":"mp3","provider_options":{"openai":{"speed":1.1}}}}

For ElevenLabs, use its model and voice IDs and put tuning under provider_options.elevenlabs.voice_settings. See the provider parameter guide. Tell users the voice is AI-generated.

Live microphone pipeline

This is the former “Realtime Voice (WebSocket)” guide, now part of Voice. stream_voice / streamVoice uses separate STT, text, and optional TTS models. It is not the native connect_realtime / connectRealtime API.

The following example simulates a microphone with a short recording, sends it in real-time chunks, receives concurrently, and saves one spoken answer. Run it after one of the HTTP examples above, using its returned response.thread_id / response.threadId as thread_id / threadId. This older pipeline still needs an existing thread; native connect_realtime can start without IDs. Convert your file with FFmpeg first:

ffmpeg -i question.wav -f s16le -acodec pcm_s16le -ac 1 -ar 16000 question-16k.pcm

import base64
from pathlib import Path

pcm = Path("question-16k.pcm").read_bytes()
thread_id = str(response.thread_id)
session = await client.stream_voice(thread_id, {
    "voice": {
        "stt": {"model": "scribe_v2_realtime", "sample_rate": 16000, "commit_strategy": "manual"},
        "tts": {"provider": "openai", "model": "tts-1", "voice": "alloy", "output_format": "mp3"},
    },
    "send_to_llm": True, "llm_provider": "openai", "llm_model": "gpt-4o-mini",
})

async def send_recording():
    for offset in range(0, len(pcm), 3200):
        chunk = pcm[offset:offset + 3200]
        await session.send_audio(chunk)
        await asyncio.sleep(len(chunk) / 32000)
    await session.commit()

async with session:
    sender = asyncio.create_task(send_recording())
    try:
        with open("reply.mp3", "wb") as output:
            async for event in session.events():
                if event["type"] == "transcript.final":
                    print("You said:", event["text"])
                elif event["type"] == "content_streaming":
                    print(event.get("content", ""), end="", flush=True)
                elif event["type"] == "tts_audio_chunk":
                    output.write(base64.b64decode(event["data"]))
                elif event["type"] == "input.resumed":
                    await session.send_json({"type": "stop"})
                    break
                elif event["type"] in ("error", "tts_error"):
                    raise RuntimeError(event)
        await sender
    finally:
        sender.cancel()
        await asyncio.gather(sender, return_exceptions=True)

For a real microphone, replace the recording sender with capture callbacks that produce 16 kHz mono PCM16 little-endian. Keep only one receive loop.

Use commit_strategy: "vad" for automatic turns; keep sending audio and silence, without calling commit(). Use manual for a push-to-talk button.
Pause microphone forwarding on input.paused. The server ignores audio until input.resumed; discard, rather than replay, audio captured during the pause.
Decode tts_audio_chunk.data into a streaming player. Do not unmute the mic while buffered assistant speech is still playing locally.
Omit voice.tts for live transcripts and a text answer only.
Pipeline configuration keeps snake_case in both SDKs, including llm_model.
session.begin_event / session.beginEvent contains setup metadata; the SDK waits for it before returning the session.
Keep keys on your backend. For browsers, relay this pipeline through your backend, or use native realtime’s single-use tickets.

See live pipeline settings and events for VAD controls, 8 kHz input, error handling, and the wire protocol.

Audio formats

HTTP audio_file / audioFile accepts supported audio files; live pipeline input is raw PCM, with no WAV header. voice.tts.output_format controls the spoken response: use mp3 for OpenAI or a supported identifier such as mp3_44100_128 for ElevenLabs. Format details.

Language codes

Set voice.stt.language to a supported two-letter code, such as en, es, or fr. Omit it to detect the language automatically. Language details.

Tools and memory

HTTP speech messages use the normal SDK tool-calling flow. For native live conversations, including built-in memory/search and your own functions, use Realtime Audio tools.

​Set up once

​Mode 1: STT + LLM

​Mode 2: LLM + TTS

​Continue the conversation

​Mode 3: STT + LLM + TTS

​Stream and save a spoken reply

​STT parameters

​TTS parameters

​Live microphone pipeline

​Audio formats

​Language codes

​Tools and memory

On this page