streamcore-server

# StreamCore ### Realtime media infrastructure for AI-powered applications **Talk to your AI over WebRTC — with interruption, streaming speech, and NAT traversal handled.**
One Go binary. Bring your own agent. [](https://github.com/streamcoreai/streamcore-server/actions/workflows/ci.yml) [](./go.mod) [](https://www.rfc-editor.org/rfc/rfc9725.html) [](./LICENSE) [](https://github.com/streamcoreai/streamcore-server/stargazers) [](https://discord.gg/xKGFaGWawT) [](https://x.com/jasonshen_) [**Quick start**](#quick-start) · [**Docs**](./docs/) · [**Demo**](#demo) · [**SDKs**](#sdks-and-examples) · [**Roadmap**](./docs/roadmap.md) · [**Discord**](https://discord.gg/xKGFaGWawT) · [简体中文](./README.zh-CN.md)

Anyone can demo a voice agent. Then a real caller talks over it, pauses mid-sentence, dials in from behind a firewall that blocks UDP, or waits three seconds for the first word — and the demo stops being a product.

StreamCore is the layer that handles all of that. It owns the latency-sensitive media path between your users and your AI: WebRTC transport, adaptive turn-taking, barge-in, streaming STT/LLM/TTS, NAT traversal, session state, and realtime events — across browsers, phones, backends, telephony, and embedded devices.

What it deliberately does not own is your agent. Keep your prompts, tools, models, and business logic exactly where they are — four supported ways, no fork required.

Built with it: voice agents, realtime copilots, live translation, AI-hosted audio rooms, embedded voice devices, and phone applications.

Demo

Quick start

Two terminals, five minutes, and you are talking to it. Needs Go 1.25+ (or Docker) and API keys for an STT, LLM, and TTS provider. No keys? Run it fully local with Ollama + VibeVoice.

cp config.toml.example config.toml   # add your provider credentials
go run .

The server listens on :8080; clients connect to http://localhost:8080/whip.

Then talk to it from a browser:

git clone https://github.com/streamcoreai/examples.git
cd examples/typescript && npm install && npm run dev

Open http://localhost:3000 and start talking.

Docker, TURN ports, and production notes: Quick start guide.

What you get

Full capability list: Capabilities.

Not built yet

Listed so the table above stays honest — unticked items are real gaps today, not soon-shipping promises. Ticked ones shipped recently and stay listed for a release or two so you can see what moved:

• [x] Session reconnection (server) — a dropped connection recovers on the same session via ICE restart, so the conversation and the running pipeline survive it
• [x] Client-driven reconnection — the TypeScript, React Native, Go and Rust SDKs recover a network change automatically: ICE restart first, then a resume redial if the connection failed
• [x] Session resume — a drop past the point ICE restart can help is recovered by redialling with a single-use token, reattaching to the running conversation. Every SDK runs restart-then-resume as one ladder, so a backgrounded phone rejoins the same conversation
• [x] Panic recovery — a panic in one call's goroutines now ends that call alone: it recovers, logs the stack, and the session is reaped like any other ended call
• [x] Session cap — server.max_sessions bounds live sessions globally; past it, POST /whip returns 503 with Retry-After. Resumes are exempt
• [x] Env-var secrets — every API key and secret can come from the environment (OPENAI_API_KEY, STREAMCORE_JWT_SECRET, …) instead of config.toml. See Configuration
• [ ] Metrics export — /health and timing events exist, no Prometheus/OpenTelemetry
• [ ] Structured logging — log.Printf text today, no JSON logs carrying session_id
• [ ] Versioned releases — a Docker image ships to GHCR on each GitHub release, but no version in the binary and no tagged standalone binaries yet
• [ ] Horizontal scaling — sessions live in process memory, so the server is single-node; reconnection and resume need sticky routing or an external store to work behind a load balancer
• [x] HTTP agent endpoint — llm.provider = "agent" POSTs each turn to an agent you host in any language; replies stream back as speech
• [ ] Persistent memory — the built-in runtime forgets callers between sessions; BYO agents can already persist their own

Full TODO list, including ecosystem items: Roadmap / TODO. Want one of these? Say so in Discord — demand reorders the list.

Bring your own agent

StreamCore starts one layer below prompt-and-tool frameworks: the media path. Your intelligence stays yours, five ways —

1. Tool call — plugins (Python/TS/JS) or native Go tools call into your existing backend
2. Your agent — set llm.provider = "agent" and each turn is POSTed to an HTTP endpoint you host, in any language
3. Your models — point llm.provider = "ollama" at any Ollama-compatible URL you run
4. Your code — implement one small Go interface; the whole media path works unchanged
5. Built in — or use StreamCore's optional agent runtime with tools, skills, RAG, and history

Details and code: Bring your own agent · Agent runtime.

Providers: Deepgram, AssemblyAI, OpenAI, Cartesia, ElevenLabs, MiniMax, Speechify, Ollama, VibeVoice (local), xAI Grok Voice (speech-to-speech), pgvector/Supabase for retrieval. See Providers.

Documentation

SDKs and examples

Connect from anywhere — every SDK speaks the same WHIP + DataChannel protocol:

React Native / Expo (@streamcore/react-native-sdk) is built but not yet published to npm.

Plugin SDKs: @streamcore/plugin and streamcore-plugin in plugin-sdk. Runnable browser, CLI, and TUI apps: examples.

Sponsors & Supporters

Thank you! Interested in sponsoring? Reach out for logo placement on GitHub + demo page.

Contributing

Read CONTRIBUTING.md first — it covers running the server locally, the four checks CI runs before you push, and the extra care the timing-sensitive media path needs. Good places to start: good first issue and help wanted.

Client SDKs, the SIP bridge, examples, and the ESP32 firmware live in their own repos under streamcoreai — send those changes there.

Security

Found a vulnerability? Don't open a public issue — report it privately through the Security tab. SECURITY.md covers scope, response targets, and the settings that matter on a public address — JWT auth on /whip above all.

Star history

License

Apache 2.0. See LICENSE.