One AI system, four channels
Serving the same AI capability over web chat, a streaming video avatar, voice, and an actual phone line — without maintaining four separate systems.
Problem
The same underlying capability needed to reach people through different surfaces. Some users want to type. Some want to talk to a face. Some want to call a phone number.
The naive approach builds a product per channel and ends up with four diverging systems, four sets of prompts and four places for behaviour to drift apart.
Constraints
- Voice and telephony are latency-sensitive in a way that text is not; a pause that reads as thoughtful in chat reads as broken on a call.
- Telephony audio is 8 kHz μ-law, not the format anything else in the stack speaks.
- Avatar streaming is billed by the minute, so idle sessions are a direct cost.
- A third-party avatar API was sunset mid-project, forcing a migration on someone else's schedule.
My role
Engineer across all four channel services and the migration between avatar providers.
Architecture
Intelligence stays in one place. Each channel is a thin presentation service that proxies to the central orchestrator over streaming HTTP and concerns itself only with capture and playback.
The avatar service pairs a streaming avatar SDK with browser-side speech-to-text, and tracks per-session cost and duration.
The voice service handles speech-to-text and text-to-speech around the same central stream, using browser-side voice activity detection to decide when a turn has ended.
The telephony service bridges Twilio Media Streams to a realtime speech model over WebSocket, translating between μ-law telephony audio and the model's expected format, with server-side turn detection.
Key decisions
Thin channels, central intelligence
Each channel service knows how to capture input and render output, and nothing about the domain. A change to reasoning or tools propagates everywhere without touching channel code. This is the decision that made the other three channels cheap to build, and it is the one I would defend hardest.
A different model tier for telephony
The phone channel uses a realtime speech-to-speech model rather than chaining speech-to-text, orchestration and text-to-speech. Chaining is more capable and too slow for a phone call, where a delay is interpreted as a dropped connection. This is a deliberate capability-for-latency trade: the phone channel is conversation-only, with no tool calls and no persistence, and it is honest to describe it as a prototype rather than as the full system with a phone number attached.
Treating the avatar provider as replaceable
Avatar rendering sat behind a narrow interface from the start. When the provider sunset the streaming API version we were on, the migration touched the avatar services and nothing else. The general lesson is to assume any vendor in a fast-moving category will change under you, and to keep the surface you depend on small.
Result
Four working channels — chat, video avatar, voice and a live phone line — sharing one orchestration layer.
The avatar provider migration was completed without changes to the reasoning layer or the other channels.
What I learned
- Building a second interface is the cheapest architecture review available. Anything you have to duplicate to do it was in the wrong place.
- Latency budgets, not capability, determine architecture at the voice and telephony end. It is better to ship a deliberately narrower phone experience than a capable one that feels broken.