~/rena
← blog
Voice AIArchitectureIntegrationsAbstraction

One Interface, Five Voice Providers

I integrated five AI voice agent vendors into a single production app. The hard part wasn't the APIs — it was deciding what the abstraction was allowed to know.

RRena Thomas·

I promised a follow-up on multi-provider voice AI in the last post. This is it.

In 2024 I integrated five AI voice agent providers — ReTELL, VAPI, Trillet, Millis, Hamsa — into one production application. One codebase, one set of conversational flows, five very different vendors underneath.

People assume the difficulty is API surface area. It isn't. Wrapping five REST APIs is a week of tedious work. The difficulty is that each provider has a different opinion about what a "conversation" is, and your abstraction has to survive all of them.

Why bother with more than one provider

Fair question. Multi-provider isn't free, and single-vendor is genuinely simpler.

We did it for three reasons, and only the third one was the real one:

  1. Latency and quality vary by workload. Some providers are noticeably better at interruption handling. Others win on cold-start time. Others have better language coverage.
  2. Pricing leverage. If migrating providers is a two-week project, you have no negotiating position. If it's a config change, you do.
  3. Vendor risk. In this market, a provider you depend on can change pricing, get acquired, deprecate the API you built on, or degrade quietly. Being able to shift traffic in an afternoon is worth real engineering cost.

That third one has aged extremely well. Over the last two years I've watched teams get stranded on abandoned APIs more than once.

Where providers actually disagree

The naive abstraction is startCall(), endCall(), onTranscript(). That covers maybe 30% of what you need. The disagreements show up in the parts nobody documents:

Turn-taking. Some providers hand you barge-in detection as a first-class event. Others let the user talk over the agent and expect you to figure out from timing whether that was an interruption or a filler word. Your abstraction needs one model of "the user interrupted" and adapters that manufacture it where it's missing.

Tool calling. Every provider has function calling now. None of them agree on where it runs. Some execute the tool call server-side and webhook you. Some stream the intent to your client and wait. Some support mid-sentence tool calls; others only between turns. This is the single biggest source of leaky abstraction.

State ownership. The critical one. Does the provider own conversation state, or do you? If the provider owns it, your handoffs and mode switches have to route through their state machine. If you own it, you're re-sending context on every turn and paying for it in latency.

We picked: we own state, always. Providers get the minimum context needed to speak the next turn. That decision cost us some latency and bought us everything else.

The shape that worked

Three layers, strictly separated:

// 1. Provider adapters — dumb, thin, no business logic
interface VoiceProvider {
  readonly id: ProviderId
  readonly capabilities: CapabilitySet

  connect(session: SessionSpec): Promise<ProviderHandle>
  speak(handle: ProviderHandle, utterance: Utterance): Promise<void>
  disconnect(handle: ProviderHandle): Promise<void>
}

// 2. Session state — ours, provider-independent, serializable
interface SessionState {
  established: Record<string, unknown>  // facts confirmed with the user
  intent: Intent | null
  mode: AgentMode
  pendingActions: Action[]
  history: Turn[]
}

// 3. Orchestrator — owns the state machine, chooses the provider

The adapters know nothing about the business. The orchestrator knows nothing about vendors. Everything vendor-specific lives in the adapter or in capabilities.

Capability flags, not feature detection

The mistake I made first was branching on provider identity:

if (provider.id === 'retell') { /* ... */ }

Within a month there were nineteen of those and adding a sixth provider meant auditing all of them.

The fix is to declare capabilities explicitly and branch on the capability:

interface CapabilitySet {
  midTurnToolCalls: boolean
  nativeBargeIn: boolean
  serverSideToolExecution: boolean
  maxContextTokens: number
  supportedLanguages: string[]
  warmPoolAvailable: boolean
}

Now the orchestrator asks "can this provider do mid-turn tool calls?" and falls back to a between-turns path when it can't. Adding a provider means filling out one struct, not touching the orchestrator. Routing logic — "this session needs Hebrew and warm start, who qualifies?" — becomes a filter over capability sets.

This is the part I'd tell anyone building a multi-provider layer to get right on day one. It's the difference between an abstraction that absorbs the fifth provider in two days and one that fights you forever.

Mode switching and handoffs

The application needed agents to switch modes mid-call — from intake to scheduling to escalation — and sometimes to hand off to a different agent entirely with a different system prompt and tool set.

With provider-owned state this is close to impossible to do cleanly. With our own SessionState it's a state machine transition plus a re-render of context:

async function transition(session: Session, next: AgentMode) {
  const spec = renderSpec(session.state, next)   // prompt + tools for the new mode
  const provider = route(spec.requirements)      // may or may not be the same vendor

  if (provider.id !== session.provider.id) {
    await session.provider.disconnect(session.handle)
    session.handle = await provider.connect(spec)
  } else {
    await provider.reconfigure(session.handle, spec)
  }

  session.state.mode = next
}

Note that a mode switch can cross providers. That fell out of the design rather than being planned, and it turned out useful — escalation paths could route to whichever vendor handled long, complex calls best.

What it cost

I don't want to oversell this. The abstraction cost us:

  • Latency. Owning state means re-sending context. We clawed most of it back with aggressive prompt caching and by keeping established facts compact, but we never fully matched a native single-provider integration.
  • Feature lag. When a provider shipped something genuinely novel, we couldn't expose it until we decided how it fit the capability model. Sometimes that took a release cycle.
  • A real test matrix. Five providers × modes × handoff paths is a lot of surface. We ended up building a recorded-session replay harness just to keep it tractable.

That harness turned out to matter more than the abstraction did, which is a topic I'll come back to.

When not to do this

If you have one product, one language, one workload, and a vendor you're happy with — don't. Build the direct integration and keep your call sites tidy enough that you could extract an adapter later. Premature abstraction over a domain you don't understand yet produces a worse boundary than no abstraction.

The signal that it's time: you're about to write your second if (provider.id === ...), or someone in a meeting says "what would it take to move off them?" and nobody can answer.

Next in this series: what the replay harness looked like, and why I now think eval infrastructure is the highest-leverage thing on an AI team.

← all posts