rtpbench

What it is

rtpbench tests voice agents that answer phone calls. It calls the agent over SIP and records the packets at both ends. From the packets, it measures how long the caller waits for each answer and how fast the agent stops when the caller interrupts. It also checks whether the agent follows the SIP protocol: hold, transfer, DTMF, codec changes, session timers, and SIP messages lost on the way. It runs each test on a clean network and on networks with packet loss and jitter (tc netem). An optional carrier, OpenSIPS with rtpengine, can sit in front of the agent, as a real trunk does.

You give it the SIP URI of your agent, or the name of one of three bundled stacks. SIP can run over TLS and the media over SRTP. It gives you a report: a verdict for each test, timings with error bars, and a timeline of every call. A GitHub Action runs it in CI and fails the build when the agent breaks your policy.

To check its own results, rtpbench has a reference agent: a simulated voice agent with no AI that logs the true time of each answer. The error bars come from a comparison with this agent. Each test must also pass and fail correctly against it before it judges a real agent.

The write-up explains how it works. The docs cover calibration, cases, targets and CI. The results site has the timeline of every published call.

What it found

rtpbench tests three open-source ways to put the OpenAI Realtime API on a SIP call: LiveKit, AI Voice Connector CE, and Asterisk with AVA. It runs 24 cases on each, with and without a carrier in front, on three network profiles, 10 times each; the 3 cases for a lost SIP message run direct on a clean network. A mock replaces the AI engine and always takes the same time to answer, so a difference in a result comes from the stack.

LiveKit1833 ms p50

engine 1501

media path 335 ms = uplink 186 + downlink 149

AI Voice Connector CE1511 ms p50

engine 1501

media path 9 ms = uplink 0 + downlink 9

Asterisk + AVA1588 ms p50

engine 1501

media path 88 ms = uplink 1 + downlink 87

0500100015002000

• LiveKit keeps sending audio to a held caller in every kind of hold. AI Voice Connector CE does it when held with c=0.0.0.0. Both answer a re-INVITE that changes codec with the old codec. AVA sends RTP only while it speaks, so whether its media comes back after a hold cannot be judged.
• LiveKit's extra latency is its own media path: 326 ms per turn more than AI Voice Connector CE's, against 322 ms between their turn latencies.
• LiveKit's RTP falls 20 ms behind its own timestamps 4 s into every call. A receiver needs a 21 ms jitter buffer for it on a clean network, and 1.1 ms for AI Voice Connector CE's.
• AI Voice Connector CE delivers the engine's speech sample for sample. LiveKit re-encodes it on the way, through its WebRTC room, and on a clean network stretches or cuts a pause by 20-40 ms in 6% of answers (10% in a second wave).
• AVA, as its golden configuration ships, cuts the last 0.8 s of every answer. It never stops for an interruption, because it keeps the caller's audio from the engine while it speaks. Over AudioSocket under the lte and lte-bad profiles, the engine sometimes got silence instead of the caller, so 5 more cases fail now and then.
• All three pass the cases for a lost SIP message. Each sends a lost REFER or BYE again after about 500 ms. AI Voice Connector CE and AVA answer a repeated INVITE with the same 200 OK. LiveKit absorbs the copy and resends its 200 OK 250 ms after the first, where RFC 3261 starts at 500 ms.
• SIP over TLS and SRTP leave the wire timing as exact as without them.

The commands, conditions, summaries and reports are in the baseline, Asterisk + AVA, lost SIP messages and TLS and SRTP; the raw runs are on GitHub releases.

What changed upstream

The findings went upstream as pull requests and issues. AI Voice Connector CE merged eight fixes on 19 and 20 October. rtpbench then ran the baseline's sets again on its main branch, at b95f5af. 4 of the 11 cases it failed in the baseline now pass: a failed transfer no longer hangs up on the caller, and compact headers, a bare a=ssrc and SIP over TCP are taken. It still refuses an INVITE without SDP, and it answers no re-INVITE without one. Its timing did not move: 1511 ms per turn at p50, against 1512 ms before.

LiveKit's pull requests and issues are still open. Contributions lists each one, with its state.

How other methods compare

Other ways of measuring turn latency have errors of their own. Measured inside the agent, a turn misses the network: 80 ms on an LTE path. An energy rule on a recording reads turns long, by 87-110 ms at the median. rtpbench measured the same calls each way against its truth: metrology.

Run it

It needs Linux (WSL2 works), Docker 28 or later with Compose, and uv.

git clone https://github.com/jjaroztegi/rtpbench
cd rtpbench
# how exactly this host times the wire
uv run rtpbench calibrate --profiles clean --turns 5
# prove every case against the reference agent
uv run --extra vad rtpbench cases
# a bundled target, no AI account needed
uv run --extra vad rtpbench target livekit --mock
# your agent, the same way
uv run --extra vad rtpbench target sip:agent@host

From a fresh clone the first three take about 1, 17 and 11 minutes. They build or pull about 2.5 GB of Docker images on first use. Each run writes results/<timestamp>/ with the captures, summary.md and, for cases, report.html: a timeline of every call's SIP, speech and RTP.

Use it in CI

- uses: jjaroztegi/rtpbench@<commit SHA>  # v0.2.0
  with:
    target: sip:agent@agent:5060
    network: ${{ job.services.agent.network }}
    profiles: clean,lte
    policy: rtpbench.toml
    previous: rtpbench-baseline.json

The action calls your agent, then rtpbench check judges the run against a policy: cases that must pass, latency and barge-in limits, and how far the median may grow from a reviewed baseline. A limit fails only when even the lowest true value its error bar allows is over it. The CI docs cover the policy and where the agent can run.