Timing voice agents from the wire

I built rtpbench to test voice agents that answer phone calls. It calls the agent over a SIP phone line, as a real caller does, and records the packets at both ends of the call. From these packets, it checks four things:

• SIP behaviour. Does the agent follow the protocol for hold, transfer, DTMF, hangup and the other parts of a call? These checks apply to any phone system, with or without AI.
• Latency. When the caller stops talking, how long until they hear the answer?
• Interruptions. When the caller talks over the agent, how fast does the agent stop?
• Codec changes. When the other side changes the codec during the call, does the agent accept the change and use the new codec?

The input is the SIP URI of the agent that you want to test, or the name of one of three bundled stacks. The output is a technical report: a verdict for each test, timings with error bars, and a timeline of the SIP messages, speech and RTP of every call. In CI, the same run passes or fails a build against a policy.

Why the packets? The caller hears only the audio that arrives on the line. Between the end of the question and the start of the answer, the audio goes through a SIP stack, a jitter buffer, a codec, a connector and an AI engine. Each part can log its own time, but no part sees the full path. The packets on the wire show the time that the caller hears.

To know that a measurement is correct, rtpbench compares it with a known value. It has a reference agent for this: a simulated voice agent with no AI. The reference agent answers each turn after a delay that it chooses, and it logs that delay. Every test runs against this agent first. The test must pass when the agent behaves correctly and fail when it does not. Only then does it judge a real agent.

When it tests a real agent, rtpbench can also replace the AI engine with a mock that always takes the same time to answer. The connector is the part that joins the phone line to the AI engine. With the engine fixed, any difference in a result comes from the connector.

Get the code

rtpbench is public under Apache-2.0. The current release is v0.2.0. Each published result set has its commands, conditions, summaries and reports in the repository and its raw runs on a release, and the results site shows the timeline of every call. The project page sums up the findings.

uv run rtpbench calibrate --profiles clean --turns 5      # how exactly this host times the wire
uv run --extra vad rtpbench cases                         # prove every case against the reference agent
uv run --extra vad rtpbench target livekit --mock         # a bundled target, no AI account needed
uv run --extra vad rtpbench target sip:agent@host         # your agent, the same way

It needs Linux (WSL2 works), Docker 28 or later with Compose, and uv. From a fresh clone, the first three commands take about 1, 17 and 11 minutes.

Overview

rtpbench has five parts:

• A caller. It places SIP calls, speaks recorded lines and listens. SIP can run over UDP, TCP or TLS, and the media over RTP or SRTP.
• A router between the caller and the agent. It runs tc netem to impair the RTP.
• An optional carrier in front of the agent: OpenSIPS 3.6 as a stateful proxy and rtpengine for the media, the way a trunk puts an SBC in front of a PBX.
• The agent. This is the reference agent that logs the truth, one of three bundled targets, or your own agent at its SIP URI. The targets are LiveKit, AI Voice Connector CE, and Asterisk with AVA.
• A mock engine. It stands in for the OpenAI Realtime API behind a target.

Every part runs in a container. The caller and the agent both capture with tcpdump, and all containers share one kernel clock. Your own agent can run anywhere. Its timings then include the real path beyond the router, which the error bars do not cover.

The wire

Time everything from packets on the wire

Every latency in rtpbench is measured on the capture timestamps of packets on the wire, not on agent logs, recordings or the host clock at the time of reading. Where the caller's speech ends comes from the labels of its own clips, to within ±17 ms.

The reference agent answers each turn after a random delay that it chooses, and it logs that delay to the sample. This gives a truth to measure against. In the baseline, the wire timing against that truth had a maximum absolute error of 0.00 ms in all seven network profiles, direct. Through the carrier it was 0.00 ms in six profiles and 0.12 ms under loss. The carrier itself adds 0.05 ms per packet at the median on a clean network.

Schedule on the monotonic clock, and log clock steps

Captures stamp the wall clock. Time sync can step the wall clock. A step in the middle of a call makes an interval read wrong by the size of the step.

Virtual machines are prone to this. On one guest, the clock stepped back 25 to 80 ms about every 30 s, so between steps it ran 800 to 2,700 ppm fast. So rtpbench senders schedule on the monotonic clock, and every run logs the steps it saw and the clock's rate between them. Summaries mark any interval that was timed across a step, and calibrate warns when the clock runs more than 100 ppm off. In the baseline, 0 steps were seen.

The senders were also on time. Across 57017 packets, both ends, sender lateness was p50 0.010 ms, p99 0.050 ms and max 2.65 ms.

Measure how many runs a host can hold

Each run brings up its own containers and network, so several runs can share a host. I measured how far this goes before timing suffers. With up to 12 reference stacks at once, wire timing against truth stayed at 0.00 ms, no packet left more than 10 ms late, and no capture dropped a packet.

Target stacks weigh more. If you publish at some concurrency, calibrate at that concurrency.

Encrypt the call, and time it again

A trunk can carry SIP over TLS and the media over SRTP. rtpbench can do both on the caller's leg, so I ran the calibrations again with them. The wire timing stayed at 0.00 ms in every profile, direct and through the carrier, except one lte-bad turn through the carrier. The network lost that turn's first packet of speech, so the wire saw the onset in the next packet, 4.1 ms later. rtpengine, which protects and opens the SRTP, adds about 0.02 ms per packet at the median.

Every case that runs over TLS also passed its proofs on the reference agent over TLS and SRTP, direct and through the carrier.

Error bars

Calibrate the detector against an agent that logs the truth

Wire timing is exact. Most of the error that remains is in speech detection: a turn starts when the agent's speech starts, and a voice activity detector decides when that is.

rtpbench calibrates the detector on clips with labelled speech boundaries. Each error bar is the p5..p95 range of the detector's error for one profile and one codec. A bar of −3.2/+44.6 means the true value lies between 3.2 ms below and 44.6 ms above the measured one.

Hold out the seeds you test on

The shipped bars come from seeds 1 to 5. I ran seeds 6 to 10 the same way and checked them against the bars. A p5..p95 bar should leave about 10% of turns outside, and it did: between 5% and 13% per bar, over 215 or 230 turns each.

The bars come from one host and serve every host. A run on a second host moved the edges no more than a second run on the same host did.

Check what the codec does to your detector

The onset bars differ a lot by codec. On a clean network, the PCMU onset bar is [−3.2, +44.6] ms. The PCMA bar is [−133.0, +11.0] ms.

The cause is the detector, webrtcvad, on A-law audio. A-law has no zero level (silence decodes to ±8), and mu-law has one. On A-law, a detector that has already heard speech fires on the quiet lead-in before a labelled onset. A fresh detector per clip does not do this, and neither does a lead-in replaced by digital silence. So an A-law agent's onsets can read early.

Measure the usual methods against the same truth

Turn latency is measured in several ways, and most tools' documentation says little about how far each is off. With the truth in hand, I measured the same calls each way: 215 turns per profile and codec.

• Inside the agent, a turn misses the network. On lte, the agent's own capture read 80 ms short at the median. On a clean network it was exact. An agent-side metric also needs its own speech detector, with the errors below.
• An energy threshold on a recording reads turns long, by 87 to 110 ms at the median and up to 274 ms at p95. It finds the agent's onset almost on time, but it ends the caller's turn early, because the soft end of a sentence is below the threshold.
• webrtcvad and Silero at 16 kHz on a recording read turns short, and they spread over 190 to 310 ms from p5 to p95. Silero at 8 kHz reads long instead, and misses some replies.
• A mean per call hides that spread. With webrtcvad on PCMU, on clean and lte, every call's mean was within 9 ms of its true mean, while its turns were off by −64 to +158 ms.
• A recorder that appends payloads as they arrive is seconds off after a few lost packets.

rtpbench's own method read −2 to +9 ms at the median, and its spread is what its error bars say. The metrology results have every method, profile and codec.

Cases

Prove every case passing and failing before it judges anyone

rtpbench has 24 conformance cases: greeting, hangup, cancel, hold, DTMF, transfer, session timers, codec changes, SIP over TCP, four that only a carrier sends, and three that lose a SIP message. The baseline ran the first 21. Each case is a scripted call with a verdict: pass, fail or inconclusive.

Before a case judges a target, it has to fail where it should. It runs against the reference agent once with a behaviour that should pass and once with each behaviour that should fail, and any unexpected verdict fails the run. In the baseline, every reference run had the expected verdict, for example 90/90 for hold and 60/60 for bye.

Give every network profile a source

The lte and lte-bad profiles stand for a mobile caller's path. Every number in them has a source. For lte, the loss is the VoLTE target of 1% per radio link (3GPP TR 26.989), in bursts of 2 packets (the 3GPP VoLTE delay and error profiles). The jitter and the delay come from published measurements of commercial LTE.

netem needs care here. Independent netem jitter with a standard deviation of 26 ms reorders 30% of packets. The 3GPP trace that lte-bad copies reorders 3%. So both profiles add a rate limit, which keeps packets in order, as a radio bearer delivers them. netem still cannot copy the trace's packet-to-packet delay correlation: 0.95 in the trace, 0.43 in netem.

Put a carrier in front

Real trunks rewrite calls. With --carrier, four more cases send what a carrier sends:

• compact-headers: f:, t:, i: and c: for From, To, Call-ID and Content-Type.
• sdp-ssrc: an offer with a bare a=ssrc:<id> line.
• late-offer: an INVITE without SDP.
• ptime-30: the caller's audio in 30 ms packets.

Some cases run without the carrier only, because rtpengine would answer for the agent. For example, it transcodes when an agent answers with the wrong G.711 law.

Lose a SIP message on purpose

SIP over UDP has its own retransmissions, and an agent has to handle them. Three cases lose one message on purpose:

• invite-retransmit: the caller loses every response for 500 ms (T1), so it sends the INVITE again.
• refer-lost: the caller leaves the agent's first REFER unanswered.
• bye-lost: the caller leaves the agent's BYE after a transfer unanswered.

These run direct only, because a carrier's own transaction would resend for the agent. All three stacks passed all three cases, 10 times out of 10. Each sent its REFER and its BYE again after about 500 ms. AI Voice Connector CE and AVA answered the repeated INVITE with the same 200 OK. LiveKit absorbed the copy, as RFC 6026 allows, and resent its 200 OK on its own, 250 ms after the first. RFC 3261 starts that interval at 500 ms.

Hold the engine fixed

A voice agent built on a connector has two parts that take time: the connector and the AI engine. A real engine adds variable time on every turn. To judge the connector, rtpbench replaces the engine with a deterministic mock.

The mock speaks the OpenAI Realtime API protocol. It ends a turn after 1 s of silence and answers 500 ms later. It logs when it heard each turn end and when it sent its first audio, on the captures' clock. So rtpbench splits each turn into three parts: the connector's uplink, the engine's fixed share, and the connector's downlink. The mock also records the audio it gets, so rtpbench can compare both directions with audio it knows sample for sample.

The target's code does not change, and the target has no route out, so no paid engine is reachable. Every number in this article about a target comes from runs on the mock.

What a mock cannot judge is the model: whether it understands, greets in time, asks for a transfer or stops for a backchannel. It judges the mechanics around those decisions, such as whether a hold stops the media, whether a codec change works, whether a transfer failure goes back to the engine, and whether a digit reaches the engine at all.

What it found

The baseline ran every case on the reference agent, LiveKit and AI Voice Connector CE, direct and through the carrier, under three profiles (clean, lte and lte-bad), with 10 repeats each. Asterisk with AVA ran the same sets later, and so did AI Voice Connector CE after its fixes (below). The table shows each stack at its latest run.

LiveKit's extra latency is its own media path

LiveKit1833 ms p50

engine 1501

media path 335 ms = uplink 186 + downlink 149

AI Voice Connector CE1511 ms p50

engine 1501

media path 9 ms = uplink 0 + downlink 9

Asterisk + AVA1588 ms p50

engine 1501

media path 88 ms = uplink 1 + downlink 87

0500100015002000

With the engine fixed at 1501 ms, LiveKit's uplink was 186 ms and its downlink 149 ms at the median on a clean network. AI Voice Connector CE's were 0 ms and 9 ms, and AVA's 1 ms and 87 ms. That is 326 ms more per turn for LiveKit's media path than for AI Voice Connector CE's, against 322 ms between their turn latencies. AVA's egress pacer adds 16 ms of its 87.

The gap grows with impairment. On lte, the turn was 1997 ms for LiveKit, 1593 ms for AI Voice Connector CE and 1663 ms for AVA. On lte-bad, it was 2134, 1604 and 1650 ms. Barge-in follows the same pattern. The mock cancels its answer on the first voiced 10 ms of the caller, so the stop time is the target's own path: LiveKit stopped in 306, 459 and 579 ms at the median across the three profiles, and AI Voice Connector CE in 24, 102 and 114 ms. AVA never stopped (below).

How each stack drives a real engine is a separate question that the mock does not answer. LiveKit and AVA convert the call audio to 24 kHz for the engine, and AI Voice Connector CE passes G.711 through.

LiveKit's audio slips 20 ms, and its speech is re-encoded

LiveKit's RTP falls 20 ms behind its own timestamps once in every call that runs long enough. At its 210th packet, 4.2 s after its 200 OK, it leaves one 20 ms slot empty without advancing the RTP timestamp, and stays 20 ms late to the end. The cause is the resampler that takes the room's 48 kHz audio to the call's 8 kHz: on one frame it returns one sample short, and the writer holds the whole frame back. A receiver needs a 21.1 ms jitter buffer for LiveKit's stream on a clean network, or plays a 20 ms gap 4 s into every call. For AI Voice Connector CE it needs 1.1 ms.

The speech itself also changes on the way. AI Voice Connector CE delivers the engine's audio sample for sample. LiveKit takes it through its room over WebRTC and back to G.711. In 6% of answers on a clean network, it made a pause in the speech 20 or 40 ms longer or shorter, and in 10% in a second wave.

Hold fails on two stacks, and the third cannot be judged

On hold, an agent should stop sending audio. LiveKit answered a sendonly hold with sendrecv and sent 125 RTP packets while held, on a clean network. It did the same in every kind of hold. AI Voice Connector CE answered an inactive hold with recvonly, and sent about 120 packets during a c=0.0.0.0 hold. Both answered a re-INVITE that changes codec with the old codec.

AVA's hold verdicts are inconclusive. AVA sends RTP only while it speaks, and it said nothing in the second before the hold, so the packets cannot show whether its media would come back.

AVA cuts every answer short and never stops for an interruption

AVA's golden configuration for the OpenAI Realtime API turns on an egress pacer. With the pacer on, AVA stopped sending each answer about 0.8 s before its end, in all 130 answers of the clean latency calls. The last packets it sent still carried speech. The pacer also made the stream uneven: over AudioSocket on a clean network, packets arrived up to 442 ms behind their timestamps (p99, median of 20 latency and barge-in calls), against 2.7 ms without the pacer.

AVA also never stopped for an interjection. Of 900, an answer ended within 2 s of one only once, and then at the time the mock's answer ended. AVA holds back the caller's audio while it plays an answer, so the mock cancelled no answer for the caller's voice. On a clean network, of the 100 lines the caller said during an answer, the mock heard only the tail of that one.

Asterisk hands AVA the call over AudioSocket, as AVA ships it, or over RTP (ExternalMedia). Under the lte and lte-bad profiles, over AudioSocket only, the engine got silence instead of the caller in 8 calls. That is why 5 more cases fail now and then.

One verdict flipped, and the packets show why

In the baseline, one verdict changed between repeats: LiveKit's refer through the carrier, under lte, repeat 6. The carrier relayed the 202 Accepted to LiveKit 0.7 ms after its REFER, and the first NOTIFY 0.2 ms after that. LiveKit logged the REFER as "transaction failed to complete (0 intermediate responses)". It answered the final NOTIFY 5 s late, after three retransmissions, and stayed on the call. In the other 29 repeats, it hung up.

LiveKit's own log reported a failed transaction. The capture showed that the 202 had reached it. That led to the cause: a race in LiveKit's SIP stack. The 202 was handled before the REFER's transaction had set its UDP timers, so the transaction was deleted with the 202 unread.

Fix it upstream, then measure again

I sent the findings to the projects as pull requests and issues.

AI Voice Connector CE merged eight fixes on 19 and 20 October. rtpbench then ran the baseline's sets again on its main branch, at b95f5af. Only the connector and its example configuration differed. 4 of the 11 cases it failed in the baseline now pass:

• A failed transfer no longer hangs up on the caller. The connector keeps the call, and the engine speaks again.
• An INVITE over 1300 bytes, sent over TCP, is taken. The example configuration now listens on TCP.
• Compact header names and a bare a=ssrc, as a carrier sends them, are understood.

Its timing did not move: 1511, 1593 and 1604 ms per turn at the median across the three profiles, against 1512, 1592 and 1605 before. It still fails hold, the c=0.0.0.0 hold, codec change, a re-INVITE without SDP, an INVITE without SDP, and both DTMF cases. An INFO now gets 501 Not Implemented, where it got no answer at all.

The re-run also found a failure the baseline had not shown. One transfer call through the carrier was refused, with Address already in use in the connector's log. It draws each call's RTP port at random from 35000-65000 and binds it without a retry. That range overlaps the kernel's ephemeral ports, which its own outbound connections also use. A deployment avoids it by setting the port range outside the ephemeral range.

LiveKit's three pull requests and two issues are still open: the c=0.0.0.0 hold, the reason phrase on a media rejection, the REFER race, and the 20 ms slip. Contributions lists each one with its state.

Publish the raw runs

rtpbench names each result set by its image tag, a hash of the source, the Dockerfile and the speech corpus, and a git tag marks the commit that ran it. The raw run directories (captures, logs and the reference agent's truth) go on a GitHub release as .tar.zst archives with their SHA-256. The TLS runs include their session keys on purpose, so anyone can open the captures.

With the archives in place, a change to how runs are judged (a check, an error bar, a summary) re-judges the archived runs. Only a change to what goes on the wire needs new runs. The results site shows each published run's report: the SIP messages, speech and RTP of every call on one time axis.

Fail CI only beyond the error bar

rtpbench check judges a finished run against a policy and exits 1 when the run fails it. A policy sets the cases that must pass, latency and barge-in limits per profile, and how far the median may grow from a baseline run. A GitHub Action runs the calls and the check, and adds the summary to the job.

A limit fails only when even the lowest true value its error bar allows is over the limit, so the detector's bias never fails a build. A regression fails only when the change in median is over the tolerance across its whole 95% interval. A tolerance of 0 still fails about 1 run in 40 of an unchanged agent, by chance, so give each regression a tolerance.

Compare with a reviewed baseline, and replace it on purpose. Compared with the last deploy instead, latency can grow by just under the tolerance every time without failing once. Pin the action by commit SHA: the checks and the error bars then change only when the pin does.

Lessons

The rules, in short:

• Time from packets on the wire, schedule senders on the monotonic clock, and log clock steps.
• Calibrate against a reference that logs the truth, and test the error bars on held-out seeds.
• Check what the codec does to your detector.
• Measure the usual methods against the same truth before you trust their numbers.
• Prove each test passing and failing before it judges anyone.
• Give every impairment a source.
• Hold the engine fixed to get verdicts about the connector.
• Send findings upstream, then measure the fixed version with the same runs.
• Publish raw runs, so judging changes re-judge them instead of re-running them.
• Fail a build only when the error bar says so.

To try it, start with the quick start in the repository. Then run it against your own agent:

uv run --extra vad rtpbench target sip:agent@host --profiles clean,lte