Veris Twin Fidelity Benchmark

The obvious objection to a vendor-run fidelity study is that the vendor wrote the tests. This study removes it by borrowing its instrument from an open-source project that has nothing to do with us: NangoHQ/integration-templates, a public repository of production integration code written for Nango’s own customers to run against live vendor APIs. It runs here unmodified through Nango’s own command-line tool: 721 template operations across fifteen services. Every test, every assertion, and every pass-or-fail decision in this study belongs to them. We supplied only the environment the tests point at.

A test written by looking at the twin can only prove that the twin behaves the way its author expected. It cannot detect the cases where the author’s understanding of the real system was wrong, and those are exactly the cases that matter. So the real system defines what counts as a correct answer, and the twin never does. Every test is first proven against the live vendor API on a dedicated account; only then is it pointed at the twin, with only the base URL changed.

Agreement is scored strictest first. The exact-match lens is the raw measurement: both worlds must return the same outcome — and when they refuse, the same error code — and the number is published before anything is attributed. Every one of the 18 mismatches is then traced to a cause with evidence: fourteen belong to our own test setup, leftover history in the real accounts, or a vendor outage on recording day; four are the twin genuinely behaving differently. The population never changes: all 721 units, nothing excluded, nothing filed under “unknown”.

Twins of large APIs are scoped to their working core, and every operation outside a twin’s published contract refuses cleanly by policy, so scope is a countable property of each twin, not an estimate.

The twin never invents a success. In every twin-attributable mismatch it refuses or under-claims: Outlook’s two mail-writing operations declare themselves unsupported instead of pretending; one Acuity answer is rejected for missing a detail rather than fabricating it; one Gmail refusal carries a different stated reason — but both worlds refuse. A permissive twin inflates every score run against it; a strict one only produces conservative results. That is what makes twin-based testing usable in an SDLC: a green integration run is never the twin flattering the agent’s code, and a red one means the agent fixes something real or hits a documented divergence.

Rev 1.0 carried one divergence that pointed the wrong way — cross-run memory of spent ids, flagged then as the structural exception. It is closed: the twin now takes seedable registries of ids its world does not own. This revision closed a different hazard on the same side: one Airtable request was found slipping past the sandbox to the real service. The hole is closed and the service re-measured at 37 of 37. What remains on the permissive side is nothing: all four open items are refusals or declared gaps, never an invented success — and where a real answer cannot be measured (an Acuity add-on nobody has bought), the unproven claim is counted against the twin, not for it.

The study produces a named divergence list rather than a score, and the twin-attributable list is now four units long — each with the mechanism behind it, read off the wire logs. Two wait on capability the twin declares it lacks: Outlook’s mail-writing paths, while every mail-reading operation already matches. One waits on a real answer nobody can measure without buying a paid Acuity add-on — until someone does, it counts against the twin, because unproven claims default against us. And one is kept open deliberately: making the twin match one Gmail refusal would break three other answers measured from the real Gmail, so it is documented rather than papered over. This revision’s ledger shrank the right way: Linear went from 33 matches to 54 after its error shapes were corrected, and twelve Acuity misses fell to a seeding fix without touching the twin.

The fourteen set-aside mismatches, named too: six are the instrument’s own checklist demanding details the real services never send — a form id Calendly doesn’t have, a payment field that is often empty, a cycle name Linear itself calls optional; corrections are being filed with the public test-suite project. Four are leftover history in the real accounts (a calendar already removed, notification channels already used up). Three are a Notion outage on the day its real answers were recorded. One is a Gmail label our own earlier recording session had already deleted. Circumstance, not behaviour: counted among the 18 mismatches, attributed to their real causes rather than the twin.

A repeat over 238 units across two freshly built deployments returned every verdict identical, down to the HTTP status, with zero flips. Over the same repeats, four verdicts on the real arm moved, on vendor-side memory of spent ids. For agents, a red test against the twin is not flake: no cycles burned retrying nondeterministic failures, and two agents testing the same change get the same answer. The oracle is held to the same standard: the recorded real-arm corpus is verified byte-stable before every scoring, and has not moved since it was taped.

A typical request is the same order of magnitude in both worlds; the twin runs about 3.6x faster at the median and its p90 stays near 130 ms on every service. The tail is where the two worlds separate: real vendors regularly take half a second and occasionally tens of seconds (Attio’s worst request sat through 32 s of rate-limit backoff), while the twin has no throttling and no multi-second stalls on someone else’s infrastructure. For agent loops this is the difference between running an integration suite once and running it hundreds of times per session. The cost is that same missing tail: timeout, retry, and backoff behavior under real-world latency is not exercised by default. Latency and fault injection exist in the product; twin behavior under injected conditions is not covered by this study.

A high-fidelity twin proves the systems around the agent behave. Simulated users carry the conversation itself, so the same rule applies to them: measured, not asserted. We scored Veris persona actors on a five-check naturalness rubric in the style of the τ-USI framework (communication style, information pattern, clarification) against a public baseline, the user simulator of τ²-bench retail (Sierra Research). One judge model graded 546 baseline traces and 200 Veris sims from four customer scenario sets with byte-identical prompts and the same aggregation; the only variable is whose user it is judging.

Veris actors lead on all five checks and all three dimensions.The table shows the two dimensions where the instrument is most stable; the communication-style dimension, under active refinement on our side, leads the baseline by +19.2 pp as well.

Rubric: five binary LLM-judge checks (v3), verdicts pass / fail / not-applicable, NAs skipped by the aggregation on both sides. Baseline: 546 fully-judged τ²-bench retail traces with a GPT-4-generation model playing the simulated user. Veris: 200 production sims, 50 per scenario set, personas from hand-curated specs. Pooled dimension rates are pass ÷ judged verdicts: 480/551 (Veris) vs 1262/1635 (baseline) across the two dimensions shown.

• The instrument probes a focused slice intensely rather than sweeping the twin’s whole declared surface; what it measures, it measures independently, but it does not measure everything.
• Twin behavior under injected latency and faults is not covered by this study.
• The real-arm answers were recorded once, on 2026-08-18, and serve as the fixed answer key; the twin roster was re-measured in one sweep on a single twin release, with two services re-measured minutes later after our own test data was corrected. The two arms were not measured on the same day, and mismatches caused by the key aging are attributed as such rather than counted against the twin.
• Salesforce was measured but its score is withheld this revision: the connector is migrating to API v67 and its real-side answers are being re-recorded. It re-enters the count next revision.
• All measurements ran through an off-the-shelf integration gateway; behavior behind a gateway with mutual TLS or DLP inspection is not evidenced here.
• Results are stamped per build; numbers from unstamped or mixed builds are not comparable.

Every number on this page is reproducible from retained artifacts: a frozen claim manifest, a per-operation ledger, captured request and response corpora from both arms, per-operation diffs, per-test logs, latency tapes, the analyzers, and an identity trail binding each result to the software build that produced it. Each figure carries the command that produced it alongside the raw output. This is the artifact set to demand from any benchmarking vendor, including us.

Pinned build

The environment is rebuilt from a pinned software build immediately before each measurement, and every result is stamped with that build. Comparisons are only ever made between stamped worlds.

Fixed oracle

The instrument and its real-arm recording are pinned; between revisions only the twin build moves, so every delta on this page belongs to the twin and nothing else.

Sealed credentials

Credentials on the twin arm are minted by the environment’s own identity twin, so no real traffic can leak into a simulated measurement.

Fresh world

Each measurement starts from a clean world with no memory, which is why determinism holds.

The benchmark re-runs on the same protocol as twins and instrument grow; new versions append here, and the tables above always show the current revision’s results. Questions, or a system you want to see measured: hello@veris.ai