Skip to main content
A benchmark answers a comparative question: not “does my checkout land?” but “does my checkout land better than the two competitors I keep losing to?” To answer it fairly, every variant has to face the same audience and the same questions. Only the thing being experienced should change. ish models that with two pieces. A reference workspace is one comparison target. A benchmark clones a study across a set of references so each one inherits the same modality, assignments, and interview questions, and differs only in the artifact participants actually see. Both live on the MCP server. There is no CLI surface for references or benchmarks.

​
A reference is a comparison workspace

A reference is a workspace in disguise. Underneath it is a workspace with a parent set and a product_type of reference, and it shares the w- alias prefix, so any tool that takes a workspace ID accepts a reference alias and resolves it. ish surfaces it in its own reference namespace anyway, because you reason about a reference differently than a top-level workspace: it is a thing you compare against, not a place you keep work. That split is deliberate, and it shows up in what the read tools return. workspace_get lists only standard workspaces, never references. reference_get lists only the references under one parent. A reference never shows up as a workspace you might accidentally start fresh work in, and a workspace never gets pulled into a benchmark cohort by mistake. A reference carries three fields that matter for a benchmark: a name, an optional description, and an optional base_url. The base_url is the one that does work later. When a benchmark clones an interactive study to a reference, that URL becomes the suggested URL for the clone’s iteration, so the competitor’s site is already half filled in.
References and the source study must share the same parent workspace. A benchmark compares variants of one body of work; it does not reach across workspaces.

​
The benchmark clones, it does not run

study_benchmark takes a source study and a list of reference IDs, and clones the study into each reference. What carries over is the question, not the answer: the name, description, assignments, and interview questions. What does not carry over is everything that makes a run concrete. Iterations, participants, and frames are not copied. That is the whole idea. The clone inherits a fixed lens, so the same audience faces the same prompts against each variant. Then you point each clone at its own artifact: your URL on the source, the competitor’s URL on its clone, the alternative copy on a third. The comparison stays honest because only the experienced thing differs. Two consequences follow from “clone, do not run”, and both trip up agents that treat a benchmark like a one-shot.
1

Clones are drafts

study_benchmark does not dispatch any participants. Each clone lands as a draft study with no run behind it. You run each one yourself with study_run once its iteration is filled in. Nothing draws simulation credits until you do.
2

Fill the placeholder, do not append

Each clone is born with one empty placeholder iteration A, returned in placeholder_iteration_ids keyed by the clone’s alias. Fill it by editing that iteration, not by adding a new one. study_add_iteration would append a second iteration B and leave the empty A in place, so the clone would carry a dead iteration alongside the real one.
A benchmark is also idempotent against a reference that already has a clone of this study, and it never silently overrides access. Any reference it could not clone to comes back in a skipped list with a reason: no_access, or already_cloned with a pointer to the existing clone. Read skipped before you assume a clean cohort, or you will run a comparison that is missing a competitor.

​
Reading the cohort head-to-head

Once each clone has run, you read the whole set in one call. study_get takes a study_ids list, and a benchmark cross-read passes the source study plus its clones. It returns a list of per-study results in the same order you asked for them, each shaped by the same view, so the variants line up side by side under one lens. The findings read as a comparison, not as five unrelated runs you have to reconcile by hand.

​
The shape of a benchmark

End to end, a benchmark is a small, ordered set of MCP calls. The mental model is worth holding even though the parameters live in the reference.
1

Create one reference per comparison target

reference_create under the parent workspace, once per competitor or alternative version. Give each a base_url so its cloned iteration starts pre-filled.
2

Clone the study across them

study_benchmark(source_study_id, reference_ids=[...]). One call fans the study out to every reference. Check skipped.
3

Fill each placeholder iteration

study_update_iteration on each placeholder_iteration_ids entry, pointing the clone at that reference’s artifact.
4

Run each clone

study_run per clone (and the source). Each draws credits only when it runs.
5

Read them together

study_get(study_ids=[source, ...clones]) for the head-to-head.
Full parameters, return shapes, and error kinds live in the reference: reference tools for reference_create / reference_get / reference_delete, and study tools for study_benchmark, study_update_iteration, study_run, and study_get.
Deleting a reference cascades to every study and iteration cloned under it, with no soft archive. Confirm before calling reference_delete. The widest destructive action, deleting the parent workspace, takes its references with it. See the reference tools.

Workspace

Why a reference is a workspace underneath, and how the namespaces stay apart.

Studies

The recipe a benchmark clones across references, and what carries over.

Iterations

The placeholder iteration each clone is born with, and how to fill it.

reference tools

The MCP reference for reference_create, reference_get, and reference_delete.

study tools

study_benchmark, study_update_iteration, study_run, and study_get.