Benchmarks and reference workspaces

A benchmark answers a comparative question: not “does my checkout land?” but “does my checkout land better than the two competitors I keep losing to?” To answer it fairly, every product has to face the same people and the same questions. Only the thing being experienced should change. ish models that with two pieces. A reference workspace is one comparison target: a competitor’s product and the URL it lives at. A benchmark clones a study onto each reference and runs it there, with the same people who took part in your version. You can drive both from the web app, the CLI (ish workspace reference, ish study benchmark, ish study comparison), or the MCP server (reference_*, study_benchmark, study_comparison).

​

A reference is a comparison workspace

A reference is a workspace in disguise. Underneath it is a workspace with a parent set and a product_type of reference, and it shares the w- alias prefix, so any tool that takes a workspace ID accepts a reference alias and resolves it. ish surfaces it in its own reference namespace anyway, because you reason about a reference differently than a top-level workspace: it is a thing you compare against, not a place you keep work. That split is deliberate, and it shows up in what the read tools return. workspace_get lists only standard workspaces, never references. reference_get lists only the references under one parent. A reference never shows up as a workspace you might accidentally start fresh work in. A reference carries a name, an optional description, and a base_url. The base_url is the URL the benchmark runs against on that reference. The web app can override it for one run; the CLI and MCP always use it. A reference with no URL is reported as failed at the no_url stage and nothing runs there. A benchmark is for comparing your product with someone else’s. To compare versions of your own work, add another iteration to the same study, or put the variants to a panel with an ask.

​

The benchmark clones and runs

One benchmark call does the whole fan-out. For each reference it reuses the clone of your study that the reference already holds, or creates one. It then opens a round on that clone (an iteration pointed at the reference’s URL) and adds the same simulated people who took part in your source iteration. Finally it dispatches their simulations. What the clone carries over is the question, not the answer: the name, description, assignments, interview questions, and evaluation dimensions. Your iterations, participants, and frames are not copied. The round gets its own participants, built from your roster, so the same people face the same prompts against each product. The comparison stays honest because only the experienced thing differs. Three consequences follow, and each one trips up agents that treat a benchmark as a free setup step. You run your own study first. The benchmark copies its people from the source iteration’s simulated participants, whatever state their runs are in. An iteration with no simulated participants is refused before anything is written. Benchmark an iteration whose run has finished, so you have your own results to compare against. It spends credits on every reference. Each reference runs the full roster, and billing is per completed participant scaled by steps, as for any run. Three references cost roughly three runs of your study. Without an explicit step cap, each participant gets the default of 15 steps. Re-running grows the round instead of starting over. Benchmark the same source iteration against the same reference again and the new participants join the round that already exists: one reference, one round, a roster that grows. Benchmarking a different source iteration opens a new round, so people who ran against different designs are never pooled into one set of metrics.

​

Read the outcome before the results

Each reference is handled on its own, so one failure never stops the rest. The response puts every reference in started, failed, or skipped, and adds a design_drift entry when a reused clone has drifted: Read failed and skipped before you trust the cohort, or you will read a comparison that is missing a competitor.

​

Reading the cohort head-to-head

The comparison read (ish study comparison, or study_comparison on MCP) returns your source iterations and each reference’s rounds in one payload, with the metrics already computed per iteration: participants, completion rate, average time to finish, average steps, friction, sentiment, and purchase intent when the study measures it. A running count above zero means people are still in flight; read again later. The web app’s benchmark page shows the same comparison and can write a comparative summary across it. For the full per-participant findings behind the numbers, read the studies together: study_get takes a study_ids list, and ish study get accepts several IDs. On MCP, study_benchmark can also create references inline from a list of {name, base_url} pairs. That is convenient once, but every call creates new references, so pass reference_ids when you re-run. Full flags, parameters, and return shapes live in the reference: ish workspace reference, ish study benchmark and ish study comparison, reference tools, and study tools.

​

Related