external_chatbot shape of the chat modality: each participant sends turns to your HTTP endpoint, reads the bot’s replies, and reports what they noticed, where they stalled, and whether they finished the assignment. For two simulated people rehearsing a conversation against each other (a pitch, a hard sell, a support escalation), see the chat-pair guide instead.
The shape of the job: save the endpoint, author a configuration on it, create a chat study, run it, read the results.
This guide assumes you have authenticated the CLI or connected an agent, and that you have an active workspace. If not, start with the CLI quickstart or connect an agent.
Before you start
You need a reachable HTTP chatbot. ish makes the call itself, so the bot has to answer a POST (or PUT) with a parseable reply. Two reachability cases:A hosted bot
A public URL ish can reach from the cloud. Nothing else to set up. If the call needs an API key, store it once with
ish secret set and reference it as {{secret:KEY}} in the endpoint config.A localhost bot
A bot running on your machine (
http://localhost:3000). Mark the endpoint tunnel-backed and open a Cloudflare tunnel with ish connect <port> so the remote fleet can reach it. The endpoint stays unreachable until a tunnel is live.Steps
Save the chatbot endpoint
An endpoint is the saved wire contract: the URL, method, headers, the request body template, and the response paths ish reads the reply from. Author it once and reuse it across studies.The fastest path is
init: hand it a curl sample, a JSON request/response sample, or a known-good template, and ish infers the ChatbotEndpointConfig for you. Templates are openai, anthropic, voiceflow, dialogflow-cx, and botframework. The MCP equivalent, chatbot_setup, infers, smoke-tests, and persists in one call.init prints the saved endpoint’s alias (Created endpoint ep-abc) and returns it as alias in the JSON. Reference the endpoint by that ep- alias (or its UUID) in every later step. --name only sets the display name; it does not resolve an endpoint. The examples below use ep-abc, the alias init returned.Templates use {{secret:NAME}} placeholders for auth. Set the matching workspace secret before you test, or the call goes out with an empty key and the bot answers 401.For a localhost bot, add --tunnel-backed (CLI) or is_tunnel_backed=true (MCP). A localhost URL is auto-detected as a tunnel hint, but the endpoint is only created tunnel-backed when you set the flag.Probe the connection
Send one synthetic turn before you commit a study to it. This is a smoke test, not a simulation: it draws no credits and tells you whether the wire contract holds.
The chatbot tools reference lists every
chatbot_setup runs this probe for you when smoke_test=True (the default) and refuses to persist the endpoint if the test fails. Branch on the returned error_kind, not the message:error_kind | What it means | What to do |
|---|---|---|
BotUnreachable | DNS or connection refused | Fix the URL, or confirm the bot is up |
TunnelInactive | Tunnel-backed endpoint, no live tunnel | Run ish connect <port> and retry |
BotAuthError | 401 or 403 | Check outgoing.headers and the {{secret:KEY}} it resolves |
BotResponseError | HTTP non-2xx | Inspect raw_excerpt |
BotInvalidResponseError | Reply missed the configured incoming.* paths | Re-infer the shape with init / chatbot_setup(paste=...) |
error_kind.Author a default configuration
A configuration captures what the bot is for this run: model, system prompt, tools, sub-agents, and custom fields. It is distinct from the endpoint’s wire contract, and it lives N-per-endpoint so you can compare bot setups across iterations.A chat study needs a default configuration on its endpoint. An endpoint alone is not enough: Each iteration freezes a snapshot of the configuration at creation, so editing it later never rewrites a past run. To change the bot setup, author a new configuration with a fresh name.
ish study create --modality chat resolves the endpoint’s default configuration at creation time. Author one with --default first.Create the chat study
Create the study against the saved endpoint. ish resolves the endpoint’s default configuration and builds the first iteration inline. Give it at least one assignment so participants have a goal.
--max-turns caps the conversation per participant; see the study create reference for the default. To embed the endpoint without saving it first, pipe a ChatbotEndpointConfig into --endpoint-config - instead of --endpoint; the two flags are mutually exclusive. Assignments may carry steps, which are honored for external_chatbot chat. See the study create reference and study tools for the full flag set.Run it and read what came back
People are selected at run time, not at create time. For a tunnel-backed bot, open the tunnel first.
--sample caps at 20 people per dispatch; split larger cohorts across slices. --wait blocks until every conversation reaches a terminal state. The results carry the full transcript per participant plus the reasoning behind each reaction, not a single score. See reactions and results and the study run reference.Variations
The bot runs on localhost
The bot runs on localhost
Mark the endpoint tunnel-backed (
--tunnel-backed), open a Cloudflare tunnel with ish connect <port> before probing or running, and keep it open for the duration of the run. A probe against a tunnel-backed endpoint with no live tunnel returns error_kind=TunnelInactive. The ish connect reference covers --detach and multi-service --proxy routing.The bot is stateful
The bot is stateful
A stateful bot tracks a session and returns a
conversation_id. The endpoint config’s outgoing.mode is stateful (versus stateless), and {{conversation_id}} threads the bot-supplied id back into each turn. When probing across invocations, pass --conversation-id (CLI) or conversation_id (MCP) to maintain the thread.The bot streams or polls instead of one request per turn
The bot streams or polls instead of one request per turn
The endpoint config dispatches on
transport: sync (one HTTP request per turn, the common case), async_poll (an initial POST returns a job id that ish polls), or streaming (SSE or chunked-stream, with eventFormat set to openai, anthropic, or raw). Infer the shape from a real sample with init / chatbot_setup(paste=...) rather than hand-writing it.You only have documentation, not a curl sample
You only have documentation, not a curl sample
ish chat endpoint map --docs ./api.md runs the same inference engine as init but fuses a draft endpoint’s URL and headers with freeform docs (a README chunk, an OpenAPI excerpt), and probes the live bot to refine the mapping when the URL is reachable. The output is read-only: pipe inferred_config into ish chat endpoint create --endpoint-config - to persist.You want to compare two bot setups
You want to compare two bot setups
Author a second configuration with a new
--name, add a second iteration referencing it, and run both. Each iteration freezes its configuration snapshot, so the comparison stays reproducible. See benchmarking.Where to go next
ish chat reference
Every flag for
chat endpoint and chat config, including init, test, map, and the config fields.chatbot tools
chatbot_setup, chatbot_test, chatbot_get, and the full error_kind vocabulary.Secrets
Store the bot’s API key once and reference it as
{{secret:KEY}} on the wire.Modalities
Where chat sits among interactive, text, image, video, audio, and document.