Score outcomes

The participant tells you how a moment felt: a comment, a sentiment, a status. That is their perception, not a judgment of whether the task actually succeeded. A participant can be confidently wrong, exactly as a real person can. If you score success off their words, you will record passes on failed tasks.

​

The participant can be confidently wrong

In a real run where a volume control had been secretly reversed, the participant declared “Volume 10 is perfect!” and finished with a completed status, while the frame plainly read VOL 12. Scoring off session_status and sentiment would have logged a clean pass on a broken control. This is not a defect in the participant; it is the point. The participant is the experience layer: did the interface feel confusing, where did they hesitate, what did they misread. A real person would have made the same mistake in front of the same reversed control. What the participant cannot be is the verdict layer, the source of truth on whether the task was met.

​

Score from your own state

Judge task outcomes from your environment’s own state, the ground truth you already hold, and never from the participant’s narration.

1. Keep a source of truth in your environment: the value the control was set to, the page the flow reached, the order that was placed.
2. After the run, assert against that state on your side. Did the volume reach the target? Did the checkout complete?
3. Use the participant’s comment, sentiment, and felt_intensity for the experience read: where they hesitated, what confused them, where friction showed up.

The two layers answer different questions. Your state answers “did it work”. The participant answers “how did it feel to try”. You want both, kept separate.

​

Never feed your state to the participant

If your web adapter exposes a state() method (see author your environment), it is for your own scoring only. Never send that state to the participant as an observation. The participant perceives only rendered senses; handing them the true state turns them from a simulated human into a solver and destroys the very signal you are running them for. Keep state() on your side of the wire, and assert against it there.

​

In intent mode, no_match is its own signal

A no_match resolution is not a failure to score against your state; it is a finding in its own right. The participant reached for something your environment does not offer, and unmet_expectation names the gap. Read those alongside your state assertions: your state tells you whether the task was met, and the no_match stream tells you where the participant expected an affordance that was not there. See intent and resolution.

​

Related