Skip to main content
Browse all guides
All guides

Evaluations

Evaluations help you test an Action against known examples and compare the actual result with the result you expected.

Use evaluations when you want confidence that an Action still behaves correctly after changing prompts, steps, tools, filters, or AI model settings.

What Evaluations Are For

An evaluation dataset is a collection of test cases for one Action. Each test case contains representative input data and the expected result.

Evaluations help answer questions such as:

  • Does the Action produce the expected final output?
  • Did the Action call the expected tools with the expected arguments?
  • Which test cases fail after a configuration change?
  • Is the behavior stable across common, edge, and failure-prone inputs?
  • Can a cheaper or faster AI model handle this Action, or does it need a more capable model?

Typical Evaluation Process

  1. Pick an Action you want to validate.
  2. Create or open an evaluation dataset for that Action.
  3. Add test cases with realistic inputs and expected outputs.
  4. Run the dataset as a dry run or live test.
  5. Review passed and failed cases.
  6. Adjust the Action, prompts, model choices, tools, or expectations.
  7. Run the dataset again and compare results.

This gives you a repeatable way to verify changes instead of relying on one manual run.

The Test Datasets panel on the Evaluations page lists every dataset in the tenant. When the list is long, it scrolls inside the panel, and the search box above it filters datasets by name, description, or Action name.

Creating Test Cases From Runs

The easiest way to create useful test cases is often from real runs.

When you inspect a run, you can add it to an evaluation dataset. Studio uses the run data to prefill the test case, including the input context and observed outputs. You can then edit the expected result to describe what should happen in future runs.

This is useful because real runs include realistic input shapes, tool responses, and edge cases that are easy to miss when writing test data manually.

What a Test Case Contains

A test case can include:

  • input parameters for the Action,
  • expected tool calls,
  • expected final output,
  • optional evaluation metrics,
  • whether the case is enabled for runs.

Expected tool calls are useful when you need to verify that the Action not only returns the right answer, but also uses the right systems in the right way.

Expected final output is useful for validating the final business result, such as extracted invoice fields, classification decisions, or generated summaries.

Evaluation Metrics

Exact output matching is strict: any difference in the final output fails the case. Evaluation metrics let you check only the fields that matter.

Each metric has a JSON path (for example $.InvoiceData.currency), a comparison type, and an expected value. When you add a test case from a run, use Pick from Output to click a field from the run's output and create a metric for it with the observed value prefilled. A field that was null in the run gets an Is Null or Empty metric.

The comparison types are grouped in the editor:

GroupComparisonPasses when the extracted value
TextExact Match / Not Equalsequals / does not equal the expected value
TextContains / Not Containscontains / does not contain the expected text
TextStarts With / Ends Withstarts / ends with the expected text
PresenceIs Null or Emptyis missing, null, blank text, or an empty list or object
PresenceIs Not Null or Emptyhas any other value
Pattern & similarityRegex Patternmatches the regular expression
Pattern & similarityString Similarityis at least as similar as the threshold (0-100%)
NumericNumeric Equals / Greater Than / Less Thancompares as a number with the expected value

Text comparisons, regular expressions, and string similarity ignore letter case unless Case Sensitive is turned on. The presence checks have no expected value. A JSON path that does not resolve counts as no value, so it passes Is Null or Empty, Not Equals, and Not Contains; use Is Not Null or Empty on the same path when the field must also exist. The run results show the comparison of each metric next to its expected and actual values.

Metrics on intermediate step outputs

By default a metric is evaluated against the final output, which is the output of the last executed step in the Action Flow. A metric can instead target the output of any executed step, which is useful when the interesting result is produced in the middle of the flow. For example, you can require that an Analysis step always extracts USD as the currency, even when a later step transforms or summarizes that result.

To create a step metric, open Pick from Output, choose the step in Pick fields from, and click a field from that step's output. The metric card shows the step name, and the Evaluate selector on the card lets you switch a metric between the final output and any step of the run. When editing a test case later, Studio loads the steps of the run the case was created from, or of its most recent evaluation run, so the same choices are available. A test case that was created manually and never evaluated has no step outputs to pick from yet; run the dataset once and the steps appear.

A step metric fails when the step did not run in the evaluation, for example because it was disabled or its switch branch was not taken. Steps are matched by their identity across draft and published versions, so renaming or reordering steps does not break existing metrics.

Turn on Evaluate metrics only when the metrics should decide the result on their own. The expected final output is then kept for reference but no longer compared.

Running a Dataset

Run a dataset when you want to verify multiple cases together.

Use a dry test when you want to avoid making live external changes where supported. Use a live test when the Action must interact with real tools or systems to validate the full Action Flow.

Select the test cases to include, choose the Action version to test, and start the run with Run Dry Test or Run Live Test. After the run finishes, Studio shows summary counts and case-level results.

Actions that trigger other Actions

When the evaluated Action contains Trigger Action steps, the triggered Actions run as well, as separate runs. A dry test stays dry for the whole chain. The Latest draft version runs the latest draft of every Action in the chain, the same way a manual run does; the Published and specific-version choices run the triggered Actions' published versions.

Only the evaluated Action's own run is compared with the test case. To check what a triggered Action received, add a step metric on the Trigger Action step and pick a field under its payload output, for example $.payload.invoiceId. To test the triggered Action itself, create a dataset for that Action and add test cases from its runs; the captured input parameters are the payload it was triggered with.

Repeating Test Cases

Agent Steps can produce a different output for the same prompt from one run to the next, so a test case that passes once is not necessarily stable. Use the Repeat selector next to the Action version to execute every selected test case up to 10 times within one run.

The run then counts each repetition as its own execution: the summary shows the total number of executions, and the results are grouped per test case. Each group shows how many repetitions passed and whether the outcome was consistent:

  • Consistent pass: every repetition passed.
  • Consistent fail: every repetition failed, which points to a wrong expectation or a real regression.
  • Inconsistent: some repetitions passed and others failed, which points to an unstable prompt, model, or expectation.

Expand a group to inspect the individual repetitions, each with its own output diff, metrics, and link to the Action run log.

Running All or Selected Datasets

Use Dry Run All or Live Run All on the Evaluations page to start a run for every dataset that has test cases. Studio then shows a batch overview with one row per dataset: its status, how many cases have finished, and the passed and failed counts. The overview updates while the runs are executing and a notification appears when every dataset has finished.

To run only some datasets, hold Ctrl (⌘ on a Mac) and click the dataset cards you want; a checkbox appears on every card, and Shift+click selects every dataset between the last one you picked and the clicked one. If a dataset is already open, the first Ctrl+click adds it to the selection. The buttons change to Dry Run Selected and Live Run Selected, and the count in the button includes only the selected datasets that have test cases. A plain click opens a dataset and ends the selection; Clear or Esc ends it without opening anything.

Click a dataset row to open that run's detailed results. If a dataset could not be started, the row shows the error and offers a retry. You can hide the overview while runs continue and reopen it from the status banner under the run buttons.

Reviewing Failed Cases

The test run view shows which cases passed and failed. For each case, you can inspect mismatches in tool calls, final output, metrics, and error messages.

If a case fails, open the linked test case from the run results and decide whether to:

  • fix the Action because the output is wrong,
  • update the expected output because the intended behavior changed,
  • split the scenario into more specific test cases,
  • adjust the model or prompt because the result is unstable (repeat the run to confirm the instability before changing anything).

Use failed cases as a focused debugging queue. They tell you exactly which input scenario needs attention.

Asking Shepherd about results

Dooap Shepherd can read test datasets, test cases, and test run results. With a test run open, ask it to summarize the results or to explain why a case failed: it reads the metric expected and actual values, the mismatched tool calls, and the linked action run, then tells you whether the Action regressed, the expected value is stale, or a metric's JSON path no longer resolves. With an Action open, ask which datasets cover it and how the latest run went.

Shepherd can also start a run for you. Ask it to run a dataset, approve the card it shows (dry or live run, Action version, number of cases), and it summarizes the results when the run finishes and compares them with the previous run. Ask "what changed since the last run" on any test run to get the cases that flipped between passing and failing and how their metric values moved.

Evaluating AI Model Choices

Evaluations are especially useful when comparing AI model configurations.

For example, you might want to know whether an Action can use a cheaper and faster model or whether it requires a more capable model. Instead of guessing, run the same dataset with different model settings and compare pass rates, output quality, latency, and cost.

Practical model comparison questions include:

  • Does the smaller model still pass the critical test cases?
  • Are failures limited to edge cases, or do they affect common scenarios?
  • Does the faster model call tools correctly?
  • Is the quality gain from a stronger model worth the added cost?
  • Can different steps use different models based on complexity?

For credit and model cost considerations, see Credits.

Good Evaluation Habits

  • Start with a small dataset of high-value scenarios.
  • Include both successful and failure-prone examples.
  • Add real cases from runs whenever possible.
  • Keep expected outputs specific enough to catch regressions.
  • Re-run evaluations after changing prompts, tools, models, or step logic.
  • Keep test cases enabled only when they represent behavior you currently want to enforce.

Evaluations work best when they become part of normal Action maintenance: change, run, review, improve, and repeat.