Agentic Evaluations

Evaluate autonomous agents end-to-end: tasks, trajectories, trace-based criteria, and aggregate agent metrics.

Agentic evaluations extend elluminate beyond single-turn LLM outputs to cover agents that plan, call tools, and work against a task description over many steps. elluminate is runtime-agnostic: you run the agent wherever it already lives — your own code, a framework like LangChain, CrewAI, or AutoGen, or a harness like Harbor — translate its output to the ATIF trajectory format, and upload the trial results, including full trajectories, to elluminate. elluminate then rates each criterion against the trajectory, and the UI surfaces the trace, per-criterion ratings, and aggregate metrics.

What agentic evaluation is

When to use agentic evaluations

Use this workflow when:

• Your system makes multiple LLM calls per task (tool use, plan/act loops, sub-agents).
• Evaluation needs to look at what the agent did, not only at its final message.
• You already have (or want to keep) an external runner, e.g. Harbor, LangChain, CrewAI, AutoGen, or your own code.

For single-turn outputs, or tool-calling patterns where elluminate generates the responses itself, see the Tool Calling guide instead.

elluminate does not execute your agent

Agentic evaluations cover uploading and rating external agent runs. You run the agent yourself — with your own code, a framework, or a harness like Harbor — and elluminate stores the trial results, renders the Agent Trace, and (optionally) rates each criterion against the trajectory.

Core concepts

• Task — one unit of work for your agent, identified by a unique task name.
• Collection — your set of tasks: one row per task, with a task (name) column and an optional instruction (input) column.
• Trajectory (trace) — the full step-by-step record of one agent run: every message, tool call, and observation, in the ATIF format.
• Criterion — a binary YES/NO quality question. elluminate's LLM judge answers each criterion against the trajectory — the same judging used for a standard experiment. Criteria in a criterion set apply to every task; task-specific criteria apply to one task only.
• Overall Rating — a platform-managed binary YES/NO criterion included automatically in every Agentic experiment. It answers whether the agent succeeded at the task overall.
• Experiment — one evaluation run: it binds a collection and a criterion set, and holds the uploaded trajectories with their ratings and aggregate metrics.

The trajectory format

elluminate consumes one input from your runner: an agent trajectory in the ATIF format (Agent Trajectory Interchange Format), an open specification defined by Harbor. Whatever your runner is, you translate its output to ATIF and upload it — either in the browser or via the SDK. There is no native importer for LangChain, CrewAI, AutoGen, or other frameworks yet. The full schema and a minimal example are in the ATIF trajectory format section.

Not to be confused with UCE (elluminate.uce/1): UCE is a conversation input elluminate runs itself (see Conversations), while ATIF is the record of a run you already executed.

UI walkthrough

Collections → New collection, with the Agentic collection type selected.

1. Create an Agentic collection. In Collections → New collection, pick the Agentic type. It comes pre-configured with two columns, and keeping these conventional names is recommended: task — a short, unique name for each task (one row per task) — and instruction — the input the agent was given for that task (for a chat agent, the first user message). Write every instruction as complete, self-contained text. instruction is optional: when you upload a trajectory captured outside elluminate, the input is already part of the trajectory, so the column can be left empty. No prompt template is required, and Agentic experiments never auto-generate responses.
2. Create a criterion set. In Criteria Library → New Criterion Set, give it a name, then open the set and use Add New Criterion to add complete, self-contained binary YES/NO questions elluminate answers against the trajectory (e.g. "Did the agent edit the correct file?"). These apply to every task; for checks that only make sense for one task, add task-specific criteria.
3. Create an Agentic experiment. On Experiments → New, choose the Agentic type, then select your collection and criterion set. No prompt template is needed — the wizard omits that step for Agentic experiments. The experiment name is auto-filled from the collection — rename it as you like.
4. Run the agent externally (your own code, a framework, or a harness like Harbor) and collect one result per task, including an ATIF trajectory.
5. Upload the results — either directly in the browser or via the SDK. When trajectories are present, elluminate automatically rates each criterion against the trajectory.
6. Review the results in the UI (see below).

Automatic Overall Rating

elluminate includes Overall Rating in the frozen evaluation definition automatically. It does not add the criterion to or otherwise modify the criterion set you selected. Every evaluated response therefore has one more rating than the number of criteria in that set. If the selected set also contains a criterion labelled Overall Rating, both remain in the frozen definition as separate criterion IDs.

Experiments → New: creating an Agentic experiment.

Task-specific criteria

A criterion set holds the checks that apply to every task. Checks that only make sense for one task — its concrete pass condition — belong on the task itself. Open your Agentic collection to reach the Tasks view: one card per task, showing its instruction and how many task-specific criteria it has.

An Agentic collection: one card per task, each with its own criteria count.

Click the criteria button on a task card to open the task criteria sheet, then use Add criterion and Save:

• Labels are optional — omitted ones default to Criterion 1, Criterion 2, … They must be unique within the task, and Overall Rating is reserved.
• Criteria can reference the task's own fields as {{field_name}}; click a field chip below the criterion to insert one. Use Add Field in the Tasks view to add a field (such as expected_outcome) to every task.

Task criteria for one task, with a criterion referencing the task's `expected_outcome` field.

What an experiment evaluates for a given task is its criterion set, plus that task's own criteria, plus Overall Rating.

Criteria are frozen when the experiment is created

An Agentic experiment freezes its evaluation definition — the criterion set, every task's criteria, and their field values — when it is created. Later edits to a criterion set or to task criteria never change an existing experiment; create a new experiment to evaluate against the updated criteria.

Reviewing results in the UI

The experiment page, Overview tab.

The experiment page has three tabs:

• Overview — aggregate metrics (overall score; plus cost per task, average duration, and tokens when your uploaded run includes them), a Criteria Performance chart (pass rate per criterion), and an AI Experiment Summary (an Executive Summary, plus sections such as Strengths, Weaknesses, Recommendations, and Notable Samples — each shown only when the summary has content for it). The chart includes Overall Rating, so the experiment's task-success rate is visible at a glance; it compares only criteria shared by all tasks, so task-specific criteria are shown on the individual responses instead. The summary is generated only after every trace is annotated, so it can take a few minutes to appear.
• Sample Navigator — per-sample detail: the Phase Timeline (a Gantt-style view of the agent's steps) and the Agent Trace (every message, tool call, and observation), alongside the per-criterion YES/NO ratings with reasoning. When a run delegates to sub-agents, each sub-agent trajectory renders nested and collapsible under the step that spawned it, so multi-agent runs stay fully inspectable.
• Responses Overview — all responses at a glance.

The experiment page, Sample Navigator — Phase Timeline and Agent Trace for one sample.

Upload results in the UI

Once your Agentic collection (the task rows) and Agentic experiment exist, you can upload results straight from the browser — no SDK required. Open the experiment and click Upload results (also offered from the empty state before any results exist).

In the dialog you:

1. Drop a results file — a .json file containing an array of trial-result objects, or a .jsonl file with one object per line.
2. Confirm the task name column — the experiment freezes this setting when it is created (usually task). Uploads use that frozen column; choosing a different one is rejected.
3. Toggle "Evaluate after upload" (default on) — when enabled, elluminate rates each criterion against the uploaded trajectory after upload.
4. Set the epoch (default 1) — bump it to upload another run of the same tasks into the same experiment.

Upload file format

Each object in the file mirrors the SDK AgentTrialResult:

• task_name (required) — must exactly match a task name frozen when the experiment was created.
• trajectory — the ATIF trajectory dict (schema_version must match ATIF-v1.x, plus agent, steps, and final_metrics).
• optional: reward, cost_usd, steps, messages, criterion_ratings.

For precomputed ratings, first call experiment.get_agent_results_definition(). Each returned task lists the exact frozen criteria that apply to it. Include criterion_id in every criterion_ratings entry; labels are display text and may repeat. If you upload only a trajectory and let elluminate evaluate it, you do not need to fetch the definition.

Experiments created before frozen definitions still require task_name_column and retain a mutable live definition. Label-only ratings remain supported when the label is unambiguous; supply criterion_id when display labels repeat. This compatibility path does not provide the historical freeze guarantee of a new experiment.

A minimal single-trial .json file (an array with one object):

[
  {
    "task_name": "write-hello-world",
    "reward": 1.0,
    "cost_usd": 0.0042,
    "trajectory": {
      "schema_version": "ATIF-v1.0",
      "session_id": "run-001/write-hello-world",
      "agent": {
        "name": "demo-agent",
        "version": "0.1.0",
        "model_name": "claude-sonnet-4-6"
      },
      "steps": [
        {
          "step_id": 1,
          "source": "user",
          "message": "Write a Python hello world script to hello.py"
        },
        {
          "step_id": 2,
          "source": "agent",
          "message": "Writing hello.py.",
          "tool_calls": [
            {
              "tool_call_id": "tc_1",
              "function_name": "write_file",
              "arguments": {"path": "hello.py", "content": "print('Hello, World!')"}
            }
          ],
          "observation": {
            "results": [{"source_call_id": "tc_1", "content": "wrote 22 bytes"}]
          },
          "metrics": {"prompt_tokens": 420, "completion_tokens": 61, "cost_usd": 0.0042}
        }
      ],
      "final_metrics": {
        "total_steps": 2,
        "total_cost_usd": 0.0042,
        "total_prompt_tokens": 420,
        "total_completion_tokens": 61
      }
    }
  }
]

For a .jsonl file, write the same objects one per line (no surrounding array, no commas).

Partial success and async evaluation

Trials are processed independently: a malformed trajectory is dropped for that trial only, and the upload reports per-trial errors so the rest still go through. When Evaluate after upload is on, rating runs asynchronously — results appear on the experiment page as they complete.

SDK walkthrough

The following script covers the full end-to-end flow: an Agentic collection, an Agentic experiment, conversion of runner output, and upload with elluminate's evaluation queued. The script is idempotent — collections, criterion sets, and experiments are reused across runs, and uploads are skipped when an experiment already contains responses.

Running the example

Set your API key (created in the elluminate UI under Project → Keys) either as an environment variable or in a .env file next to the script; the example calls load_dotenv():

# option 1: shell
export ELLUMINATE_API_KEY=<your-key>
# optionally, if you run elluminate on a non-default host:
# export ELLUMINATE_BASE_URL=https://your-instance.example.com

# option 2: .env in elluminate_sdk/examples/
echo "ELLUMINATE_API_KEY=<your-key>" > elluminate_sdk/examples/.env

# run
uv run --directory elluminate_sdk python examples/example_harbor_agentic_upload.py

"""Harbor-based Agentic Evaluation: end-to-end upload example.

This example shows the full workflow for evaluating an agent that was run
externally with Harbor (or any other agent framework), and uploading the
results, including ATIF trajectories, to elluminate for inspection and
automatic per-criterion evaluation.

Workflow:

1. Create an AGENTIC collection whose rows are the agent's tasks. The task
   name lives in a single TEXT `task` column (matched against each uploaded
   result's `task_name`); no prompt template is needed because AGENTIC
   experiments do not auto-generate responses.
2. Create a criterion set describing what counts as success.
3. Create an AGENTIC experiment (no auto-generation; results are uploaded).
4. Run the agent externally (Harbor CLI, LangChain, CrewAI, custom code).
5. Read Harbor's per-task output and convert it into `AgentTrialResult` objects.
6. Upload via `experiment.upload_agent_results(...)`.
7. The backend stores the trajectories and, when `evaluate=True` and
   trajectories are present, elluminate automatically rates each criterion
   against the trajectory.

The script is idempotent: collections, criterion sets, and experiments are
reused across runs; uploads are skipped when an experiment already contains
responses.

For a self-contained demo this script uses a small in-memory stand-in
(`HARBOR_RUN`) for the runner's output. In a real run you would replace it
with your own loader that reads your runner's per-task output from disk — for
Harbor, the run directory at `~/.harbor/runs/<run_name>/tasks/<task>/...`.
"""

from typing import Any

from dotenv import load_dotenv
from elluminate import AgentTrialResult, Client
from elluminate.schemas import CollectionColumn, ColumnTypeEnum
from elluminate.schemas.criterion import CriterionIn
from elluminate.schemas.experiments import Experiment

load_dotenv(override=True)

client = Client()  # (1)!
llm_config = client.get_llm_config(name="Claude Sonnet 4.6")

# Mock "Harbor output"; in a real integration this is read from disk.  # (2)!
# Each entry is what a Harbor run produces per task: a short task identifier,
# the instruction text, final messages, aggregate metrics, and an ATIF
# trajectory describing every step the agent took.
HARBOR_RUN: list[dict[str, Any]] = [
    {
        "task_name": "write-hello-world",
        "instruction": "Write a Python hello world script to hello.py",
        "reward": 1.0,
        "steps": 2,
        "cost_usd": 0.0042,
        "input_tokens": 420,
        "output_tokens": 61,
        "duration_seconds": 3.2,
        "messages": [
            {"role": "user", "content": "Write a Python hello world script to hello.py"},
            {"role": "assistant", "content": "Wrote hello.py: print('Hello, World!')"},
        ],
        "trajectory": {
            "schema_version": "ATIF-v1.0",
            "session_id": "harbor-run-001/write-hello-world",
            "agent": {
                "name": "harbor-demo-agent",
                "version": "0.1.0",
                "model_name": "claude-sonnet-4-6",
            },
            "steps": [
                {
                    "step_id": 1,
                    "source": "user",
                    "message": "Write a Python hello world script to hello.py",
                },
                {
                    "step_id": 2,
                    "source": "agent",
                    "message": "Writing hello.py.",
                    "tool_calls": [
                        {
                            "tool_call_id": "tc_1",
                            "function_name": "write_file",
                            "arguments": {"path": "hello.py", "content": "print('Hello, World!')"},
                        }
                    ],
                    "observation": {
                        "results": [{"source_call_id": "tc_1", "content": "wrote 22 bytes"}],
                    },
                    "metrics": {"prompt_tokens": 420, "completion_tokens": 61, "cost_usd": 0.0042},
                },
            ],
            "final_metrics": {
                "total_steps": 2,
                "total_cost_usd": 0.0042,
                "total_prompt_tokens": 420,
                "total_completion_tokens": 61,
            },
        },
    },
    {
        "task_name": "reverse-string-function",
        "instruction": "Create a Python function that reverses a string in reverse.py",
        "reward": 0.5,
        "steps": 2,
        "cost_usd": 0.0031,
        "input_tokens": 310,
        "output_tokens": 42,
        "duration_seconds": 2.1,
        "messages": [
            {"role": "user", "content": "Create a Python function that reverses a string in reverse.py"},
            {"role": "assistant", "content": "Wrote reverse.py with a one-line slice-based reverse."},
        ],
        "trajectory": {
            "schema_version": "ATIF-v1.0",
            "session_id": "harbor-run-001/reverse-string-function",
            "agent": {
                "name": "harbor-demo-agent",
                "version": "0.1.0",
                "model_name": "claude-sonnet-4-6",
            },
            "steps": [
                {
                    "step_id": 1,
                    "source": "user",
                    "message": "Create a Python function that reverses a string in reverse.py",
                },
                {
                    "step_id": 2,
                    "source": "agent",
                    "message": "Writing reverse.py.",
                    "tool_calls": [
                        {
                            "tool_call_id": "tc_1",
                            "function_name": "write_file",
                            "arguments": {
                                "path": "reverse.py",
                                "content": "def reverse(s: str) -> str:\n    return s[::-1]\n",
                            },
                        }
                    ],
                    "observation": {
                        "results": [{"source_call_id": "tc_1", "content": "wrote 42 bytes"}],
                    },
                    "metrics": {"prompt_tokens": 310, "completion_tokens": 42, "cost_usd": 0.0031},
                },
            ],
            "final_metrics": {
                "total_steps": 2,
                "total_cost_usd": 0.0031,
                "total_prompt_tokens": 310,
                "total_completion_tokens": 42,
            },
        },
    },
]


def harbor_to_agent_trial(task_output: dict[str, Any]) -> AgentTrialResult:  # (3)!
    """Map one Harbor per-task output dict to an `AgentTrialResult`.

    `task_name` on `AgentTrialResult` is what elluminate matches against the
    collection's `task_name_column` value, so here we set it to the full
    instruction text (which is also what the `task` column row holds).
    """
    return AgentTrialResult(
        task_name=task_output["instruction"],
        messages=task_output["messages"],
        reward=task_output["reward"],
        steps=task_output["steps"],
        cost_usd=task_output["cost_usd"],
        input_tokens=task_output["input_tokens"],
        output_tokens=task_output["output_tokens"],
        duration_seconds=task_output["duration_seconds"],
        trajectory=task_output["trajectory"],
        metadata={"run_name": "harbor-run-001", "task_id": task_output["task_name"]},
    )


# Step 1: AGENTIC collection with a single TEXT `task` column.  # (4)!
# The `task` column must be TEXT so elluminate auto-selects it as the
# task-name column that uploaded results are matched against. No prompt
# template is required because AGENTIC experiments never auto-generate;
# responses are supplied by `upload_agent_results`.
collection, _ = client.get_or_create_collection(
    name="Harbor Demo Tasks",
    defaults={
        "collection_type": "AGENTIC",
        "columns": [CollectionColumn(name="task", column_type=ColumnTypeEnum.TEXT)],
        "variables": [{"task": h["instruction"]} for h in HARBOR_RUN],
    },
)

# Step 2: criterion set defining what success looks like for these tasks.  # (5)!
# Labels are display text and may repeat; frozen criterion IDs identify ratings.
criterion_set, _ = client.get_or_create_criterion_set(
    name="Harbor Demo Criteria",
    defaults={
        "criteria": [
            CriterionIn(
                criterion_str="Did the agent correctly complete the requested task?",
                label="task-complete",
            ),
            CriterionIn(
                criterion_str="Did the agent use tools appropriately?",
                label="uses-tools",
            ),
            CriterionIn(
                criterion_str="Is the agent's final output correct?",
                label="output-correct",
            ),
        ],
    },
)


def get_or_create_agentic_experiment(name: str, description: str) -> tuple[Experiment, bool]:  # (6)!
    """Return an AGENTIC experiment, creating it if missing.

    Also reports whether the experiment already has uploaded responses so the
    caller can skip a redundant upload on re-runs (avoids epoch conflicts).
    """
    try:
        experiment = client.get_experiment(name=name, fetch_responses=False)
        populated = experiment.results is not None and experiment.results.completed_epochs > 0
        return experiment, populated
    except ValueError:
        experiment = client.create_experiment(
            name=name,
            collection=collection,
            prompt_template=None,
            criterion_set=criterion_set,
            description=description,
            evaluation_mode="AGENTIC",
            llm_config=llm_config,
        )
        return experiment, False


# Step 3: AGENTIC experiment. No auto-generation; results come from Harbor.  # (7)!
experiment, experiment_populated = get_or_create_agentic_experiment(
    "Harbor Demo — Agent Run",
    "Harbor-run coding agent with ATIF trajectories.",
)
print(f"Experiment: {experiment.name} (id={experiment.id})")

# Step 4: Convert Harbor output to `AgentTrialResult` objects.  # (8)!
results = [harbor_to_agent_trial(task_output) for task_output in HARBOR_RUN]

# Step 5: upload with `evaluate=True`. elluminate rates every  # (9)!
# criterion against the trajectory and fills in per-criterion ratings.
if experiment_populated:
    print("Experiment already has responses; skipping upload.")
else:
    upload = experiment.upload_agent_results(
        results=results,
        task_name_column="task",
        evaluate=True,
    )
    print(
        f"Uploaded: {upload.created_responses} responses, "
        f"{upload.created_ratings} ratings, "
        f"{upload.pending_evaluations} pending trace evaluations"
    )
    if upload.errors:
        print(f"Errors: {upload.errors}")

# Step 6: Verify the trajectories are queryable from the SDK.  # (10)!
experiment.fetch_responses()
for resp in experiment.responses():
    task = resp.prompt.template_variables.input_values.get("task", "?")
    steps = len(resp.trajectory["steps"]) if resp.trajectory else 0
    print(f"  [{task[:50]}] trajectory_steps={steps}")

1. Initialize the SDK client (uses ELLUMINATE_API_KEY) and pick an LLMConfig to associate with the experiment (metadata only; Agentic experiments never invoke it).
2. Stand-in for your runner's on-disk output. Replace with code that reads Harbor's task_result.json + trajectory.json per task.
3. Translate one runner output into an AgentTrialResult. This is the only integration-specific code you need; task_name must match the value in the collection row's task column.
4. Create an Agentic collection with a single TEXT task column, one row per task. No prompt template is required.
5. Create the criterion set that defines success. Labels are display text and may repeat; frozen criterion IDs identify ratings in uploads.
6. Idempotent helper that returns an Agentic experiment and whether it already holds uploaded responses.
7. Get-or-create the main experiment. evaluation_mode="AGENTIC" disables auto-generation; responses are supplied via upload.
8. Convert every runner output into an AgentTrialResult.
9. Upload with evaluate=True so elluminate rates every criterion against the trajectory. Skipped on re-runs when the experiment is already populated.
10. Re-fetch the experiment and confirm trajectories are queryable from the SDK.

Growing the dev set (the eval flywheel)

Coverage grows over time: when your agent fails in production, feed that case back into the dev set so every future run is checked against it. To append new tasks to an existing collection, fetch the collection and call add_many() with one dict per task, keyed by the collection's task-name column (task):

# Grow an existing dev-set collection with newly discovered failure cases
collection = client.get_collection(name="my-agent-dev-set")
collection.add_many([
    {"task": "New failing scenario discovered in prod ..."},
    {"task": "Another regression case ..."},
])
# Re-run the agent on the (now larger) collection and upload the new trajectories.

This is different from bumping the epoch at upload time: the epoch re-runs the same tasks for another sample, whereas appending rows widens what you evaluate. Once the tasks exist, run your agent over the enlarged collection and upload the new trajectories as usual.

AgentTrialResult fields

Each trial your runner produces maps to one AgentTrialResult. Required and optional fields:

Field	Required	Description
task_name	yes	Must exactly match a task name frozen when the experiment was created.
messages	no	Final OpenAI-format message list (shown on the response page).
reward	no	Primary reward score (0.0–1.0).
steps	no	Number of agent steps / LLM calls.
cost_usd	no	Total USD cost for the trial.
duration_seconds	no	Wall-clock duration.
input_tokens	no	Aggregate input tokens.
output_tokens	no	Aggregate output tokens.
cached_tokens	no	Aggregate cached input tokens.
error	no	Error message if the trial failed.
metadata	no	Free-form dict surfaced on the response page.
trajectory	no	Raw ATIF trajectory (validated by the backend; see ATIF format).
criterion_ratings	no	Precomputed YES/NO ratings. For frozen experiments, use criterion_id from get_agent_results_definition(); labels are display-only.

ATIF trajectory format

Trajectories use the Agent Trajectory Interchange Format (ATIF), an open trajectory specification defined by Harbor.

A minimal ATIF v1 trajectory:

{
  "schema_version": "ATIF-v1.0",
  "session_id": "harbor-run-001/write-hello-world",
  "agent": {
    "name": "harbor-demo-agent",
    "version": "0.1.0",
    "model_name": "claude-sonnet-4-6"
  },
  "steps": [
    {
      "step_id": 1,
      "source": "user",
      "message": "Write a Python hello world script to hello.py"
    },
    {
      "step_id": 2,
      "source": "agent",
      "message": "Writing hello.py.",
      "tool_calls": [
        {
          "tool_call_id": "tc_1",
          "function_name": "write_file",
          "arguments": {"path": "hello.py", "content": "print('Hello, World!')"}
        }
      ],
      "observation": {
        "results": [{"source_call_id": "tc_1", "content": "wrote 22 bytes"}]
      },
      "metrics": {"prompt_tokens": 420, "completion_tokens": 61, "cost_usd": 0.0042}
    }
  ],
  "final_metrics": {
    "total_steps": 2,
    "total_cost_usd": 0.0042,
    "total_prompt_tokens": 420,
    "total_completion_tokens": 61
  }
}