Agentic Evaluations

Evaluate autonomous agents end-to-end: tasks, trajectories, trace-based criteria, and aggregate agent metrics.

Agentic evaluations extend elluminate beyond single-turn LLM outputs to cover agents that plan, call tools, and work against a task description over many steps. elluminate is runtime-agnostic: you run the agent wherever it already lives — your own code, a framework like LangChain, CrewAI, or AutoGen, or a harness like Harbor — translate its output to the ATIF trajectory format, and upload the trial results, including full trajectories, to elluminate. elluminate then rates each criterion against the trajectory, and the UI surfaces the trace, per-criterion ratings, and aggregate metrics.

What agentic evaluation is

When to use agentic evaluations

Use this workflow when:

• Your system makes multiple LLM calls per task (tool use, plan/act loops, sub-agents).
• Evaluation needs to look at what the agent did, not only at its final message.
• You already have (or want to keep) an external runner, e.g. Harbor, LangChain, CrewAI, AutoGen, or your own code.

For single-turn outputs, or tool-calling patterns where elluminate generates the responses itself, see the Tool Calling guide instead.

elluminate does not execute your agent

Agentic evaluations cover uploading and rating external agent runs. You run the agent yourself — with your own code, a framework, or a harness like Harbor — and elluminate stores the trial results, renders the Agent Trace, and (optionally) rates each criterion against the trajectory.

Core concepts

• Task — one unit of work for your agent, identified by a unique task name.
• Collection — your set of tasks: one row per task, with a task (name) column and an optional instruction (input) column.
• Trajectory (trace) — the full step-by-step record of one agent run: every message, tool call, and observation, in the ATIF format.
• Criterion — a binary YES/NO quality question. elluminate's LLM judge answers each criterion against the trajectory — the same judging used for a standard experiment.
• Experiment — one evaluation run: it binds a collection and a criterion set, and holds the uploaded trajectories with their ratings and aggregate metrics.

The trajectory format

elluminate consumes one input from your runner: an agent trajectory in the ATIF format (Agent Trajectory Interchange Format), an open specification defined by Harbor. Whatever your runner is, you translate its output to ATIF and upload it — either in the browser or via the SDK. There is no native importer for LangChain, CrewAI, AutoGen, or other frameworks yet. The full schema and a minimal example are in the ATIF trajectory format section.

UI walkthrough

Collections → New collection, with the Agentic collection type selected.

1. Create an Agentic collection. In Collections → New collection, pick the Agentic type. It comes pre-configured with two columns, and keeping these conventional names is recommended: task — a short, unique name for each task (one row per task) — and instruction — the input the agent was given for that task (for a chat agent, the first user message). instruction is optional: when you upload a trajectory captured outside elluminate, the input is already part of the trajectory, so the column can be left empty. No prompt template is required, and Agentic experiments never auto-generate responses.
2. Create a criterion set. In Criteria Library → New Criterion Set, give it a name, then open the set and use Add New Criterion to add binary YES/NO questions elluminate answers against the trajectory (e.g. "Did the agent edit the correct file?"). To anchor a criterion to a collection's ground-truth data, select Use variables from \<your collection> in the set, then use the Add Variable dropdown to insert a {{column_name}} placeholder for any column in that collection (e.g. {{expected_legal_area}}) — the criterion is then checked against that column's value.
3. Create an Agentic experiment. On Experiments → New, choose the Agentic type, then select your collection and criterion set. No prompt template is needed — the wizard omits that step for Agentic experiments. The experiment name is auto-filled from the collection — rename it as you like.
4. Run the agent externally (your own code, a framework, or a harness like Harbor) and collect one result per task, including an ATIF trajectory.
5. Upload the results — either directly in the browser or via the SDK. When trajectories are present, elluminate automatically rates each criterion against the trajectory.
6. Review the results in the UI (see below).

Experiments → New: creating an Agentic experiment.

Reviewing results in the UI

The experiment page, Overview tab.

The experiment page has three tabs:

• Overview — aggregate metrics (overall score; plus cost per task, average duration, and tokens when your uploaded run includes them), a Criteria Performance chart (pass rate per criterion), and an AI Experiment Summary (an Executive Summary, plus sections such as Strengths, Weaknesses, Recommendations, and Notable Samples — each shown only when the summary has content for it). The summary is generated only after every trace is annotated, so it can take a few minutes to appear.
• Sample Navigator — per-sample detail: the Phase Timeline (a Gantt-style view of the agent's steps) and the Agent Trace (every message, tool call, and observation), alongside the per-criterion YES/NO ratings with reasoning. When a run delegates to sub-agents, each sub-agent trajectory renders nested and collapsible under the step that spawned it, so multi-agent runs stay fully inspectable.
• Responses Overview — all responses at a glance.

The experiment page, Sample Navigator — Phase Timeline and Agent Trace for one sample.

Upload results in the UI

Once your Agentic collection (the task rows) and Agentic experiment exist, you can upload results straight from the browser — no SDK required. Open the experiment and click Upload results (also offered from the empty state before any results exist).

In the dialog you:

1. Drop a results file — a .json file containing an array of trial-result objects, or a .jsonl file with one object per line.
2. Pick the task name column — the collection column whose values each trial's task_name is matched against (usually task).
3. Toggle "Evaluate after upload" (default on) — when enabled, elluminate rates each criterion against the uploaded trajectory after upload.
4. Set the epoch (default 1) — bump it to upload another run of the same tasks into the same experiment.

Upload file format

Each object in the file mirrors the SDK AgentTrialResult:

• task_name (required) — must exactly match a value in the collection's task name column.
• trajectory — the ATIF trajectory dict (schema_version must match ATIF-v1.x, plus agent, steps, and final_metrics).
• optional: reward, cost_usd, steps, messages, criterion_ratings.

A minimal single-trial .json file (an array with one object):

[
  {
    "task_name": "write-hello-world",
    "reward": 1.0,
    "cost_usd": 0.0042,
    "trajectory": {
      "schema_version": "ATIF-v1.0",
      "session_id": "run-001/write-hello-world",
      "agent": {
        "name": "demo-agent",
        "version": "0.1.0",
        "model_name": "claude-sonnet-4-6"
      },
      "steps": [
        {
          "step_id": 1,
          "source": "user",
          "message": "Write a Python hello world script to hello.py"
        },
        {
          "step_id": 2,
          "source": "agent",
          "message": "Writing hello.py.",
          "tool_calls": [
            {
              "tool_call_id": "tc_1",
              "function_name": "write_file",
              "arguments": {"path": "hello.py", "content": "print('Hello, World!')"}
            }
          ],
          "observation": {
            "results": [{"source_call_id": "tc_1", "content": "wrote 22 bytes"}]
          },
          "metrics": {"prompt_tokens": 420, "completion_tokens": 61, "cost_usd": 0.0042}
        }
      ],
      "final_metrics": {
        "total_steps": 2,
        "total_cost_usd": 0.0042,
        "total_prompt_tokens": 420,
        "total_completion_tokens": 61
      }
    }
  }
]

For a .jsonl file, write the same objects one per line (no surrounding array, no commas).

Partial success and async evaluation

Trials are processed independently: a malformed trajectory is dropped for that trial only, and the upload reports per-trial errors so the rest still go through. When Evaluate after upload is on, rating runs asynchronously — results appear on the experiment page as they complete.

SDK walkthrough

The following script covers the full end-to-end flow: an Agentic collection, an Agentic experiment, conversion of runner output, and upload with elluminate's evaluation queued. The script is idempotent — collections, criterion sets, and experiments are reused across runs, and uploads are skipped when an experiment already contains responses.

Running the example

Set your API key (created in the elluminate UI under Project → Keys) either as an environment variable or in a .env file next to the script; the example calls load_dotenv():

# option 1: shell
export ELLUMINATE_API_KEY=<your-key>
# optionally, if you run elluminate on a non-default host:
# export ELLUMINATE_BASE_URL=https://your-instance.example.com

# option 2: .env in elluminate_sdk/examples/
echo "ELLUMINATE_API_KEY=<your-key>" > elluminate_sdk/examples/.env

# run
uv run --directory elluminate_sdk python examples/example_harbor_agentic_upload.py

"""Harbor-based Agentic Evaluation: end-to-end upload example.

This example shows the full workflow for evaluating an agent that was run
externally with Harbor (or any other agent framework), and uploading the
results, including ATIF trajectories, to elluminate for inspection and
automatic per-criterion evaluation.

Workflow:

1. Create an AGENTIC collection whose rows are the agent's tasks. The task
   description lives in a single RAW_INPUT column; no prompt template is
   needed because AGENTIC experiments do not auto-generate responses.
2. Create a criterion set describing what counts as success.
3. Create an AGENTIC experiment (no auto-generation; results are uploaded).
4. Run the agent externally (Harbor CLI, LangChain, CrewAI, custom code).
5. Read Harbor's per-task output and convert it into `AgentTrialResult` objects.
6. Upload via `experiment.upload_agent_results(...)`.
7. The backend stores the trajectories and, when `evaluate=True` and
   trajectories are present, elluminate automatically rates each criterion
   against the trajectory.

The script is idempotent: collections, criterion sets, and experiments are
reused across runs; uploads are skipped when an experiment already contains
responses.

For a self-contained demo this script uses a small in-memory stand-in
(`HARBOR_RUN`) for the runner's output. In a real run you would replace it
with your own loader that reads your runner's per-task output from disk — for
Harbor, the run directory at `~/.harbor/runs/<run_name>/tasks/<task>/...`.
"""

from typing import Any

from dotenv import load_dotenv
from elluminate import AgentTrialResult, Client
from elluminate.schemas import CollectionColumn, ColumnTypeEnum
from elluminate.schemas.criterion import CriterionIn
from elluminate.schemas.experiments import Experiment

load_dotenv(override=True)

client = Client()  # (1)!
llm_config = client.get_llm_config(name="Claude Sonnet 4.6")

# Mock "Harbor output"; in a real integration this is read from disk.  # (2)!
# Each entry is what a Harbor run produces per task: a short task identifier,
# the instruction text, final messages, aggregate metrics, and an ATIF
# trajectory describing every step the agent took.
HARBOR_RUN: list[dict[str, Any]] = [
    {
        "task_name": "write-hello-world",
        "instruction": "Write a Python hello world script to hello.py",
        "reward": 1.0,
        "steps": 2,
        "cost_usd": 0.0042,
        "input_tokens": 420,
        "output_tokens": 61,
        "duration_seconds": 3.2,
        "messages": [
            {"role": "user", "content": "Write a Python hello world script to hello.py"},
            {"role": "assistant", "content": "Wrote hello.py: print('Hello, World!')"},
        ],
        "trajectory": {
            "schema_version": "ATIF-v1.0",
            "session_id": "harbor-run-001/write-hello-world",
            "agent": {
                "name": "harbor-demo-agent",
                "version": "0.1.0",
                "model_name": "claude-sonnet-4-6",
            },
            "steps": [
                {
                    "step_id": 1,
                    "source": "user",
                    "message": "Write a Python hello world script to hello.py",
                },
                {
                    "step_id": 2,
                    "source": "agent",
                    "message": "Writing hello.py.",
                    "tool_calls": [
                        {
                            "tool_call_id": "tc_1",
                            "function_name": "write_file",
                            "arguments": {"path": "hello.py", "content": "print('Hello, World!')"},
                        }
                    ],
                    "observation": {
                        "results": [{"source_call_id": "tc_1", "content": "wrote 22 bytes"}],
                    },
                    "metrics": {"prompt_tokens": 420, "completion_tokens": 61, "cost_usd": 0.0042},
                },
            ],
            "final_metrics": {
                "total_steps": 2,
                "total_cost_usd": 0.0042,
                "total_prompt_tokens": 420,
                "total_completion_tokens": 61,
            },
        },
    },
    {
        "task_name": "reverse-string-function",
        "instruction": "Create a Python function that reverses a string in reverse.py",
        "reward": 0.5,
        "steps": 2,
        "cost_usd": 0.0031,
        "input_tokens": 310,
        "output_tokens": 42,
        "duration_seconds": 2.1,
        "messages": [
            {"role": "user", "content": "Create a Python function that reverses a string in reverse.py"},
            {"role": "assistant", "content": "Wrote reverse.py with a one-line slice-based reverse."},
        ],
        "trajectory": {
            "schema_version": "ATIF-v1.0",
            "session_id": "harbor-run-001/reverse-string-function",
            "agent": {
                "name": "harbor-demo-agent",
                "version": "0.1.0",
                "model_name": "claude-sonnet-4-6",
            },
            "steps": [
                {
                    "step_id": 1,
                    "source": "user",
                    "message": "Create a Python function that reverses a string in reverse.py",
                },
                {
                    "step_id": 2,
                    "source": "agent",
                    "message": "Writing reverse.py.",
                    "tool_calls": [
                        {
                            "tool_call_id": "tc_1",
                            "function_name": "write_file",
                            "arguments": {
                                "path": "reverse.py",
                                "content": "def reverse(s: str) -> str:\n    return s[::-1]\n",
                            },
                        }
                    ],
                    "observation": {
                        "results": [{"source_call_id": "tc_1", "content": "wrote 42 bytes"}],
                    },
                    "metrics": {"prompt_tokens": 310, "completion_tokens": 42, "cost_usd": 0.0031},
                },
            ],
            "final_metrics": {
                "total_steps": 2,
                "total_cost_usd": 0.0031,
                "total_prompt_tokens": 310,
                "total_completion_tokens": 42,
            },
        },
    },
]


def harbor_to_agent_trial(task_output: dict[str, Any]) -> AgentTrialResult:  # (3)!
    """Map one Harbor per-task output dict to an `AgentTrialResult`.

    `task_name` on `AgentTrialResult` is what elluminate matches against the
    collection's `task_name_column` value, so here we set it to the full
    instruction text (which is also what the `task` column row holds).
    """
    return AgentTrialResult(
        task_name=task_output["instruction"],
        messages=task_output["messages"],
        reward=task_output["reward"],
        steps=task_output["steps"],
        cost_usd=task_output["cost_usd"],
        input_tokens=task_output["input_tokens"],
        output_tokens=task_output["output_tokens"],
        duration_seconds=task_output["duration_seconds"],
        trajectory=task_output["trajectory"],
        metadata={"run_name": "harbor-run-001", "task_id": task_output["task_name"]},
    )


# Step 1: AGENTIC collection with a single RAW_INPUT `task` column.  # (4)!
# No prompt template is required because AGENTIC experiments never
# auto-generate; responses are supplied by `upload_agent_results`.
collection, _ = client.get_or_create_collection(
    name="Harbor Demo Tasks",
    defaults={
        "collection_type": "AGENTIC",
        "columns": [CollectionColumn(name="task", column_type=ColumnTypeEnum.RAW_INPUT)],
        "variables": [{"task": h["instruction"]} for h in HARBOR_RUN],
    },
)

# Step 2: criterion set defining what success looks like for these tasks.  # (5)!
# Labels are explicit identifiers for each criterion.
criterion_set, _ = client.get_or_create_criterion_set(
    name="Harbor Demo Criteria",
    defaults={
        "criteria": [
            CriterionIn(
                criterion_str="Did the agent correctly complete the requested task?",
                label="task-complete",
            ),
            CriterionIn(
                criterion_str="Did the agent use tools appropriately?",
                label="uses-tools",
            ),
            CriterionIn(
                criterion_str="Is the agent's final output correct?",
                label="output-correct",
            ),
        ],
    },
)


def get_or_create_agentic_experiment(name: str, description: str) -> tuple[Experiment, bool]:  # (6)!
    """Return an AGENTIC experiment, creating it if missing.

    Also reports whether the experiment already has uploaded responses so the
    caller can skip a redundant upload on re-runs (avoids epoch conflicts).
    """
    try:
        experiment = client.get_experiment(name=name, fetch_responses=False)
        populated = experiment.results is not None and experiment.results.completed_epochs > 0
        return experiment, populated
    except ValueError:
        experiment = client.create_experiment(
            name=name,
            collection=collection,
            prompt_template=None,
            criterion_set=criterion_set,
            description=description,
            evaluation_mode="AGENTIC",
            llm_config=llm_config,
        )
        return experiment, False


# Step 3: AGENTIC experiment. No auto-generation; results come from Harbor.  # (7)!
experiment, experiment_populated = get_or_create_agentic_experiment(
    "Harbor Demo — Agent Run",
    "Harbor-run coding agent with ATIF trajectories.",
)
print(f"Experiment: {experiment.name} (id={experiment.id})")

# Step 4: Convert Harbor output to `AgentTrialResult` objects.  # (8)!
results = [harbor_to_agent_trial(task_output) for task_output in HARBOR_RUN]

# Step 5: upload with `evaluate=True`. elluminate rates every  # (9)!
# criterion against the trajectory and fills in per-criterion ratings.
if experiment_populated:
    print("Experiment already has responses; skipping upload.")
else:
    upload = experiment.upload_agent_results(
        results=results,
        task_name_column="task",
        evaluate=True,
    )
    print(
        f"Uploaded: {upload.created_responses} responses, "
        f"{upload.created_ratings} ratings, "
        f"{upload.pending_evaluations} pending trace evaluations"
    )
    if upload.errors:
        print(f"Errors: {upload.errors}")

# Step 6: Verify the trajectories are queryable from the SDK.  # (10)!
experiment.fetch_responses()
for resp in experiment.responses():
    task = resp.prompt.template_variables.input_values.get("task", "?")
    steps = len(resp.trajectory["steps"]) if resp.trajectory else 0
    print(f"  [{task[:50]}] trajectory_steps={steps}")

1. Initialize the SDK client (uses ELLUMINATE_API_KEY) and pick an LLMConfig to associate with the experiment (metadata only; Agentic experiments never invoke it).
2. Stand-in for your runner's on-disk output. Replace with code that reads Harbor's task_result.json + trajectory.json per task.
3. Translate one runner output into an AgentTrialResult. This is the only integration-specific code you need; task_name must match the value in the collection row's task column.
4. Create an Agentic collection with a single task column (RAW_INPUT), one row per task. No prompt template is required.
5. Create the criterion set that defines success. Labels are explicit identifiers for each criterion.
6. Idempotent helper that returns an Agentic experiment and whether it already holds uploaded responses.
7. Get-or-create the main experiment. evaluation_mode="AGENTIC" disables auto-generation; responses are supplied via upload.
8. Convert every runner output into an AgentTrialResult.
9. Upload with evaluate=True so elluminate rates every criterion against the trajectory. Skipped on re-runs when the experiment is already populated.
10. Re-fetch the experiment and confirm trajectories are queryable from the SDK.

AgentTrialResult fields

Each trial your runner produces maps to one AgentTrialResult. Required and optional fields:

Field	Required	Description
task_name	yes	Must exactly match a value in the collection column given as task_name_column.
messages	no	Final OpenAI-format message list (shown on the response page).
reward	no	Primary reward score (0.0–1.0).
steps	no	Number of agent steps / LLM calls.
cost_usd	no	Total USD cost for the trial.
duration_seconds	no	Wall-clock duration.
input_tokens	no	Aggregate input tokens.
output_tokens	no	Aggregate output tokens.
cached_tokens	no	Aggregate cached input tokens.
error	no	Error message if the trial failed.
metadata	no	Free-form dict surfaced on the response page.
trajectory	no	Raw ATIF trajectory (validated by the backend; see ATIF format).

ATIF trajectory format

Trajectories use the Agent Trajectory Interchange Format (ATIF), an open trajectory specification defined by Harbor.

A minimal ATIF v1 trajectory:

{
  "schema_version": "ATIF-v1.0",
  "session_id": "harbor-run-001/write-hello-world",
  "agent": {
    "name": "harbor-demo-agent",
    "version": "0.1.0",
    "model_name": "claude-sonnet-4-6"
  },
  "steps": [
    {
      "step_id": 1,
      "source": "user",
      "message": "Write a Python hello world script to hello.py"
    },
    {
      "step_id": 2,
      "source": "agent",
      "message": "Writing hello.py.",
      "tool_calls": [
        {
          "tool_call_id": "tc_1",
          "function_name": "write_file",
          "arguments": {"path": "hello.py", "content": "print('Hello, World!')"}
        }
      ],
      "observation": {
        "results": [{"source_call_id": "tc_1", "content": "wrote 22 bytes"}]
      },
      "metrics": {"prompt_tokens": 420, "completion_tokens": 61, "cost_usd": 0.0042}
    }
  ],
  "final_metrics": {
    "total_steps": 2,
    "total_cost_usd": 0.0042,
    "total_prompt_tokens": 420,
    "total_completion_tokens": 61
  }
}