Experiments

Learn to systematically test and compare your LLM applications

Experiments help you systematically compare different prompts, evaluate their effectiveness, and track performance over time. They are the foundation of data-driven LLM development, allowing you to test multiple variations and analyze which performs best for your specific use case.

What is an Experiment?

An experiment is one evaluation run of a given evaluation type. Every experiment combines a set of test cases, a source of responses and a Criterion Set, and rates each response against the criteria. What the test cases and responses are depends on the type:

Evaluation type	Test cases	Responses	Setup details
Single-turn	Single-turn collection	Generated from a prompt template and an LLM config, or imported	This page
Persona	Scenario set	Conversations between the personas of a persona group and your bot (an LLM config)	Running a Persona Experiment
Agent	Agent collection	Traces of your agent, uploaded or imported	Agent experiment setup

The walkthrough below uses a Single-turn experiment. Within Single-turn, a Generate & Evaluate run uses a collection, a prompt template, an LLM config and a criterion set, while Evaluate Imported Responses skips the prompt template and model entirely and scores responses you already have. See Choose an experiment type for the setup steps of every type.

Creating Your First Experiment

Step 1: Navigate to Experiments

Click Experiments in the sidebar. You'll see your experiment runs list with options to create new experiments or view the timeline. The Evaluation type filter narrows the list to Single-turn, Persona, or Agent experiments; it follows the Evaluation switcher, and All types shows everything.

Step 2: Choose an experiment type

New Experiment sits in the Evaluation panel of the sidebar and starts an experiment of the type selected in the Evaluation switcher (see Navigation). With Single-turn selected, elluminate first asks which kind of experiment you want to create; Persona and Agent go straight to their setup. The setup is type-aware — it reveals only the steps relevant to the type.

Experiment type	What it does	Setup steps
Generate & Evaluate	Run a prompt template through an LLM against your collection. Variables are filled in for each row, the model is called, and responses are evaluated against your criteria.	Collection → Prompt template → Model → Criteria
Evaluate Imported Responses	Evaluate externally generated responses already stored in your collection. Your criteria run directly against them — no LLM inference.	Collection → Response column → Criteria
Persona	Simulate multi-turn conversations between LLM-driven personas and your chatbot. Every persona in the group runs against every scenario in a scenario set, and each resulting dialogue is evaluated.	Choose scenarios → Choose a persona group → Choose a model → Choose evaluation criteria
Agent	Evaluate the full trajectories of an agent you run yourself across the tasks in your Agent collection, including its tool calls, reasoning steps, and final output. elluminate does not run the agent; see Agent Evaluations.	Data source → Choose a collection → Choose evaluation criteria

In Single-turn, the chosen type collapses to a summary row; use Change type to switch at any point.

Type-specific collections

Persona experiments run on scenarios instead of a collection, and Agent experiments use Agent collections, while Generate & Evaluate and Evaluate Imported Responses use Single-turn collections.

Step 3: Configure the setup flow

elluminate guides you through a step accordion — one step at a time, each unlocking the next. Completed steps collapse to a summary you can reopen to change a selection. For a Generate & Evaluate experiment the steps are:

1. Choose a collection - Your test inputs dataset.
2. Choose a prompt template - The versioned template with {{variables}}. Optional when the collection already contains full prompts (a Conversation or Raw Input column).
3. Choose a model - The LLM config used to generate responses.
4. Choose evaluation criteria - The criterion set responses are evaluated against.

Below the accordion you set the Experiment Name and an optional Description.

Compatibility rules

The system will only offer mutually compatible Collections, Prompt Templates and Criterion Sets. If your desired selection does not appear, check for issues on your placeholders and column naming.

Advanced Settings

• Rating Mode - Detailed mode includes reasoning for each criterion and is recommended for better interpretability. Fast mode provides only ratings without explanations and may be slightly faster.
• Epochs - Epochs allow you to run the same evaluation multiple times. This makes the statistics more reliable but can take some time.
• Rating Version - Select the rating model version to use for evaluating responses. Defaults to the version set in Settings → Project Settings.

Step 4: Run the Experiment

After clicking "Create Experiment", elluminate will:

1. Generate Responses - Create LLM responses for each test input (skipped for Evaluate Imported Responses, which uses the responses already in your collection)
2. Rate Responses - Evaluate each response against your criteria
3. Calculate Results - Compute overall scores and criterion performance

You can monitor progress in real-time from the experiment detail page.

Evaluating Imported Responses

When you already have model outputs — from a production system, another tool, or a previous run — you can evaluate them directly without regenerating anything. Pick the Evaluate Imported Responses type and follow its three-step flow:

1. Choose a collection - Only collections with at least one text column are shown.
2. Choose a response column - The text column that holds the outputs you want to evaluate.
3. Choose evaluation criteria - The criterion set applied to those responses.

elluminate runs your criteria directly against the stored responses — no prompt template, model, or LLM inference is involved. This makes it ideal for benchmarking external systems or scoring responses collected elsewhere.

Understanding Experiment Results

Once your experiment completes, you can analyze results using two main views:

Detailed Analysis Tab

This tab provides aggregate insights and sample-by-sample inspection:

• Overall Performance - Success rates, score distributions, token usage
• Criterion Breakdown - Which criteria pass/fail most frequently
• Sample Navigator - Browse individual responses with full rating details
• Performance Charts - Visual analysis of score patterns and distributions

Individual Responses Tab

This tab offers a tabular view of all results:

• Response Table - Sortable list of all prompt-response pairs
• Rating Details - Pass/fail status for each criterion per response
• Export Options - Download results as CSV with full data
• Filtering - Focus on specific score ranges or criterion results

For detailed analysis techniques, see the Response Analysis Guide.

Filtering by Categorical Columns

When your collection includes categorical columns, you can filter experiment responses to analyze performance across different input categories.

Setup: Configure columns as Category type in your collection. See Column Types for details.

Usage:

• Click the filter icon in the experiment header
• Select categorical columns and values to include
• Multiple values within a column use OR logic (show responses matching any value)
• Multiple columns use AND logic (show responses matching all conditions)
• Metrics and charts update to reflect only filtered responses

This enables targeted analysis - for example, filtering by user_type being Enterprise or SMB to compare performance across customer segments, or region being US or EU to analyze regional effectiveness.

Comparing Experiments

elluminate supports comparing experiments to identify improvements and track performance changes.

Two-Experiment Comparison

Compare two experiments side-by-side to understand differences:

1. Select two experiments from your list
2. Click "Compare" to access detailed comparison view
3. Analyze side-by-side responses, criterion performance deltas, and overall score changes

Multi-Experiment Comparison

For high-level comparison across multiple experiments:

1. Select 3+ experiments from your list
2. Use multi-comparison view for aggregate performance trends
3. Identify which experiments perform best across different criteria

Monitoring Performance Over Time

Timeline View

The Timeline provides historical performance tracking:

• Performance Trends - See how your experiments perform over time
• Filtering Options - Filter by date range, prompt template, collection, or model
• Data Point Details - Hover for experiment information, click to view details

Scheduled Experiments

It is possible to automate regular evaluations with scheduled experiments.

Creating Schedules:

1. From Timeline view, click "Schedule Run"
2. Configure frequency (daily, weekly, monthly)
3. Set experiment parameters (template, collection, model)
4. Configure notifications and thresholds

Schedule Management:

To manage your scheduled experiments, navigate to the Timeline view and scroll to the bottom. There you can:

• View All Schedules - Monitor active schedules and their settings using the filter function
• Edit Schedules - Modify frequency, thresholds, or disable schedules
• Delete Schedules - Remove scheduled experiments without affecting the previous runs

Advanced Features

Retrying Failed Responses

When a run finishes with some responses errored — a provider timeout, a rate limit, a content filter, malformed output — Retry failed on the experiment regenerates only those items. The responses that succeeded keep their ratings and are not paid for again. The button appears on the error banner and shows how many responses will be retried. Prompts the run never reached at all, which is what a cancelled or interrupted run leaves behind, are retried along with them.

A retry deletes the failed responses, so any annotation or manual rating on one of them goes with it. When there are any, the retry asks first and only proceeds once you confirm.

Two limits are worth knowing:

• A multi-epoch experiment can only be retried when its failures cover the same leading epochs for every affected prompt. Failures scattered unevenly across epochs are refused; re-run the experiment instead.
• A retry cannot start while the experiment is still generating. Wait for the run to finish.

From the SDK:

experiment.retry_failed()

# The failed responses carry annotations or manual ratings you are willing to lose
experiment.retry_failed(discard_annotations=True)

Structured Outputs and Tool Calling

Experiments work seamlessly with advanced LLM features:

• Structured Outputs - Evaluate JSON responses with schema validation criteria
• Tool Calling - Rate tool usage and parameter correctness in agentic applications
• Multi-Step Workflows - Evaluate complex interaction patterns

For details, see Structured Outputs and Tool Calling.

Integration with Other Components

Experiments leverage your entire elluminate setup:

• Collections - Reuse test datasets across different experiments
• Prompt Templates - Version control ensures experiment reproducibility
• Criterion Sets - Apply consistent evaluation standards
• LLM Configs - Test different models and parameters systematically

Best Practices

Experiment Design

• Clear Objectives - Define what you're testing before running experiments
• Representative Data - Ensure your collection covers real-world scenarios
• Appropriate Criteria - Choose evaluation criteria that match your use case
• Controlled Variables - Change one thing at a time for clear insights

Performance Optimization

• Batch Testing - Run multiple variations together for faster iteration
• Strategic Sampling - Use smaller collections for rapid prototyping
• Cost Management - Use Fast rating mode when detailed explanations aren't needed
• Historical Context - Compare against previous experiments to track improvement

Quality Assurance

• Manual Review - Spot-check automated ratings for accuracy
• Edge Case Testing - Include challenging scenarios in your collections
• Consistent Evaluation - Use the same criterion sets for comparable results
• Documentation - Record experiment goals and findings in descriptions

SDK Integration

For programmatic experiment creation and management, you can use the elluminate SDK:

from elluminate import Client
from elluminate.schemas import RatingMode

client = Client()  # Uses ELLUMINATE_API_KEY env var

For complete SDK documentation, see the API Reference.

Troubleshooting

Common Issues

• Generation Failures - Check LLM config settings and rate limits
• Rating Errors - Verify criterion set compatibility with response format
• Performance Issues - Consider using Fast rating mode for large collections
• Missing Results - Ensure experiment completed successfully before analysis

Getting Help

When experiments don't behave as expected:

• Check Logs - Review experiment logs for specific error messages
• Validate Components - Test prompt templates and collections independently
• Review Settings - Confirm LLM config parameters are appropriate
• Contact Support - Reach out with experiment IDs for detailed assistance