Async Operations

Optimize performance with asynchronous operations for large-scale prompt evaluations and batch processing

elluminate supports asynchronous operations for improved performance when working with multiple prompts or evaluations. This is particularly useful when:

• Batch Processing - Handling multiple prompts or responses simultaneously
• Performance Optimization - Reducing overall execution time for large-scale evaluations
• Resource Efficiency - Better utilization of system resources during I/O operations

The following example demonstrates how to use async operations with collections in elluminate:

import asyncio

from dotenv import load_dotenv
from elluminate import Client
from elluminate.schemas import RatingMode

load_dotenv(override=True)


async def run_eval() -> None:
    client = Client()

    prompt_template, _ = await client.prompt_templates.aget_or_create(
        "Please write a short summary of the life of {{person}} from {{state}}.",
        name="Summary of Person",
    )

    values = [
        {"person": "Paul Revere", "state": "Massachusetts"},
        {"person": "George Washington", "state": "Virginia"},
    ]
    collection, _ = await client.collections.aget_or_create(
        name="Founding Fathers",
        description="A collection of Founding Fathers",
        variables=values,
    )  # (1)!

    await client.criteria.aget_or_generate_many(prompt_template)

    # Create an experiment for tracking responses and ratings
    experiment, _ = await client.experiments.aget_or_create(
        "Founding Fathers Analysis",
        prompt_template=prompt_template,
        collection=collection,
        description="Evaluating summaries of Founding Fathers",
    )

    responses = await client.responses.agenerate_many(experiment=experiment, collection=collection)
    criteria_ratings = await client.ratings.arate_many(responses, rating_mode=RatingMode.FAST)

    for ratings in criteria_ratings:
        for rating in ratings:
            print(f"Criteria: {rating.criterion.criterion_str}")
            print(f"Rating: {rating.rating}\n")


asyncio.run(run_eval())

1. Async methods in elluminate are prefixed with a. For example, aget_or_create_collection instead of get_or_create_collection.

2. Using asyncio.gather allows for concurrent execution of multiple async operations, significantly improving performance when adding multiple entries.

Most elluminate SDK operations have async counterparts, making it easy to convert existing synchronous code to async. Simply:

1. Prefix method calls with a (e.g., arate instead of rate)
2. Use await with async method calls
3. Wrap your code in an async function
4. Use asyncio.run() to execute the async function

When working with multiple independent async operations, consider using asyncio.gather() to run them concurrently. Although this example runs tasks sequentially for clarity, you can parallelize multiple async calls for improved performance.