Skip to content

Batch Operations

Maximize efficiency by processing multiple responses and evaluations in parallel with batch operations

Batch operations allow you to process multiple items in a single API call rather than making separate calls for each item. In elluminate, this means you can generate multiple responses or perform multiple ratings in parallel, significantly improving efficiency and reducing overhead.

For example, instead of making 10 separate API calls to generate responses for 10 different sets of template variables, you can send all 10 sets in a single batch request. The server processes these requests concurrently and returns all results together.

Batch operations are particularly useful when you need to:

  • Bulk Process Responses - Generate responses for multiple prompts efficiently
  • Batch Evaluate - Rate multiple responses against criteria in parallel
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
"""v1.0 API version of example_sdk_usage_batch.py

Key changes from v0.x:
- Uses collection.add_many() for batch variable addition (single call)
- Uses client.run_experiment() which handles batch generation and rating
- No need for manual generate_many() + rate_many() calls

The v1.0 API simplifies batch operations significantly because:
1. collection.add_many() replaces the loop with add_to_collection()
2. run_experiment() handles all responses and ratings automatically
"""

from dotenv import load_dotenv
from elluminate import Client

load_dotenv(override=True)

client = Client()

# v1.0: get_or_create_prompt_template - messages is part of lookup
template, _ = client.get_or_create_prompt_template(
    name="Programming Quirks",
    messages="Explain why {{language}}'s {{quirk}} is considered surprising or counterintuitive, "
    "and demonstrate it with a short code example that might trip up even experienced developers.",
)

# v1.0: Rich model method - auto-generate criteria from template if none exist
template.get_or_generate_criteria()

# v1.0: get_or_create_collection
collection, _ = client.get_or_create_collection(
    name="Programming Quirks Variables", defaults={"description": "Decide on defaults dict vs regular params"}
)

# v1.0: Batch add with collection.add_many()
# OLD: Loop with client.template_variables.add_to_collection() for each item
# NEW: Single call with list of variable dicts
values = [
    {"language": "Python", "quirk": "mutable default arguments"},
    {"language": "JavaScript", "quirk": "type coercion"},
]
collection.add_many(variables=values)

# v1.0: run_experiment() handles ALL batch operations internally:
# - Creates the experiment
# - Generates responses for ALL collection items (batch)
# - Rates ALL responses (batch)
# OLD: Required separate generate_many() + rate_many() calls
experiment = client.run_experiment(
    name="Programming Quirks Analysis",
    prompt_template=template,
    collection=collection,
    description="Evaluating explanations of programming language quirks",
)

# Results are already populated - no additional API calls needed
for response in experiment.responses():
    print(f"\n\nResponse: {response.response_str[:200]}...")
    for rating in response.ratings:
        print(f"  Criteria: {rating.criterion.criterion_str}")
        print(f"  Rating: {rating.rating}")

  1. Create an experiment to track the batch responses and ratings. This experiment will be used to associate all generated responses with this evaluation run.

  2. Efficiently generates multiple responses from a single prompt template with different template variables. The experiment parameter links all generated responses to the experiment.

  3. Efficiently rate multiple responses against their respective criteria. The ratings are automatically collected in the experiment because the responses were linked to it during generation. The rating mode determines the type of rating strategy to employ.

Performance Considerations

Batch operations offer several performance benefits:

  • Reduced network overhead
  • Parallel processing on the server
  • Lower total latency for multiple operations
  • More efficient resource utilization

However, be mindful of:

  • Memory usage with large batches
  • Timeout limits for long-running operations
  • API rate limits and quotas
  • Error handling complexity