Skip to content

Batch Operations

Batch operations allow you to process multiple items in a single API call rather than making separate calls for each item. In Elluminate, this means you can generate multiple responses or perform multiple ratings in parallel, significantly improving efficiency and reducing overhead.

For example, instead of making 10 separate API calls to generate responses for 10 different sets of template variables, you can send all 10 sets in a single batch request. The server processes these requests concurrently and returns all results together.

Batch operations are particularly useful when you need to:

  • Bulk Process Responses: Generate responses for multiple prompts efficiently
  • Batch Evaluate: Rate multiple responses against criteria in parallel
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
from dotenv import load_dotenv
from elluminate import Client

load_dotenv(override=True)

client = Client()

prompt_template, _ = client.prompt_templates.get_or_create(
    "Explain why {{language}}'s {{quirk}} is considered surprising or counterintuitive, "
    + "and demonstrate it with a short code example that might trip up even experienced developers.",
    name="Programming Quirks",
)

client.criteria.get_or_generate_many(prompt_template)

values = [
    {"language": "Python", "quirk": "mutable default arguments"},
    {"language": "JavaScript", "quirk": "type coercion"},
    {"language": "Java", "quirk": "string pool internment"},
]

template_variables = []
for value in values:
    variables = client.template_variables.add_to_collection(
        template_variables=value,
        collection=prompt_template.default_template_variables_collection,
    )
    template_variables.append(variables)

responses = client.responses.generate_many(
    prompt_template=prompt_template, template_variables=template_variables
)  # (1)!

# Generate ratings for the `responses`
batch_ratings = client.ratings.rate_many(responses)  # (2)!

for i, response in enumerate(responses):
    ratings = batch_ratings[i]
    print(f"\n\nResponse: {response.response}")
    for rating in ratings:
        print(f"Criteria: {rating.criterion.criterion_str}")
        print(f"Rating: {rating.rating}\n")

1. Efficiently generates multiple responses from a single prompt template with different template variables. Note we could also use `generate_many(prompt_template, collection=collection)` here.

2. Efficiently evaluates multiple responses against their respective criteria. Like the `rate` method, you can specify a rating mode or reference an experiment.

Performance Considerations

Batch operations offer several performance benefits:

  • Reduced network overhead
  • Parallel processing on the server
  • Lower total latency for multiple operations
  • More efficient resource utilization

However, be mindful of:

  • Memory usage with large batches
  • Timeout limits for long-running operations
  • API rate limits and quotas
  • Error handling complexity