Criterion Sets

Criterion sets are collections of evaluation criteria that can be applied to multiple prompt templates. They provide a way to organize and reuse criteria across your evaluation workflows, making it easier to maintain consistent evaluation standards.

Key Benefits

• Reusable Evaluation Standards: Create a set of criteria once and apply it to multiple prompt templates
• Consistent Assessment: Ensure all prompt templates are evaluated using the same quality standards
• Modular Organization: Group related criteria into logical sets for better management
• Efficient Workflow: Reduce duplication and streamline your evaluation process

Working with Criterion Sets

Criterion sets allow you to organize related criteria into logical groups. For instance, you might create sets like "Content Quality", "Technical Accuracy", or "Brand Voice Standards" that contain specific evaluation criteria relevant to each category. They can be applied to multiple prompt templates, enabling consistent evaluation across different contexts.

from dotenv import load_dotenv
from elluminate import Client

load_dotenv(override=True)

client = Client()

prompt_template, _ = client.prompt_templates.get_or_create(
    name="Product Review Template",
    user_prompt_template="Please review this product: {{product_name}}. {{product_description}}",
)

generated_criteria = client.criteria.generate_many(
    prompt_template=prompt_template,
    delete_existing=True,
)  # (1)!


criterion_set, _ = client.criterion_sets.get_or_create(
    name="Shipping-related Criteria",
    prompt_template=prompt_template,
)  # (2)!

client.criteria.add_many(
    [
        "Does the review mention that the shipping was less than $10?",
        "Are shipping timelines mentioned in the review?",
    ],
    criterion_set=criterion_set,
)

another_template, _ = client.prompt_templates.get_or_create(
    name="Customer Service Template",
    user_prompt_template="Please respond to this customer inquiry: {{inquiry}}",
)  # (3)!

client.criterion_sets.add_prompt_template(
    criterion_set=criterion_set,
    prompt_template=another_template,
)  # (4)!

1. Generate criteria for the prompt template using delete_existing=True to replace any existing criteria
2. Create a criterion set and associate it with the prompt template
3. Create another prompt template with different template variables
4. Link your existing criterion set to the new template, allowing the same criteria to be used for both templates

When to Use Criterion Sets

Criterion sets are particularly valuable when:

• Standardizing Evaluation: You want to ensure consistent evaluation across multiple prompt templates
• Organizing Criteria: You need to group related criteria into logical categories
• Scaling Evaluation: You're working with many templates that share similar evaluation needs
• Collaborative Workflows: Multiple team members need to apply the same criteria to different templates