Skip to content

LLM Configurations

Elluminate allows you to customize the behavior of language models through LLM configurations. This is particularly useful when you need:

  • Different Configurations: Control configuration parameters like temperature, top_p, max_tokens, etc.
  • Multiple Model Variants: Test different models or model versions
  • Custom API Endpoints: Use different API endpoints or providers

The following example demonstrates how to use different LLM configurations:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
import os

from dotenv import load_dotenv
from elluminate import Client
from elluminate.schemas import RatingMode

load_dotenv(override=True)

# Initialize the client
client = Client()

# Create a low temperature LLM config for more deterministic responses
low_temp_config, _ = client.llm_configs.get_or_create(
    name="Low Temperature GPT-4o mini",
    llm_model_name="gpt-4o-mini",
    temperature=0.1,
    api_key=os.getenv("AZURE_OPENAI_API_KEY"),
    llm_base_url=os.getenv("AZURE_OPENAI_ENDPOINT"),
    api_version=os.getenv("OPENAI_API_VERSION"),
)  # (1)!

# Create a prompt template
prompt_template, _ = client.prompt_templates.get_or_create(
    "Create a {{length}} story about a programmer solving the {{problem}} problem.",
    name="Programming Stories",
)

# Generate evaluation criteria
client.criteria.generate_many(prompt_template, delete_existing=True)

# Create a collection for our variables
collection, _ = client.collections.get_or_create(
    name="Programming Stories Variables", description="Template variables for programming story generation"
)

template_variables = client.template_variables.add_to_collection(
    template_variables={"length": "short", "problem": "fizzbuzz"},
    collection=collection,
)

# Create experiments for tracking responses and ratings
default_experiment, _ = client.experiments.get_or_create(
    "Programming Stories - Default Config",
    prompt_template=prompt_template,
    collection=collection,
    description="Evaluating programming stories with default LLM configuration",
)

low_temp_experiment, _ = client.experiments.get_or_create(
    "Programming Stories - Low Temperature",
    prompt_template=prompt_template,
    collection=collection,
    llm_config=low_temp_config,
    description="Evaluating programming stories with low temperature configuration",
)

# Generate response using default configuration
response = client.responses.generate(
    prompt_template,
    template_variables=template_variables,
    experiment=default_experiment,
)

# Generate response using the low temperature configuration
low_temp_response = client.responses.generate(
    prompt_template,
    template_variables=template_variables,
    experiment=low_temp_experiment,
    llm_config=low_temp_config,
)  # (2)!

# Rate responses
print("\nRating default response:")
ratings = client.ratings.rate(response, rating_mode=RatingMode.FAST)
print("Response: ", response.messages[-1].content, "\n")
for rating in ratings:
    print(f"Criterion: {rating.criterion.criterion_str}")
    print(f"Rating: {rating.rating}")

print("\nRating low temperature response:")
ratings = client.ratings.rate(low_temp_response, rating_mode=RatingMode.FAST)
print("Response: ", low_temp_response.messages[-1].content, "\n")
for rating in ratings:
    print(f"Criterion: {rating.criterion.criterion_str}")
    print(f"Rating: {rating.rating}")

1. Creating an LLM configuration with specific parameters allows you to customize the model's behavior. You can pass any credentials for a LLM provider like Azure OpenAI, OpenAI, Anthropic, etc.

2. When generating responses, you can pass the LLM configuration as an optional parameter to use these custom settings instead of the default configuration.

The following are the key configuration parameters:

  • name: A unique identifier for your configuration
  • llm_model_name: The specific model to use (e.g., "gpt-4", "gpt-3.5-turbo")
  • api_key: Authentication key for the LLM provider
  • llm_base_url: API endpoint URL
  • api_version: API version to use

Usage in the Frontend

In LLM Configurations tag you can view, search and manage your LLM configurations.

LLM Configurations

You can also create a new LLM configuration by clicking the New Configuration button. Here you can select the LLM provider and enter the required credentials.

LLM Configurations

Settings can be adjusted both during creation and afterwards for the configurations.

LLM Configurations