ExperimentsResource

ExperimentsResource ¶

ExperimentsResource(client: Client)

Bases: BaseResource

create ¶

create(name: str, prompt_template: PromptTemplate, collection: TemplateVariablesCollection, llm_config: LLMConfig | None = None, description: str = '', generate: bool = False, rating_mode: RatingMode = DETAILED, n_epochs: int = 1) -> Experiment

Creates a new experiment.

Parameters:

Name	Type	Description	Default
`name`	`str`	The name of the experiment.	required
`prompt_template`	`PromptTemplate`	The prompt template to use for the experiment.	required
`collection`	`TemplateVariablesCollection`	The collection of template variables to use for the experiment.	required
`llm_config`	`LLMConfig \| None`	Optional LLMConfig to use for the experiment. Uses platform default if not specified.	`None`
`description`	`str`	Optional description for the experiment.	`''`
`generate`	`bool`	Whether to generate responses and ratings immediately. Defaults to False.	`False`
`rating_mode`	`RatingMode`	The rating mode to use if generating responses (Only used if generate=True). Defaults to RatingMode.DETAILED.	`DETAILED`
`n_epochs`	`int`	Number of times to run the experiment for each input. Defaults to 1.	`1`

Returns:

Name	Type	Description
`Experiment`	`Experiment`	The newly created experiment object. If generate=True,
	`Experiment`	responses and ratings will be generated. The returned experiment object will
	`Experiment`	then include a generation task ID that can be used to check the status of the
	`Experiment`	generation.

Raises:

Type	Description
`HTTPStatusError`	If the experiment with the same name already exists

create_and_run ¶

create_and_run(name: str, prompt_template: PromptTemplate, collection: TemplateVariablesCollection, llm_config: LLMConfig | None = None, description: str = '', rating_mode: RatingMode = DETAILED, timeout: float | None = None, n_epochs: int = 1) -> Experiment

Creates a new experiment and runs it (generates responses and ratings) in one go.

Parameters:

Name	Type	Description	Default
`name`	`str`	The name of the experiment.	required
`prompt_template`	`PromptTemplate`	The prompt template to use for the experiment.	required
`collection`	`TemplateVariablesCollection`	The collection of template variables to use for the experiment.	required
`llm_config`	`LLMConfig \| None`	Optional LLMConfig to use for the experiment.	`None`
`description`	`str`	Optional description for the experiment.	`''`
`rating_mode`	`RatingMode`	The rating mode to use for evaluating responses. Defaults to DETAILED.	`DETAILED`
`timeout`	`float \| None`	Optional timeout in seconds. Defaults to None (no timeout).	`None`
`n_epochs`	`int`	Number of times to run the experiment for each input. Defaults to 1.	`1`

Returns:

Name	Type	Description
`ExperimentOut`	`Experiment`	The experiment with generated responses and ratings.

Raises:

Type	Description
`RuntimeError`	If generation fails or returns no result.
`TimeoutError`	If the operation times out.

delete ¶

delete(experiment: Experiment) -> None

Deletes an experiment.

Parameters:

Name	Type	Description	Default
`experiment`	`Experiment`	The experiment to delete.	required

Raises:

Type	Description
`HTTPStatusError`	If the experiment doesn't exist or belongs to a different project.

get ¶

get(name: str) -> Experiment

Get the experiment with the given name.

Parameters:

Name	Type	Description	Default
`name`	`str`	The name of the experiment to get.	required

Returns:

Name	Type	Description
`Experiment`	`Experiment`	The experiment object.

Raises:

Type	Description
`ValueError`	If no experiment is found with the given name.

get_or_create ¶

get_or_create(name: str, prompt_template: PromptTemplate, collection: TemplateVariablesCollection, llm_config: LLMConfig | None = None, description: str = '', generate: bool = False, rating_mode: RatingMode = DETAILED, n_epochs: int = 1) -> tuple[Experiment, bool]

Gets an existing experiment by name or creates a new one if it doesn't exist.

The existence of an experiment is determined solely by its name. If an experiment with the given name exists, it will be returned regardless of its other properties. If no experiment exists with that name, a new one will be created with the provided parameters.

Parameters:

Name	Type	Description	Default
`name`	`str`	The name of the experiment to get or create.	required
`prompt_template`	`PromptTemplate`	The prompt template to use if creating a new experiment.	required
`collection`	`TemplateVariablesCollection`	The collection of template variables to use if creating a new experiment.	required
`llm_config`	`LLMConfig \| None`	Optional LLMConfig to use if creating a new experiment.	`None`
`description`	`str`	Optional description if creating a new experiment.	`''`
`generate`	`bool`	Whether to generate responses and ratings immediately. Defaults to False.	`False`
`rating_mode`	`RatingMode`	The rating mode to use if generating responses. Defaults to RatingMode.DETAILED.	`DETAILED`
`n_epochs`	`int`	Number of times to run the experiment for each input. Defaults to 1.	`1`

Returns:

Type	Description
`tuple[Experiment, bool]`	tuple[Experiment \| ExperimentGenerationStatus, bool]: A tuple containing: - The experiment object (either existing or newly created) - Boolean indicating if a new experiment was created (True) or existing one returned (False)

list ¶

list(prompt_template: PromptTemplate | None = None, collection: TemplateVariablesCollection | None = None, llm_config: LLMConfig | None = None) -> list[Experiment]

Get a list of experiments sorted by creation date.

Parameters:

Name	Type	Description	Default
`prompt_template`	`PromptTemplate \| None`	The prompt template to filter by.	`None`
`collection`	`TemplateVariablesCollection \| None`	The collection to filter by.	`None`
`llm_config`	`LLMConfig \| None`	The LLM config to filter by.	`None`

Returns:

Type	Description
`list[Experiment]`	list[Experiment]: A list of experiments.