CriteriaResource

CriteriaResource ¶

CriteriaResource(client: Client)

Bases: BaseResource

add_many ¶

add_many(criteria: List[CriterionIn], prompt_template: PromptTemplate, *, delete_existing: bool = False) -> List[Criterion]

add_many(criteria: List[CriterionIn], *, criterion_set: CriterionSet, delete_existing: bool = False) -> List[Criterion]

add_many(criteria: List[CriterionIn], prompt_template: PromptTemplate | None = None, criterion_set: CriterionSet | None = None, delete_existing: bool = False) -> List[Criterion]

Adds custom evaluation criteria to a prompt template or criterion set.

If criteria with the same strings already exist, they will be reused rather than duplicated.

There are two ways to use this method: 1. With a prompt template: add criteria to the default criterion set associated with the template 2. With a criterion set: add criteria directly to the specified criterion set

Parameters:

Name	Type	Description	Default
`criteria`	`list[CriterionIn]`	List of criterion objects to add.	required
`prompt_template`	`PromptTemplate`	The prompt template to add criteria to.	`None`
`criterion_set`	`CriterionSet`	A CriterionSet object to add criteria to directly.	`None`
`delete_existing`	`bool`	If True, deletes any existing criteria before adding new ones. Defaults to False.	`False`

Returns:

Type	Description
`List[Criterion]`	list[Criterion]: List of created and/or existing criterion objects.

Raises:

Type	Description
`ValueError`	If both prompt_template and criterion_set are provided, or if neither is provided.
`HTTPStatusError`	If the provided objects don't belong to the project or other API errors occur.

delete ¶

delete(criterion: Criterion) -> None

Delete a criterion.

Parameters:

Name	Type	Description	Default
`criterion`	`Criterion`	The criterion to delete.	required

generate_many ¶

generate_many(prompt_template: PromptTemplate, delete_existing: bool = False) -> List[Criterion]

Automatically generates evaluation criteria for the prompt template using an LLM.

This method uses the project's default LLM to analyze the prompt template and generate appropriate evaluation criteria. The criteria will be added to a criterion set which is associated with the prompt template.

Note: Unlike add_many, this method only works with prompt templates and not directly with criterion sets, as generation requires a prompt template to analyze.

Parameters:

Name	Type	Description	Default
`prompt_template`	`PromptTemplate`	The prompt template to generate criteria for.	required
`delete_existing`	`bool`	If True, deletes any existing criteria before generating new ones. If False and criteria exist, raises an error. Defaults to False.	`False`

Returns:

Type	Description
`List[Criterion]`	list[Criterion]: List of generated criterion objects. Each criterion includes the generation metadata from the LLM.

Raises:

Type	Description
`HTTPStatusError`	If criteria already exist and delete_existing is False, if the template variables are not found in the project

get_or_generate_many ¶

get_or_generate_many(prompt_template: PromptTemplate) -> Tuple[List[Criterion], bool]

Gets existing criteria or generates new ones if none exist.

This method generates new criteria if none exist, otherwise it returns the existing criteria. The criteria are associated with the prompt template via a criterion set.

Parameters:

Name	Type	Description	Default
`prompt_template`	`PromptTemplate`	The prompt template to get or generate criteria for.	required

Returns:

Type	Description
`Tuple[List[Criterion], bool]`	tuple[list[Criterion], bool]: A tuple containing: - List of criterion objects, either existing or newly generated - Boolean indicating if criteria were generated (True) or existing ones returned (False)

Raises:

Type	Description
`HTTPStatusError`	If criteria already exist and delete_existing is False, if the template variables are not found in the project

list ¶

list(prompt_template: PromptTemplate) -> List[Criterion]

Get the evaluation criteria for a prompt template.

This method retrieves all criteria associated with the prompt template via criterion sets.

Parameters:

Name	Type	Description	Default
`prompt_template`	`PromptTemplate`	The prompt template to get criteria for.	required

Returns:

Type	Description
`List[Criterion]`	list[Criterion]: List of criterion objects, ordered by creation date.