ResponsesResource

ResponsesResource ¶

ResponsesResource(client: Union[Client, AsyncClient])

Bases: BaseResource

add ¶

add(response: str | List[ChatCompletionMessageParam], template_variables: TemplateVariables, experiment: Experiment, epoch: int = 1, metadata: LLMConfig | GenerationMetadata | None = None) -> PromptResponse

Add a response to an experiment.

Parameters:

Name	Type	Description	Default
`response`	`str \| List[ChatCompletionMessageParam]`	The response to add.	required
`template_variables`	`TemplateVariables`	The template variables to use for the response.	required
`experiment`	`Experiment`	The experiment this response belongs to.	required
`epoch`	`int`	The epoch for the response within the experiment. Defaults to 1.	`1`
`metadata`	`LLMConfig \| GenerationMetadata \| None`	Optional metadata to associate with the response.	`None`

Returns:

Name	Type	Description
`PromptResponse`	`PromptResponse`	The newly created prompt response object.

add_many ¶

add_many(responses: List[str | List[ChatCompletionMessageParam]], template_variables: List[TemplateVariables], experiment: Experiment, epoch: int = 1, metadata: List[LLMConfig | GenerationMetadata | None] | None = None, timeout: float | None = None, polling_interval: float = 3.0) -> List[PromptResponse]

Add multiple responses to an experiment in bulk.

Use this method when you have a list of responses to add, instead of adding them one by one with the add() method.

Parameters:

Name	Type	Description	Default
`responses`	`list[str \| List[ChatCompletionMessageParam]]`	List of responses to add.	required
`template_variables`	`list[TemplateVariables]`	List of template variables for each response.	required
`experiment`	`Experiment`	The experiment these responses belong to.	required
`epoch`	`int`	The epoch for the responses within the experiment. Defaults to 1.	`1`
`metadata`	`list[LLMConfig \| GenerationMetadata \| None] \| None`	Optional list of metadata for each response.	`None`
`timeout`	`float \| None`	Timeout in seconds for API requests. Defaults to no timeout.	`None`
`polling_interval`	`float`	Time between status checks in seconds. Defaults to 3.0.	`3.0`

Returns:

Type	Description
`List[PromptResponse]`	list[PromptResponse]: List of newly created prompt response objects.

aupdate_annotation `async` ¶

aupdate_annotation(prompt_response: PromptResponse | int, annotation: str) -> PromptResponse

Update the annotation for a prompt response (async).

delete ¶

delete(prompt_response: PromptResponse) -> None

Delete a prompt response.

Parameters:

Name	Type	Description	Default
`prompt_response`	`PromptResponse`	The prompt response to delete.	required

generate ¶

generate(template_variables: TemplateVariables, experiment: Experiment, llm_config: LLMConfig | None = None) -> PromptResponse

Generate a response using an LLM.

This method sends the prompt to an LLM for generation. If no LLM config is provided, the project's default LLM config will be used.

Parameters:

Name	Type	Description	Default
`template_variables`	`TemplateVariables`	The template variables to use for the response.	required
`experiment`	`Experiment`	The experiment this response belongs to.	required
`llm_config`	`LLMConfig \| None`	Optional LLM configuration to use for generation. If not provided, the project's default config will be used.	`None`

Returns:

Name	Type	Description
`PromptResponse`	`PromptResponse`	The generated response object

generate_many ¶

generate_many(experiment: Experiment, *, template_variables: List[TemplateVariables], llm_config: LLMConfig | None = None, timeout: float | None = None) -> List[PromptResponse]

generate_many(experiment: Experiment, *, collection: TemplateVariablesCollection, llm_config: LLMConfig | None = None, timeout: float | None = None) -> List[PromptResponse]

generate_many(experiment: Experiment, *, template_variables: List[TemplateVariables] | None = None, collection: TemplateVariablesCollection | None = None, llm_config: LLMConfig | None = None, timeout: float | None = None, polling_interval: float = 3.0) -> List[PromptResponse]

Generate multiple responses for an experiment.

Use this method when you have a list of responses to generate, instead of generating them one by one with the generate() method.

Either template_variables or collection can be provided: - If template_variables is given, it will use the provided list of template variables for each response. - If collection is given, it will use the template variables from the specified collection.

Parameters:

Name	Type	Description	Default
`experiment`	`Experiment`	The experiment these responses belong to.	required
`template_variables`	`list[TemplateVariables] \| None`	List of template variables for each response.	`None`
`collection`	`TemplateVariablesCollection \| None`	The collection to use for the template variables.	`None`
`llm_config`	`LLMConfig \| None`	Optional LLMConfig to use for generation.	`None`
`timeout`	`float`	Timeout in seconds for API requests. Defaults to no timeout.	`None`
`polling_interval`	`float`	Time between status checks in seconds. Defaults to 3.0.	`3.0`

Returns:

Type	Description
`List[PromptResponse]`	list[PromptResponse]: List of newly created prompt response objects.

get_stats ¶

get_stats(llm_config: LLMConfig | None = None, days: int = 30) -> ResponsesStats

Get usage statistics for responses in a project with optional LLM config filtering.

Parameters:

Name	Type	Description	Default
`llm_config`	`LLMConfig \| None`	The LLM config to get stats of. If not provided, the project's default LLM config will be used.	`None`
`days`	`int`	The number of days to get stats for. Defaults to 30. Must be between 1 and 90.	`30`

Returns:

Name	Type	Description
`ResponsesStats`	`ResponsesStats`	The stats of the LLM config.

list ¶

list(prompt_template: PromptTemplate | None = None, template_variables: TemplateVariables | None = None, experiment: Experiment | None = None, collection: TemplateVariablesCollection | None = None, filters: PromptResponseFilter | None = None) -> list[PromptResponse]

Returns the responses belonging to a prompt template, template variables, experiment, or collection.

Parameters:

Name	Type	Description	Default
`prompt_template`	`PromptTemplate \| None`	The prompt template to get responses for.	`None`
`template_variables`	`TemplateVariables \| None`	The template variables to get responses for.	`None`
`experiment`	`Experiment \| None`	The experiment to get responses for.	`None`
`collection`	`TemplateVariablesCollection \| None`	The collection to get responses for.	`None`
`filters`	`PromptResponseFilter \| None`	The filters to apply to the responses.	`None`

Returns:

Type	Description
`list[PromptResponse]`	list[PromptResponse]: The list of prompt responses.

list_comparison_samples ¶

list_comparison_samples(experiment_a: Experiment, experiment_b: Experiment, filter_samples_by: Literal['improved', 'regressed'] | None = None, criterion_ids: list[int] | None = None, categorical_column_names: list[str] | None = None, categorical_column_values: list[str] | None = None) -> List[ResponsesSample]

List comparison samples between two experiments.

This returns representative response samples from both experiments grouped by template variables, enabling side-by-side comparison of results.

Parameters:

Name	Type	Description	Default
`experiment_a`	`Experiment`	The first experiment (baseline).	required
`experiment_b`	`Experiment`	The second experiment to compare against baseline.	required
`filter_samples_by`	`str \| None`	Filter samples by performance difference. - "improved": Only show samples where B performs better than A - "regressed": Only show samples where B performs worse than A	`None`
`criterion_ids`	`list[int] \| None`	Optional list of criterion IDs to consider when computing scores. If not provided, all criteria are used.	`None`
`categorical_column_names`	`list[str] \| None`	Names of categorical columns to filter by.	`None`
`categorical_column_values`	`list[str] \| None`	Values to match for the specified columns.	`None`

Returns:

Type	Description
`List[ResponsesSample]`	List[ResponsesSample]: Samples from both experiments for comparison.

list_samples ¶

list_samples(experiment: Experiment, exclude_perfect_responses: bool = False, show_only_annotated_responses: bool = False, filters: ResponsesSampleFilter | None = None, sort_by: ResponsesSampleSortBy | None = None) -> List[ResponsesSample]

List samples for an experiment.

Parameters:

Name	Type	Description	Default
`experiment`	`Experiment`	The experiment to get samples for.	required
`exclude_perfect_responses`	`bool`	Whether to exclude perfect responses.	`False`
`show_only_annotated_responses`	`bool`	Whether to show only annotated responses.	`False`
`filters`	`ResponsesSampleFilter \| None`	The filters to apply to the samples.	`None`
`sort_by`	`ResponsesSampleSortBy \| None`	The sort order for the samples.	`None`

Returns:

Type	Description
`List[ResponsesSample]`	List[ResponsesSample]: The list of samples.

update_annotation ¶

update_annotation(prompt_response: PromptResponse | int, annotation: str) -> PromptResponse

Update the annotation of a prompt response.

Annotations are useful for categorizing, labeling, or adding notes to responses, especially when reviewing failed responses or building golden answer sets.

Parameters:

Name	Type	Description	Default
`prompt_response`	`PromptResponse \| int`	The prompt response object or its ID	required
`annotation`	`str`	The annotation text to set (empty string to clear)	required

Returns:

Name	Type	Description
`PromptResponse`	`PromptResponse`	The updated prompt response

Example

# Annotate a failed response
response = client.responses.update_annotation(
    response_id=123,
    annotation="Failed due to incorrect entity extraction"
)

# Clear an annotation
response = client.responses.update_annotation(response, "")

ResponsesResource