Package dev.braintrust.eval

Interface Scorer<INPUT,OUTPUT>

Type Parameters:

INPUT - type of the input data

OUTPUT - type of the output data

All Known Implementing Classes:

ScorerBrainstoreImpl

public interface Scorer<INPUT,OUTPUT>

A scorer evaluates the result of a test case with a score between 0 (inclusive) and 1 (inclusive).

Method Summary

Modifier and Type	Method	Description
static <INPUT, OUTPUT> Scorer<INPUT,OUTPUT>	fetchFromBraintrust(BraintrustApiClient apiClient, String projectName, String scorerSlug, String version)	Fetch a scorer from Braintrust by project name and slug.
String	getName()
static <INPUT, OUTPUT> Scorer<INPUT,OUTPUT>	of(String scorerName, BiFunction<OUTPUT,OUTPUT,Double> scorerFn)
static <INPUT, OUTPUT> Scorer<INPUT,OUTPUT>	of(String scorerName, Function<TaskResult<INPUT,OUTPUT>,Double> scorerFn)
List<Score>	score(TaskResult<INPUT,OUTPUT> taskResult)

Method Details

getName

String getName()

score

List<Score> score(TaskResult<INPUT,OUTPUT> taskResult)

of

static <INPUT,
OUTPUT> Scorer<INPUT,OUTPUT> of(String scorerName,
 Function<TaskResult<INPUT,OUTPUT>,Double> scorerFn)

of

static <INPUT,
OUTPUT> Scorer<INPUT,OUTPUT> of(String scorerName,
 BiFunction<OUTPUT,OUTPUT,Double> scorerFn)

fetchFromBraintrust

static <INPUT,
OUTPUT> Scorer<INPUT,OUTPUT> fetchFromBraintrust(BraintrustApiClient apiClient,
 String projectName,
 String scorerSlug,
 @Nullable
 String version)

Fetch a scorer from Braintrust by project name and slug.

Parameters:

apiClient - the API client to use

projectName - the name of the project containing the scorer

scorerSlug - the unique slug identifier for the scorer

version - optional version of the scorer to fetch

Returns:

a Scorer that invokes the remote function

Throws:

RuntimeException - if the scorer is not found