LLM Evaluators

Use LLM-backed evaluators in @arizeai/phoenix-evals

The llm entrypoint provides reusable evaluator factories that call an AI SDK model and return structured evaluation results.

Relevant Source Files

• src/llm/index.ts

Example

import { openai } from "@ai-sdk/openai";
import { createFaithfulnessEvaluator } from "@arizeai/phoenix-evals";

const faithfulness = createFaithfulnessEvaluator({
  model: openai("gpt-4o-mini"),
});

const result = await faithfulness.evaluate({
  input: "What is the capital of France?",
  context: "France is a country in Europe. Paris is its capital city.",
  output: "The capital of France is Paris.",
});

Built-In Evaluator Factories

• createConcisenessEvaluator
• createCorrectnessEvaluator
• createDocumentRelevanceEvaluator
• createFaithfulnessEvaluator
• createRefusalEvaluator
• createClassificationEvaluator
• createToolSelectionEvaluator
• createToolInvocationEvaluator
• createToolResponseHandlingEvaluator

import { openai } from "@ai-sdk/openai";
import {
  createCorrectnessEvaluator,
  createRefusalEvaluator,
} from "@arizeai/phoenix-evals";

const model = openai("gpt-4o-mini");

const correctness = createCorrectnessEvaluator({ model });
const refusal = createRefusalEvaluator({ model });

Source Map

• src/llm/createClassificationEvaluator.ts
• src/llm/ClassificationEvaluator.ts
• src/llm/LLMEvaluator.ts
• src/llm/createFaithfulnessEvaluator.ts
• src/types/evals.ts