12.04.2025: Evaluator Message Formats

Available in phoenix-evals 0.22+ (Python) and @arizeai/phoenix-evals 2.0+ (TypeScript)

Evaluator Message Formats

Phoenix evaluators now support flexible prompt formats in both Python and TypeScript, giving you full control over how you structure prompts for LLM-based evaluations.

Supported Formats

String Templates - Simple templates with variable placeholders:

Python

from phoenix.evals import ClassificationEvaluator, LLM

evaluator = ClassificationEvaluator(
    name="sentiment",
    llm=LLM(provider="openai", model="gpt-4o-mini"),
    prompt_template="Classify the sentiment: {text}",
    choices=["positive", "negative", "neutral"]
)

TypeScript

import { createClassificationEvaluator } from "@arizeai/phoenix-evals";
import { openai } from "@ai-sdk/openai";

const evaluator = createClassificationEvaluator({
  name: "sentiment",
  model: openai("gpt-4o-mini"),
  promptTemplate: "Classify the sentiment: {{text}}",
  choices: { positive: 1, negative: 0, neutral: 0.5 },
});

Message Lists - OpenAI-style arrays with role and content fields for multi-turn prompts:

Python

evaluator = ClassificationEvaluator(
    name="helpfulness",
    llm=llm,
    prompt_template=[
        {"role": "system", "content": "You evaluate response helpfulness."},
        {"role": "user", "content": "Question: {question}\nAnswer: {answer}"}
    ],
    choices=["helpful", "somewhat_helpful", "not_helpful"]
)

TypeScript

const evaluator = createClassificationEvaluator({
  name: "helpfulness",
  model,
  promptTemplate: [
    { role: "system", content: "You evaluate response helpfulness." },
    { role: "user", content: "Question: {{question}}\nAnswer: {{answer}}" },
  ],
  choices: { helpful: 1, somewhat_helpful: 0.5, not_helpful: 0 },
});

Template Variable Syntax

• Python: Supports both f-string ({variable}) and mustache ({{variable}}) syntax with auto-detection
• TypeScript: Uses mustache syntax ({{variable}})

Provider Compatibility

Adapters handle provider-specific message transformations automatically:

Provider	Transformation
OpenAI	System role converted to developer role for reasoning models
Anthropic	System messages extracted to system parameter
Google GenAI	System messages passed via system_instruction
LiteLLM	Messages passed in OpenAI format (LiteLLM handles conversion)
LangChain	Converted to LangChain message objects

More Information:

Eval Prompt Templates