RAG & LLM Evaluation (EvalSuite)
Mutant now includes a first-class Evaluation Framework (EvalSuite) for running deterministic and LLM-as-a-judge metrics against your applications. This allows you to evaluate your RAG pipelines and AI agents using industry-standard metrics like Correctness, Faithfulness, Context Precision, and more.
1. Define the Metrics
Metrics define the criteria you are evaluating your AI system against. Many industry-standard metrics (such as Correctness and Faithfulness) use an LLM-as-a-judge, which means they require an initialized LLM Provider.
| Parameter | Type | Description |
|---|---|---|
provider |
BaseLLMProvider |
Required for LLM metrics. Your configured LLM provider instance. |
strict |
bool |
Optional. Enforce strict schema validation on the LLM judge. Default False. |
from mutant.eval import Correctness, Faithfulness
from mutant.providers.ollama import OllamaProvider
provider = OllamaProvider(model="llama3.1")
# Define the metrics you want to evaluate against
metrics = [
Correctness(provider=provider),
Faithfulness(provider=provider)
]
2. Initialize the EvalSuite
The EvalSuite object orchestrates the evaluation process, applying your selected metrics across your dataset.
| Parameter | Type | Description |
|---|---|---|
metrics |
list |
Required. A list of metric instances you defined in Step 1. |
concurrency |
int |
Optional. Number of concurrent LLM calls allowed during evaluation. Default 5. |
from mutant.eval import EvalSuite
# Initialize the suite with your chosen metrics
suite = EvalSuite(metrics=metrics)
3. Create Test Cases
A TestCase represents a single evaluation scenario containing the data required for the metrics to run. Different metrics require different fields.
| Parameter | Type | Description |
|---|---|---|
input |
str |
Required. The original user prompt, question, or scenario. |
actual_output |
str |
Required. The actual response generated by your target application. |
expected_output |
str |
Optional. The ideal "ground truth" reference answer (used by metrics like Correctness). |
retrieval_context |
list[str] |
Optional. The documents/context retrieved by your RAG pipeline (used by metrics like Faithfulness). |
from mutant.eval import TestCase
test_case = TestCase(
input="What is the refund policy?",
actual_output="You can get a refund within 30 days.",
expected_output="Refunds are available within 30 days of purchase.",
retrieval_context=["Our store policy allows refunds within 30 days of purchase."]
)
4. Run Evaluation & Export
Once everything is defined, run .evaluate() on your suite by passing a list of test cases. You can then export the results to various formats.
| Method | Description |
|---|---|
to_html(path) |
Exports a visually rich, standalone HTML report. |
to_json(path) |
Exports the raw evaluation data for CI/CD integration. |