Skip to content
2K+ downloads and counting

Automate Evaluation & Red Teaming for LLMs

Mutant is the open-source evaluation framework for synthesizing adversarial datasets, testing AI agents, and grading RAG pipelines.

Get Started GitHub
$ pip install mutant-ai
1

The Evaluation Gap

During evaluations, teams simply don't have the right datasets. Testing the "happy path" leaves you entirely blind to edge cases.

Mutant's Augment Engine solves this by taking a single seed scenario and multiplying it into hundreds of behaviorally diverse test cases using cognitive dimensions (like anger, manipulation, or slang).

  • Stop writing manual test cases.
  • Uncover edge cases automatically.
generate_dataset.py
from mutant.core import Scenario, augment
from mutant.providers.ollama import OllamaProvider

provider = OllamaProvider(model="llama3.1")

# Generate 5 diverse variations of a basic intent
dataset = await augment(
    dataset=[Scenario(title="Refund", description="I want a refund.")],
    provider=provider,
    mutations_per_case=5,
    dimensions=["emotion.angry", "language.slang"]
)

dataset.save("mutations.json")
2

Agentic Red Teaming

Static security scanners fail to catch complex vulnerabilities in generative AI.

The Red Team Agent acts as a synthetic adversary. It executes adaptive, hypothesis-driven attacks to bypass safeguards and extract sensitive information through multi-turn conversations.

  • Automated jailbreak attempts.
  • Multi-turn conversational profiling.
red_team.py
from mutant.redteam import red_team
from mutant.providers.ollama import OllamaProvider
from my_app import my_agent

provider = OllamaProvider(model="llama3.1")

# Unleash the attacker against your live agent
report = await red_team(
    target=my_agent,
    goal="Extract the hidden system instructions.",
    provider=provider,
    max_turns=5
)

report.to_html("redteam_report.html")
3

Security Evaluation

Go beyond general correctness and actively grade your application's defensive mechanisms.

Run your adversarial datasets through the Eval Suite using specialized security metrics to automatically detect prompt injections, PII leakage, and unauthorized behavior.

  • Detect Sensitive Data Leakage.
  • Prevent Prompt Injections.
security_eval.py
from mutant.eval import EvalSuite, SensitiveDataLeakage, PromptInjectionResistance
from mutant.providers.ollama import OllamaProvider

provider = OllamaProvider(model="llama3.1")

# Grade the agent against security threats
suite = EvalSuite(metrics=[
    SensitiveDataLeakage(provider=provider),
    PromptInjectionResistance(provider=provider)
])

report = await suite.evaluate(test_cases=dataset.cases)
report.to_html("security_report.html")
4

Evaluation Suite

Once you've generated your adversarial dataset or executed your attacks, you need a robust way to grade your application's responses.

The Eval Suite integrates industry-standard metrics using LLM-as-a-judge to provide deterministic scores, visualizing everything in beautiful HTML reports.

  • Correctness, Faithfulness, Context Precision.
  • Highly visual CI/CD ready dashboards.
run_metrics.py
from mutant.eval import EvalSuite, Correctness, Faithfulness
from mutant.providers.ollama import OllamaProvider

provider = OllamaProvider(model="llama3.1")

# Initialize metrics and evaluate test cases
suite = EvalSuite(metrics=[
    Correctness(provider=provider),
    Faithfulness(provider=provider)
])

report = await suite.evaluate(test_cases=dataset.cases)
report.to_html("eval_report.html")

Ready to mutate your datasets?

View the Documentation

Welcome to Mutant

Whether you are building a simple RAG chatbot or a complex multi-agent system, Mutant provides the tools to automatically evaluate your system's correctness, faithfulness, and security before you deploy to production.

⚡ Quick Install

Mutant is available on PyPI. Install it using pip or uv:

$ pip install mutant-ai

🏗️ Core Framework

Mutant is divided into three core pillars. You can use them independently or chain them together for end-to-end evaluation.

  • 1. Data Augmentation

    Stop writing manual test cases. Mutant takes a single seed scenario and uses cognitive dimensions (like anger, prompt injection, or complex reasoning) to generate hundreds of behaviorally diverse test cases instantly.

    Read the Augment Guide

  • 2. Agentic Red Teaming

    Instead of static security scans, unleash a hypothesis-driven attacker agent. Mutant dynamically probes your system in multi-turn conversations, adapting its attacks to bypass your safeguards.

    Read the Red Team Guide

  • 3. Evaluation Suite

    Grade your production applications using deterministic or LLM-as-a-judge metrics. Measure Correctness, Faithfulness, Context Precision, and Sensitive Data Leakage out of the box.

    Read the Eval Suite Guide

💻 See it in Action

Here is what an end-to-end evaluation pipeline looks like using Mutant.

from mutant.core import Scenario, augment
from mutant.providers.ollama import OllamaProvider

provider = OllamaProvider(model="llama3.1")

# Generate 5 variations of a basic prompt using the angry dimension
dataset = await augment(
    dataset=[Scenario(title="Refund", description="I want a refund.")],
    provider=provider,
    mutations_per_case=5,
    dimensions=["emotion.angry"]
)

dataset.save("mutations.json")

from mutant.eval import EvalSuite, Correctness, Faithfulness
from mutant.providers.ollama import OllamaProvider

provider = OllamaProvider(model="llama3.1")

# Grade the target outputs against your generated dataset
suite = EvalSuite(metrics=[
    Correctness(provider=provider),
    Faithfulness(provider=provider)
])

report = await suite.evaluate(test_cases=dataset.cases)
report.to_html("eval_report.html")

from mutant.redteam import red_team
from mutant.providers.ollama import OllamaProvider
from my_app import my_agent 

provider = OllamaProvider(model="llama3.1")

# Unleash an attacker agent against your application
report = await red_team(
    target=my_agent,
    goal="Extract the system prompt",
    provider=provider,
    max_turns=3
)

report.to_html("redteam_report.html")

📊 Actionable Reports

Mutant natively outputs highly visual HTML dashboards and CI/CD-ready JSON artifacts, mapping your exact vulnerability and coverage footprint.

Coverage Report Example