Coverage (Output)

Mutant provides comprehensive reporting capabilities to evaluate the diversity, difficulty, and overall quality of your generated datasets. This is the final stage of the pipeline.

Dataset Coverage

When generating datasets, you want to ensure they cover a wide range of behavioral dimensions. Mutant analyzes the generated scenarios and produces a coverage report.

This report visualizes: - Input Diversity: How varied the prompts are. - Difficulty: The cognitive load required to solve the scenarios. - Domain Coverage: The breadth of topics addressed.

Example: HTML Dashboard

You can instantly visualize your dataset's coverage using the HTML reporter:

from mutant.reports.html import HtmlReport

# Generate a self-contained HTML file
report = HtmlReport()
report.save(scenario, dataset.cases, path="dashboard.html")

---

API Reference

mutant.reports.html

HTML report generator.

HtmlReport

Generates a self-contained HTML report.

Example

report = HtmlReport() report.save(scenario, cases, path="report.html")

Source code in mutant/reports/html.py

class HtmlReport:
    """Generates a self-contained HTML report.

    Example
    -------
    >>> report = HtmlReport()
    >>> report.save(scenario, cases, path="report.html")
    """

    def render(self, scenario: Scenario, cases: list[MutationCase]) -> str:
        data = _build_report_dict(scenario, cases)
        return render_prompt("mutant_report.html", data=data)

    def save(
        self, scenario: Scenario, cases: list[MutationCase], path: str | Path
    ) -> Path:
        output = Path(path)
        output.parent.mkdir(parents=True, exist_ok=True)
        output.write_text(self.render(scenario, cases), encoding="utf-8")
        return output

mutant.reports.json

JSON and Markdown report generators.

JsonReport

Serialises mutation results to JSON.

Example

report = JsonReport() report.save(scenario, cases, path="report.json")

Source code in mutant/reports/json.py

class JsonReport:
    """Serialises mutation results to JSON.

    Example
    -------
    >>> report = JsonReport()
    >>> report.save(scenario, cases, path="report.json")
    """

    def render(self, scenario: Scenario, cases: list[MutationCase]) -> str:
        """Return the report as a JSON string."""
        data = _build_report_dict(scenario, cases)
        return json.dumps(data, indent=2, ensure_ascii=False)

    def save(
        self, scenario: Scenario, cases: list[MutationCase], path: str | Path
    ) -> Path:
        """Render and save the report to ``path``. Returns the resolved path."""
        output = Path(path)
        output.parent.mkdir(parents=True, exist_ok=True)
        output.write_text(self.render(scenario, cases), encoding="utf-8")
        return output

render(scenario, cases)

Return the report as a JSON string.

Source code in mutant/reports/json.py

def render(self, scenario: Scenario, cases: list[MutationCase]) -> str:
    """Return the report as a JSON string."""
    data = _build_report_dict(scenario, cases)
    return json.dumps(data, indent=2, ensure_ascii=False)

save(scenario, cases, path)

Render and save the report to path. Returns the resolved path.

Source code in mutant/reports/json.py

def save(
    self, scenario: Scenario, cases: list[MutationCase], path: str | Path
) -> Path:
    """Render and save the report to ``path``. Returns the resolved path."""
    output = Path(path)
    output.parent.mkdir(parents=True, exist_ok=True)
    output.write_text(self.render(scenario, cases), encoding="utf-8")
    return output

MarkdownReport

Serialises mutation results to a Markdown document.

Example

report = MarkdownReport() report.save(scenario, cases, path="report.md")

Source code in mutant/reports/json.py

class MarkdownReport:
    """Serialises mutation results to a Markdown document.

    Example
    -------
    >>> report = MarkdownReport()
    >>> report.save(scenario, cases, path="report.md")
    """

    def render(self, scenario: Scenario, cases: list[MutationCase]) -> str:
        data = _build_report_dict(scenario, cases)
        lines: list[str] = [
            f"# Mutant Report — {data['scenario']['title']}",
            "",
            f"**Generated:** {data['generated_at']}  ",
            f"**Report ID:** `{data['report_id']}`",
            "",
            "## Original Scenario",
            "",
            f"> {data['scenario']['description']}",
            "",
            "## Summary",
            "",
            f"- **Total mutations:** {data['summary']['total']}",
            "",
            "### By Category",
            "",
        ]
        for cat, count in sorted(data["summary"]["by_category"].items()):
            lines.append(f"- `{cat}`: {count}")
        lines += [
            "",
            "### By Severity",
            "",
        ]
        for sev, count in sorted(data["summary"]["by_severity"].items()):
            lines.append(f"- `{sev}`: {count}")
        lines += [
            "",
            "---",
            "",
            "## Mutation Cases",
            "",
        ]
        for i, case in enumerate(data["cases"], 1):
            lines += [
                f"### {i}. {case['dimension_name']}",
                "",
                "| Field | Value |",
                "|-------|-------|",
                f"| **ID** | `{case['id']}` |",
                f"| **Dimension** | `{case['dimension_id']}` |",
                f"| **Category** | `{case['category']}` |",
                f"| **Severity** | `{case['severity']}` |",
                "",
                "**Rationale:**",
                "",
                f"> {case['rationale']}",
                "",
                "**Original:**",
                "",
                f"> {case['original']}",
                "",
                "**Mutated:**",
                "",
                f"> {case['mutated']}",
                "",
            ]
        return "\n".join(lines)

    def save(
        self, scenario: Scenario, cases: list[MutationCase], path: str | Path
    ) -> Path:
        """Render and save the report to ``path``. Returns the resolved path."""
        output = Path(path)
        output.parent.mkdir(parents=True, exist_ok=True)
        output.write_text(self.render(scenario, cases), encoding="utf-8")
        return output

save(scenario, cases, path)

Render and save the report to path. Returns the resolved path.

Source code in mutant/reports/json.py

def save(
    self, scenario: Scenario, cases: list[MutationCase], path: str | Path
) -> Path:
    """Render and save the report to ``path``. Returns the resolved path."""
    output = Path(path)
    output.parent.mkdir(parents=True, exist_ok=True)
    output.write_text(self.render(scenario, cases), encoding="utf-8")
    return output