Ensemble¶

Multi-judge evaluation with configurable aggregation strategies.

Overview¶

Ensemble judging combines verdicts from multiple LLM judges to improve robustness and reduce individual model biases. All graders return EnsembleEvaluationReport for a consistent interface (single LLM is treated as "ensemble of 1").

Research Background

Verga et al. (2024) demonstrate in "Replacing Judges with Juries" that aggregating independent judgments from diverse models reduces systematic errors. Cross-family judging (using models from different providers) is particularly effective at mitigating self-preference bias documented by He et al. (2025).

Quick Example¶

from autorubric import LLMConfig
from autorubric.graders import CriterionGrader, JudgeSpec

# Ensemble with multiple judges
grader = CriterionGrader(
    judges=[
        JudgeSpec(LLMConfig(model="gemini/gemini-3-flash-preview"), "gemini", weight=1.0),
        JudgeSpec(LLMConfig(model="anthropic/claude-sonnet-4-5-20250929"), "claude", weight=1.2),
        JudgeSpec(LLMConfig(model="openai/gpt-4.1-mini"), "gpt", weight=1.0),
    ],
    aggregation="weighted",
)

result = await rubric.grade(to_grade=response, grader=grader)

# Ensemble-specific fields. score / mean_agreement are `float | None`: score is None if
# the grade failed or no criterion was left to score, mean_agreement if nothing was
# measured (an empty rubric). Guard before formatting.
print(f"Score: {result.score:.3f}" if result.score is not None else "Score: n/a")
print(
    f"Mean Agreement: {result.mean_agreement:.1%}"
    if result.mean_agreement is not None
    else "Mean Agreement: n/a"
)
print(f"Judge Scores: {result.judge_scores}")

# Per-criterion vote breakdown
for cr in result.report:
    print(f"{cr.criterion.requirement}")
    for vote in cr.votes:
        print(f"  {vote.judge_id}: {vote.verdict} ({vote.reason[:50]}...)")

Keyword Form¶

JudgeSpec takes the judge's config, its judge_id, and an optional weight, in that order. A weight must be a positive, finite number (default 1.0); zero, negative, NaN and infinite weights raise ValueError, because the weighted and threshold strategies sum and compare judge weights. When you pass the config by keyword, call it judge_model_config:

spec = JudgeSpec(
    judge_model_config=LLMConfig(model="openai/gpt-4.1-mini"),
    judge_id="gpt",
    weight=1.0,
)
spec.judge_model_config  # read/write; the same object as spec.llm_config

The underlying dataclass field is still named llm_config, so JudgeSpec(llm_config=...) and spec.llm_config keep working without a warning. Passing both keywords raises ValueError.

Judge IDs¶

Give every judge in judges= its own judge_id. A judge_id is the key that separates judges, so judges that share one are merged. Judges of the same kind all call the model of the last one. They also share a single judge_scores entry, one set of per-judge metrics, and the same option order and few-shot examples. CriterionGrader still accepts a repeated judge_id for now, but it emits a FutureWarning naming the repeated ids. A repeated judge_id will raise ValueError in the next major version. A cascade already rejects repeats (see Decision Models).

To poll one model several times, LLM or decision model, give each copy its own id. Each copy then gets its own response-cache entries with cache_enabled=True, so a rerun replays each copy's own answers, and each copy of an LLM judge also gets its own option shuffle:

judges = [JudgeSpec(LLMConfig(model="openai/gpt-4.1-mini"), f"gpt-{i}") for i in range(3)]

Aggregation Strategies¶

Strategy	Description
`majority`	Head count of MET vs UNMET votes; the larger count wins
`weighted`	Weighted vote using judge weights
`unanimous`	All non-abstaining judges must vote MET
`any`	Any judge voting MET results in MET

These apply to binary criteria only and are independent of multi-choice aggregation (ordinal_aggregation / nominal_aggregation). Conceptually, binary unanimous ≡ the min over the {0, 1} option values and any ≡ the max; the ordinal analogs are the min / max strategies (see the multi-choice cookbook).

Every strategy counts only MET and UNMET votes. CANNOT_ASSESS votes, including those of judge calls that failed with an API or parse error, are set aside first, and a criterion whose votes all abstain is CANNOT_ASSESS. A majority or weighted tie goes to the verdict that scores lowest for the criterion's weight sign: UNMET for a positive (or zero) weight, MET for a negative one.

A judge whose every call on an item failed judged nothing there, so its judge_scores entry for that item is None, whatever its stand-in verdicts would score. When every judge failed on every criterion, the item has no score at all: score and raw_score are None, and the report's error begins Every criterion's judgment failed:.

AggregationStrategy¶

Enum for binary verdict aggregation strategies.

AggregationStrategy `module-attribute` ¶

AggregationStrategy = Literal['majority', 'weighted', 'unanimous', 'any']

Strategy for aggregating votes from multiple judges (binary criteria).

majority: Head count of MET vs UNMET votes; the larger count wins
weighted: Weighted vote based on judge weights
unanimous: All non-abstaining judges must vote MET
any: Any judge voting MET results in MET

Only MET and UNMET votes count: CANNOT_ASSESS votes (including failed judge calls) are set aside first, and a criterion whose votes all abstain is CANNOT_ASSESS.

Tie-breaking (majority head-count tie or weighted equal-weight tie) resolves to the score-minimizing verdict by weight sign: UNMET for weight ≥ 0 (earns 0), MET for weight < 0 (applies the full penalty) — the binary analog of Criterion.worst_scored_option. unanimous/any are thresholds, not ties.

EnsembleEvaluationReport¶

Evaluation result from ensemble grading with per-judge breakdown.

EnsembleEvaluationReport ¶

Bases: BaseModel

Evaluation report with ensemble voting details.

Extends EvaluationReport with per-judge breakdown and agreement metrics.

ATTRIBUTE	DESCRIPTION
`score`	The final aggregated score (0-1 if normalized). `None` when there is no score: grading FAILED (an error report: no judge results, or every criterion's judgment failed), or no criterion is left to score (under the SKIP strategy, every final verdict abstains). TYPE: `float \| None`
`raw_score`	The unnormalized weighted sum. `None` exactly when `score` is. TYPE: `float \| None`
`llm_raw_score`	Same as raw_score (for compatibility with EvaluationReport). TYPE: `float \| None`
`report`	Per-criterion breakdown with ensemble voting details. TYPE: `list[EnsembleCriterionReport] \| None`
`judge_scores`	Each judge's own score over the whole rubric (from its own verdicts), keyed by `judge_id`. `None` when that judge's whole-rubric score is undefined: by role, a judge that votes only on some criteria (e.g. an escalation judge) is `None` on every item, even an item where it happened to vote on every criterion, so the entry's meaning never depends on the item; and on an item where every vote of the judge failed (it judged nothing) or none of its verdicts is left to score. TYPE: `dict[str, float \| None]`
`mean_agreement`	Average agreement across all criteria, or None when there are no criteria to agree on (empty rubric) / agreement was not measured. TYPE: `float \| None`
`cannot_assess_count`	Number of criteria with CANNOT_ASSESS final verdict. TYPE: `int`
`token_usage`	Total token usage across all judges. TYPE: `TokenUsage \| None`
`completion_cost`	Total cost across all judges. TYPE: `float \| None`
`error`	Error message if grading failed: no judge results, or "Every criterion's judgment failed: ..." when each criterion's verdict stands in for failed calls (each keeps its own error). When set, `score`, `raw_score` and `llm_raw_score` are `None`. TYPE: `str \| None`

EnsembleCriterionReport¶

Per-criterion result with individual judge votes.

EnsembleCriterionReport ¶

Bases: BaseModel

A criterion report with ensemble voting details.

Supports both binary and multi-choice criteria: - Binary: Use final_verdict and votes (list of JudgeVote) - Multi-choice: Use final_multi_choice_verdict and multi_choice_votes

ATTRIBUTE	DESCRIPTION
`criterion`	The criterion being evaluated. TYPE: `Criterion`
`final_verdict`	Aggregated binary verdict from all judges. None for multi-choice. TYPE: `CriterionVerdict \| None`
`final_reason`	Combined reasoning from judges: each aggregated (non-`superseded`) vote's reason rendered as `"judge_id: reason"`, joined with `" \| "`, so an escalated criterion's comes from the escalation judges alone. Votes whose `reason` is `None` (the judge gives no explanation) are skipped. `None` only when no aggregated vote carried a reason. The fixed reasons are kept whatever the votes' reasons: `"No votes"` (an empty vote list) and, for binary criteria, `"All judges could not assess"` (every aggregated vote `CANNOT_ASSESS`). TYPE: `str \| None`
`votes`	Individual binary votes from each judge. Empty for multi-choice. TYPE: `list[JudgeVote]`
`agreement`	Proportion of the aggregated (non-`superseded`) votes agreeing with the final verdict (0-1). TYPE: `float`
`final_multi_choice_verdict`	Aggregated multi-choice verdict. None for binary. TYPE: `AggregatedMultiChoiceVerdict \| None`
`multi_choice_votes`	Individual multi-choice votes. Empty for binary. TYPE: `list[MultiChoiceJudgeVote]`
`error`	Set (with a category prefix) when the final verdict was driven entirely by judge-call failures (every aggregated, non-`superseded` vote errored). None when at least one genuine judgment was aggregated. See `is_error`. TYPE: `str \| None`
`escalated`	True when this criterion was escalated: the vote first cast on it (a decision model's) was judged insufficient because it errored, abstained, or its `confidence` fell below the escalation threshold. That vote stays in the vote list with `superseded=True` and the final verdict is aggregated from the escalation judges' votes alone, even when every one of them failed. False otherwise, including every report of a grader that does not escalate. TYPE: `bool`

score_value `property` ¶

score_value: float

Get the score contribution (0-1) for this criterion.

is_na `property` ¶

is_na: bool

Check if this criterion was marked NA or CANNOT_ASSESS.

is_error `property` ¶

is_error: bool

Whether the final verdict was driven entirely by judge-call failures.

JudgeVote¶

Individual judge's verdict for a criterion.

JudgeVote ¶

Bases: BaseModel

A single judge's vote on a criterion.

ATTRIBUTE	DESCRIPTION
`judge_id`	Identifier for the judge (e.g., "gpt-4", "claude-sonnet"). TYPE: `str`
`verdict`	The judge's verdict (MET/UNMET). TYPE: `CriterionVerdict`
`reason`	The judge's brief justification for the verdict (the conclusion distilled from `reasoning` when thinking is enabled). `None` means the judge gives no explanation (a judge that returns probabilities instead of text); an LLM judge always sets a string, possibly empty. TYPE: `str \| None`
`weight`	Judge's voting weight (default 1.0). TYPE: `float`
`error`	Set (with a category prefix) when this vote's verdict was synthesized because the judge call failed. None for genuine votes. TYPE: `str \| None`
`reasoning`	The judge's verbose extended-thinking deliberation trace (populated only when thinking is enabled; None otherwise). `reason` is the conclusion distilled from it. Carried from this judge's `CriterionReport.reasoning`. TYPE: `str \| None`
`probabilities`	The judge's probability for each verdict it was offered, keyed by verdict value (`"MET"`, `"UNMET"`, and `"CANNOT_ASSESS"` when offered), when the judge returns a distribution (a decision model); `None` for LLM judges. Same scheme as `CriterionReport.probabilities`. TYPE: `dict[str, float] \| None`
`confidence`	The judge's support for the verdict it selected, in [0, 1]: `clamp((K * p - 1) / (K - 1), 0, 1)`, where `K` is the number of verdicts offered and `p` is the selected verdict's probability. `None` for LLM judges. TYPE: `float \| None`
`superseded`	True when this vote was recorded but not aggregated: a vote judged insufficient (e.g. a decision model's low-confidence vote on a criterion that was escalated to other judges) whose criterion's final verdict comes from the other votes. It is kept for diagnostics and excluded from `agreement`. TYPE: `bool`

is_error `property` ¶

is_error: bool

Whether this vote's verdict was synthesized due to a judge-call failure.

Use this instead of inspecting reason to distinguish error-induced votes from genuine judgments.

References¶

He, J., Shi, J., Zhuo, T. Y., Treude, C., Sun, J., Xing, Z., Du, X., and Lo, D. (2025). LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead. arXiv:2510.24367.

Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., and Lewis, P. (2024). Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv:2404.18796.

Ensemble¶

Overview¶

Quick Example¶

Keyword Form¶

Judge IDs¶

Aggregation Strategies¶

AggregationStrategy¶

AggregationStrategy module-attribute ¶

EnsembleEvaluationReport¶

EnsembleEvaluationReport ¶

EnsembleCriterionReport¶

EnsembleCriterionReport ¶

score_value property ¶

is_na property ¶

is_error property ¶

JudgeVote¶

JudgeVote ¶

is_error property ¶

References¶

AggregationStrategy `module-attribute` ¶

score_value `property` ¶

is_na `property` ¶

is_error `property` ¶

is_error `property` ¶