Graders

Grader implementations for evaluating responses against rubrics.

Overview

Graders evaluate responses against rubrics and return structured reports. The main implementation is CriterionGrader, which supports single LLM, ensemble, and few-shot modes. All combinations work orthogonally.

Quick Example

from autorubric import LLMConfig, FewShotConfig
from autorubric.graders import CriterionGrader, JudgeSpec, Grader

# Single LLM mode
grader = CriterionGrader(
    judge_model_config=LLMConfig(model="openai/gpt-4.1-mini"),
)

# One call per item: the judge grades the whole rubric in a single call
# instead of one call per criterion
grader = CriterionGrader(
    judge_model_config=LLMConfig(model="openai/gpt-4.1-mini"),
    llm_calls="per_item",
)

# With custom system prompt
grader = CriterionGrader(
    judge_model_config=LLMConfig(model="openai/gpt-4.1-mini"),
    system_prompt="You are evaluating technical documentation...",
)

# Ensemble mode
grader = CriterionGrader(
    judges=[
        JudgeSpec(LLMConfig(model="gemini/gemini-3-flash-preview"), "gemini", weight=1.0),
        JudgeSpec(LLMConfig(model="anthropic/claude-sonnet-4-5-20250929"), "claude", weight=1.2),
    ],
    aggregation="weighted",
)

# Single LLM + few-shot
grader = CriterionGrader(
    judge_model_config=LLMConfig(model="openai/gpt-4.1-mini"),
    training_data=train_data,
    few_shot_config=FewShotConfig(n_examples=3, balance_verdicts=True),
)

# Grade
result = await rubric.grade(to_grade=response, grader=grader)

judge_model_config sets up a single judge and judges sets up an ensemble. Pass exactly one of them.

Either accepts a DecisionModelConfig as well as an LLMConfig: a decision-model judge grades each item with one request for the whole rubric. escalation=EscalationConfig(...) makes a grader with one decision-model judge a confidence cascade that escalates uncertain criteria to LLM judges. See Decision Models for the API, and the LLM Judges and Decision-Model Judges chapters for guided introductions to the two judge kinds.

llm_calls="per_item" makes every LLM judge of the grader — the judge of judge_model_config, each judge of judges, and a cascade's escalation judges — grade an item's whole rubric in one call instead of one call per criterion, the way a decision-model judge always does. The default, "per_criterion", is unchanged. See Grading a whole rubric in one call for what the single call looks like, and Decision-Model Judges for what it shares with a decision model's request.

llm_config is deprecated

llm_config is a deprecated alias of judge_model_config. It builds the same judge but emits a DeprecationWarning. To migrate, rename the keyword; nothing else changes. Passing both raises ValueError. llm_config will not be removed before the next major version.

Grading Options

grader = CriterionGrader(
    judge_model_config=LLMConfig(model="openai/gpt-4.1-mini"),

    # Score normalization
    normalize=True,          # True: 0-1 range, False: raw weighted sum

    # CANNOT_ASSESS handling
    cannot_assess_config=CannotAssessConfig(strategy=CannotAssessStrategy.SKIP),

    # Length penalty
    length_penalty=LengthPenalty(free_budget=6000, max_cap=8000),

    # Position bias mitigation (for multi-choice)
    shuffle_options=True,    # Default: enabled

    # Multi-choice abstain channel
    auto_na_option=True,     # Default: True — guarantees every multi-choice criterion a
                             # first-class NA/abstain option (auto-injected if absent).
                             # Set False for forced-choice (no auto NA option).

    # Reproducibility — pins all non-LLM randomness (shuffles, few-shot selection)
    seed=42,                 # Default: auto-generated
)

With auto_na_option=True, an auto-injected NA option is appended at the end (highest index), so existing option indices are preserved; an author-supplied NA option is never stripped.

---

CriterionGrader

Main grader with support for single LLM, ensemble, and few-shot modes.

CriterionGrader

CriterionGrader(*, judge_model_config: LLMConfig | DecisionModelConfig | None = None, judges: list[JudgeSpec] | None = None, aggregation: AggregationStrategy = 'majority', ordinal_aggregation: OrdinalAggregation = 'mean', nominal_aggregation: NominalAggregation = 'mode', training_data: RubricDataset | None = None, few_shot_config: FewShotConfig | None = None, system_prompt: str | None = None, multi_choice_system_prompt: str | None = None, length_penalty: LengthPenalty | None = None, normalize: bool = True, cannot_assess_config: CannotAssessConfig | None = None, shuffle_options: bool = True, auto_na_option: bool = True, seed: int | None = None, binary_response_format: type[BaseModel] | None = None, multi_choice_response_format: type[BaseModel] | None = None, escalation: EscalationConfig | None = None, llm_calls: LLMCalls = 'per_criterion', llm_config: LLMConfig | DecisionModelConfig | None = None)

Bases: Grader[EnsembleEvaluationReport]

Unified criterion-based grader with compositional few-shot and ensemble support.

This grader evaluates each criterion independently and supports: - Single-judge mode (via judge_model_config) - Ensemble mode with multiple judges (via judges) - Few-shot prompting (via training_data + few_shot_config)

All combinations work: single judge, single + few-shot, ensemble, ensemble + few-shot.

Parameters are orthogonal: - judge_model_config OR judges: Choose single-judge or ensemble mode - training_data + few_shot_config: Enable few-shot prompting (applies to every LLM judge)

A judge is an LLM (LLMConfig) or a decision model (DecisionModelConfig), alone or mixed in one ensemble. An LLM judge is asked about each criterion in its own call. A decision-model judge is asked about the whole rubric in one request per item: the submission goes once as shared state and every criterion it can express becomes one question; a criterion it cannot express fails alone, and a failed request fails every criterion posed in it, through the same failure routing as an LLM judge's failed call. Its votes carry probabilities and confidence and no explanation (reason is None). In a mixed ensemble every judge votes on every criterion, and the votes aggregate exactly as an all-LLM ensemble's do.

With llm_calls="per_item", each LLM judge of judge_model_config or judges is asked about the whole rubric in one call per item instead, as a decision model is: the submission and its context are sent once; the system prompt embeds the binary and multi-choice system prompts, for the kinds of criteria the rubric has, as guides to apply to each criterion separately and independently; every criterion is listed under its decision-model question id (c0, c1, ...), posed as in a per-criterion call and with its options shuffled exactly as there; and the answer (RubricJudgment) maps back to one result per criterion. A missing or unusable answer fails only its criterion (parse), a failed call fails every criterion, and the call's usage and cost ride on the item's first result. With few-shot examples, the call shows whole training items, each once with its ground truth for every criterion, instead of examples drawn for each criterion. Votes, aggregation and scoring are the same in both modes. A cascade's escalation judges make that call too, but only for an item with an escalated criterion, and keep only the escalated criteria's results (see below).

A confidence cascade (escalation=EscalationConfig(...)) puts one decision model first and LLM judges behind it. The decision model judges every criterion in its one request per item; a criterion it errored or abstained on, or answered with a confidence below the criterion's threshold, is escalated, and each escalation judge votes on the escalated criteria only. Called per criterion, an escalation judge is asked about each escalated criterion in a call of its own. Under llm_calls="per_item" it is asked about the whole rubric in one call, made only for an item with an escalated criterion; its judgments of the other criteria are discarded, a failed call fails its votes on the escalated criteria, and the call's usage and cost ride on the first escalated criterion's result. Either way each call is the one the judge would make in an otherwise identical grader without the cascade (the same seed, judge_id and LLM settings, llm_calls included). An escalated criterion's report is marked escalated=True, keeps the decision model's vote with superseded=True, and takes its verdict from the escalation judges' votes alone, aggregated as any ensemble's; it never falls back to the decision model's vote. The decision model's judge_scores entry is its own score over every criterion; an escalation judge's is None by role, as it votes only where the decision model escalated, even on an item where every criterion escalated.

Some settings exist for LLM judges only. Few-shot examples, the system prompts and shuffle_options apply to the LLM judges of a mixed ensemble or a cascade and never reach a decision model; so does llm_calls. A grader whose judges are all decision models rejects few-shot examples and warns when a system prompt is given, shuffle_options is False or llm_calls is "per_item", since it has no judge to apply them to. A custom response format describes a generated judgment, which only an LLM judge produces, so it cannot be combined with any decision-model judge; it describes one criterion's judgment, so it cannot be combined with llm_calls="per_item" either.

Example

from autorubric import DecisionModelConfig, LLMConfig, FewShotConfig, RubricDataset from autorubric.graders import CriterionGrader, JudgeSpec

Single LLM

grader = CriterionGrader( ... judge_model_config=LLMConfig(model="gemini/gemini-3-flash-preview") ... )

Single LLM + few-shot

train, test = dataset.split_train_test(n_train=100) grader = CriterionGrader( ... judge_model_config=LLMConfig(model="gemini/gemini-3-flash-preview"), ... training_data=train, ... few_shot_config=FewShotConfig(n_examples=3), ... )

Ensemble

grader = CriterionGrader( ... judges=[ ... JudgeSpec(LLMConfig(model="gemini/gemini-3-flash-preview"), "gemini"), ... JudgeSpec(LLMConfig(model="anthropic/claude-sonnet-4-5-20250929"), "claude"), ... ], ... aggregation="majority", ... )

Ensemble + few-shot

grader = CriterionGrader( ... judges=[JudgeSpec(...), JudgeSpec(...)], ... aggregation="majority", ... training_data=train, ... few_shot_config=FewShotConfig(n_examples=3), ... )

Single LLM, one call per item for the whole rubric

grader = CriterionGrader( ... judge_model_config=LLMConfig(model="gemini/gemini-3-flash-preview"), ... llm_calls="per_item", ... )

Decision model: one request per item for the whole rubric

grader = CriterionGrader(judge_model_config=DecisionModelConfig(model="jev-latest"))

Mixed ensemble of a decision model and an LLM

grader = CriterionGrader( ... judges=[ ... JudgeSpec(DecisionModelConfig(model="jev-latest"), "jev"), ... JudgeSpec(LLMConfig(model="gemini/gemini-3-flash-preview"), "gemini"), ... ], ... )

Cascade: the decision model first, an LLM for what it is unsure of

grader = CriterionGrader( ... judge_model_config=DecisionModelConfig(model="jev-latest"), ... escalation=EscalationConfig( ... judges=LLMConfig(model="gemini/gemini-3-flash-preview"), threshold=0.72 ... ), ... )

Initialize the criterion grader.

PARAMETER	DESCRIPTION
judge_model_config	Configuration of the single judge (single-judge mode), which is given judge_id="default": an LLMConfig for an LLM judge or a DecisionModelConfig for a decision-model judge. Mutually exclusive with judges. TYPE: LLMConfig | DecisionModelConfig | None DEFAULT: None
judges	List of JudgeSpec for ensemble mode, each an LLM or a decision model, each with its own judge_id. Mutually exclusive with judge_model_config. TYPE: list[JudgeSpec] | None DEFAULT: None
aggregation	Strategy for aggregating votes in ensemble mode (binary criteria). TYPE: AggregationStrategy DEFAULT: 'majority'
ordinal_aggregation	Strategy for aggregating ordinal multi-choice votes. Central tendency: "mean", "median", "weighted_mean", "mode". Conservative/ permissive (analogs of binary unanimous/any): "min" (lowest selected option) / "max" (highest selected option). TYPE: OrdinalAggregation DEFAULT: 'mean'
nominal_aggregation	Strategy for aggregating nominal multi-choice votes. Options: "mode", "weighted_mode", "unanimous". "unanimous" abstains via the NA option on disagreement, or falls back to mode + warns if there is no NA option. TYPE: NominalAggregation DEFAULT: 'mode'
training_data	Dataset for few-shot examples. If provided, enables few-shot prompting for the LLM judges; examples are selected for, and sent to, LLM judges only, never to a decision-model judge, whose request is exactly what it would be without them. An example shows its item's ground truth for the criterion and, when few_shot_config.include_reason is True, the item's written reason for it (DataItem.ground_truth_reasons), if any. Called per criterion, a judge is shown examples drawn for each criterion separately. Under llm_calls="per_item" an example is a whole training item, shown once in the judge's one call per item with its ground truth (and reason) for every criterion; only an item whose ground truth resolves for every criterion (a verdict for a binary criterion, an option for a multi-choice one) is eligible. Either way each judge draws its own examples, and the training data needs a global rubric, which says what each label means. TYPE: RubricDataset | None DEFAULT: None
few_shot_config	Configuration for few-shot example selection (LLM judges only). Under llm_calls="per_item", n_examples counts the training items each call shows and balance_verdicts selects them by greedy label coverage (see FewShotConfig). TYPE: FewShotConfig | None DEFAULT: None
system_prompt	Custom system prompt for binary criteria. Applies to LLM judges only (a decision model's request has no system prompt); in a mixed ensemble the LLM judges use it. TYPE: str | None DEFAULT: None
multi_choice_system_prompt	Custom system prompt for multi-choice criteria. Applies to LLM judges only, like system_prompt. TYPE: str | None DEFAULT: None
length_penalty	Optional length penalty configuration. TYPE: LengthPenalty | None DEFAULT: None
normalize	If True, normalize score to [0, 1]. If False, return raw sum. TYPE: bool DEFAULT: True
cannot_assess_config	Configuration for handling CANNOT_ASSESS verdicts. TYPE: CannotAssessConfig | None DEFAULT: None
shuffle_options	If True (default), randomize the order of multi-choice options presented to the LLM to mitigate position bias. Each judge/call sees a different random order, and responses are mapped back to original indices. Disable for deterministic behavior in tests. Applies to LLM judges only: a decision model's question offers the options in the rubric's order, and in a mixed ensemble the LLM judges shuffle as set here. TYPE: bool DEFAULT: True
auto_na_option	If True (default), auto-inject a canonical NA / "cannot assess" option into any multi-choice criterion that lacks one, giving the judge a first-class abstain channel analogous to binary CANNOT_ASSESS. The injected option is appended at the end (highest index) so existing option indices are preserved. Set False for forced-choice classification (the judge must pick a scored option). Never strips an author-supplied NA option — author intent wins. TYPE: bool DEFAULT: True
seed	Master seed for all non-LLM randomness (option shuffling, few-shot example selection). Auto-generated when None so that randomness is always pinned and reproducible. Inspect via the seed property after construction. TYPE: int | None DEFAULT: None
binary_response_format	Pydantic model to use as the structured output schema for binary criterion judgments. Must be a subclass of (or compatible with) CriterionJudgment. If the model includes an affected_criteria field (list[int]), matching indices are injected as an [Affects: ...] tag into the reason string. Defaults to CriterionJudgment. LLM judges only: it describes a generated judgment, which a decision model does not produce, so it cannot be combined with any decision-model judge. TYPE: type[BaseModel] | None DEFAULT: None
multi_choice_response_format	Pydantic model to use as the structured output schema for multi-choice criterion judgments. Must be a subclass of (or compatible with) MultiChoiceJudgment. If the model includes an affected_criteria field (list[int]), matching indices are injected as an [Affects: ...] tag into the reason string (same convention as binary_response_format). Defaults to MultiChoiceJudgment. LLM judges only, like binary_response_format. TYPE: type[BaseModel] | None DEFAULT: None
escalation	Makes the grader a confidence cascade (see the class docstring): the escalation judges, all LLMs, vote only on the criteria the decision model errored or abstained on or answered with a confidence below the criterion's threshold. The grader's one judge (judge_model_config, or a single JudgeSpec in judges) must then be a decision model. The escalation judges count as judges for the LLM-only settings: few-shot examples, the system prompts, shuffle_options and llm_calls apply to them. The grader validates and keeps its own copy of the config (dataclasses.replace), so changing the config's lists in place later does not affect it. TYPE: EscalationConfig | None DEFAULT: None
llm_calls	How each LLM judge, a cascade's escalation judges included, is called for an item: "per_criterion" (the default), one call per criterion, or "per_item", one call for the whole rubric (see the class docstring). An escalation judge calls only where the decision model escalated: once per escalated criterion, or once, about the whole rubric, for an item with any escalated criterion, keeping its judgments of the escalated criteria only. "per_item" sends the submission once instead of once per criterion but generates every criterion's judgment in one reply, so max_tokens and timeout must allow for the whole rubric's answer: a reply cut off at max_tokens fails every criterion. It also shapes few-shot examples: under "per_item" they are whole training items, each labelled on every criterion (see training_data). Applies to LLM judges only; a decision model always makes one request per item. The name counts LLM calls per item; it is unrelated to compute_metrics(per_item_metrics=...) and to per-item rubrics. TYPE: LLMCalls DEFAULT: 'per_criterion'
llm_config	Deprecated alias of judge_model_config; builds the same single judge and emits a DeprecationWarning. TYPE: LLMConfig | DecisionModelConfig | None DEFAULT: None

RAISES	DESCRIPTION
ValueError	If neither judge_model_config nor judges is provided, if both are provided, or if both judge_model_config and its deprecated alias llm_config are provided; if escalation is given while the grader's judges are not exactly one decision model, with a judge_id that repeats among the decision model and the escalation judges, or with a config that no longer validates (its judges list or per_criterion dict changed in place since it was built, e.g. emptied); if llm_calls is not "per_criterion" or "per_item"; if few_shot_config or training_data is given while every judge is a decision model (few-shot examples apply to LLM judges only), or training_data has no global rubric; if binary_response_format or multi_choice_response_format is given while any judge is a decision model, or with llm_calls="per_item"; or if a decision-model judge has no API key, has a key the SDK would reject (anything but printable ASCII without whitespace), or has a resolved base URL (api_base or the TYPESAFE_BASE_URL environment variable) that is not an absolute http(s) URL free of whitespace and control characters, query, fragment and credentials, or whose host the SDK's HTTP stack cannot encode (e.g. a non-ASCII name that is not a valid internationalized domain name). Such a judge fails here, never as an abstention or a worst case on every item.
ImportError	If a judge is a decision model and the TypeSafe SDK is not installed (pip install 'autorubric[typesafe]').

WARNS	DESCRIPTION
UserWarning	If system_prompt or multi_choice_system_prompt is not None, shuffle_options is False, or llm_calls is "per_item", while every judge is a decision model: these settings apply to LLM judges only, so they have no effect. One warning per setting that differs from its default (None, None, True and "per_criterion"); a cascade's escalation judges are LLM judges, so a cascade never warns.
FutureWarning	If a judge_id repeats among judges. Judges that share one are conflated: those of one kind (LLM or decision model) all call the model of the last of them, and all share one judge_scores entry, one set of per-judge metrics, and the same option shuffles and few-shot examples. Grading is otherwise unchanged; a repeat will be a ValueError in the next major version, as it already is in a cascade.

Source code in src/autorubric/graders/criterion_grader.py

def __init__(
    self,
    *,
    # Single-judge mode
    judge_model_config: LLMConfig | DecisionModelConfig | None = None,
    # Ensemble mode (mutually exclusive with judge_model_config)
    judges: list[JudgeSpec] | None = None,
    aggregation: AggregationStrategy = "majority",
    # Multi-choice aggregation strategies
    ordinal_aggregation: OrdinalAggregation = "mean",
    nominal_aggregation: NominalAggregation = "mode",
    # Few-shot mode (orthogonal - applies to every LLM judge)
    training_data: RubricDataset | None = None,
    few_shot_config: FewShotConfig | None = None,
    # Common parameters
    system_prompt: str | None = None,
    multi_choice_system_prompt: str | None = None,
    length_penalty: LengthPenalty | None = None,
    normalize: bool = True,
    cannot_assess_config: CannotAssessConfig | None = None,
    # Position bias mitigation
    shuffle_options: bool = True,
    # Multi-choice abstain channel
    auto_na_option: bool = True,
    # Reproducibility
    seed: int | None = None,
    # Structured output override for binary criteria
    binary_response_format: type[BaseModel] | None = None,
    # Structured output override for multi-choice criteria
    multi_choice_response_format: type[BaseModel] | None = None,
    # Confidence cascade: LLM judges behind a decision model
    escalation: EscalationConfig | None = None,
    # LLM calls per item: one per criterion, or one for the whole rubric
    llm_calls: LLMCalls = "per_criterion",
    # Deprecated alias of judge_model_config (every parameter is keyword-only, so
    # its position at the end changes no call)
    llm_config: LLMConfig | DecisionModelConfig | None = None,
):
    """Initialize the criterion grader.

    Args:
        judge_model_config: Configuration of the single judge (single-judge mode), which
            is given ``judge_id="default"``: an ``LLMConfig`` for an LLM judge or a
            ``DecisionModelConfig`` for a decision-model judge. Mutually exclusive with
            judges.
        judges: List of JudgeSpec for ensemble mode, each an LLM or a decision model,
            each with its own ``judge_id``. Mutually exclusive with judge_model_config.
        aggregation: Strategy for aggregating votes in ensemble mode (binary criteria).
        ordinal_aggregation: Strategy for aggregating ordinal multi-choice votes.
            Central tendency: "mean", "median", "weighted_mean", "mode". Conservative/
            permissive (analogs of binary unanimous/any): "min" (lowest selected
            option) / "max" (highest selected option).
        nominal_aggregation: Strategy for aggregating nominal multi-choice votes.
            Options: "mode", "weighted_mode", "unanimous". "unanimous" abstains via the
            NA option on disagreement, or falls back to mode + warns if there is no NA
            option.
        training_data: Dataset for few-shot examples. If provided, enables few-shot
            prompting for the LLM judges; examples are selected for, and sent to, LLM
            judges only, never to a decision-model judge, whose request is exactly what
            it would be without them. An example shows its item's ground truth for the
            criterion and, when ``few_shot_config.include_reason`` is True, the item's
            written reason for it (``DataItem.ground_truth_reasons``), if any. Called per
            criterion, a judge is shown examples drawn for each criterion separately.
            Under ``llm_calls="per_item"`` an example is a whole training item, shown
            once in the judge's one call per item with its ground truth (and reason) for
            every criterion; only an item whose ground truth resolves for every
            criterion (a verdict for a binary criterion, an option for a multi-choice
            one) is eligible. Either way each judge draws its own examples, and the
            training data needs a global rubric, which says what each label means.
        few_shot_config: Configuration for few-shot example selection (LLM judges only).
            Under ``llm_calls="per_item"``, ``n_examples`` counts the training items
            each call shows and ``balance_verdicts`` selects them by greedy label
            coverage (see ``FewShotConfig``).
        system_prompt: Custom system prompt for binary criteria. Applies to LLM judges
            only (a decision model's request has no system prompt); in a mixed ensemble
            the LLM judges use it.
        multi_choice_system_prompt: Custom system prompt for multi-choice criteria. Applies
            to LLM judges only, like ``system_prompt``.
        length_penalty: Optional length penalty configuration.
        normalize: If True, normalize score to [0, 1]. If False, return raw sum.
        cannot_assess_config: Configuration for handling CANNOT_ASSESS verdicts.
        shuffle_options: If True (default), randomize the order of multi-choice options
            presented to the LLM to mitigate position bias. Each judge/call sees a
            different random order, and responses are mapped back to original indices.
            Disable for deterministic behavior in tests. Applies to LLM judges only: a
            decision model's question offers the options in the rubric's order, and in a
            mixed ensemble the LLM judges shuffle as set here.
        auto_na_option: If True (default), auto-inject a canonical NA / "cannot assess"
            option into any multi-choice criterion that lacks one, giving the judge a
            first-class abstain channel analogous to binary CANNOT_ASSESS. The injected
            option is appended at the end (highest index) so existing option indices are
            preserved. Set False for forced-choice classification (the judge must pick a
            scored option). Never strips an author-supplied NA option — author intent wins.
        seed: Master seed for all non-LLM randomness (option shuffling, few-shot
            example selection). Auto-generated when None so that randomness is always
            pinned and reproducible. Inspect via the ``seed`` property after construction.
        binary_response_format: Pydantic model to use as the structured output schema
            for binary criterion judgments. Must be a subclass of (or compatible with)
            CriterionJudgment. If the model includes an ``affected_criteria`` field
            (list[int]), matching indices are injected as an ``[Affects: ...]`` tag
            into the reason string. Defaults to CriterionJudgment. LLM judges only: it
            describes a generated judgment, which a decision model does not produce, so
            it cannot be combined with any decision-model judge.
        multi_choice_response_format: Pydantic model to use as the structured output
            schema for multi-choice criterion judgments. Must be a subclass of (or
            compatible with) MultiChoiceJudgment. If the model includes an
            ``affected_criteria`` field (list[int]), matching indices are injected as
            an ``[Affects: ...]`` tag into the reason string (same convention as
            binary_response_format). Defaults to MultiChoiceJudgment. LLM judges only,
            like binary_response_format.
        escalation: Makes the grader a confidence cascade (see the class docstring):
            the escalation judges, all LLMs, vote only on the criteria the decision
            model errored or abstained on or answered with a confidence below the
            criterion's threshold. The grader's one judge (``judge_model_config``, or a
            single ``JudgeSpec`` in ``judges``) must then be a decision model. The
            escalation judges count as judges for the LLM-only settings: few-shot
            examples, the system prompts, ``shuffle_options`` and ``llm_calls`` apply to
            them. The grader validates and keeps its own copy of the config
            (``dataclasses.replace``), so changing the config's lists in place later
            does not affect it.
        llm_calls: How each LLM judge, a cascade's escalation judges included, is called
            for an item: ``"per_criterion"`` (the default), one call per criterion, or
            ``"per_item"``, one call for the whole rubric (see the class docstring). An
            escalation judge calls only where the decision model escalated: once per
            escalated criterion, or once, about the whole rubric, for an item with any
            escalated criterion, keeping its judgments of the escalated criteria only.
            ``"per_item"`` sends the submission once instead of once per criterion but
            generates every criterion's judgment in one reply, so ``max_tokens`` and
            ``timeout`` must allow for the whole rubric's answer: a reply cut off at
            ``max_tokens`` fails every criterion. It also shapes few-shot examples: under
            ``"per_item"`` they are whole training items, each labelled on every
            criterion (see ``training_data``). Applies to LLM judges only; a decision
            model always makes one request per item. The name counts LLM calls per item;
            it is unrelated to ``compute_metrics(per_item_metrics=...)`` and to per-item
            rubrics.
        llm_config: Deprecated alias of ``judge_model_config``; builds the same single
            judge and emits a ``DeprecationWarning``.

    Raises:
        ValueError: If neither judge_model_config nor judges is provided, if both are
            provided, or if both judge_model_config and its deprecated alias
            llm_config are provided; if escalation is given while the grader's judges
            are not exactly one decision model, with a ``judge_id`` that repeats among
            the decision model and the escalation judges, or with a config that no
            longer validates (its ``judges`` list or ``per_criterion`` dict changed in
            place since it was built, e.g. emptied); if llm_calls is not
            ``"per_criterion"`` or ``"per_item"``; if few_shot_config or training_data is
            given while every judge is a decision model (few-shot examples apply to LLM
            judges only), or training_data has no global rubric; if
            binary_response_format or multi_choice_response_format is given while any
            judge is a decision model, or with ``llm_calls="per_item"``;
            or if a decision-model judge has no API key, has a key the SDK would reject
            (anything but printable ASCII without whitespace), or has a resolved base
            URL (``api_base`` or the ``TYPESAFE_BASE_URL`` environment variable) that is
            not an absolute http(s) URL free of whitespace and control characters,
            query, fragment and credentials, or whose host the SDK's HTTP stack cannot
            encode (e.g. a non-ASCII name that is not a valid internationalized domain
            name). Such a judge fails here, never as an abstention or a worst case on
            every item.
        ImportError: If a judge is a decision model and the TypeSafe SDK is not
            installed (``pip install 'autorubric[typesafe]'``).

    Warns:
        UserWarning: If system_prompt or multi_choice_system_prompt is not None,
            shuffle_options is False, or llm_calls is ``"per_item"``, while every judge
            is a decision model: these settings apply to LLM judges only, so they have
            no effect. One warning per setting that differs from its default (None,
            None, True and ``"per_criterion"``); a cascade's escalation judges are LLM
            judges, so a cascade never warns.
        FutureWarning: If a ``judge_id`` repeats among ``judges``. Judges that share
            one are conflated: those of one kind (LLM or decision model) all call the
            model of the last of them, and all share one ``judge_scores`` entry, one set
            of per-judge metrics, and the same option shuffles and few-shot examples.
            Grading is otherwise unchanged; a repeat will be a ``ValueError`` in the
            next major version, as it already is in a cascade.
    """
    if llm_config is not None:
        if judge_model_config is not None:
            raise ValueError(
                "Pass only one of judge_model_config and llm_config "
                "(llm_config is a deprecated alias of judge_model_config)"
            )
        warnings.warn(
            "llm_config is deprecated; use judge_model_config",
            DeprecationWarning,
            stacklevel=2,
        )
        judge_model_config = llm_config

    super().__init__(length_penalty=length_penalty, normalize=normalize)

    # Validate: must have either judge_model_config or judges, not both, not neither
    if judge_model_config is None and judges is None:
        raise ValueError("Must provide either judge_model_config or judges")
    if judge_model_config is not None and judges is not None:
        raise ValueError("Cannot provide both judge_model_config and judges")

    # Normalize to ensemble representation (single judge = ensemble of 1).
    # The validation above guarantees exactly one of judge_model_config/judges is set,
    # so by this point `judges` in the else branch is a non-None list[JudgeSpec]; the
    # explicit attribute annotation lets the type checker see that without a
    # suppression comment.
    self._judges: list[JudgeSpec]
    if judge_model_config is not None:
        self._judges = [
            JudgeSpec(
                judge_model_config=judge_model_config, judge_id=_SINGLE_JUDGE_ID, weight=1.0
            )
        ]
    else:
        assert judges is not None
        self._judges = judges

    # The grader's own copy of the cascade config, validated again: the config's list and
    # dict can have been changed in place since it was built, and can be again later.
    # Everything the cascade does reads this copy. Its escalation judges
    # (``_escalation_judges``) are kept apart from ``_judges``, which judge every
    # criterion: they judge only the criteria the decision model escalates. Every other
    # per-judge concern (clients, few-shot examples, the LLM-only settings, the
    # manifest, ``judge_ids``) covers both lists.
    self._escalation = dataclasses.replace(escalation) if escalation is not None else None
    if self._escalation is not None:
        primary = self._judges
        if len(primary) != 1 or not isinstance(primary[0].llm_config, DecisionModelConfig):
            described = [
                f"{j.judge_id!r} (a decision model)"
                if isinstance(j.llm_config, DecisionModelConfig)
                else f"{j.judge_id!r} (an LLM)"
                for j in primary
            ]
            raise ValueError(
                "escalation needs exactly one primary judge, a decision model "
                "(judge_model_config=DecisionModelConfig(...), or judges=[JudgeSpec("
                "DecisionModelConfig(...), ...)]), which judges every criterion and "
                f"escalates the ones it is unsure of; got {', '.join(described)}"
            )
        repeated = _repeated_judge_ids((*primary, *self._escalation_judges))
        if repeated:
            raise ValueError(
                "judge_ids must be unique across the decision model and the escalation "
                f"judges; repeated: {', '.join(repr(judge_id) for judge_id in repeated)}"
            )
    # A panel's judge_ids should be unique too, but a repeat has always been accepted,
    # so it warns until the next major version makes it the cascade's ValueError. A
    # cascade's primary is one judge, so this never warns for a cascade.
    repeated = _repeated_judge_ids(self._judges)
    if repeated:
        warnings.warn(
            f"judge_ids should be unique; repeated: {', '.join(map(repr, repeated))}. "
            "Judges that share a judge_id are conflated: those of one kind (LLM or "
            "decision model) all call the model of the last of them, and all share one "
            "judge_scores entry, one set of per-judge metrics, and the same option "
            "shuffles and few-shot examples. Give each judge its own judge_id; a "
            "repeated judge_id will be a ValueError in the next major version.",
            FutureWarning,
            stacklevel=2,
        )
    all_judges = [*self._judges, *self._escalation_judges]

    # Checked before any client is built, as the settings below are.
    _check_llm_calls(llm_calls)

    # Settings a decision-model judge cannot use, checked before any client is built so
    # that a configuration error is reported as one.
    decision_model_ids = [
        j.judge_id for j in all_judges if isinstance(j.llm_config, DecisionModelConfig)
    ]
    every_judge_is_decision_model = 0 < len(decision_model_ids) == len(all_judges)
    if every_judge_is_decision_model and (
        few_shot_config is not None or training_data is not None
    ):
        raise ValueError(
            "few-shot examples apply to LLM judges only, and every judge of this "
            "grader is a decision model: remove few_shot_config and training_data, "
            "or add an LLM judge"
        )
    response_formats = [
        name
        for name, value in (
            ("binary_response_format", binary_response_format),
            ("multi_choice_response_format", multi_choice_response_format),
        )
        if value is not None
    ]
    if response_formats and decision_model_ids:
        raise ValueError(
            f"{' and '.join(response_formats)} cannot be combined with decision-model "
            f"judges ({', '.join(repr(judge_id) for judge_id in decision_model_ids)}): "
            "a response format describes a generated judgment, which only an LLM judge "
            "produces"
        )
    if response_formats and llm_calls == "per_item":
        raise ValueError(
            f"{' and '.join(response_formats)} cannot be combined with "
            "llm_calls='per_item': a response format describes one criterion's judgment, "
            "and a per_item call answers every criterion at once"
        )
    if every_judge_is_decision_model:
        # A setting that differs from its default asks for something no judge here
        # can do: a prompt (None is "no prompt"), shuffle_options=False (a decision
        # model never shuffles, so True, the default, is what it does anyway), or
        # llm_calls="per_item" (it changes how LLM judges are called, and there are none).
        changed = (
            ("system_prompt", system_prompt is not None),
            ("multi_choice_system_prompt", multi_choice_system_prompt is not None),
            ("shuffle_options", not shuffle_options),
            ("llm_calls", llm_calls != "per_criterion"),
        )
        for name, is_changed in changed:
            if is_changed:
                warnings.warn(
                    f"{name} applies to LLM judges only; this grader has none (every "
                    "judge is a decision model), so it has no effect",
                    UserWarning,
                    stacklevel=2,
                )

    self._aggregation = aggregation
    self._ordinal_aggregation = ordinal_aggregation
    self._nominal_aggregation = nominal_aggregation
    self._training_data = training_data
    self._cannot_assess_config = cannot_assess_config or CannotAssessConfig()
    self._shuffle_options = shuffle_options
    self._auto_na_option = auto_na_option
    self._llm_calls = llm_calls
    self._seed = seed if seed is not None else random.randint(0, 2**31 - 1)

    # Coordinate few-shot seed with master seed when unset
    fsc = few_shot_config or FewShotConfig()
    if fsc.seed is None and training_data is not None:
        fsc = dataclasses.replace(fsc, seed=self._seed)
    self._few_shot_config = fsc
    self._binary_response_format = binary_response_format or CriterionJudgment
    self._multi_choice_response_format = multi_choice_response_format or MultiChoiceJudgment

    # Build system prompts (separate for binary and multi-choice)
    if system_prompt is None:
        self._system_prompt = GRADER_SYSTEM_PROMPT_DEFAULT
        if training_data is not None:
            self._system_prompt += FEW_SHOT_SYSTEM_PROMPT_ADDITION
    else:
        self._system_prompt = system_prompt

    if multi_choice_system_prompt is None:
        self._multi_choice_system_prompt = MULTI_CHOICE_SYSTEM_PROMPT
        if training_data is not None:
            self._multi_choice_system_prompt += MULTI_CHOICE_FEW_SHOT_ADDITION
    else:
        self._multi_choice_system_prompt = multi_choice_system_prompt

    # Create each judge's client: an LLMClient for an LLM judge, a DecisionModelClient
    # for a decision-model judge. Building one resolves its credentials, so a missing
    # SDK or API key fails here rather than on every item.
    self._clients: dict[str, LLMClient] = {}
    self._decision_clients: dict[str, DecisionModelClient] = {}
    for judge in all_judges:
        # A spec is mutable, so check the weight it has now, not only the one it was built with
        _check_judge_weight(judge.weight, judge.judge_id)
        config = judge.llm_config
        # Each judge of an ensemble or cascade caches under its own id, so judges sending
        # identical requests never read one another's answers; the lone judge keeps the
        # plain key, and with it the caches of earlier single-judge runs.
        namespace = None if judge.judge_id == _SINGLE_JUDGE_ID else judge.judge_id
        if isinstance(config, DecisionModelConfig):
            self._decision_clients[judge.judge_id] = DecisionModelClient(
                config, cache_namespace=namespace
            )
        else:
            self._clients[judge.judge_id] = LLMClient(config, cache_namespace=namespace)

    # Pre-compute few-shot examples if training data provided
    # Note: For multi-choice, examples are stored as (submission, selected_index, reason)
    self._criterion_examples: dict[tuple[int, str], list[FewShotExample]] = {}
    self._multi_choice_examples: dict[tuple[int, str], list[tuple[str, int, str | None]]] = {}
    # Under llm_calls="per_item": each LLM judge's examples, whole training items in the
    # order its one call per item shows them, each as (submission, labels, reasons): a
    # label per criterion (_item_labels) and the item's reason for it, if any
    self._item_examples: dict[
        str, list[tuple[str, list[CriterionVerdict | int], list[str | None]]]
    ] = {}
    if training_data is not None:
        self._prepare_examples()

is_ensemble property

is_ensemble: bool

Whether this grader has several judges that each judge every criterion.

A cascade is not an ensemble: its escalation judges vote only on the criteria its decision model escalates.

judge_ids property

judge_ids: list[str]

The judge_id of every judge, in order: the judges that judge every criterion (judges, or the single "default" judge), then a cascade's escalation judges.

has_few_shot property

has_few_shot: bool

Whether this grader uses few-shot prompting.

seed property

seed: int

The master seed governing all non-LLM randomness in this grader.

judge async

judge(to_grade: str, rubric: list[Criterion], query: str | None = None, reference_submission: str | None = None, *, guidelines: str | None = None) -> list[JudgeCriterionResults]

Judge all criteria with all judges (parallel across judges).

In a cascade the decision model judges every criterion first; then every escalation judge judges the escalated criteria (_escalates: an error, an abstention, or a confidence below EscalationConfig.threshold_for the criterion), all at once, and none when nothing is escalated: in a call per escalated criterion, or, under llm_calls="per_item", in one call about the whole rubric whose judgments of the other criteria are discarded (_judge_escalated_criteria).

PARAMETER	DESCRIPTION
to_grade	The submission, as Grader.grade passes it. TYPE: str
rubric	The criteria. TYPE: list[Criterion]
query	Optional input/query that prompted the submission. TYPE: str | None DEFAULT: None
reference_submission	Optional exemplar response for grading context. TYPE: str | None DEFAULT: None
guidelines	Optional rubric guidelines (Grader.grade passes them when the rubric has any; blank text means none). Every judge sees them: each LLM user prompt starts with GUIDELINES_BLOCK, and a decision model's state holds them as its first field, with the framed binary questions naming them. With none, every prompt and request is exactly what it is without guidelines. TYPE: str | None DEFAULT: None

RETURNS	DESCRIPTION
list[JudgeCriterionResults]	One JudgeCriterionResults per judge, in judge order: in a cascade, the
list[JudgeCriterionResults]	decision model's (role="primary"), then each escalation judge's
list[JudgeCriterionResults]	(role="escalation", None at the criteria not escalated).

RAISES	DESCRIPTION
TypeError	If guidelines is neither a str nor None.

WARNS	DESCRIPTION
UserWarning	Once per grader, when an EscalationConfig.per_criterion name matches no criterion of rubric (not while EvalRunner or fill_ground_truth grades a dataset, which checks the names against every item's rubric instead).

Source code in src/autorubric/graders/criterion_grader.py

async def judge(
    self,
    to_grade: str,
    rubric: list[Criterion],
    query: str | None = None,
    reference_submission: str | None = None,
    *,
    guidelines: str | None = None,
) -> list[JudgeCriterionResults]:
    """Judge all criteria with all judges (parallel across judges).

    In a cascade the decision model judges every criterion first; then every escalation
    judge judges the escalated criteria (``_escalates``: an error, an abstention, or a
    confidence below ``EscalationConfig.threshold_for`` the criterion), all at once, and
    none when nothing is escalated: in a call per escalated criterion, or, under
    ``llm_calls="per_item"``, in one call about the whole rubric whose judgments of the
    other criteria are discarded (``_judge_escalated_criteria``).

    Args:
        to_grade: The submission, as ``Grader.grade`` passes it.
        rubric: The criteria.
        query: Optional input/query that prompted the submission.
        reference_submission: Optional exemplar response for grading context.
        guidelines: Optional rubric guidelines (``Grader.grade`` passes them when the
            rubric has any; blank text means none). Every judge sees them: each LLM user
            prompt starts with ``GUIDELINES_BLOCK``, and a decision model's state holds
            them as its first field, with the framed binary questions naming them. With
            none, every prompt and request is exactly what it is without guidelines.

    Returns:
        One ``JudgeCriterionResults`` per judge, in judge order: in a cascade, the
        decision model's (``role="primary"``), then each escalation judge's
        (``role="escalation"``, ``None`` at the criteria not escalated).

    Raises:
        TypeError: If ``guidelines`` is neither a ``str`` nor ``None``.

    Warns:
        UserWarning: Once per grader, when an ``EscalationConfig.per_criterion`` name
            matches no criterion of ``rubric`` (not while ``EvalRunner`` or
            ``fill_ground_truth`` grades a dataset, which checks the names against
            every item's rubric instead).
    """
    # Checked before any judge runs; blank guidelines become None.
    guidelines = _normalize_guidelines(guidelines)
    if not self._escalation_names_warned and not _ESCALATION_NAMES_CHECKED.get():
        # The flag is set only once the warning is issued, so under a
        # warnings-as-errors filter it raises on every call instead of only once.
        if self._check_escalation_names([rubric], "the rubric being graded"):
            self._escalation_names_warned = True
    # Normalize once to the effective rubric (abstain channel guaranteed for
    # multi-choice under auto_na_option). Same length/order, so criterion_idx — and
    # thus the shuffle RNG key — stays aligned; the user's rubric is never mutated.
    effective_rubric = [self._effective_criterion(c) for c in rubric]
    tasks = [
        self._judge_all_criteria_for_judge(
            judge,
            effective_rubric,
            to_grade,
            query,
            reference_submission,
            guidelines=guidelines,
        )
        for judge in self._judges
    ]
    results = list(await asyncio.gather(*tasks))
    escalation = self._escalation
    if escalation is None:
        return results

    # A cascade: ``_judges`` is its one decision model, which judged every criterion.
    (primary,) = results
    escalated = [
        criterion_idx
        for criterion_idx, (criterion, report) in enumerate(
            zip(effective_rubric, primary.reports, strict=True)
        )
        if _escalates(report, escalation.threshold_for(criterion))
    ]
    escalation_results = await asyncio.gather(
        *(
            self._judge_escalated_criteria(
                judge,
                effective_rubric,
                escalated,
                to_grade,
                query,
                reference_submission,
                guidelines=guidelines,
            )
            for judge in self._escalation_judges
        )
    )
    return [primary, *escalation_results]

aggregate async

aggregate(judge_results: list[JudgeCriterionResults], *, normalize: bool = True) -> EnsembleEvaluationReport

Aggregate results from all judges into final report.

Handles both binary and multi-choice criteria: - Binary: Uses JudgeVote and _aggregate_votes() - Multi-choice: Uses MultiChoiceJudgeVote and _aggregate_multi_choice_votes()

In a cascade the escalated criteria are the ones the escalation judges have results for (their results are None everywhere else). On those, the primary judge's vote is marked superseded=True and the final verdict, reason and error come from the votes that are not superseded, which are the escalation judges'; the report is marked escalated=True. judge_scores holds each primary judge's score over its own verdicts (superseded ones included) and None for each escalation judge, by role: it votes only where the decision model escalated, even on an item where every criterion escalated.

Source code in src/autorubric/graders/criterion_grader.py

async def aggregate(
    self, judge_results: list[JudgeCriterionResults], *, normalize: bool = True
) -> EnsembleEvaluationReport:
    """Aggregate results from all judges into final report.

    Handles both binary and multi-choice criteria:
    - Binary: Uses JudgeVote and _aggregate_votes()
    - Multi-choice: Uses MultiChoiceJudgeVote and _aggregate_multi_choice_votes()

    In a cascade the escalated criteria are the ones the escalation judges have results
    for (their results are ``None`` everywhere else). On those, the primary judge's vote
    is marked ``superseded=True`` and the final verdict, reason and error come from the
    votes that are not superseded, which are the escalation judges'; the report is
    marked ``escalated=True``. ``judge_scores`` holds each primary judge's score over its
    own verdicts (superseded ones included) and ``None`` for each escalation judge, by
    role: it votes only where the decision model escalated, even on an item where every
    criterion escalated.
    """
    if not judge_results:
        # Empty/failed aggregation has no score: emit None, not a fabricated 0.0
        # (which is a valid catastrophic score, indistinguishable from a real zero).
        return EnsembleEvaluationReport(
            score=None,
            raw_score=None,
            llm_raw_score=None,
            error="No judge results to aggregate",
        )

    n_criteria = len(judge_results[0].criterion_results)
    # A cascade's escalated criteria: an escalation judge has a result at each of them,
    # and only there (none at all without a cascade).
    escalated = {
        criterion_idx
        for judge_result in judge_results
        if judge_result.role == "escalation"
        for criterion_idx, cr in enumerate(judge_result.criterion_results)
        if cr is not None
    }

    # Build ensemble criterion reports
    ensemble_reports: list[EnsembleCriterionReport] = []
    for criterion_idx in range(n_criteria):
        is_escalated = criterion_idx in escalated
        # Each judge's result on this criterion, in judge order; an escalation judge
        # has one only where the criterion was escalated.
        judged: list[tuple[JudgeCriterionResults, CriterionResult]] = [
            (judge_result, cr)
            for judge_result in judge_results
            if (cr := judge_result.criterion_results[criterion_idx]) is not None
        ]
        # Get criterion from the first judge's result
        criterion_report = judged[0][1].report

        if criterion_report.is_multi_choice:
            # Multi-choice: build MultiChoiceJudgeVote list
            mc_votes: list[MultiChoiceJudgeVote] = []
            # Each MultiChoiceJudgeVote carries its own .error (parity with JudgeVote),
            # so the ensemble error is derived from the votes via _aggregate_error,
            # mirroring the binary path. Every judge call synthesizes a verdict, so the
            # `mcv is not None` guard below never drops an errored vote.
            for judge_result, cr in judged:
                mcv = cr.report.multi_choice_verdict
                if mcv is not None:
                    mc_votes.append(
                        MultiChoiceJudgeVote(
                            judge_id=judge_result.judge_id,
                            selected_index=mcv.selected_index,
                            selected_label=mcv.selected_label,
                            value=mcv.value,
                            reason=cr.report.reason,
                            weight=judge_result.weight,
                            na=mcv.na,
                            shuffle_order=cr.report.shuffle_order,
                            error=cr.report.error,
                            reasoning=cr.report.reasoning,
                            probabilities=cr.report.probabilities,
                            confidence=cr.report.confidence,
                            superseded=is_escalated and judge_result.role == "primary",
                        )
                    )
            aggregated_mc_votes = [v for v in mc_votes if not v.superseded]

            # Aggregate multi-choice votes
            final_mc_verdict, final_reason = self._aggregate_multi_choice_votes(
                aggregated_mc_votes, criterion_report
            )

            ensemble_reports.append(
                EnsembleCriterionReport(
                    criterion=Criterion(
                        weight=criterion_report.weight,
                        requirement=criterion_report.requirement,
                        name=criterion_report.name,
                        options=criterion_report.options,
                        scale_type=criterion_report.scale_type,
                        aggregation=criterion_report.aggregation,
                    ),
                    final_verdict=None,  # Binary verdict is None for multi-choice
                    final_reason=final_reason,
                    votes=[],  # Binary votes empty for multi-choice
                    final_multi_choice_verdict=final_mc_verdict,
                    multi_choice_votes=mc_votes,
                    error=_aggregate_error(aggregated_mc_votes),
                    escalated=is_escalated,
                )
            )
        else:
            # Binary: build JudgeVote list
            votes: list[JudgeVote] = []
            for judge_result, cr in judged:
                votes.append(
                    JudgeVote(
                        judge_id=judge_result.judge_id,
                        verdict=cr.report.verdict,
                        reason=cr.report.reason,
                        weight=judge_result.weight,
                        error=cr.report.error,
                        reasoning=cr.report.reasoning,
                        probabilities=cr.report.probabilities,
                        confidence=cr.report.confidence,
                        superseded=is_escalated and judge_result.role == "primary",
                    )
                )
            aggregated_votes = [v for v in votes if not v.superseded]

            final_verdict, final_reason = self._aggregate_votes(
                aggregated_votes, criterion_report.weight
            )

            ensemble_reports.append(
                EnsembleCriterionReport(
                    criterion=Criterion(
                        weight=criterion_report.weight,
                        requirement=criterion_report.requirement,
                        name=criterion_report.name,
                    ),
                    final_verdict=final_verdict,
                    final_reason=final_reason,
                    votes=votes,
                    error=_aggregate_error(aggregated_votes),
                    escalated=is_escalated,
                )
            )

    # Calculate per-judge scores: a primary judge's over its own verdicts on every
    # criterion; an escalation judge's is undefined, None by role on every item. A
    # judge none of whose votes on the item it judged (every call failed) has no score
    # for it either, whatever its stand-in verdicts would score.
    judge_scores: dict[str, float | None] = {}
    for judge_result in judge_results:
        judge_scores[judge_result.judge_id] = (
            self._calculate_score_from_reports(judge_result.reports, normalize)
            if judge_result.role == "primary"
            and not _every_judgment_failed(judge_result.reports)
            else None
        )

    # Aggregate token usage and cost
    total_usage = TokenUsage()
    total_cost = 0.0
    for jr in judge_results:
        if jr.total_usage:
            total_usage = total_usage + jr.total_usage
        if jr.total_cost:
            total_cost += jr.total_cost

    return _ensemble_evaluation_report(
        ensemble_reports,
        judge_scores,
        self._calculate_score_from_reports,
        normalize=normalize,
        token_usage=total_usage if total_usage.total_tokens > 0 else None,
        completion_cost=total_cost if total_cost > 0 else None,
    )

---

Grader

Abstract base class for grader implementations.

Grader is generic in the report type it returns. CriterionGrader is a Grader[EnsembleEvaluationReport], so a type checker types await rubric.grade(..., grader=grader) as EnsembleEvaluationReport. A custom grader declares its own, for example class MyGrader(Grader[EvaluationReport]); one that names no type still works.

Grader

Grader(*, length_penalty: LengthPenalty | None = None, normalize: bool = True)

Bases: ABC, Generic[ReportT]

Base class for LLM-backed grading implementations.

All graders require an LLMConfig for the LLM client. Subclasses must implement judge() and aggregate() methods.

Rubric guidelines reach judge only if it accepts them: a judge with a guidelines parameter that can be passed by keyword (e.g. keyword-only guidelines: str | None = None) or with **kwargs receives them as guidelines=. A judge with the abstract signature keeps working unchanged; the grader then grades without the guidelines and warns once (see grade).

PARAMETER	DESCRIPTION
length_penalty	Optional configuration for penalizing overly long outputs. When provided, a penalty based on the token/word count is subtracted from the final score. TYPE: LengthPenalty | None DEFAULT: None
normalize	If True (default), scores are normalized to 0-1. If False, raw weighted sums are returned, which is useful for RL training scenarios. TYPE: bool DEFAULT: True

Source code in src/autorubric/graders/base.py

def __init__(
    self,
    *,
    length_penalty: LengthPenalty | None = None,
    normalize: bool = True,
):
    self.length_penalty: LengthPenalty | None = length_penalty
    self.normalize: bool = normalize

judge abstractmethod async

judge(to_grade: str, rubric: list[Criterion], query: str | None = None, reference_submission: str | None = None) -> Any

Collect raw judge results for the provided submission.

An implementation may also take the rubric's guidelines, as a guidelines parameter that can be passed by keyword or through **kwargs; grade passes them, when there are any, only to an implementation that does.

PARAMETER	DESCRIPTION
to_grade	The text to evaluate. TYPE: str
rubric	List of criteria to evaluate against. TYPE: list[Criterion]
query	Optional input/query that prompted the response. TYPE: str | None DEFAULT: None
reference_submission	Optional exemplar response for grading context. TYPE: str | None DEFAULT: None

RETURNS	DESCRIPTION
Any	Raw judge results (format depends on implementation).

Source code in src/autorubric/graders/base.py

@abstractmethod
async def judge(
    self,
    to_grade: str,
    rubric: list[Criterion],
    query: str | None = None,
    reference_submission: str | None = None,
) -> Any:
    """Collect raw judge results for the provided submission.

    An implementation may also take the rubric's guidelines, as a ``guidelines``
    parameter that can be passed by keyword or through ``**kwargs``; ``grade`` passes
    them, when there are any, only to an implementation that does.

    Args:
        to_grade: The text to evaluate.
        rubric: List of criteria to evaluate against.
        query: Optional input/query that prompted the response.
        reference_submission: Optional exemplar response for grading context.

    Returns:
        Raw judge results (format depends on implementation).
    """
    pass

aggregate abstractmethod async

aggregate(judge_results: Any, *, normalize: bool = True) -> ReportT

Transform judge results into an EvaluationReport.

PARAMETER	DESCRIPTION
judge_results	Raw results from judge(). TYPE: Any
normalize	If True, normalize score to 0-1. If False, return raw weighted sum. TYPE: bool DEFAULT: True

RETURNS	DESCRIPTION
ReportT	EvaluationReport with score and optional per-criterion breakdown.

Source code in src/autorubric/graders/base.py

@abstractmethod
async def aggregate(self, judge_results: Any, *, normalize: bool = True) -> ReportT:
    """Transform judge results into an EvaluationReport.

    Args:
        judge_results: Raw results from judge().
        normalize: If True, normalize score to 0-1. If False, return raw weighted sum.

    Returns:
        EvaluationReport with score and optional per-criterion breakdown.
    """
    pass

grade async

grade(to_grade: ToGradeInput, rubric: list[Criterion], query: str | None = None, reference_submission: str | None = None, *, guidelines: str | None = None) -> ReportT

Grade the submission against the rubric.

This is the main entry point for the grader.

PARAMETER	DESCRIPTION
to_grade	The text to evaluate. Can be either: - A string (optionally with / markers) - A dict with 'thinking' and 'output' keys TYPE: ToGradeInput
rubric	List of criteria to evaluate against. TYPE: list[Criterion]
query	Optional input/query that prompted the response. TYPE: str | None DEFAULT: None
reference_submission	Optional exemplar response for grading context. TYPE: str | None DEFAULT: None
guidelines	Optional rubric guidelines: free text that applies to every criterion (Rubric.grade passes Rubric.guidelines). Blank text means none. They are passed to judge as guidelines= only if judge accepts that keyword (a guidelines parameter or **kwargs; checked once per grader instance). Otherwise, and whenever there are no guidelines, judge is called exactly as without them: judge(to_grade, rubric, query, reference_submission). TYPE: str | None DEFAULT: None

RETURNS	DESCRIPTION
ReportT	EvaluationReport with score and optional per-criterion breakdown.
ReportT	If normalize=True (default), score is 0-1. If normalize=False, score is raw
ReportT	weighted sum. If length_penalty was configured, the penalty is subtracted from
ReportT	the score. The raw_score field contains the unnormalized weighted sum before
ReportT	length penalty.

RAISES	DESCRIPTION
TypeError	If guidelines is neither a str nor None.

WARNS	DESCRIPTION
UserWarning	" ignores rubric guidelines", once per grader instance, when guidelines are given and judge does not accept them.

Source code in src/autorubric/graders/base.py

async def grade(
    self,
    to_grade: ToGradeInput,
    rubric: list[Criterion],
    query: str | None = None,
    reference_submission: str | None = None,
    *,
    guidelines: str | None = None,
) -> ReportT:
    """Grade the submission against the rubric.

    This is the main entry point for the grader.

    Args:
        to_grade: The text to evaluate. Can be either:
            - A string (optionally with <thinking>/<output> markers)
            - A dict with 'thinking' and 'output' keys
        rubric: List of criteria to evaluate against.
        query: Optional input/query that prompted the response.
        reference_submission: Optional exemplar response for grading context.
        guidelines: Optional rubric guidelines: free text that applies to every
            criterion (``Rubric.grade`` passes ``Rubric.guidelines``). Blank text means
            none. They are passed to ``judge`` as ``guidelines=`` only if ``judge``
            accepts that keyword (a ``guidelines`` parameter or ``**kwargs``; checked
            once per grader instance). Otherwise, and whenever there are no guidelines,
            ``judge`` is called exactly as without them:
            ``judge(to_grade, rubric, query, reference_submission)``.

    Returns:
        EvaluationReport with score and optional per-criterion breakdown.
        If normalize=True (default), score is 0-1. If normalize=False, score is raw
        weighted sum. If length_penalty was configured, the penalty is subtracted from
        the score. The raw_score field contains the unnormalized weighted sum before
        length penalty.

    Raises:
        TypeError: If ``guidelines`` is neither a ``str`` nor ``None``.

    Warns:
        UserWarning: "<GraderClass> ignores rubric guidelines", once per grader
            instance, when guidelines are given and ``judge`` does not accept them.
    """
    guidelines = _normalize_guidelines(guidelines)
    if guidelines is not None and not _takes_guidelines(self, "judge"):
        _warn_ignores_guidelines(self)
        guidelines = None

    # Convert to_grade to string for judge() call (maintains compatibility)
    if isinstance(to_grade, str):
        to_grade_str = to_grade
    else:
        # Dict format - reconstruct string with markers for judge()
        thinking = to_grade.get("thinking", "")
        output = to_grade.get("output", "")
        parts = []
        if thinking:
            parts.append(f"<thinking>{thinking}</thinking>")
        if output:
            parts.append(f"<output>{output}</output>")
        to_grade_str = "\n".join(parts) if parts else ""

    # Call judge with string format (maintains compatibility). Guidelines go as a
    # keyword, and only to a judge that accepts it (checked above); the abstract
    # signature does not declare it, hence the cast.
    judge = cast(Callable[..., Awaitable[Any]], self.judge)
    if guidelines is None:
        judge_results = await judge(to_grade_str, rubric, query, reference_submission)
    else:
        judge_results = await judge(
            to_grade_str, rubric, query, reference_submission, guidelines=guidelines
        )
    report = await self.aggregate(judge_results, normalize=self.normalize)

    if self.length_penalty is not None:
        # A grade-FAILURE has no score (errored/empty report): there is nothing to
        # penalize, so return it unchanged rather than subtracting from None.
        if report.score is None:
            return report

        # Normalize to_grade to dict format for penalty calculation
        to_grade_normalized = normalize_to_grade_input(to_grade)

        # Compute penalty
        penalty = compute_length_penalty(to_grade_normalized, self.length_penalty)

        # Apply penalty (penalty is always non-negative, so we subtract)
        adjusted_score = report.score - penalty
        if self.normalize:
            adjusted_score = max(0.0, adjusted_score)

        # The same report, of the same type, with the adjusted score
        return report.model_copy(update={"score": adjusted_score})

    return report

---

JudgeSpec

Configuration for a single judge in an ensemble.

JudgeSpec dataclass

JudgeSpec(llm_config: LLMConfig | DecisionModelConfig, judge_id: str, weight: float = 1.0)

JudgeSpec(*, judge_model_config: LLMConfig | DecisionModelConfig, judge_id: str, weight: float = 1.0)

JudgeSpec(llm_config: Any = MISSING, judge_id: Any = MISSING, weight: float = 1.0, *, judge_model_config: Any = MISSING)

Specification for a single judge in an ensemble.

judge_model_config is the preferred keyword for the judge's configuration, and the positional form is unchanged. llm_config= keeps working without a warning: it is the name of the stored dataclass field, which is deliberately not renamed so that dataclasses.fields/asdict/replace, the repr, equality and pickles stay exactly as they were. dataclasses.replace therefore takes llm_config=, and passing both judge_model_config= and llm_config= raises ValueError.

The configuration decides the judge's kind: an LLMConfig makes an LLM judge, which is asked about each criterion in its own call (or about every criterion of an item in one call, when the grader's llm_calls is "per_item"), and a DecisionModelConfig makes a decision-model judge, which is asked about every criterion of an item in one request. Both kinds mix freely in one ensemble.

ATTRIBUTE	DESCRIPTION
llm_config	The judge's model configuration, an LLMConfig or a DecisionModelConfig (the stored field; also readable and writable as the judge_model_config property). TYPE: LLMConfig | DecisionModelConfig
judge_id	Unique identifier for this judge (e.g., "gpt-4", "claude-sonnet"). TYPE: str
weight	Voting weight for weighted aggregation (default 1.0); a positive, finite number. TYPE: float

Example

gemini = LLMConfig(model="gemini/gemini-3-flash-preview") JudgeSpec(judge_model_config=gemini, judge_id="gemini") JudgeSpec(gemini, "gemini", weight=2.0) # positional form JudgeSpec(DecisionModelConfig(model="jev-latest"), "jev") # a decision model

Initialize from judge_model_config (or its stored-field name llm_config).

PARAMETER	DESCRIPTION
llm_config	The judge's model configuration, positionally or by the stored field name. Mutually exclusive with judge_model_config. TYPE: Any DEFAULT: MISSING
judge_id	Unique identifier for this judge. TYPE: Any DEFAULT: MISSING
weight	Voting weight for weighted aggregation (default 1.0); a positive, finite number. TYPE: float DEFAULT: 1.0
judge_model_config	The judge's model configuration (preferred keyword). TYPE: Any DEFAULT: MISSING

RAISES	DESCRIPTION
ValueError	If both judge_model_config and llm_config are passed, or if weight is not a positive, finite number.
TypeError	If the configuration or judge_id is missing.

Source code in src/autorubric/graders/criterion_grader.py

def __init__(
    self,
    llm_config: Any = dataclasses.MISSING,
    judge_id: Any = dataclasses.MISSING,
    weight: float = 1.0,
    *,
    judge_model_config: Any = dataclasses.MISSING,
) -> None:
    """Initialize from ``judge_model_config`` (or its stored-field name ``llm_config``).

    Args:
        llm_config: The judge's model configuration, positionally or by the stored
            field name. Mutually exclusive with ``judge_model_config``.
        judge_id: Unique identifier for this judge.
        weight: Voting weight for weighted aggregation (default 1.0); a positive,
            finite number.
        judge_model_config: The judge's model configuration (preferred keyword).

    Raises:
        ValueError: If both ``judge_model_config`` and ``llm_config`` are passed, or if
            ``weight`` is not a positive, finite number.
        TypeError: If the configuration or ``judge_id`` is missing.
    """
    if judge_model_config is not dataclasses.MISSING:
        if llm_config is not dataclasses.MISSING:
            raise ValueError(
                "Pass only one of judge_model_config and llm_config "
                "(llm_config is the stored field name of judge_model_config)"
            )
        llm_config = judge_model_config
    missing = []
    if llm_config is dataclasses.MISSING:
        missing.append("'judge_model_config' (or its alias 'llm_config')")
    if judge_id is dataclasses.MISSING:
        missing.append("'judge_id'")
    if missing:
        # Mirror Python's own wording; a lone judge_id is exactly the positional
        # argument the generated dataclass __init__ used to report.
        kind = "positional argument" if missing == ["'judge_id'"] else "argument"
        plural = "s" if len(missing) > 1 else ""
        raise TypeError(
            f"JudgeSpec.__init__() missing {len(missing)} required {kind}{plural}: "
            + " and ".join(missing)
        )
    _check_judge_weight(weight, judge_id)
    self.llm_config = llm_config
    self.judge_id = judge_id
    self.weight = weight

judge_model_config property writable

judge_model_config: LLMConfig | DecisionModelConfig

The judge's model configuration (read/write alias of the llm_config field).

---

LLMCalls

How CriterionGrader calls each LLM judge for an item: "per_criterion" (the default) or "per_item".

LLMCalls module-attribute

LLMCalls = Literal['per_criterion', 'per_item']

How CriterionGrader calls each LLM judge for an item (its llm_calls).

• per_criterion: One call per criterion (the default).
• per_item: One call for the whole rubric: the submission and its context are sent once, every criterion is listed under its id (c0, c1, ...), and the answer holds one judgment per criterion (RubricJudgment).

A cascade's escalation judges are called the same way, only where the decision model escalated: per_item makes one full-rubric call for an item with any escalated criterion. replay_escalation and calibrate_escalation take it to know how an LLM run was called, which sets how they estimate a cascade's cost and time from that run's.

A decision-model judge always makes one request per item, whatever this is. The name counts LLM calls per item; it is unrelated to compute_metrics(per_item_metrics=...) and to per-item rubrics.