Skip to content

Graders

Grader implementations for evaluating responses against rubrics.

Overview

Graders evaluate responses against rubrics and return structured reports. The main implementation is CriterionGrader, which supports single LLM, ensemble, and few-shot modes. All combinations work orthogonally.

Quick Example

from autorubric import LLMConfig, FewShotConfig
from autorubric.graders import CriterionGrader, JudgeSpec, Grader

# Single LLM mode
grader = CriterionGrader(
    judge_model_config=LLMConfig(model="openai/gpt-5.6-luna"),
)

# One call per item: the judge grades the whole rubric in a single call
# instead of one call per criterion
grader = CriterionGrader(
    judge_model_config=LLMConfig(model="openai/gpt-5.6-luna"),
    llm_calls="per_item",
)

# With custom system prompt
grader = CriterionGrader(
    judge_model_config=LLMConfig(model="openai/gpt-5.6-luna"),
    system_prompt="You are evaluating technical documentation...",
)

# Ensemble mode
grader = CriterionGrader(
    judges=[
        JudgeSpec(LLMConfig(model="gemini/gemini-3-flash-preview"), "gemini", weight=1.0),
        JudgeSpec(LLMConfig(model="anthropic/claude-sonnet-4-5-20250929"), "claude", weight=1.2),
    ],
    aggregation="weighted",
)

# Single LLM + few-shot
grader = CriterionGrader(
    judge_model_config=LLMConfig(model="openai/gpt-5.6-luna"),
    training_data=train_data,
    few_shot_config=FewShotConfig(n_examples=3, balance_verdicts=True),
)

# Grade
result = await rubric.grade(to_grade=response, grader=grader)

judge_model_config sets up a single judge and judges sets up an ensemble. Pass exactly one of them.

Either accepts a DecisionModelConfig as well as an LLMConfig: a decision-model judge grades each item with one request for the whole rubric. escalation=EscalationConfig(...) makes a grader with one decision-model judge a confidence cascade that escalates uncertain criteria to LLM judges. See Decision Models for the API, and the LLM Judges and Decision-Model Judges chapters for guided introductions to the two judge kinds.

llm_calls="per_item" makes every LLM judge of the grader — the judge of judge_model_config, each judge of judges, and a cascade's escalation judges — grade an item's whole rubric in one call instead of one call per criterion, the way a decision-model judge always does. The default, "per_criterion", is unchanged. See Grading a whole rubric in one call for what the single call looks like, and Decision-Model Judges for what it shares with a decision model's request.

llm_config is deprecated

llm_config is a deprecated alias of judge_model_config. It builds the same judge but emits a DeprecationWarning. To migrate, rename the keyword; nothing else changes. Passing both raises ValueError. llm_config will not be removed before the next major version.

Grading Options

grader = CriterionGrader(
    judge_model_config=LLMConfig(model="openai/gpt-5.6-luna"),

    # Score normalization
    normalize=True,          # True: 0-1 range, False: raw weighted sum

    # CANNOT_ASSESS handling
    cannot_assess_config=CannotAssessConfig(strategy=CannotAssessStrategy.SKIP),

    # Length penalty
    length_penalty=LengthPenalty(free_budget=6000, max_cap=8000),

    # Position bias mitigation (for multi-choice)
    shuffle_options=True,    # Default: enabled

    # Multi-choice abstain channel
    auto_na_option=True,     # Default: True — guarantees every multi-choice criterion a
                             # first-class NA/abstain option (auto-injected if absent).
                             # Set False for forced-choice (no auto NA option).

    # Reproducibility — pins all non-LLM randomness (shuffles, few-shot selection)
    seed=42,                 # Default: auto-generated
)

With auto_na_option=True, an auto-injected NA option is appended at the end (highest index), so existing option indices are preserved; an author-supplied NA option is never stripped.

seed must be None or an integer. Numpy integers are accepted and kept as a plain int, so seed=np.int64(42) shuffles and selects examples exactly as seed=42 does, and the experiment manifest records it as master_seed. The grader raises ValueError for a bool, a float such as 1.0, or a string. The seed is hashed as text, so earlier releases accepted a bool or a float and silently produced different shuffles and few-shot draws from the equal integer (True from 1, 1.0 from 1).


CriterionGrader

Main grader with support for single LLM, ensemble, and few-shot modes.

CriterionGrader

CriterionGrader(*, judge_model_config: LLMConfig | DecisionModelConfig | None = None, judges: list[JudgeSpec] | None = None, aggregation: AggregationStrategy = 'majority', ordinal_aggregation: OrdinalAggregation = 'mean', nominal_aggregation: NominalAggregation = 'mode', training_data: RubricDataset | None = None, few_shot_config: FewShotConfig | None = None, system_prompt: str | None = None, multi_choice_system_prompt: str | None = None, length_penalty: LengthPenalty | None = None, normalize: bool = True, cannot_assess_config: CannotAssessConfig | None = None, shuffle_options: bool = True, auto_na_option: bool = True, seed: int | None = None, binary_response_format: type[BaseModel] | None = None, multi_choice_response_format: type[BaseModel] | None = None, escalation: EscalationConfig | None = None, llm_calls: LLMCalls = 'per_criterion', llm_config: LLMConfig | DecisionModelConfig | None = None)

Bases: Grader[EnsembleEvaluationReport]

Unified criterion-based grader with compositional few-shot and ensemble support.

This grader evaluates each criterion independently and supports: - Single-judge mode (via judge_model_config) - Ensemble mode with multiple judges (via judges) - Few-shot prompting (via training_data + few_shot_config)

All combinations work: single judge, single + few-shot, ensemble, ensemble + few-shot.

Parameters are orthogonal: - judge_model_config OR judges: Choose single-judge or ensemble mode - training_data + few_shot_config: Enable few-shot prompting (applies to every LLM judge)

A judge is an LLM (LLMConfig) or a decision model (DecisionModelConfig), alone or mixed in one ensemble. An LLM judge is asked about each criterion in its own call. A decision-model judge is asked about the whole rubric in one request per item: the submission goes once as shared state and every criterion it can express becomes one question; a criterion it cannot express fails alone, and a failed request fails every criterion posed in it, through the same failure routing as an LLM judge's failed call. Its votes carry probabilities and confidence and no explanation (reason is None). In a mixed ensemble every judge votes on every criterion, and the votes aggregate exactly as an all-LLM ensemble's do.

With llm_calls="per_item", each LLM judge of judge_model_config or judges is asked about the whole rubric in one call per item instead, as a decision model is: the submission and its context are sent once; the system prompt embeds the binary and multi-choice system prompts, for the kinds of criteria the rubric has, as guides to apply to each criterion separately and independently; every criterion is listed under its decision-model question id (c0, c1, ...), posed as in a per-criterion call and with its options shuffled exactly as there; and the answer (RubricJudgment) maps back to one result per criterion. A missing or unusable answer fails only its criterion (parse), a failed call fails every criterion, and the call's usage and cost ride on the item's first result. With few-shot examples, the call shows whole training items, each once with its ground truth for every criterion, instead of examples drawn for each criterion. Votes, aggregation and scoring are the same in both modes. A cascade's escalation judges make that call too, but only for an item with an escalated criterion, and keep only the escalated criteria's results (see below).

A confidence cascade (escalation=EscalationConfig(...)) puts one decision model first and LLM judges behind it. The decision model judges every criterion in its one request per item; a criterion it errored or abstained on, or answered with a confidence below the criterion's threshold, is escalated, and each escalation judge votes on the escalated criteria only. Called per criterion, an escalation judge is asked about each escalated criterion in a call of its own. Under llm_calls="per_item" it is asked about the whole rubric in one call, made only for an item with an escalated criterion; its judgments of the other criteria are discarded, a failed call fails its votes on the escalated criteria, and the call's usage and cost ride on the first escalated criterion's result. Either way each call is the one the judge would make in an otherwise identical grader without the cascade (the same seed, judge_id and LLM settings, llm_calls included). An escalated criterion's report is marked escalated=True, keeps the decision model's vote with superseded=True, and takes its verdict from the escalation judges' votes alone, aggregated as any ensemble's; it never falls back to the decision model's vote. The decision model's judge_scores entry is its own score over every criterion; an escalation judge's is None by role, as it votes only where the decision model escalated, even on an item where every criterion escalated.

Some settings exist for LLM judges only. Few-shot examples, the system prompts and shuffle_options apply to the LLM judges of a mixed ensemble or a cascade and never reach a decision model; so does llm_calls. A grader whose judges are all decision models rejects few-shot examples and warns when a system prompt is given, shuffle_options is False or llm_calls is "per_item", since it has no judge to apply them to. A custom response format describes a generated judgment, which only an LLM judge produces, so it cannot be combined with any decision-model judge; it describes one criterion's judgment, so it cannot be combined with llm_calls="per_item" either.

Example

from autorubric import DecisionModelConfig, LLMConfig, FewShotConfig, RubricDataset from autorubric.graders import CriterionGrader, JudgeSpec

Single LLM

grader = CriterionGrader( ... judge_model_config=LLMConfig(model="gemini/gemini-3-flash-preview") ... )

Single LLM + few-shot

train, test = dataset.split_train_test(n_train=100) grader = CriterionGrader( ... judge_model_config=LLMConfig(model="gemini/gemini-3-flash-preview"), ... training_data=train, ... few_shot_config=FewShotConfig(n_examples=3), ... )

Ensemble

grader = CriterionGrader( ... judges=[ ... JudgeSpec(LLMConfig(model="gemini/gemini-3-flash-preview"), "gemini"), ... JudgeSpec(LLMConfig(model="anthropic/claude-sonnet-4-5-20250929"), "claude"), ... ], ... aggregation="majority", ... )

Ensemble + few-shot

grader = CriterionGrader( ... judges=[JudgeSpec(...), JudgeSpec(...)], ... aggregation="majority", ... training_data=train, ... few_shot_config=FewShotConfig(n_examples=3), ... )

Single LLM, one call per item for the whole rubric

grader = CriterionGrader( ... judge_model_config=LLMConfig(model="gemini/gemini-3-flash-preview"), ... llm_calls="per_item", ... )

Decision model: one request per item for the whole rubric

grader = CriterionGrader(judge_model_config=DecisionModelConfig(model="jev-latest"))

Mixed ensemble of a decision model and an LLM

grader = CriterionGrader( ... judges=[ ... JudgeSpec(DecisionModelConfig(model="jev-latest"), "jev"), ... JudgeSpec(LLMConfig(model="gemini/gemini-3-flash-preview"), "gemini"), ... ], ... )

Cascade: the decision model first, an LLM for what it is unsure of

grader = CriterionGrader( ... judge_model_config=DecisionModelConfig(model="jev-latest"), ... escalation=EscalationConfig( ... judges=LLMConfig(model="gemini/gemini-3-flash-preview"), threshold=0.72 ... ), ... )

Initialize the criterion grader.

PARAMETER DESCRIPTION
judge_model_config

Configuration of the single judge (single-judge mode), which is given judge_id="default": an LLMConfig for an LLM judge or a DecisionModelConfig for a decision-model judge. Mutually exclusive with judges.

TYPE: LLMConfig | DecisionModelConfig | None DEFAULT: None

judges

List of JudgeSpec for ensemble mode, each an LLM or a decision model, each with its own judge_id. Mutually exclusive with judge_model_config.

TYPE: list[JudgeSpec] | None DEFAULT: None

aggregation

Strategy for aggregating votes in ensemble mode (binary criteria).

TYPE: AggregationStrategy DEFAULT: 'majority'

ordinal_aggregation

Strategy for aggregating ordinal multi-choice votes. Central tendency: "mean", "median", "weighted_mean", "mode". Conservative/ permissive (analogs of binary unanimous/any): "min" (lowest selected option) / "max" (highest selected option).

TYPE: OrdinalAggregation DEFAULT: 'mean'

nominal_aggregation

Strategy for aggregating nominal multi-choice votes. Options: "mode", "weighted_mode", "unanimous". "unanimous" abstains via the NA option on disagreement, or falls back to mode + warns if there is no NA option.

TYPE: NominalAggregation DEFAULT: 'mode'

training_data

Dataset for few-shot examples. If provided, enables few-shot prompting for the LLM judges; examples are selected for, and sent to, LLM judges only, never to a decision-model judge, whose request is exactly what it would be without them. An example shows its item's ground truth for the criterion and, when few_shot_config.include_reason is True, the item's written reason for it (DataItem.ground_truth_reasons), if any. Called per criterion, a judge is shown examples drawn for each criterion separately. Under llm_calls="per_item" an example is a whole training item, shown once in the judge's one call per item with its ground truth (and reason) for every criterion it has a label for; only an item with at least one label, each of which resolves (a verdict for a binary criterion, an option for a multi-choice one), is eligible. Either way each judge draws its own examples, and the training data needs a global rubric, which says what each label means. Examples are drawn for that rubric's criteria and shown for the criterion at the same index of the rubric being graded only when the two pose the same question, as the judge is shown each: the same kind and requirement, for a binary criterion the same criterion type (a negative weight or not), for a multi-choice criterion the same options (labels and NA flags) in the same order, the NA option auto_na_option adds included. Any other graded criterion, or one beyond the training rubric, gets no examples (no judgment in a whole-rubric example); weight magnitude, name, option values, scale_type and aggregation do not matter. Likewise a training item with its own rubric (DataItem.rubric) lends its label for a criterion only when its own criterion at that index poses the training rubric's question (under llm_calls="per_item", its example has no judgment for the others). Both rubrics are read as they are when the grader is built: changing the training rubric afterwards changes no prompt. Grading a criterion shown no examples this way warns (UserWarning, once per grader per criterion index and graded question), naming the criterion and why; a training item's own rubric that lends no label warns nothing.

TYPE: RubricDataset | None DEFAULT: None

few_shot_config

Configuration for few-shot example selection (LLM judges only). The grader validates and keeps its own copy, so changing the config after the grader is built does not affect it. Under llm_calls="per_item", n_examples counts the training items each call shows and balance_verdicts selects them by greedy label coverage (see FewShotConfig).

TYPE: FewShotConfig | None DEFAULT: None

system_prompt

Custom system prompt for binary criteria. Applies to LLM judges only (a decision model's request has no system prompt); in a mixed ensemble the LLM judges use it.

TYPE: str | None DEFAULT: None

multi_choice_system_prompt

Custom system prompt for multi-choice criteria. Applies to LLM judges only, like system_prompt.

TYPE: str | None DEFAULT: None

length_penalty

Optional length penalty configuration.

TYPE: LengthPenalty | None DEFAULT: None

normalize

If True, normalize score to [0, 1]. If False, return raw sum.

TYPE: bool DEFAULT: True

cannot_assess_config

Configuration for handling CANNOT_ASSESS verdicts.

TYPE: CannotAssessConfig | None DEFAULT: None

shuffle_options

If True (default), randomize the order of multi-choice options presented to the LLM to mitigate position bias. Each judge/call sees a different random order, and responses are mapped back to original indices. Disable for deterministic behavior in tests. Applies to LLM judges only: a decision model's question offers the options in the rubric's order, and in a mixed ensemble the LLM judges shuffle as set here.

TYPE: bool DEFAULT: True

auto_na_option

If True (default), auto-inject a canonical NA / "cannot assess" option into any multi-choice criterion that lacks one, giving the judge a first-class abstain channel analogous to binary CANNOT_ASSESS. The injected option is appended at the end (highest index) so existing option indices are preserved. Set False for forced-choice classification (the judge must pick a scored option). Never strips an author-supplied NA option — author intent wins.

TYPE: bool DEFAULT: True

seed

Master seed for all non-LLM randomness (option shuffling, few-shot example selection). Auto-generated when None so that randomness is always pinned and reproducible. Any integer is accepted (numpy integers too) and kept as a plain int; a bool or a float is refused. Inspect via the seed property after construction.

TYPE: int | None DEFAULT: None

binary_response_format

Pydantic model to use as the structured output schema for binary criterion judgments. Must be a subclass of (or compatible with) CriterionJudgment. If the model includes an affected_criteria field (list[int]), matching indices are injected as an [Affects: ...] tag into the reason string. Defaults to CriterionJudgment. LLM judges only: it describes a generated judgment, which a decision model does not produce, so it cannot be combined with any decision-model judge.

TYPE: type[BaseModel] | None DEFAULT: None

multi_choice_response_format

Pydantic model to use as the structured output schema for multi-choice criterion judgments. Must be a subclass of (or compatible with) MultiChoiceJudgment. If the model includes an affected_criteria field (list[int]), matching indices are injected as an [Affects: ...] tag into the reason string (same convention as binary_response_format). Defaults to MultiChoiceJudgment. LLM judges only, like binary_response_format.

TYPE: type[BaseModel] | None DEFAULT: None

escalation

Makes the grader a confidence cascade (see the class docstring): the escalation judges, all LLMs, vote only on the criteria the decision model errored or abstained on or answered with a confidence below the criterion's threshold. The grader's one judge (judge_model_config, or a single JudgeSpec in judges) must then be a decision model. The escalation judges count as judges for the LLM-only settings: few-shot examples, the system prompts, shuffle_options and llm_calls apply to them. The grader validates and keeps its own copy of the config (dataclasses.replace), so changing the config's lists in place later does not affect it.

TYPE: EscalationConfig | None DEFAULT: None

llm_calls

How each LLM judge, a cascade's escalation judges included, is called for an item: "per_criterion" (the default), one call per criterion, or "per_item", one call for the whole rubric (see the class docstring). An escalation judge calls only where the decision model escalated: once per escalated criterion, or once, about the whole rubric, for an item with any escalated criterion, keeping its judgments of the escalated criteria only. "per_item" sends the submission once instead of once per criterion but generates every criterion's judgment in one reply, so max_tokens and timeout must allow for the whole rubric's answer: a reply cut off at max_tokens fails every criterion. It also shapes few-shot examples: under "per_item" they are whole training items, each labelled on every criterion (see training_data). Applies to LLM judges only; a decision model always makes one request per item. The name counts LLM calls per item; it is unrelated to compute_metrics(per_item_metrics=...) and to per-item rubrics.

TYPE: LLMCalls DEFAULT: 'per_criterion'

llm_config

Deprecated alias of judge_model_config; builds the same single judge and emits a DeprecationWarning.

TYPE: LLMConfig | DecisionModelConfig | None DEFAULT: None

RAISES DESCRIPTION
ValueError

If neither judge_model_config nor judges is provided, if both are provided, or if both judge_model_config and its deprecated alias llm_config are provided; if escalation is given while the grader's judges are not exactly one decision model, with a judge_id that repeats among the decision model and the escalation judges, or with a config that no longer validates (its judges list or per_criterion dict changed in place since it was built, e.g. emptied); if llm_calls is not "per_criterion" or "per_item"; if seed is neither None nor an integer (a bool or a float such as 1.0 is refused, since the seed keys the RNGs as text); if few_shot_config or training_data is given while every judge is a decision model (few-shot examples apply to LLM judges only), or training_data has no global rubric; if few_shot_config no longer validates (a field set after it was built, e.g. a negative n_examples; see FewShotConfig); if binary_response_format or multi_choice_response_format is given while any judge is a decision model, or with llm_calls="per_item"; or if a decision-model judge has no API key, has a key the SDK would reject (anything but printable ASCII without whitespace), or has a resolved base URL (api_base or the TYPESAFE_BASE_URL environment variable) that is not an absolute http(s) URL free of whitespace and control characters, query, fragment and credentials, or whose host the SDK's HTTP stack cannot encode (e.g. a non-ASCII name that is not a valid internationalized domain name). Such a judge fails here, never as an abstention or a worst case on every item.

ImportError

If a judge is a decision model and the TypeSafe SDK is not installed (pip install 'autorubric[typesafe]').

WARNS DESCRIPTION
UserWarning

If system_prompt or multi_choice_system_prompt is not None, shuffle_options is False, or llm_calls is "per_item", while every judge is a decision model: these settings apply to LLM judges only, so they have no effect. One warning per setting that differs from its default (None, None, True and "per_criterion"); a cascade's escalation judges are LLM judges, so a cascade never warns.

FutureWarning

If a judge_id repeats among judges. Judges that share one are conflated: those of one kind (LLM or decision model) all call the model of the last of them, and all share one judge_scores entry, one set of per-judge metrics, and the same option shuffles and few-shot examples. Grading is otherwise unchanged; a repeat will be a ValueError in the next major version, as it already is in a cascade.

Source code in src/autorubric/graders/criterion_grader.py
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
1694
1695
1696
1697
1698
1699
1700
1701
1702
1703
1704
1705
1706
1707
1708
1709
1710
1711
1712
1713
1714
1715
1716
1717
1718
1719
1720
1721
1722
1723
1724
1725
1726
1727
1728
1729
1730
1731
1732
1733
1734
1735
1736
1737
1738
1739
1740
1741
1742
def __init__(
    self,
    *,
    # Single-judge mode
    judge_model_config: LLMConfig | DecisionModelConfig | None = None,
    # Ensemble mode (mutually exclusive with judge_model_config)
    judges: list[JudgeSpec] | None = None,
    aggregation: AggregationStrategy = "majority",
    # Multi-choice aggregation strategies
    ordinal_aggregation: OrdinalAggregation = "mean",
    nominal_aggregation: NominalAggregation = "mode",
    # Few-shot mode (orthogonal - applies to every LLM judge)
    training_data: RubricDataset | None = None,
    few_shot_config: FewShotConfig | None = None,
    # Common parameters
    system_prompt: str | None = None,
    multi_choice_system_prompt: str | None = None,
    length_penalty: LengthPenalty | None = None,
    normalize: bool = True,
    cannot_assess_config: CannotAssessConfig | None = None,
    # Position bias mitigation
    shuffle_options: bool = True,
    # Multi-choice abstain channel
    auto_na_option: bool = True,
    # Reproducibility
    seed: int | None = None,
    # Structured output override for binary criteria
    binary_response_format: type[BaseModel] | None = None,
    # Structured output override for multi-choice criteria
    multi_choice_response_format: type[BaseModel] | None = None,
    # Confidence cascade: LLM judges behind a decision model
    escalation: EscalationConfig | None = None,
    # LLM calls per item: one per criterion, or one for the whole rubric
    llm_calls: LLMCalls = "per_criterion",
    # Deprecated alias of judge_model_config (every parameter is keyword-only, so
    # its position at the end changes no call)
    llm_config: LLMConfig | DecisionModelConfig | None = None,
):
    """Initialize the criterion grader.

    Args:
        judge_model_config: Configuration of the single judge (single-judge mode), which
            is given ``judge_id="default"``: an ``LLMConfig`` for an LLM judge or a
            ``DecisionModelConfig`` for a decision-model judge. Mutually exclusive with
            judges.
        judges: List of JudgeSpec for ensemble mode, each an LLM or a decision model,
            each with its own ``judge_id``. Mutually exclusive with judge_model_config.
        aggregation: Strategy for aggregating votes in ensemble mode (binary criteria).
        ordinal_aggregation: Strategy for aggregating ordinal multi-choice votes.
            Central tendency: "mean", "median", "weighted_mean", "mode". Conservative/
            permissive (analogs of binary unanimous/any): "min" (lowest selected
            option) / "max" (highest selected option).
        nominal_aggregation: Strategy for aggregating nominal multi-choice votes.
            Options: "mode", "weighted_mode", "unanimous". "unanimous" abstains via the
            NA option on disagreement, or falls back to mode + warns if there is no NA
            option.
        training_data: Dataset for few-shot examples. If provided, enables few-shot
            prompting for the LLM judges; examples are selected for, and sent to, LLM
            judges only, never to a decision-model judge, whose request is exactly what
            it would be without them. An example shows its item's ground truth for the
            criterion and, when ``few_shot_config.include_reason`` is True, the item's
            written reason for it (``DataItem.ground_truth_reasons``), if any. Called per
            criterion, a judge is shown examples drawn for each criterion separately.
            Under ``llm_calls="per_item"`` an example is a whole training item, shown
            once in the judge's one call per item with its ground truth (and reason) for
            every criterion it has a label for; only an item with at least one label,
            each of which resolves (a verdict for a binary criterion, an option for a
            multi-choice one), is eligible. Either way each judge draws its own
            examples, and the training data needs a global rubric, which says what each
            label means.
            Examples are drawn for that rubric's criteria and shown for the criterion at
            the same index of the rubric being graded only when the two pose the same
            question, as the judge is shown each: the same kind and requirement, for a
            binary criterion the same criterion type (a negative weight or not), for a
            multi-choice criterion the same options (labels and NA flags) in the same
            order, the NA option ``auto_na_option`` adds included. Any other graded
            criterion, or one beyond the training rubric, gets no examples (no judgment
            in a whole-rubric example); weight magnitude, name, option values,
            ``scale_type`` and ``aggregation`` do not matter. Likewise a training item
            with its own rubric (``DataItem.rubric``) lends its label for a criterion
            only when its own criterion at that index poses the training rubric's
            question (under ``llm_calls="per_item"``, its example has no judgment for
            the others). Both rubrics are read as they are when the grader is built:
            changing the training rubric afterwards changes no prompt. Grading a
            criterion shown no examples this way warns (``UserWarning``, once per grader
            per criterion index and graded question), naming the criterion and why; a
            training item's own rubric that lends no label warns nothing.
        few_shot_config: Configuration for few-shot example selection (LLM judges only).
            The grader validates and keeps its own copy, so changing the config after
            the grader is built does not affect it.
            Under ``llm_calls="per_item"``, ``n_examples`` counts the training items
            each call shows and ``balance_verdicts`` selects them by greedy label
            coverage (see ``FewShotConfig``).
        system_prompt: Custom system prompt for binary criteria. Applies to LLM judges
            only (a decision model's request has no system prompt); in a mixed ensemble
            the LLM judges use it.
        multi_choice_system_prompt: Custom system prompt for multi-choice criteria. Applies
            to LLM judges only, like ``system_prompt``.
        length_penalty: Optional length penalty configuration.
        normalize: If True, normalize score to [0, 1]. If False, return raw sum.
        cannot_assess_config: Configuration for handling CANNOT_ASSESS verdicts.
        shuffle_options: If True (default), randomize the order of multi-choice options
            presented to the LLM to mitigate position bias. Each judge/call sees a
            different random order, and responses are mapped back to original indices.
            Disable for deterministic behavior in tests. Applies to LLM judges only: a
            decision model's question offers the options in the rubric's order, and in a
            mixed ensemble the LLM judges shuffle as set here.
        auto_na_option: If True (default), auto-inject a canonical NA / "cannot assess"
            option into any multi-choice criterion that lacks one, giving the judge a
            first-class abstain channel analogous to binary CANNOT_ASSESS. The injected
            option is appended at the end (highest index) so existing option indices are
            preserved. Set False for forced-choice classification (the judge must pick a
            scored option). Never strips an author-supplied NA option — author intent wins.
        seed: Master seed for all non-LLM randomness (option shuffling, few-shot
            example selection). Auto-generated when None so that randomness is always
            pinned and reproducible. Any integer is accepted (numpy integers too) and
            kept as a plain ``int``; a ``bool`` or a float is refused. Inspect via the
            ``seed`` property after construction.
        binary_response_format: Pydantic model to use as the structured output schema
            for binary criterion judgments. Must be a subclass of (or compatible with)
            CriterionJudgment. If the model includes an ``affected_criteria`` field
            (list[int]), matching indices are injected as an ``[Affects: ...]`` tag
            into the reason string. Defaults to CriterionJudgment. LLM judges only: it
            describes a generated judgment, which a decision model does not produce, so
            it cannot be combined with any decision-model judge.
        multi_choice_response_format: Pydantic model to use as the structured output
            schema for multi-choice criterion judgments. Must be a subclass of (or
            compatible with) MultiChoiceJudgment. If the model includes an
            ``affected_criteria`` field (list[int]), matching indices are injected as
            an ``[Affects: ...]`` tag into the reason string (same convention as
            binary_response_format). Defaults to MultiChoiceJudgment. LLM judges only,
            like binary_response_format.
        escalation: Makes the grader a confidence cascade (see the class docstring):
            the escalation judges, all LLMs, vote only on the criteria the decision
            model errored or abstained on or answered with a confidence below the
            criterion's threshold. The grader's one judge (``judge_model_config``, or a
            single ``JudgeSpec`` in ``judges``) must then be a decision model. The
            escalation judges count as judges for the LLM-only settings: few-shot
            examples, the system prompts, ``shuffle_options`` and ``llm_calls`` apply to
            them. The grader validates and keeps its own copy of the config
            (``dataclasses.replace``), so changing the config's lists in place later
            does not affect it.
        llm_calls: How each LLM judge, a cascade's escalation judges included, is called
            for an item: ``"per_criterion"`` (the default), one call per criterion, or
            ``"per_item"``, one call for the whole rubric (see the class docstring). An
            escalation judge calls only where the decision model escalated: once per
            escalated criterion, or once, about the whole rubric, for an item with any
            escalated criterion, keeping its judgments of the escalated criteria only.
            ``"per_item"`` sends the submission once instead of once per criterion but
            generates every criterion's judgment in one reply, so ``max_tokens`` and
            ``timeout`` must allow for the whole rubric's answer: a reply cut off at
            ``max_tokens`` fails every criterion. It also shapes few-shot examples: under
            ``"per_item"`` they are whole training items, each labelled on every
            criterion (see ``training_data``). Applies to LLM judges only; a decision
            model always makes one request per item. The name counts LLM calls per item;
            it is unrelated to ``compute_metrics(per_item_metrics=...)`` and to per-item
            rubrics.
        llm_config: Deprecated alias of ``judge_model_config``; builds the same single
            judge and emits a ``DeprecationWarning``.

    Raises:
        ValueError: If neither judge_model_config nor judges is provided, if both are
            provided, or if both judge_model_config and its deprecated alias
            llm_config are provided; if escalation is given while the grader's judges
            are not exactly one decision model, with a ``judge_id`` that repeats among
            the decision model and the escalation judges, or with a config that no
            longer validates (its ``judges`` list or ``per_criterion`` dict changed in
            place since it was built, e.g. emptied); if llm_calls is not
            ``"per_criterion"`` or ``"per_item"``; if seed is neither None nor an
            integer (a ``bool`` or a float such as ``1.0`` is refused, since the seed
            keys the RNGs as text); if few_shot_config or training_data is
            given while every judge is a decision model (few-shot examples apply to LLM
            judges only), or training_data has no global rubric; if few_shot_config
            no longer validates (a field set after it was built, e.g. a negative
            ``n_examples``; see ``FewShotConfig``); if
            binary_response_format or multi_choice_response_format is given while any
            judge is a decision model, or with ``llm_calls="per_item"``;
            or if a decision-model judge has no API key, has a key the SDK would reject
            (anything but printable ASCII without whitespace), or has a resolved base
            URL (``api_base`` or the ``TYPESAFE_BASE_URL`` environment variable) that is
            not an absolute http(s) URL free of whitespace and control characters,
            query, fragment and credentials, or whose host the SDK's HTTP stack cannot
            encode (e.g. a non-ASCII name that is not a valid internationalized domain
            name). Such a judge fails here, never as an abstention or a worst case on
            every item.
        ImportError: If a judge is a decision model and the TypeSafe SDK is not
            installed (``pip install 'autorubric[typesafe]'``).

    Warns:
        UserWarning: If system_prompt or multi_choice_system_prompt is not None,
            shuffle_options is False, or llm_calls is ``"per_item"``, while every judge
            is a decision model: these settings apply to LLM judges only, so they have
            no effect. One warning per setting that differs from its default (None,
            None, True and ``"per_criterion"``); a cascade's escalation judges are LLM
            judges, so a cascade never warns.
        FutureWarning: If a ``judge_id`` repeats among ``judges``. Judges that share
            one are conflated: those of one kind (LLM or decision model) all call the
            model of the last of them, and all share one ``judge_scores`` entry, one set
            of per-judge metrics, and the same option shuffles and few-shot examples.
            Grading is otherwise unchanged; a repeat will be a ``ValueError`` in the
            next major version, as it already is in a cascade.
    """
    if llm_config is not None:
        if judge_model_config is not None:
            raise ValueError(
                "Pass only one of judge_model_config and llm_config "
                "(llm_config is a deprecated alias of judge_model_config)"
            )
        warnings.warn(
            "llm_config is deprecated; use judge_model_config",
            DeprecationWarning,
            stacklevel=2,
        )
        judge_model_config = llm_config

    super().__init__(length_penalty=length_penalty, normalize=normalize)

    # Validate: must have either judge_model_config or judges, not both, not neither
    if judge_model_config is None and judges is None:
        raise ValueError("Must provide either judge_model_config or judges")
    if judge_model_config is not None and judges is not None:
        raise ValueError("Cannot provide both judge_model_config and judges")

    # Normalize to ensemble representation (single judge = ensemble of 1).
    # The validation above guarantees exactly one of judge_model_config/judges is set,
    # so by this point `judges` in the else branch is a non-None list[JudgeSpec]; the
    # explicit attribute annotation lets the type checker see that without a
    # suppression comment.
    self._judges: list[JudgeSpec]
    if judge_model_config is not None:
        self._judges = [
            JudgeSpec(
                judge_model_config=judge_model_config, judge_id=_SINGLE_JUDGE_ID, weight=1.0
            )
        ]
    else:
        assert judges is not None
        self._judges = judges

    # The grader's own copy of the cascade config, validated again: the config's list and
    # dict can have been changed in place since it was built, and can be again later.
    # Everything the cascade does reads this copy. Its escalation judges
    # (``_escalation_judges``) are kept apart from ``_judges``, which judge every
    # criterion: they judge only the criteria the decision model escalates. Every other
    # per-judge concern (clients, few-shot examples, the LLM-only settings, the
    # manifest, ``judge_ids``) covers both lists.
    self._escalation = dataclasses.replace(escalation) if escalation is not None else None
    if self._escalation is not None:
        primary = self._judges
        if len(primary) != 1 or not isinstance(primary[0].llm_config, DecisionModelConfig):
            described = [
                f"{j.judge_id!r} (a decision model)"
                if isinstance(j.llm_config, DecisionModelConfig)
                else f"{j.judge_id!r} (an LLM)"
                for j in primary
            ]
            raise ValueError(
                "escalation needs exactly one primary judge, a decision model "
                "(judge_model_config=DecisionModelConfig(...), or judges=[JudgeSpec("
                "DecisionModelConfig(...), ...)]), which judges every criterion and "
                f"escalates the ones it is unsure of; got {', '.join(described)}"
            )
        repeated = _repeated_judge_ids((*primary, *self._escalation_judges))
        if repeated:
            raise ValueError(
                "judge_ids must be unique across the decision model and the escalation "
                f"judges; repeated: {', '.join(repr(judge_id) for judge_id in repeated)}"
            )
    # A panel's judge_ids should be unique too, but a repeat has always been accepted,
    # so it warns until the next major version makes it the cascade's ValueError. A
    # cascade's primary is one judge, so this never warns for a cascade.
    repeated = _repeated_judge_ids(self._judges)
    if repeated:
        warnings.warn(
            f"judge_ids should be unique; repeated: {', '.join(map(repr, repeated))}. "
            "Judges that share a judge_id are conflated: those of one kind (LLM or "
            "decision model) all call the model of the last of them, and all share one "
            "judge_scores entry, one set of per-judge metrics, and the same option "
            "shuffles and few-shot examples. Give each judge its own judge_id; a "
            "repeated judge_id will be a ValueError in the next major version.",
            FutureWarning,
            stacklevel=2,
        )
    all_judges = [*self._judges, *self._escalation_judges]

    # Checked before any client is built, as the settings below are.
    _check_llm_calls(llm_calls)
    # The seed enters every shuffle and few-shot RNG key as text, so a bool or a float
    # would key other draws than the equal integer. Any integer is kept as a plain int
    # (as ``FewShotConfig.seed`` is), so a numpy integer keys what the int does and the
    # manifest records it.
    if seed is not None:
        checked_seed = _as_int(seed)
        if checked_seed is None:
            raise ValueError(f"seed must be None or an integer; got {seed!r}")
        seed = checked_seed

    # Settings a decision-model judge cannot use, checked before any client is built so
    # that a configuration error is reported as one.
    decision_model_ids = [
        j.judge_id for j in all_judges if isinstance(j.llm_config, DecisionModelConfig)
    ]
    every_judge_is_decision_model = 0 < len(decision_model_ids) == len(all_judges)
    if every_judge_is_decision_model and (
        few_shot_config is not None or training_data is not None
    ):
        raise ValueError(
            "few-shot examples apply to LLM judges only, and every judge of this "
            "grader is a decision model: remove few_shot_config and training_data, "
            "or add an LLM judge"
        )
    response_formats = [
        name
        for name, value in (
            ("binary_response_format", binary_response_format),
            ("multi_choice_response_format", multi_choice_response_format),
        )
        if value is not None
    ]
    if response_formats and decision_model_ids:
        raise ValueError(
            f"{' and '.join(response_formats)} cannot be combined with decision-model "
            f"judges ({', '.join(repr(judge_id) for judge_id in decision_model_ids)}): "
            "a response format describes a generated judgment, which only an LLM judge "
            "produces"
        )
    if response_formats and llm_calls == "per_item":
        raise ValueError(
            f"{' and '.join(response_formats)} cannot be combined with "
            "llm_calls='per_item': a response format describes one criterion's judgment, "
            "and a per_item call answers every criterion at once"
        )
    if every_judge_is_decision_model:
        # A setting that differs from its default asks for something no judge here
        # can do: a prompt (None is "no prompt"), shuffle_options=False (a decision
        # model never shuffles, so True, the default, is what it does anyway), or
        # llm_calls="per_item" (it changes how LLM judges are called, and there are none).
        changed = (
            ("system_prompt", system_prompt is not None),
            ("multi_choice_system_prompt", multi_choice_system_prompt is not None),
            ("shuffle_options", not shuffle_options),
            ("llm_calls", llm_calls != "per_criterion"),
        )
        for name, is_changed in changed:
            if is_changed:
                warnings.warn(
                    f"{name} applies to LLM judges only; this grader has none (every "
                    "judge is a decision model), so it has no effect",
                    UserWarning,
                    stacklevel=2,
                )

    self._aggregation = aggregation
    self._ordinal_aggregation = ordinal_aggregation
    self._nominal_aggregation = nominal_aggregation
    self._training_data = training_data
    self._cannot_assess_config = cannot_assess_config or CannotAssessConfig()
    self._shuffle_options = shuffle_options
    self._auto_na_option = auto_na_option
    self._llm_calls = llm_calls
    self._seed = seed if seed is not None else random.randint(0, 2**31 - 1)

    # The grader's own copy, validated again (``dataclasses.replace`` reruns
    # ``__post_init__``): a field set on the caller's config after it was built is
    # checked here, and later changes to it do not reach this grader.
    fsc = (
        dataclasses.replace(few_shot_config) if few_shot_config is not None else FewShotConfig()
    )
    # Coordinate few-shot seed with master seed when unset
    if fsc.seed is None and training_data is not None:
        # Set on the copy, not through ``dataclasses.replace``: the check in
        # ``FewShotConfig.__post_init__`` is about a seed the caller set on the config,
        # and the grader's own ``seed``, a plain int, was checked above.
        fsc.seed = self._seed
    self._few_shot_config = fsc
    self._binary_response_format = binary_response_format or CriterionJudgment
    self._multi_choice_response_format = multi_choice_response_format or MultiChoiceJudgment

    # Build system prompts (separate for binary and multi-choice)
    if system_prompt is None:
        self._system_prompt = GRADER_SYSTEM_PROMPT_DEFAULT
        if training_data is not None:
            self._system_prompt += FEW_SHOT_SYSTEM_PROMPT_ADDITION
    else:
        self._system_prompt = system_prompt

    if multi_choice_system_prompt is None:
        self._multi_choice_system_prompt = MULTI_CHOICE_SYSTEM_PROMPT
        if training_data is not None:
            self._multi_choice_system_prompt += MULTI_CHOICE_FEW_SHOT_ADDITION
    else:
        self._multi_choice_system_prompt = multi_choice_system_prompt

    # Create each judge's client: an LLMClient for an LLM judge, a DecisionModelClient
    # for a decision-model judge. Building one resolves its credentials, so a missing
    # SDK or API key fails here rather than on every item.
    self._clients: dict[str, LLMClient] = {}
    self._decision_clients: dict[str, DecisionModelClient] = {}
    for judge in all_judges:
        # A spec is mutable, so check the weight it has now, not only the one it was built with
        _check_judge_weight(judge.weight, judge.judge_id)
        config = judge.llm_config
        # Each judge of an ensemble or cascade caches under its own id, so judges sending
        # identical requests never read one another's answers; the lone judge keeps the
        # plain key, and with it the caches of earlier single-judge runs.
        namespace = None if judge.judge_id == _SINGLE_JUDGE_ID else judge.judge_id
        if isinstance(config, DecisionModelConfig):
            self._decision_clients[judge.judge_id] = DecisionModelClient(
                config, cache_namespace=namespace
            )
        else:
            self._clients[judge.judge_id] = LLMClient(config, cache_namespace=namespace)

    # Pre-compute few-shot examples if training data provided
    # Note: For multi-choice, examples are stored as (submission, selected_index, reason)
    self._criterion_examples: dict[tuple[int, str], list[FewShotExample]] = {}
    self._multi_choice_examples: dict[
        tuple[int, str], list[tuple[Content, int, str | None]]
    ] = {}
    # Under llm_calls="per_item": each LLM judge's examples, whole training items in the
    # order its one call per item shows them, each as (submission, labels, reasons): a
    # label per criterion of the training rubric (_item_labels; None where the item has
    # none) and the item's reason for it, if any
    self._item_examples: dict[
        str, list[tuple[Content, list[CriterionVerdict | int | None], list[str | None]]]
    ] = {}
    if training_data is not None:
        self._prepare_examples()
        # Training images go out with every call of the judges that show them, text
        # items' calls included, so those judges are checked now, once.
        self._check_examples_reach_judges_accepting_images()

is_ensemble property

is_ensemble: bool

Whether this grader has several judges that each judge every criterion.

A cascade is not an ensemble: its escalation judges vote only on the criteria its decision model escalates.

judge_ids property

judge_ids: list[str]

The judge_id of every judge, in order: the judges that judge every criterion (judges, or the single "default" judge), then a cascade's escalation judges.

has_few_shot property

has_few_shot: bool

Whether this grader uses few-shot prompting.

seed property

seed: int

The master seed governing all non-LLM randomness in this grader.

judge async

judge(to_grade: Content, rubric: list[Criterion], query: Content | None = None, reference_submission: Content | None = None, *, guidelines: str | None = None) -> list[JudgeCriterionResults]

Judge all criteria with all judges (parallel across judges).

In a cascade the decision model judges every criterion first; then every escalation judge judges the escalated criteria (_escalates: an error, an abstention, or a confidence below EscalationConfig.threshold_for the criterion), all at once, and none when nothing is escalated: in a call per escalated criterion, or, under llm_calls="per_item", in one call about the whole rubric whose judgments of the other criteria are discarded (_judge_escalated_criteria).

The submission, query and reference submission are put in canonical form and their image files read (_load_content) before any judge is called, idempotently: content Grader.grade has loaded reads no file again, so a grade reads each file once, and a direct call to judge is as safe as one through grade. Every judge gets the same loaded content; an LLM judge receives a document's images as content parts in its user prompt, and a decision model as its request's assets, each fitted into the envelope the model accepts.

PARAMETER DESCRIPTION
to_grade

The submission, as Grader.grade passes it: text, or a document of text and images.

TYPE: Content

rubric

The criteria.

TYPE: list[Criterion]

query

Optional input/query that prompted the submission.

TYPE: Content | None DEFAULT: None

reference_submission

Optional exemplar response for grading context.

TYPE: Content | None DEFAULT: None

guidelines

Optional rubric guidelines (Grader.grade passes them when the rubric has any; blank text means none). Every judge sees them: each LLM user prompt starts with GUIDELINES_BLOCK, and a decision model's state holds them as its first field, with the framed binary questions naming them. With none, every prompt and request is exactly what it is without guidelines.

TYPE: str | None DEFAULT: None

RETURNS DESCRIPTION
list[JudgeCriterionResults]

One JudgeCriterionResults per judge, in judge order: in a cascade, the

list[JudgeCriterionResults]

decision model's (role="primary"), then each escalation judge's

list[JudgeCriterionResults]

(role="escalation", None at the criteria not escalated).

RAISES DESCRIPTION
TypeError

If guidelines is neither a str nor None, or the submission, query or reference submission is not Content.

OSError

If an image file cannot be read (FileNotFoundError when it is missing); raised before any judge is called.

ImportError

If a decision-model judge is given content holding an image and the installed TypeSafe SDK cannot send images (before its request).

ValueError

If an image file is not a supported image, or the content holds an image and a judge that could be sent it does not accept images (_check_judges_accept_images), before any judge is called: an LLM judge whose LLMConfig(images=False) says so, or whose model LiteLLM's model map does not list as accepting images with images unset; or a decision-model judge whose model is neither in autorubric's decision-model registry nor given capabilities, or does not accept images; if a decision-model judge is given an image it cannot fit for its model, or images over its model's max_request_bytes together (before its request); or if a judge's call or request fails with a configuration error, which would fail every call it makes (#40): LiteLLM's UnsupportedParamsError, or a 401, 403 or 404 from the provider (litellm.AuthenticationError, PermissionDeniedError, NotFoundError: e.g. an invalid API key or an unknown model; a 403 also when LiteLLM reports it as another API error, as it does for OpenAI-compatible providers, Anthropic and Gemini, but not OpenRouter's, which flags an input by moderation) or from TypeSafe (TypeSafeAuthenticationError, TypeSafePermissionDeniedError, TypeSafeNotFoundError), or the OpenAI SDK's refusal to build its client before any request (e.g. no OpenAI or Azure API key), which LiteLLM wraps in an InternalServerError (an APIError for Azure). The message names the judge, and the provider's exception is its __cause__. This holds for every judge (a single judge, a panel member, a decision model, a cascade's escalation judge), never as an abstention; the judge's other calls for the item are cancelled. Every other failure is routed as before (classify_grading_error), item-specific 400s such as ContextWindowExceededError and ContentPolicyViolationError included.

WARNS DESCRIPTION
UserWarning

Once per grader, when an EscalationConfig.per_criterion name matches no criterion of rubric (not while EvalRunner or fill_ground_truth grades a dataset, which checks the names against every item's rubric instead).

UserWarning

With training data, once per grader per (criterion index, graded question), when a criterion of rubric is shown no few-shot examples because it does not pose the training rubric's question at its index (_warn_examples_not_shown).

Source code in src/autorubric/graders/criterion_grader.py
async def judge(
    self,
    to_grade: Content,
    rubric: list[Criterion],
    query: Content | None = None,
    reference_submission: Content | None = None,
    *,
    guidelines: str | None = None,
) -> list[JudgeCriterionResults]:
    """Judge all criteria with all judges (parallel across judges).

    In a cascade the decision model judges every criterion first; then every escalation
    judge judges the escalated criteria (``_escalates``: an error, an abstention, or a
    confidence below ``EscalationConfig.threshold_for`` the criterion), all at once, and
    none when nothing is escalated: in a call per escalated criterion, or, under
    ``llm_calls="per_item"``, in one call about the whole rubric whose judgments of the
    other criteria are discarded (``_judge_escalated_criteria``).

    The submission, query and reference submission are put in canonical form and their
    image files read (``_load_content``) before any judge is called, idempotently:
    content ``Grader.grade`` has loaded reads no file again, so a grade reads each file
    once, and a direct call to ``judge`` is as safe as one through ``grade``. Every
    judge gets the same loaded content; an LLM judge receives a document's images as
    content parts in its user prompt, and a decision model as its request's assets,
    each fitted into the envelope the model accepts.

    Args:
        to_grade: The submission, as ``Grader.grade`` passes it: text, or a document
            of text and images.
        rubric: The criteria.
        query: Optional input/query that prompted the submission.
        reference_submission: Optional exemplar response for grading context.
        guidelines: Optional rubric guidelines (``Grader.grade`` passes them when the
            rubric has any; blank text means none). Every judge sees them: each LLM user
            prompt starts with ``GUIDELINES_BLOCK``, and a decision model's state holds
            them as its first field, with the framed binary questions naming them. With
            none, every prompt and request is exactly what it is without guidelines.

    Returns:
        One ``JudgeCriterionResults`` per judge, in judge order: in a cascade, the
        decision model's (``role="primary"``), then each escalation judge's
        (``role="escalation"``, ``None`` at the criteria not escalated).

    Raises:
        TypeError: If ``guidelines`` is neither a ``str`` nor ``None``, or the
            submission, query or reference submission is not ``Content``.
        OSError: If an image file cannot be read (``FileNotFoundError`` when it is
            missing); raised before any judge is called.
        ImportError: If a decision-model judge is given content holding an image and the
            installed TypeSafe SDK cannot send images (before its request).
        ValueError: If an image file is not a supported image, or the content holds an
            image and a judge that could be sent it does not accept images
            (``_check_judges_accept_images``), before any judge is called: an LLM judge
            whose ``LLMConfig(images=False)`` says so, or whose model LiteLLM's model
            map does not list as accepting images with ``images`` unset; or a
            decision-model judge whose model is neither in autorubric's decision-model
            registry nor given ``capabilities``, or does not accept images; if a
            decision-model judge is given an image it cannot fit for its model, or
            images over its model's ``max_request_bytes`` together (before its
            request); or if a judge's call or request
            fails with a configuration error, which would fail every call it makes
            (#40): LiteLLM's ``UnsupportedParamsError``, or a 401, 403 or 404 from
            the provider (``litellm.AuthenticationError``, ``PermissionDeniedError``,
            ``NotFoundError``: e.g. an invalid API key or an unknown model; a 403 also
            when LiteLLM reports it as another API error, as it does for
            OpenAI-compatible providers, Anthropic and Gemini, but not OpenRouter's,
            which flags an input by moderation) or from
            TypeSafe (``TypeSafeAuthenticationError``, ``TypeSafePermissionDeniedError``,
            ``TypeSafeNotFoundError``), or the OpenAI SDK's refusal to build its client
            before any request (e.g. no OpenAI or Azure API key), which LiteLLM wraps
            in an ``InternalServerError`` (an ``APIError`` for Azure). The message
            names the judge, and the provider's exception is its ``__cause__``. This
            holds for every judge (a single judge, a panel member, a decision model, a
            cascade's escalation judge), never as an abstention; the judge's other
            calls for the item are cancelled. Every other failure is routed as before
            (``classify_grading_error``), item-specific 400s such as
            ``ContextWindowExceededError`` and ``ContentPolicyViolationError`` included.

    Warns:
        UserWarning: Once per grader, when an ``EscalationConfig.per_criterion`` name
            matches no criterion of ``rubric`` (not while ``EvalRunner`` or
            ``fill_ground_truth`` grades a dataset, which checks the names against
            every item's rubric instead).
        UserWarning: With training data, once per grader per (criterion index, graded
            question), when a criterion of ``rubric`` is shown no few-shot examples
            because it does not pose the training rubric's question at its index
            (``_warn_examples_not_shown``).
    """
    # Checked before any judge runs; blank guidelines become None.
    guidelines = _normalize_guidelines(guidelines)
    # Every judge, a cascade's escalation judges included, gets this one loaded copy.
    # Content Grader.grade passes is loaded already, so nothing is read again here; a
    # direct call reads each image file once, here, before any judge is called.
    to_grade = await _load_content_async(to_grade)
    if query is not None:
        query = await _load_content_async(query)
    if reference_submission is not None:
        reference_submission = await _load_content_async(reference_submission)
    # Before any judge is called, so that no request at all is sent, by any judge, for
    # images one of the judges that could be sent them does not accept.
    self._check_judges_accept_images(to_grade, query, reference_submission)
    if not self._escalation_names_warned and not _ESCALATION_NAMES_CHECKED.get():
        # The flag is set only once the warning is issued, so under a
        # warnings-as-errors filter it raises on every call instead of only once.
        if self._check_escalation_names([rubric], "the rubric being graded"):
            self._escalation_names_warned = True
    # Normalize once to the effective rubric (abstain channel guaranteed for
    # multi-choice under auto_na_option). Same length/order, so criterion_idx — and
    # thus the shuffle RNG key — stays aligned; the user's rubric is never mutated.
    effective_rubric = [self._effective_criterion(c) for c in rubric]
    self._warn_examples_not_shown(effective_rubric)
    tasks = [
        self._judge_all_criteria_for_judge(
            judge,
            effective_rubric,
            to_grade,
            query,
            reference_submission,
            guidelines=guidelines,
        )
        for judge in self._judges
    ]
    results = list(await _gather_or_cancel(*tasks))
    escalation = self._escalation
    if escalation is None:
        return results

    # A cascade: ``_judges`` is its one decision model, which judged every criterion.
    (primary,) = results
    escalated = [
        criterion_idx
        for criterion_idx, (criterion, report) in enumerate(
            zip(effective_rubric, primary.reports, strict=True)
        )
        if _escalates(report, escalation.threshold_for(criterion))
    ]
    escalation_results = await _gather_or_cancel(
        *(
            self._judge_escalated_criteria(
                judge,
                effective_rubric,
                escalated,
                to_grade,
                query,
                reference_submission,
                guidelines=guidelines,
            )
            for judge in self._escalation_judges
        )
    )
    return [primary, *escalation_results]

aggregate async

aggregate(judge_results: list[JudgeCriterionResults], *, normalize: bool = True) -> EnsembleEvaluationReport

Aggregate results from all judges into final report.

Handles both binary and multi-choice criteria: - Binary: Uses JudgeVote and _aggregate_votes() - Multi-choice: Uses MultiChoiceJudgeVote and _aggregate_multi_choice_votes()

In a cascade the escalated criteria are the ones the escalation judges have results for (their results are None everywhere else). On those, the primary judge's vote is marked superseded=True and the final verdict, reason and error come from the votes that are not superseded, which are the escalation judges'; the report is marked escalated=True. judge_scores holds each primary judge's score over its own verdicts (superseded ones included) and None for each escalation judge, by role: it votes only where the decision model escalated, even on an item where every criterion escalated.

Source code in src/autorubric/graders/criterion_grader.py
async def aggregate(
    self, judge_results: list[JudgeCriterionResults], *, normalize: bool = True
) -> EnsembleEvaluationReport:
    """Aggregate results from all judges into final report.

    Handles both binary and multi-choice criteria:
    - Binary: Uses JudgeVote and _aggregate_votes()
    - Multi-choice: Uses MultiChoiceJudgeVote and _aggregate_multi_choice_votes()

    In a cascade the escalated criteria are the ones the escalation judges have results
    for (their results are ``None`` everywhere else). On those, the primary judge's vote
    is marked ``superseded=True`` and the final verdict, reason and error come from the
    votes that are not superseded, which are the escalation judges'; the report is
    marked ``escalated=True``. ``judge_scores`` holds each primary judge's score over its
    own verdicts (superseded ones included) and ``None`` for each escalation judge, by
    role: it votes only where the decision model escalated, even on an item where every
    criterion escalated.
    """
    if not judge_results:
        # Empty/failed aggregation has no score: emit None, not a fabricated 0.0
        # (which is a valid catastrophic score, indistinguishable from a real zero).
        return EnsembleEvaluationReport(
            score=None,
            raw_score=None,
            llm_raw_score=None,
            error="No judge results to aggregate",
        )

    n_criteria = len(judge_results[0].criterion_results)
    # A cascade's escalated criteria: an escalation judge has a result at each of them,
    # and only there (none at all without a cascade).
    escalated = {
        criterion_idx
        for judge_result in judge_results
        if judge_result.role == "escalation"
        for criterion_idx, cr in enumerate(judge_result.criterion_results)
        if cr is not None
    }

    # Build ensemble criterion reports
    ensemble_reports: list[EnsembleCriterionReport] = []
    for criterion_idx in range(n_criteria):
        is_escalated = criterion_idx in escalated
        # Each judge's result on this criterion, in judge order; an escalation judge
        # has one only where the criterion was escalated.
        judged: list[tuple[JudgeCriterionResults, CriterionResult]] = [
            (judge_result, cr)
            for judge_result in judge_results
            if (cr := judge_result.criterion_results[criterion_idx]) is not None
        ]
        # Get criterion from the first judge's result
        criterion_report = judged[0][1].report

        if criterion_report.is_multi_choice:
            # Multi-choice: build MultiChoiceJudgeVote list
            mc_votes: list[MultiChoiceJudgeVote] = []
            # Each MultiChoiceJudgeVote carries its own .error (parity with JudgeVote),
            # so the ensemble error is derived from the votes via _aggregate_error,
            # mirroring the binary path. Every judge call synthesizes a verdict, so the
            # `mcv is not None` guard below never drops an errored vote.
            for judge_result, cr in judged:
                mcv = cr.report.multi_choice_verdict
                if mcv is not None:
                    mc_votes.append(
                        MultiChoiceJudgeVote(
                            judge_id=judge_result.judge_id,
                            selected_index=mcv.selected_index,
                            selected_label=mcv.selected_label,
                            value=mcv.value,
                            reason=cr.report.reason,
                            weight=judge_result.weight,
                            na=mcv.na,
                            shuffle_order=cr.report.shuffle_order,
                            error=cr.report.error,
                            reasoning=cr.report.reasoning,
                            probabilities=cr.report.probabilities,
                            confidence=cr.report.confidence,
                            superseded=is_escalated and judge_result.role == "primary",
                        )
                    )
            aggregated_mc_votes = [v for v in mc_votes if not v.superseded]

            # Aggregate multi-choice votes
            final_mc_verdict, final_reason = self._aggregate_multi_choice_votes(
                aggregated_mc_votes, criterion_report
            )

            ensemble_reports.append(
                EnsembleCriterionReport(
                    criterion=Criterion(
                        weight=criterion_report.weight,
                        requirement=criterion_report.requirement,
                        name=criterion_report.name,
                        options=criterion_report.options,
                        scale_type=criterion_report.scale_type,
                        aggregation=criterion_report.aggregation,
                    ),
                    final_verdict=None,  # Binary verdict is None for multi-choice
                    final_reason=final_reason,
                    votes=[],  # Binary votes empty for multi-choice
                    final_multi_choice_verdict=final_mc_verdict,
                    multi_choice_votes=mc_votes,
                    error=_aggregate_error(aggregated_mc_votes),
                    escalated=is_escalated,
                )
            )
        else:
            # Binary: build JudgeVote list
            votes: list[JudgeVote] = []
            for judge_result, cr in judged:
                votes.append(
                    JudgeVote(
                        judge_id=judge_result.judge_id,
                        verdict=cr.report.verdict,
                        reason=cr.report.reason,
                        weight=judge_result.weight,
                        error=cr.report.error,
                        reasoning=cr.report.reasoning,
                        probabilities=cr.report.probabilities,
                        confidence=cr.report.confidence,
                        superseded=is_escalated and judge_result.role == "primary",
                    )
                )
            aggregated_votes = [v for v in votes if not v.superseded]

            final_verdict, final_reason = self._aggregate_votes(
                aggregated_votes, criterion_report.weight
            )

            ensemble_reports.append(
                EnsembleCriterionReport(
                    criterion=Criterion(
                        weight=criterion_report.weight,
                        requirement=criterion_report.requirement,
                        name=criterion_report.name,
                    ),
                    final_verdict=final_verdict,
                    final_reason=final_reason,
                    votes=votes,
                    error=_aggregate_error(aggregated_votes),
                    escalated=is_escalated,
                )
            )

    # Calculate per-judge scores: a primary judge's over its own verdicts on every
    # criterion; an escalation judge's is undefined, None by role on every item. A
    # judge none of whose votes on the item it judged (every call failed) has no score
    # for it either, whatever its stand-in verdicts would score.
    judge_scores: dict[str, float | None] = {}
    for judge_result in judge_results:
        judge_scores[judge_result.judge_id] = (
            self._calculate_score_from_reports(judge_result.reports, normalize)
            if judge_result.role == "primary"
            and not _every_judgment_failed(judge_result.reports)
            else None
        )

    # Aggregate token usage and cost
    total_usage = TokenUsage()
    total_cost = 0.0
    for jr in judge_results:
        if jr.total_usage:
            total_usage = total_usage + jr.total_usage
        if jr.total_cost:
            total_cost += jr.total_cost

    return _ensemble_evaluation_report(
        ensemble_reports,
        judge_scores,
        self._calculate_score_from_reports,
        normalize=normalize,
        token_usage=total_usage if total_usage.total_tokens > 0 else None,
        completion_cost=total_cost if total_cost > 0 else None,
    )

Grader

Abstract base class for grader implementations.

Grader is generic in the report type it returns. CriterionGrader is a Grader[EnsembleEvaluationReport], so a type checker types await rubric.grade(..., grader=grader) as EnsembleEvaluationReport. A custom grader declares its own, for example class MyGrader(Grader[EvaluationReport]); one that names no type still works.

Changed: judge takes Content

The abstract judge annotates to_grade, query and reference_submission as Content (a string, an Image, or a sequence of both) instead of str, because a submission may hold images. A subclass whose judge still annotates them as str runs unchanged, but a type checker reports its override as incompatible with the base class; annotate them as Content. grade now raises TypeError for a query or reference submission that is not Content (for example a number), where earlier releases passed it through. See Submissions with Images.

Grader

Grader(*, length_penalty: LengthPenalty | None = None, normalize: bool = True)

Bases: ABC, Generic[ReportT]

Base class for LLM-backed grading implementations.

All graders require an LLMConfig for the LLM client. Subclasses must implement judge() and aggregate() methods.

Rubric guidelines reach judge only if it accepts them: a judge with a guidelines parameter that can be passed by keyword (e.g. keyword-only guidelines: str | None = None) or with **kwargs receives them as guidelines=. A judge with the abstract signature keeps working unchanged; the grader then grades without the guidelines and warns once (see grade).

PARAMETER DESCRIPTION
length_penalty

Optional configuration for penalizing overly long outputs. When provided, a penalty based on the token/word count is subtracted from the final score.

TYPE: LengthPenalty | None DEFAULT: None

normalize

If True (default), scores are normalized to 0-1. If False, raw weighted sums are returned, which is useful for RL training scenarios.

TYPE: bool DEFAULT: True

Source code in src/autorubric/graders/base.py
def __init__(
    self,
    *,
    length_penalty: LengthPenalty | None = None,
    normalize: bool = True,
):
    self.length_penalty: LengthPenalty | None = length_penalty
    self.normalize: bool = normalize

judge abstractmethod async

judge(to_grade: Content, rubric: list[Criterion], query: Content | None = None, reference_submission: Content | None = None) -> Any

Collect raw judge results for the provided submission.

An implementation may also take the rubric's guidelines, as a guidelines parameter that can be passed by keyword or through **kwargs; grade passes them, when there are any, only to an implementation that does.

PARAMETER DESCRIPTION
to_grade

The submission to evaluate, as grade passes it: text, or a document of text and images in canonical form (a tuple of parts) whose images hold their bytes. A text submission is the str given to grade (a sequence of strings alone is joined, a {"thinking", "output"} dict flattened into marked sections), so a judge written for text receives text, as before.

TYPE: Content

rubric

List of criteria to evaluate against.

TYPE: list[Criterion]

query

Optional input/query that prompted the response, in the same form.

TYPE: Content | None DEFAULT: None

reference_submission

Optional exemplar response for grading context, in the same form.

TYPE: Content | None DEFAULT: None

RETURNS DESCRIPTION
Any

Raw judge results (format depends on implementation).

Source code in src/autorubric/graders/base.py
@abstractmethod
async def judge(
    self,
    to_grade: Content,
    rubric: list[Criterion],
    query: Content | None = None,
    reference_submission: Content | None = None,
) -> Any:
    """Collect raw judge results for the provided submission.

    An implementation may also take the rubric's guidelines, as a ``guidelines``
    parameter that can be passed by keyword or through ``**kwargs``; ``grade`` passes
    them, when there are any, only to an implementation that does.

    Args:
        to_grade: The submission to evaluate, as ``grade`` passes it: text, or a
            document of text and images in canonical form (a tuple of parts) whose
            images hold their bytes. A text submission is the ``str`` given to
            ``grade`` (a sequence of strings alone is joined, a ``{"thinking",
            "output"}`` dict flattened into marked sections), so a ``judge`` written
            for text receives text, as before.
        rubric: List of criteria to evaluate against.
        query: Optional input/query that prompted the response, in the same form.
        reference_submission: Optional exemplar response for grading context, in the
            same form.

    Returns:
        Raw judge results (format depends on implementation).
    """
    pass

aggregate abstractmethod async

aggregate(judge_results: Any, *, normalize: bool = True) -> ReportT

Transform judge results into an EvaluationReport.

PARAMETER DESCRIPTION
judge_results

Raw results from judge().

TYPE: Any

normalize

If True, normalize score to 0-1. If False, return raw weighted sum.

TYPE: bool DEFAULT: True

RETURNS DESCRIPTION
ReportT

EvaluationReport with score and optional per-criterion breakdown.

Source code in src/autorubric/graders/base.py
@abstractmethod
async def aggregate(self, judge_results: Any, *, normalize: bool = True) -> ReportT:
    """Transform judge results into an EvaluationReport.

    Args:
        judge_results: Raw results from judge().
        normalize: If True, normalize score to 0-1. If False, return raw weighted sum.

    Returns:
        EvaluationReport with score and optional per-criterion breakdown.
    """
    pass

grade async

grade(to_grade: ToGradeInput, rubric: list[Criterion], query: Content | None = None, reference_submission: Content | None = None, *, guidelines: str | None = None) -> ReportT

Grade the submission against the rubric.

This is the main entry point for the grader.

Before judge is called, the submission, query and reference submission are put in canonical form (_normalize_content) and every image file they refer to is read, once per grade (_load_content), in a worker thread (asyncio.to_thread); content without such a file reads nothing and is passed on as it is (text as the same str). A missing, unreadable or unsupported image file therefore raises from grade before any judge is called: it is a fault of the item's data, never a judge's failed judgment, so it is not routed into a verdict.

PARAMETER DESCRIPTION
to_grade

The submission to evaluate. Can be either: - Content: a string (optionally with / markers), an Image, or a sequence of strings and images read in order - A dict with 'thinking' and 'output' keys, both text

TYPE: ToGradeInput

rubric

List of criteria to evaluate against.

TYPE: list[Criterion]

query

Optional input/query that prompted the response (Content).

TYPE: Content | None DEFAULT: None

reference_submission

Optional exemplar response for grading context (Content).

TYPE: Content | None DEFAULT: None

guidelines

Optional rubric guidelines: free text that applies to every criterion (Rubric.grade passes Rubric.guidelines). Blank text means none. They are passed to judge as guidelines= only if judge accepts that keyword (a guidelines parameter or **kwargs; checked once per grader instance). Otherwise, and whenever there are no guidelines, judge is called exactly as without them: judge(to_grade, rubric, query, reference_submission).

TYPE: str | None DEFAULT: None

RETURNS DESCRIPTION
ReportT

EvaluationReport with score and optional per-criterion breakdown.

ReportT

If normalize=True (default), score is 0-1. If normalize=False, score is raw

ReportT

weighted sum. If length_penalty was configured, the penalty is subtracted from

ReportT

the score. The raw_score field contains the unnormalized weighted sum before

ReportT

length penalty.

RAISES DESCRIPTION
TypeError

If guidelines is neither a str nor None; if the submission is a sequence holding a part that is neither a str nor an Image, or the query or reference submission is not Content; or if a section of a {"thinking", "output"} submission holds an image or a sequence. Any other submission that is not Content is read as that dict form, as before images existed: a mapping with neither key is graded as "", and a value with no get (a number) raises AttributeError.

OSError

If an image file cannot be read (FileNotFoundError when it is missing).

ValueError

If an image file is empty, not an image, or not PNG, JPEG, GIF or WebP.

WARNS DESCRIPTION
UserWarning

" ignores rubric guidelines", once per grader instance, when guidelines are given and judge does not accept them.

Source code in src/autorubric/graders/base.py
async def grade(
    self,
    to_grade: ToGradeInput,
    rubric: list[Criterion],
    query: Content | None = None,
    reference_submission: Content | None = None,
    *,
    guidelines: str | None = None,
) -> ReportT:
    """Grade the submission against the rubric.

    This is the main entry point for the grader.

    Before ``judge`` is called, the submission, query and reference submission are put
    in canonical form (``_normalize_content``) and every image file they refer to is
    read, once per grade (``_load_content``), in a worker thread
    (``asyncio.to_thread``); content without such a file reads nothing and is passed
    on as it is (text as the same ``str``). A missing, unreadable or unsupported image
    file therefore raises from ``grade`` before any judge is called: it is a fault of
    the item's data, never a judge's failed judgment, so it is not routed into a
    verdict.

    Args:
        to_grade: The submission to evaluate. Can be either:
            - ``Content``: a string (optionally with <thinking>/<output> markers), an
              ``Image``, or a sequence of strings and images read in order
            - A dict with 'thinking' and 'output' keys, both text
        rubric: List of criteria to evaluate against.
        query: Optional input/query that prompted the response (``Content``).
        reference_submission: Optional exemplar response for grading context
            (``Content``).
        guidelines: Optional rubric guidelines: free text that applies to every
            criterion (``Rubric.grade`` passes ``Rubric.guidelines``). Blank text means
            none. They are passed to ``judge`` as ``guidelines=`` only if ``judge``
            accepts that keyword (a ``guidelines`` parameter or ``**kwargs``; checked
            once per grader instance). Otherwise, and whenever there are no guidelines,
            ``judge`` is called exactly as without them:
            ``judge(to_grade, rubric, query, reference_submission)``.

    Returns:
        EvaluationReport with score and optional per-criterion breakdown.
        If normalize=True (default), score is 0-1. If normalize=False, score is raw
        weighted sum. If length_penalty was configured, the penalty is subtracted from
        the score. The raw_score field contains the unnormalized weighted sum before
        length penalty.

    Raises:
        TypeError: If ``guidelines`` is neither a ``str`` nor ``None``; if the
            submission is a sequence holding a part that is neither a ``str`` nor an
            ``Image``, or the query or reference submission is not ``Content``; or if a
            section of a ``{"thinking", "output"}`` submission holds an image or a
            sequence. Any other submission that is not ``Content`` is read as that dict
            form, as before images existed: a mapping with neither key is graded as
            ``""``, and a value with no ``get`` (a number) raises ``AttributeError``.
        OSError: If an image file cannot be read (``FileNotFoundError`` when it is
            missing).
        ValueError: If an image file is empty, not an image, or not PNG, JPEG, GIF or
            WebP.

    Warns:
        UserWarning: "<GraderClass> ignores rubric guidelines", once per grader
            instance, when guidelines are given and ``judge`` does not accept them.
    """
    guidelines = _normalize_guidelines(guidelines)
    if guidelines is not None and not _takes_guidelines(self, "judge"):
        _warn_ignores_guidelines(self)
        guidelines = None

    # The dict form becomes text with section markers, as before Content admitted
    # images; Content is read once here, its image files off the event loop.
    submission: Content
    if isinstance(to_grade, str | Image | Sequence):
        submission = await _load_content_async(to_grade)
    else:
        submission = _thinking_output_text(to_grade)
    if query is not None:
        query = await _load_content_async(query)
    if reference_submission is not None:
        reference_submission = await _load_content_async(reference_submission)

    # Guidelines go as a keyword, and only to a judge that accepts it (checked above);
    # the abstract signature does not declare it, hence the cast.
    judge = cast(Callable[..., Awaitable[Any]], self.judge)
    if guidelines is None:
        judge_results = await judge(submission, rubric, query, reference_submission)
    else:
        judge_results = await judge(
            submission, rubric, query, reference_submission, guidelines=guidelines
        )
    report = await self.aggregate(judge_results, normalize=self.normalize)

    if self.length_penalty is not None:
        # A grade-FAILURE has no score (errored/empty report): there is nothing to
        # penalize, so return it unchanged rather than subtracting from None.
        if report.score is None:
            return report

        # Normalize to_grade to dict format for penalty calculation (text only: an
        # image counts as the newline _content_text puts in its place)
        to_grade_normalized = normalize_to_grade_input(to_grade)

        # Compute penalty
        penalty = compute_length_penalty(to_grade_normalized, self.length_penalty)

        # Apply penalty (penalty is always non-negative, so we subtract)
        adjusted_score = report.score - penalty
        if self.normalize:
            adjusted_score = max(0.0, adjusted_score)

        # The same report, of the same type, with the adjusted score
        return report.model_copy(update={"score": adjusted_score})

    return report

JudgeSpec

Configuration for a single judge in an ensemble.

JudgeSpec dataclass

JudgeSpec(llm_config: LLMConfig | DecisionModelConfig, judge_id: str, weight: float = 1.0)
JudgeSpec(*, judge_model_config: LLMConfig | DecisionModelConfig, judge_id: str, weight: float = 1.0)
JudgeSpec(llm_config: Any = MISSING, judge_id: Any = MISSING, weight: float = 1.0, *, judge_model_config: Any = MISSING)

Specification for a single judge in an ensemble.

judge_model_config is the preferred keyword for the judge's configuration, and the positional form is unchanged. llm_config= keeps working without a warning: it is the name of the stored dataclass field, which is deliberately not renamed so that dataclasses.fields/asdict/replace, the repr, equality and pickles stay exactly as they were. dataclasses.replace therefore takes llm_config=, and passing both judge_model_config= and llm_config= raises ValueError.

The configuration decides the judge's kind: an LLMConfig makes an LLM judge, which is asked about each criterion in its own call (or about every criterion of an item in one call, when the grader's llm_calls is "per_item"), and a DecisionModelConfig makes a decision-model judge, which is asked about every criterion of an item in one request. Both kinds mix freely in one ensemble.

ATTRIBUTE DESCRIPTION
llm_config

The judge's model configuration, an LLMConfig or a DecisionModelConfig (the stored field; also readable and writable as the judge_model_config property).

TYPE: LLMConfig | DecisionModelConfig

judge_id

Unique identifier for this judge (e.g., "gpt-4", "claude-sonnet").

TYPE: str

weight

Voting weight for weighted aggregation (default 1.0); a positive, finite number.

TYPE: float

Example

gemini = LLMConfig(model="gemini/gemini-3-flash-preview") JudgeSpec(judge_model_config=gemini, judge_id="gemini") JudgeSpec(gemini, "gemini", weight=2.0) # positional form JudgeSpec(DecisionModelConfig(model="jev-latest"), "jev") # a decision model

Initialize from judge_model_config (or its stored-field name llm_config).

PARAMETER DESCRIPTION
llm_config

The judge's model configuration, positionally or by the stored field name. Mutually exclusive with judge_model_config.

TYPE: Any DEFAULT: MISSING

judge_id

Unique identifier for this judge.

TYPE: Any DEFAULT: MISSING

weight

Voting weight for weighted aggregation (default 1.0); a positive, finite number.

TYPE: float DEFAULT: 1.0

judge_model_config

The judge's model configuration (preferred keyword).

TYPE: Any DEFAULT: MISSING

RAISES DESCRIPTION
ValueError

If both judge_model_config and llm_config are passed, or if weight is not a positive, finite number.

TypeError

If the configuration or judge_id is missing.

Source code in src/autorubric/graders/criterion_grader.py
def __init__(
    self,
    llm_config: Any = dataclasses.MISSING,
    judge_id: Any = dataclasses.MISSING,
    weight: float = 1.0,
    *,
    judge_model_config: Any = dataclasses.MISSING,
) -> None:
    """Initialize from ``judge_model_config`` (or its stored-field name ``llm_config``).

    Args:
        llm_config: The judge's model configuration, positionally or by the stored
            field name. Mutually exclusive with ``judge_model_config``.
        judge_id: Unique identifier for this judge.
        weight: Voting weight for weighted aggregation (default 1.0); a positive,
            finite number.
        judge_model_config: The judge's model configuration (preferred keyword).

    Raises:
        ValueError: If both ``judge_model_config`` and ``llm_config`` are passed, or if
            ``weight`` is not a positive, finite number.
        TypeError: If the configuration or ``judge_id`` is missing.
    """
    if judge_model_config is not dataclasses.MISSING:
        if llm_config is not dataclasses.MISSING:
            raise ValueError(
                "Pass only one of judge_model_config and llm_config "
                "(llm_config is the stored field name of judge_model_config)"
            )
        llm_config = judge_model_config
    missing = []
    if llm_config is dataclasses.MISSING:
        missing.append("'judge_model_config' (or its alias 'llm_config')")
    if judge_id is dataclasses.MISSING:
        missing.append("'judge_id'")
    if missing:
        # Mirror Python's own wording; a lone judge_id is exactly the positional
        # argument the generated dataclass __init__ used to report.
        kind = "positional argument" if missing == ["'judge_id'"] else "argument"
        plural = "s" if len(missing) > 1 else ""
        raise TypeError(
            f"JudgeSpec.__init__() missing {len(missing)} required {kind}{plural}: "
            + " and ".join(missing)
        )
    _check_judge_weight(weight, judge_id)
    self.llm_config = llm_config
    self.judge_id = judge_id
    self.weight = weight

judge_model_config property writable

judge_model_config: LLMConfig | DecisionModelConfig

The judge's model configuration (read/write alias of the llm_config field).


LLMCalls

How CriterionGrader calls each LLM judge for an item: "per_criterion" (the default) or "per_item".

LLMCalls module-attribute

LLMCalls = Literal['per_criterion', 'per_item']

How CriterionGrader calls each LLM judge for an item (its llm_calls).

  • per_criterion: One call per criterion (the default).
  • per_item: One call for the whole rubric: the submission and its context are sent once, every criterion is listed under its id (c0, c1, ...), and the answer holds one judgment per criterion (RubricJudgment).

A cascade's escalation judges are called the same way, only where the decision model escalated: per_item makes one full-rubric call for an item with any escalated criterion. replay_escalation and calibrate_escalation take it to know how an LLM run was called, which sets how they estimate a cascade's cost and time from that run's.

A decision-model judge always makes one request per item, whatever this is. The name counts LLM calls per item; it is unrelated to compute_metrics(per_item_metrics=...) and to per-item rubrics.