Graders¶
Grader implementations for evaluating responses against rubrics.
Overview¶
Graders evaluate responses against rubrics and return structured reports. The main implementation is CriterionGrader, which supports single LLM, ensemble, and few-shot modes. All combinations work orthogonally.
Quick Example¶
from autorubric import LLMConfig, FewShotConfig
from autorubric.graders import CriterionGrader, JudgeSpec, Grader
# Single LLM mode
grader = CriterionGrader(
judge_model_config=LLMConfig(model="openai/gpt-5.6-luna"),
)
# One call per item: the judge grades the whole rubric in a single call
# instead of one call per criterion
grader = CriterionGrader(
judge_model_config=LLMConfig(model="openai/gpt-5.6-luna"),
llm_calls="per_item",
)
# With custom system prompt
grader = CriterionGrader(
judge_model_config=LLMConfig(model="openai/gpt-5.6-luna"),
system_prompt="You are evaluating technical documentation...",
)
# Ensemble mode
grader = CriterionGrader(
judges=[
JudgeSpec(LLMConfig(model="gemini/gemini-3-flash-preview"), "gemini", weight=1.0),
JudgeSpec(LLMConfig(model="anthropic/claude-sonnet-4-5-20250929"), "claude", weight=1.2),
],
aggregation="weighted",
)
# Single LLM + few-shot
grader = CriterionGrader(
judge_model_config=LLMConfig(model="openai/gpt-5.6-luna"),
training_data=train_data,
few_shot_config=FewShotConfig(n_examples=3, balance_verdicts=True),
)
# Grade
result = await rubric.grade(to_grade=response, grader=grader)
judge_model_config sets up a single judge and judges sets up an ensemble. Pass exactly one of them.
Either accepts a DecisionModelConfig as well as an LLMConfig: a decision-model judge grades each item with one request for the whole rubric. escalation=EscalationConfig(...) makes a grader with one decision-model judge a confidence cascade that escalates uncertain criteria to LLM judges. See Decision Models for the API, and the LLM Judges and Decision-Model Judges chapters for guided introductions to the two judge kinds.
llm_calls="per_item" makes every LLM judge of the grader — the judge of judge_model_config, each judge of judges, and a cascade's escalation judges — grade an item's whole rubric in one call instead of one call per criterion, the way a decision-model judge always does. The default, "per_criterion", is unchanged. See Grading a whole rubric in one call for what the single call looks like, and Decision-Model Judges for what it shares with a decision model's request.
llm_config is deprecated
llm_config is a deprecated alias of judge_model_config. It builds the same judge but emits a
DeprecationWarning. To migrate, rename the keyword; nothing else changes. Passing both raises
ValueError. llm_config will not be removed before the next major version.
Grading Options¶
grader = CriterionGrader(
judge_model_config=LLMConfig(model="openai/gpt-5.6-luna"),
# Score normalization
normalize=True, # True: 0-1 range, False: raw weighted sum
# CANNOT_ASSESS handling
cannot_assess_config=CannotAssessConfig(strategy=CannotAssessStrategy.SKIP),
# Length penalty
length_penalty=LengthPenalty(free_budget=6000, max_cap=8000),
# Position bias mitigation (for multi-choice)
shuffle_options=True, # Default: enabled
# Multi-choice abstain channel
auto_na_option=True, # Default: True — guarantees every multi-choice criterion a
# first-class NA/abstain option (auto-injected if absent).
# Set False for forced-choice (no auto NA option).
# Reproducibility — pins all non-LLM randomness (shuffles, few-shot selection)
seed=42, # Default: auto-generated
)
With auto_na_option=True, an auto-injected NA option is appended at the end (highest index), so existing option indices are preserved; an author-supplied NA option is never stripped.
seed must be None or an integer. Numpy integers are accepted and kept as a plain int, so seed=np.int64(42) shuffles and selects examples exactly as seed=42 does, and the experiment manifest records it as master_seed. The grader raises ValueError for a bool, a float such as 1.0, or a string. The seed is hashed as text, so earlier releases accepted a bool or a float and silently produced different shuffles and few-shot draws from the equal integer (True from 1, 1.0 from 1).
CriterionGrader¶
Main grader with support for single LLM, ensemble, and few-shot modes.
CriterionGrader
¶
CriterionGrader(*, judge_model_config: LLMConfig | DecisionModelConfig | None = None, judges: list[JudgeSpec] | None = None, aggregation: AggregationStrategy = 'majority', ordinal_aggregation: OrdinalAggregation = 'mean', nominal_aggregation: NominalAggregation = 'mode', training_data: RubricDataset | None = None, few_shot_config: FewShotConfig | None = None, system_prompt: str | None = None, multi_choice_system_prompt: str | None = None, length_penalty: LengthPenalty | None = None, normalize: bool = True, cannot_assess_config: CannotAssessConfig | None = None, shuffle_options: bool = True, auto_na_option: bool = True, seed: int | None = None, binary_response_format: type[BaseModel] | None = None, multi_choice_response_format: type[BaseModel] | None = None, escalation: EscalationConfig | None = None, llm_calls: LLMCalls = 'per_criterion', llm_config: LLMConfig | DecisionModelConfig | None = None)
Bases: Grader[EnsembleEvaluationReport]
Unified criterion-based grader with compositional few-shot and ensemble support.
This grader evaluates each criterion independently and supports: - Single-judge mode (via judge_model_config) - Ensemble mode with multiple judges (via judges) - Few-shot prompting (via training_data + few_shot_config)
All combinations work: single judge, single + few-shot, ensemble, ensemble + few-shot.
Parameters are orthogonal: - judge_model_config OR judges: Choose single-judge or ensemble mode - training_data + few_shot_config: Enable few-shot prompting (applies to every LLM judge)
A judge is an LLM (LLMConfig) or a decision model (DecisionModelConfig), alone
or mixed in one ensemble. An LLM judge is asked about each criterion in its own call. A
decision-model judge is asked about the whole rubric in one request per item: the
submission goes once as shared state and every criterion it can express becomes one
question; a criterion it cannot express fails alone, and a failed request fails every
criterion posed in it, through the same failure routing as an LLM judge's failed call.
Its votes carry probabilities and confidence and no explanation (reason is
None). In a mixed ensemble every judge votes on every criterion, and the votes
aggregate exactly as an all-LLM ensemble's do.
With llm_calls="per_item", each LLM judge of judge_model_config or judges is
asked about the whole rubric in one call per item instead, as a decision model is:
the submission and its context are sent once; the system prompt embeds the binary and
multi-choice system prompts, for the kinds of criteria the rubric has, as guides to
apply to each criterion separately and independently; every criterion is listed under
its decision-model question id (c0, c1, ...), posed as in a per-criterion call
and with its options shuffled exactly as there; and the answer (RubricJudgment) maps
back to one result per criterion. A missing or unusable answer fails only its criterion
(parse), a failed call fails every criterion, and the call's usage and cost ride on
the item's first result. With few-shot examples, the call shows whole training items,
each once with its ground truth for every criterion, instead of examples drawn for each
criterion. Votes, aggregation and scoring are the same in both modes. A cascade's
escalation judges make that call too, but only for an item with an escalated criterion,
and keep only the escalated criteria's results (see below).
A confidence cascade (escalation=EscalationConfig(...)) puts one decision model
first and LLM judges behind it. The decision model judges every criterion in its one
request per item; a criterion it errored or abstained on, or answered with a
confidence below the criterion's threshold, is escalated, and each escalation judge
votes on the escalated criteria only. Called per criterion, an escalation judge is
asked about each escalated criterion in a call of its own. Under
llm_calls="per_item" it is asked about the whole rubric in one call, made only for
an item with an escalated criterion; its judgments of the other criteria are
discarded, a failed call fails its votes on the escalated criteria, and the call's
usage and cost ride on the first escalated criterion's result. Either way each call is
the one the judge would make in an otherwise identical grader without the cascade (the
same seed, judge_id and LLM settings, llm_calls included). An escalated
criterion's report is marked escalated=True, keeps the decision model's vote with
superseded=True, and takes its verdict from the escalation judges' votes alone,
aggregated as any ensemble's; it never falls back to the decision model's vote. The
decision model's judge_scores entry is its own score over every criterion; an
escalation judge's is None by role, as it votes only where the decision model
escalated, even on an item where every criterion escalated.
Some settings exist for LLM judges only. Few-shot examples, the system prompts and
shuffle_options apply to the LLM judges of a mixed ensemble or a cascade and never
reach a decision model; so does llm_calls. A grader whose judges are all decision
models rejects few-shot examples and warns when a system prompt is given,
shuffle_options is False or llm_calls is "per_item", since it has no judge
to apply them to. A custom response format describes a generated judgment, which only an
LLM judge produces, so it cannot be combined with any decision-model judge; it describes
one criterion's judgment, so it cannot be combined with llm_calls="per_item" either.
Example
from autorubric import DecisionModelConfig, LLMConfig, FewShotConfig, RubricDataset from autorubric.graders import CriterionGrader, JudgeSpec
Single LLM¶
grader = CriterionGrader( ... judge_model_config=LLMConfig(model="gemini/gemini-3-flash-preview") ... )
Single LLM + few-shot¶
train, test = dataset.split_train_test(n_train=100) grader = CriterionGrader( ... judge_model_config=LLMConfig(model="gemini/gemini-3-flash-preview"), ... training_data=train, ... few_shot_config=FewShotConfig(n_examples=3), ... )
Ensemble¶
grader = CriterionGrader( ... judges=[ ... JudgeSpec(LLMConfig(model="gemini/gemini-3-flash-preview"), "gemini"), ... JudgeSpec(LLMConfig(model="anthropic/claude-sonnet-4-5-20250929"), "claude"), ... ], ... aggregation="majority", ... )
Ensemble + few-shot¶
grader = CriterionGrader( ... judges=[JudgeSpec(...), JudgeSpec(...)], ... aggregation="majority", ... training_data=train, ... few_shot_config=FewShotConfig(n_examples=3), ... )
Single LLM, one call per item for the whole rubric¶
grader = CriterionGrader( ... judge_model_config=LLMConfig(model="gemini/gemini-3-flash-preview"), ... llm_calls="per_item", ... )
Decision model: one request per item for the whole rubric¶
grader = CriterionGrader(judge_model_config=DecisionModelConfig(model="jev-latest"))
Mixed ensemble of a decision model and an LLM¶
grader = CriterionGrader( ... judges=[ ... JudgeSpec(DecisionModelConfig(model="jev-latest"), "jev"), ... JudgeSpec(LLMConfig(model="gemini/gemini-3-flash-preview"), "gemini"), ... ], ... )
Cascade: the decision model first, an LLM for what it is unsure of¶
grader = CriterionGrader( ... judge_model_config=DecisionModelConfig(model="jev-latest"), ... escalation=EscalationConfig( ... judges=LLMConfig(model="gemini/gemini-3-flash-preview"), threshold=0.72 ... ), ... )
Initialize the criterion grader.
| PARAMETER | DESCRIPTION |
|---|---|
judge_model_config
|
Configuration of the single judge (single-judge mode), which
is given
TYPE:
|
judges
|
List of JudgeSpec for ensemble mode, each an LLM or a decision model,
each with its own
TYPE:
|
aggregation
|
Strategy for aggregating votes in ensemble mode (binary criteria).
TYPE:
|
ordinal_aggregation
|
Strategy for aggregating ordinal multi-choice votes. Central tendency: "mean", "median", "weighted_mean", "mode". Conservative/ permissive (analogs of binary unanimous/any): "min" (lowest selected option) / "max" (highest selected option).
TYPE:
|
nominal_aggregation
|
Strategy for aggregating nominal multi-choice votes. Options: "mode", "weighted_mode", "unanimous". "unanimous" abstains via the NA option on disagreement, or falls back to mode + warns if there is no NA option.
TYPE:
|
training_data
|
Dataset for few-shot examples. If provided, enables few-shot
prompting for the LLM judges; examples are selected for, and sent to, LLM
judges only, never to a decision-model judge, whose request is exactly what
it would be without them. An example shows its item's ground truth for the
criterion and, when
TYPE:
|
few_shot_config
|
Configuration for few-shot example selection (LLM judges only).
The grader validates and keeps its own copy, so changing the config after
the grader is built does not affect it.
Under
TYPE:
|
system_prompt
|
Custom system prompt for binary criteria. Applies to LLM judges only (a decision model's request has no system prompt); in a mixed ensemble the LLM judges use it.
TYPE:
|
multi_choice_system_prompt
|
Custom system prompt for multi-choice criteria. Applies
to LLM judges only, like
TYPE:
|
length_penalty
|
Optional length penalty configuration.
TYPE:
|
normalize
|
If True, normalize score to [0, 1]. If False, return raw sum.
TYPE:
|
cannot_assess_config
|
Configuration for handling CANNOT_ASSESS verdicts.
TYPE:
|
shuffle_options
|
If True (default), randomize the order of multi-choice options presented to the LLM to mitigate position bias. Each judge/call sees a different random order, and responses are mapped back to original indices. Disable for deterministic behavior in tests. Applies to LLM judges only: a decision model's question offers the options in the rubric's order, and in a mixed ensemble the LLM judges shuffle as set here.
TYPE:
|
auto_na_option
|
If True (default), auto-inject a canonical NA / "cannot assess" option into any multi-choice criterion that lacks one, giving the judge a first-class abstain channel analogous to binary CANNOT_ASSESS. The injected option is appended at the end (highest index) so existing option indices are preserved. Set False for forced-choice classification (the judge must pick a scored option). Never strips an author-supplied NA option — author intent wins.
TYPE:
|
seed
|
Master seed for all non-LLM randomness (option shuffling, few-shot
example selection). Auto-generated when None so that randomness is always
pinned and reproducible. Any integer is accepted (numpy integers too) and
kept as a plain
TYPE:
|
binary_response_format
|
Pydantic model to use as the structured output schema
for binary criterion judgments. Must be a subclass of (or compatible with)
CriterionJudgment. If the model includes an
TYPE:
|
multi_choice_response_format
|
Pydantic model to use as the structured output
schema for multi-choice criterion judgments. Must be a subclass of (or
compatible with) MultiChoiceJudgment. If the model includes an
TYPE:
|
escalation
|
Makes the grader a confidence cascade (see the class docstring):
the escalation judges, all LLMs, vote only on the criteria the decision
model errored or abstained on or answered with a confidence below the
criterion's threshold. The grader's one judge (
TYPE:
|
llm_calls
|
How each LLM judge, a cascade's escalation judges included, is called
for an item:
TYPE:
|
llm_config
|
Deprecated alias of
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If neither judge_model_config nor judges is provided, if both are
provided, or if both judge_model_config and its deprecated alias
llm_config are provided; if escalation is given while the grader's judges
are not exactly one decision model, with a |
ImportError
|
If a judge is a decision model and the TypeSafe SDK is not
installed ( |
| WARNS | DESCRIPTION |
|---|---|
UserWarning
|
If system_prompt or multi_choice_system_prompt is not None,
shuffle_options is False, or llm_calls is |
FutureWarning
|
If a |
Source code in src/autorubric/graders/criterion_grader.py
1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 1323 1324 1325 1326 1327 1328 1329 1330 1331 1332 1333 1334 1335 1336 1337 1338 1339 1340 1341 1342 1343 1344 1345 1346 1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 1694 1695 1696 1697 1698 1699 1700 1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 1720 1721 1722 1723 1724 1725 1726 1727 1728 1729 1730 1731 1732 1733 1734 1735 1736 1737 1738 1739 1740 1741 1742 | |
is_ensemble
property
¶
Whether this grader has several judges that each judge every criterion.
A cascade is not an ensemble: its escalation judges vote only on the criteria its decision model escalates.
judge_ids
property
¶
The judge_id of every judge, in order: the judges that judge every criterion
(judges, or the single "default" judge), then a cascade's escalation judges.
judge
async
¶
judge(to_grade: Content, rubric: list[Criterion], query: Content | None = None, reference_submission: Content | None = None, *, guidelines: str | None = None) -> list[JudgeCriterionResults]
Judge all criteria with all judges (parallel across judges).
In a cascade the decision model judges every criterion first; then every escalation
judge judges the escalated criteria (_escalates: an error, an abstention, or a
confidence below EscalationConfig.threshold_for the criterion), all at once, and
none when nothing is escalated: in a call per escalated criterion, or, under
llm_calls="per_item", in one call about the whole rubric whose judgments of the
other criteria are discarded (_judge_escalated_criteria).
The submission, query and reference submission are put in canonical form and their
image files read (_load_content) before any judge is called, idempotently:
content Grader.grade has loaded reads no file again, so a grade reads each file
once, and a direct call to judge is as safe as one through grade. Every
judge gets the same loaded content; an LLM judge receives a document's images as
content parts in its user prompt, and a decision model as its request's assets,
each fitted into the envelope the model accepts.
| PARAMETER | DESCRIPTION |
|---|---|
to_grade
|
The submission, as
TYPE:
|
rubric
|
The criteria.
TYPE:
|
query
|
Optional input/query that prompted the submission.
TYPE:
|
reference_submission
|
Optional exemplar response for grading context.
TYPE:
|
guidelines
|
Optional rubric guidelines (
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
list[JudgeCriterionResults]
|
One |
list[JudgeCriterionResults]
|
decision model's ( |
list[JudgeCriterionResults]
|
( |
| RAISES | DESCRIPTION |
|---|---|
TypeError
|
If |
OSError
|
If an image file cannot be read ( |
ImportError
|
If a decision-model judge is given content holding an image and the installed TypeSafe SDK cannot send images (before its request). |
ValueError
|
If an image file is not a supported image, or the content holds an
image and a judge that could be sent it does not accept images
( |
| WARNS | DESCRIPTION |
|---|---|
UserWarning
|
Once per grader, when an |
UserWarning
|
With training data, once per grader per (criterion index, graded
question), when a criterion of |
Source code in src/autorubric/graders/criterion_grader.py
3427 3428 3429 3430 3431 3432 3433 3434 3435 3436 3437 3438 3439 3440 3441 3442 3443 3444 3445 3446 3447 3448 3449 3450 3451 3452 3453 3454 3455 3456 3457 3458 3459 3460 3461 3462 3463 3464 3465 3466 3467 3468 3469 3470 3471 3472 3473 3474 3475 3476 3477 3478 3479 3480 3481 3482 3483 3484 3485 3486 3487 3488 3489 3490 3491 3492 3493 3494 3495 3496 3497 3498 3499 3500 3501 3502 3503 3504 3505 3506 3507 3508 3509 3510 3511 3512 3513 3514 3515 3516 3517 3518 3519 3520 3521 3522 3523 3524 3525 3526 3527 3528 3529 3530 3531 3532 3533 3534 3535 3536 3537 3538 3539 3540 3541 3542 3543 3544 3545 3546 3547 3548 3549 3550 3551 3552 3553 3554 3555 3556 3557 3558 3559 3560 3561 3562 3563 3564 3565 3566 3567 3568 3569 3570 3571 3572 3573 3574 3575 3576 3577 | |
aggregate
async
¶
aggregate(judge_results: list[JudgeCriterionResults], *, normalize: bool = True) -> EnsembleEvaluationReport
Aggregate results from all judges into final report.
Handles both binary and multi-choice criteria: - Binary: Uses JudgeVote and _aggregate_votes() - Multi-choice: Uses MultiChoiceJudgeVote and _aggregate_multi_choice_votes()
In a cascade the escalated criteria are the ones the escalation judges have results
for (their results are None everywhere else). On those, the primary judge's vote
is marked superseded=True and the final verdict, reason and error come from the
votes that are not superseded, which are the escalation judges'; the report is
marked escalated=True. judge_scores holds each primary judge's score over its
own verdicts (superseded ones included) and None for each escalation judge, by
role: it votes only where the decision model escalated, even on an item where every
criterion escalated.
Source code in src/autorubric/graders/criterion_grader.py
3771 3772 3773 3774 3775 3776 3777 3778 3779 3780 3781 3782 3783 3784 3785 3786 3787 3788 3789 3790 3791 3792 3793 3794 3795 3796 3797 3798 3799 3800 3801 3802 3803 3804 3805 3806 3807 3808 3809 3810 3811 3812 3813 3814 3815 3816 3817 3818 3819 3820 3821 3822 3823 3824 3825 3826 3827 3828 3829 3830 3831 3832 3833 3834 3835 3836 3837 3838 3839 3840 3841 3842 3843 3844 3845 3846 3847 3848 3849 3850 3851 3852 3853 3854 3855 3856 3857 3858 3859 3860 3861 3862 3863 3864 3865 3866 3867 3868 3869 3870 3871 3872 3873 3874 3875 3876 3877 3878 3879 3880 3881 3882 3883 3884 3885 3886 3887 3888 3889 3890 3891 3892 3893 3894 3895 3896 3897 3898 3899 3900 3901 3902 3903 3904 3905 3906 3907 3908 3909 3910 3911 3912 3913 3914 3915 3916 3917 3918 3919 3920 3921 3922 3923 3924 3925 3926 3927 3928 3929 3930 3931 3932 3933 3934 3935 3936 3937 3938 3939 3940 3941 3942 3943 3944 | |
Grader¶
Abstract base class for grader implementations.
Grader is generic in the report type it returns. CriterionGrader is a
Grader[EnsembleEvaluationReport], so a type checker types await rubric.grade(..., grader=grader)
as EnsembleEvaluationReport. A custom grader declares its own, for example
class MyGrader(Grader[EvaluationReport]); one that names no type still works.
Changed: judge takes Content
The abstract judge annotates to_grade, query and reference_submission as Content (a
string, an Image, or a sequence of both) instead of str, because a submission may hold
images. A subclass whose judge still annotates them as str runs unchanged, but a type
checker reports its override as incompatible with the base class; annotate them as Content.
grade now raises TypeError for a query or reference submission that is not Content
(for example a number), where earlier releases passed it through. See
Submissions with Images.
Grader
¶
Grader(*, length_penalty: LengthPenalty | None = None, normalize: bool = True)
Bases: ABC, Generic[ReportT]
Base class for LLM-backed grading implementations.
All graders require an LLMConfig for the LLM client. Subclasses must implement judge() and aggregate() methods.
Rubric guidelines reach judge only if it accepts them: a judge with a
guidelines parameter that can be passed by keyword (e.g. keyword-only
guidelines: str | None = None) or with **kwargs receives them as
guidelines=. A judge with the abstract signature keeps working unchanged; the
grader then grades without the guidelines and warns once (see grade).
| PARAMETER | DESCRIPTION |
|---|---|
length_penalty
|
Optional configuration for penalizing overly long outputs. When provided, a penalty based on the token/word count is subtracted from the final score.
TYPE:
|
normalize
|
If True (default), scores are normalized to 0-1. If False, raw weighted sums are returned, which is useful for RL training scenarios.
TYPE:
|
Source code in src/autorubric/graders/base.py
judge
abstractmethod
async
¶
judge(to_grade: Content, rubric: list[Criterion], query: Content | None = None, reference_submission: Content | None = None) -> Any
Collect raw judge results for the provided submission.
An implementation may also take the rubric's guidelines, as a guidelines
parameter that can be passed by keyword or through **kwargs; grade passes
them, when there are any, only to an implementation that does.
| PARAMETER | DESCRIPTION |
|---|---|
to_grade
|
The submission to evaluate, as
TYPE:
|
rubric
|
List of criteria to evaluate against.
TYPE:
|
query
|
Optional input/query that prompted the response, in the same form.
TYPE:
|
reference_submission
|
Optional exemplar response for grading context, in the same form.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Any
|
Raw judge results (format depends on implementation). |
Source code in src/autorubric/graders/base.py
aggregate
abstractmethod
async
¶
Transform judge results into an EvaluationReport.
| PARAMETER | DESCRIPTION |
|---|---|
judge_results
|
Raw results from judge().
TYPE:
|
normalize
|
If True, normalize score to 0-1. If False, return raw weighted sum.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ReportT
|
EvaluationReport with score and optional per-criterion breakdown. |
Source code in src/autorubric/graders/base.py
grade
async
¶
grade(to_grade: ToGradeInput, rubric: list[Criterion], query: Content | None = None, reference_submission: Content | None = None, *, guidelines: str | None = None) -> ReportT
Grade the submission against the rubric.
This is the main entry point for the grader.
Before judge is called, the submission, query and reference submission are put
in canonical form (_normalize_content) and every image file they refer to is
read, once per grade (_load_content), in a worker thread
(asyncio.to_thread); content without such a file reads nothing and is passed
on as it is (text as the same str). A missing, unreadable or unsupported image
file therefore raises from grade before any judge is called: it is a fault of
the item's data, never a judge's failed judgment, so it is not routed into a
verdict.
| PARAMETER | DESCRIPTION |
|---|---|
to_grade
|
The submission to evaluate. Can be either:
-
TYPE:
|
rubric
|
List of criteria to evaluate against.
TYPE:
|
query
|
Optional input/query that prompted the response (
TYPE:
|
reference_submission
|
Optional exemplar response for grading context
(
TYPE:
|
guidelines
|
Optional rubric guidelines: free text that applies to every
criterion (
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ReportT
|
EvaluationReport with score and optional per-criterion breakdown. |
ReportT
|
If normalize=True (default), score is 0-1. If normalize=False, score is raw |
ReportT
|
weighted sum. If length_penalty was configured, the penalty is subtracted from |
ReportT
|
the score. The raw_score field contains the unnormalized weighted sum before |
ReportT
|
length penalty. |
| RAISES | DESCRIPTION |
|---|---|
TypeError
|
If |
OSError
|
If an image file cannot be read ( |
ValueError
|
If an image file is empty, not an image, or not PNG, JPEG, GIF or WebP. |
| WARNS | DESCRIPTION |
|---|---|
UserWarning
|
" |
Source code in src/autorubric/graders/base.py
240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 | |
JudgeSpec¶
Configuration for a single judge in an ensemble.
JudgeSpec
dataclass
¶
JudgeSpec(llm_config: LLMConfig | DecisionModelConfig, judge_id: str, weight: float = 1.0)
JudgeSpec(*, judge_model_config: LLMConfig | DecisionModelConfig, judge_id: str, weight: float = 1.0)
JudgeSpec(llm_config: Any = MISSING, judge_id: Any = MISSING, weight: float = 1.0, *, judge_model_config: Any = MISSING)
Specification for a single judge in an ensemble.
judge_model_config is the preferred keyword for the judge's configuration, and the
positional form is unchanged. llm_config= keeps working without a warning: it is
the name of the stored dataclass field, which is deliberately not renamed so that
dataclasses.fields/asdict/replace, the repr, equality and pickles stay
exactly as they were. dataclasses.replace therefore takes llm_config=, and
passing both judge_model_config= and llm_config= raises ValueError.
The configuration decides the judge's kind: an LLMConfig makes an LLM judge, which
is asked about each criterion in its own call (or about every criterion of an item in
one call, when the grader's llm_calls is "per_item"), and a
DecisionModelConfig makes a decision-model judge, which is asked about every
criterion of an item in one request. Both kinds mix freely in one ensemble.
| ATTRIBUTE | DESCRIPTION |
|---|---|
llm_config |
The judge's model configuration, an
TYPE:
|
judge_id |
Unique identifier for this judge (e.g., "gpt-4", "claude-sonnet").
TYPE:
|
weight |
Voting weight for weighted aggregation (default 1.0); a positive, finite number.
TYPE:
|
Example
gemini = LLMConfig(model="gemini/gemini-3-flash-preview") JudgeSpec(judge_model_config=gemini, judge_id="gemini") JudgeSpec(gemini, "gemini", weight=2.0) # positional form JudgeSpec(DecisionModelConfig(model="jev-latest"), "jev") # a decision model
Initialize from judge_model_config (or its stored-field name llm_config).
| PARAMETER | DESCRIPTION |
|---|---|
llm_config
|
The judge's model configuration, positionally or by the stored
field name. Mutually exclusive with
TYPE:
|
judge_id
|
Unique identifier for this judge.
TYPE:
|
weight
|
Voting weight for weighted aggregation (default 1.0); a positive, finite number.
TYPE:
|
judge_model_config
|
The judge's model configuration (preferred keyword).
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If both |
TypeError
|
If the configuration or |
Source code in src/autorubric/graders/criterion_grader.py
judge_model_config
property
writable
¶
judge_model_config: LLMConfig | DecisionModelConfig
The judge's model configuration (read/write alias of the llm_config field).
LLMCalls¶
How CriterionGrader calls each LLM judge for an item: "per_criterion" (the default) or "per_item".
LLMCalls
module-attribute
¶
How CriterionGrader calls each LLM judge for an item (its llm_calls).
- per_criterion: One call per criterion (the default).
- per_item: One call for the whole rubric: the submission and its context are sent once,
every criterion is listed under its id (
c0,c1, ...), and the answer holds one judgment per criterion (RubricJudgment).
A cascade's escalation judges are called the same way, only where the decision model
escalated: per_item makes one full-rubric call for an item with any escalated criterion.
replay_escalation and calibrate_escalation take it to know how an LLM run was
called, which sets how they estimate a cascade's cost and time from that run's.
A decision-model judge always makes one request per item, whatever this is. The name counts
LLM calls per item; it is unrelated to compute_metrics(per_item_metrics=...) and to
per-item rubrics.