Decision Models

Decision-model judges, the confidence cascade, and the offline tools that calibrate and replay a cascade.

Overview

A decision model answers typed questions with probabilities instead of generating text, and grades an item with one request for the whole rubric. DecisionModelConfig configures one; it is accepted wherever a judge config is (CriterionGrader(judge_model_config=...), JudgeSpec). EscalationConfig turns a grader with one decision-model judge into a confidence cascade that escalates uncertain criteria to LLM judges. replay_escalation, calibrate_escalation and escalation_stats choose and check a cascade's threshold offline, without new calls.

The decision-model client needs the optional TypeSafe SDK: pip install 'autorubric[typesafe]'.

For a guided introduction see Decision-Model Judges; for the end-to-end calibration workflow see the cascade recipe.

Quick Example

from autorubric import (
    DecisionModelConfig, EscalationConfig, LLMConfig,
    calibrate_escalation, replay_escalation,
)
from autorubric.graders import CriterionGrader, JudgeSpec

jev = DecisionModelConfig(model="jev-latest", cache_enabled=True)
gemini = LLMConfig(model="gemini/gemini-3-flash-preview")

# Standalone
grader = CriterionGrader(judge_model_config=jev)

# Cascade
grader = CriterionGrader(
    judge_model_config=jev,
    escalation=EscalationConfig(judges=gemini, threshold=0.72),
    seed=7,
)

# Offline calibration from a decision-model run and an LLM run over the same items
curve = calibrate_escalation(calib, dm_calib, llm_calib)
best = curve.best(tolerance=0.005)
hybrid = replay_escalation(dm_test, llm_test, best)

---

DecisionModelConfig

Configuration of a decision-model judge: endpoint, framings, threshold, cache and price.

DecisionModelConfig dataclass

DecisionModelConfig(model: str, *, api_key: str | None = None, api_base: str | None = None, timeout: float = 60.0, max_retries: int = 3, max_parallel_requests: int | None = None, cache_enabled: bool = False, cache_dir: str | Path = '.autorubric_cache', cache_ttl: int | None = None, extra_headers: dict[str, str] = dict(), binary_framing: BinaryFraming = 'noul_framed', ordinal_framing: OrdinalFraming = 'choice', decision_threshold: float = 0.5, input_cost_per_token: float | None = None)

Configuration of a decision-model judge.

A decision model grades an item with one request for the whole rubric: the submission (with the prompt, a reference submission and rubric guidelines when present) is sent once as shared state, and each criterion becomes one typed question. It returns probabilities, not text, so its votes carry probabilities and confidence and have no explanation (reason is None).

Use it wherever a judge config is accepted, e.g. CriterionGrader(judge_model_config=DecisionModelConfig(model="jev-latest")) or JudgeSpec(DecisionModelConfig(...), "jev") in an ensemble. Building the grader builds the client, which needs the TypeSafe SDK (pip install 'autorubric[typesafe]'); constructing this config does not. The client's HTTP requests honour the environment's TLS and proxy settings (SSL_CERT_FILE, HTTPS_PROXY and the like), and settings no HTTP client can be built with (e.g. a CA file that does not exist) fail the grader's construction with ValueError. Field names match LLMConfig wherever the concept is the same. Every field after model is keyword-only.

Few-shot examples (FewShotConfig, training_data) apply to LLM judges only and are never sent to a decision model; a grader whose only judges are decision models rejects them. A decision model's calibration channels are rubric guidelines and, in a cascade, the escalation threshold.

Decision-model answers are not bit-deterministic: repeating an identical request can return slightly different probabilities, so a verdict whose probability lies near decision_threshold can change between runs. Enable cache_enabled to make re-runs reproducible: a cached request returns the stored answers, usage included.

ATTRIBUTE	DESCRIPTION
model	Model name or alias sent with each request, e.g. "jev-latest" or the model id a self-hosted endpoint serves. Required. TYPE: str
api_key	API key, sent as a bearer token. None (default) reads the TYPESAFE_API_KEY environment variable, which a .env file may set, as for LLM provider keys. Surrounding whitespace is stripped, as the SDK strips it, so a key read with its trailing newline works. Building a client without either, or with a key the SDK rejects (anything but printable ASCII without whitespace), raises ValueError, so a grader fails at construction rather than abstaining on every item. No message repeats the key, and it is never recorded. TYPE: str | None
api_base	Base URL of a System One-compatible endpoint; requests go to {api_base}/v1/systemone. None (default) reads TYPESAFE_BASE_URL (surrounding whitespace stripped), then falls back to TypeSafe's API, https://api.typesafe.ai. The URL is resolved and checked once, when a client (and so the grader) is built: the request URL is parsed with the SDK's HTTP stack, which refuses, e.g., a host that is not a valid internationalized domain name, and the URL must be absolute http(s), with no whitespace (a trailing newline included), query or fragment (the path is appended to the URL) and no credentials (they would replace the bearer api_key). Anything else raises ValueError, never repeating a URL that holds an @, so a base URL no request could use fails when the grader is built. The resolved URL is part of the response cache key, and its host (with any non-default port) names the rate-limit bucket. The URL is not a secret: a failed request's error, recorded in the reports it fails, shows it with its path, as the SDK's logs do. Only its host goes into an experiment's manifest. TYPE: str | None
timeout	Per-request timeout in seconds, applied to each HTTP operation, as in LLMConfig. It also sets the retry budget (see max_retries). A request's wait for its turn (see max_parallel_requests) is not timed. TYPE: float
max_retries	Total attempts per request, including the first, as in LLMConfig (which stops after that many attempts). Rate limits (429), request timeouts (408), server errors (5xx), connection failures and timeouts are retried with exponential backoff, honouring the endpoint's Retry-After. Retries stay within a budget, counted from the first attempt, of max_retries * (timeout + b) seconds, where b is the SDK's longest backoff wait (5 s): room for every attempt's full timeout and a backoff wait before each, so backoff retries are never cut short. A retry whose wait would end past the budget is not made, and the request fails with its last error (its criteria abstain), so a Retry-After beyond the budget fails the request at once instead of stalling the run, much as LLMConfig caps each wait at retry_max_wait. TYPE: int
max_parallel_requests	Maximum concurrent requests to the endpoint, as in LLMConfig. The limit is shared by every decision-model config whose resolved base URL has the same host, whatever its model name; the strictest limit wins. The limit applies within one event loop; each asyncio.run call has its own. None (default) sets no shared limit: the judge then keeps at most 100 requests in flight per event loop, one per connection of its HTTP client (the size of the SDK's default pool). A limit sizes the pool instead, so one above 100 is honoured. A request waits for its turn before it is sent, so the wait, however long, counts against neither timeout nor the retry budget: those measure the endpoint. TYPE: int | None
cache_enabled	Cache responses on disk, as in LLMConfig. There is one entry per request, keyed by the model, the resolved base URL, the state and the questions, plus the judge_id of any judge other than a lone judge_model_config judge, so that judges of one decision model keep their own answers. A hit returns the stored response, usage included, without a request. Failed requests are never cached. TYPE: bool
cache_dir	Directory of the response cache. It is the same store that LLM judges use when their cache_dir matches. TYPE: str | Path
cache_ttl	Lifetime of a cache entry in seconds; None (default) never expires. TYPE: int | None
extra_headers	Additional HTTP headers sent with every request, header names to values, both strings. Building a client checks each header with the HTTP/1.1 layer's own rule (h11): a name that is not a token (letters, digits and !#$%&'*+-.^_`|~), or a value that is not an ASCII string or has a line break or surrounding whitespace, such as a secret read from a file with its trailing newline, raises ValueError naming the header but never its value: the HTTP layer would refuse it on every request. Unlike LLMConfig, a few headers cannot be set: the TypeSafe SDK sets Authorization (the bearer api_key), Accept, Content-Type, User-Agent, X-TypeSafe-SDK, X-TypeSafe-Runtime and X-TypeSafe-Retry-Count itself on every request, replacing any value given, so naming one of them, in any letter case, raises ValueError too. TYPE: dict[str, str]
binary_framing	How a binary criterion is posed. The criterion's requirement is always sent verbatim; only the structure around it differs. "noul_framed" (default): a yes/no question. The requirement is wrapped in a fixed sentence asking whether the submission satisfies it, and the yes and no outcomes carry the same MET/UNMET definitions as the LLM judge's system prompt (the negative-criterion definitions for negative weights). P(MET) is compared with decision_threshold. The fixed text adapts to the state: a structured submission is judged by its output, with the thinking as context only, and a reference submission comes with the LLM judge's rule for using it (calibrate expectations, judge the submission on its own merits). "noul": a yes/no question whose instructions are the bare requirement, with no wrapper and no definitions. Thresholded like "noul_framed". It sees the input, reference and thinking only through the state. "choice": a choice among MET, UNMET and CANNOT_ASSESS with the same definitions and fixed text as "noul_framed"; the selected option is the verdict. This is the only binary framing that can abstain. TYPE: BinaryFraming
ordinal_framing	How an ordinal multi-choice criterion is posed. The requirement is the bare instructions and option labels are always sent verbatim, in the rubric's order. Nominal criteria are always posed as "choice". Option shuffling (CriterionGrader(shuffle_options=...)) does not apply to decision models. "choice" (default): a choice over the option labels, the NA option included; the selected option is the verdict. It needs unique labels (they are the choice keys) and at most 255 options. "score": an ordered scale over the non-NA options, which needs 2 to 10 of them. The expected level s is snapped to level round(s), clamped to the scale; round is Python's, half to even, so a level exactly halfway between two snaps to the even one (0.5 to 0, 1.5 to 2), whatever the criterion's weight sign. The NA option is not among the levels offered, so a Score-framed criterion cannot abstain. A criterion its framing cannot express is left out of the item's request and fails alone, as a parse error; the other criteria are still asked. TYPE: OrdinalFraming
decision_threshold	Cut on P(MET) for the Noul framings, in [0, 1]. Above it the verdict is MET, below it UNMET; at exactly the threshold the verdict is the worst case for the criterion's weight sign, the same tie rule as ensemble aggregation. Choice and Score answers ignore it. It is not part of the cache key: changing it re-reads cached answers instead of asking again. TYPE: float
input_cost_per_token	Price in USD per input token, e.g. 0.042e-6. A request's cost is its input tokens times this price. There is no built-in price table, because vendor prices change: set it whenever cost matters. None (default) means the cost is unknown, and the decision model then contributes nothing to a report's completion_cost. TYPE: float | None

Examples:

>>> jev = DecisionModelConfig(model="jev-latest", cache_enabled=True)
>>> self_hosted = DecisionModelConfig(
...     model="my-org/rubric-dm-7b",
...     api_base="https://xyz.endpoints.huggingface.cloud",
...     api_key="hf_...",
... )

---

EscalationConfig

Configuration of a confidence cascade: the LLM escalation judges and the thresholds.

EscalationConfig dataclass

EscalationConfig(judges: LLMConfig | Sequence[JudgeSpec], threshold: float, per_criterion: Mapping[str, float] | None = None)

Configuration of a confidence cascade: which LLM judges take over, and when.

Passed as CriterionGrader(escalation=...) to a grader whose one judge is a decision model. The decision model answers every criterion of an item, in its one request per item; a criterion is escalated when its vote errored, abstained (CANNOT_ASSESS or an NA option), or has a confidence below the criterion's threshold (threshold_for). The escalation judges vote on the escalated criteria only, and only their votes decide such a criterion's verdict (aggregated with the grader's aggregation / ordinal_aggregation / nominal_aggregation); the decision model's vote is kept with superseded=True and never used as a fallback. An escalation judge is called as the grader's llm_calls says: by default once per escalated criterion, about that criterion alone; under llm_calls="per_item" once per item with any escalated criterion, about the whole rubric, keeping its judgments of the escalated criteria only. Either way each call is the one the judge would make in the same grader without the cascade.

Validated at construction (dataclasses.replace validates again) and frozen: its fields cannot be reassigned. The judges list and the per_criterion dict can still be changed in place, so a grader validates and keeps its own copy (dataclasses.replace) when it is built; later changes to the config do not reach it.

ATTRIBUTE	DESCRIPTION
judges	The escalation judges, LLM judges only, in order. A bare LLMConfig given to the constructor becomes [JudgeSpec(config, "escalation")]. To compare a cascade with a separate LLM run judge for judge (the same prompts, option shuffles and few-shot examples included), grade that run with these judges (judges=escalation.judges, their judge_ids included) and the cascade's seed, llm_calls and other LLM settings (replay_escalation lists them). TYPE: list[JudgeSpec]
threshold	A criterion is escalated when the decision model's confidence is below it, in [0, 1]. 0.0 escalates only errors and abstentions; 1.0 everything short of certainty. TYPE: float
per_criterion	Thresholds for particular criteria, keyed by criterion name (each in [0, 1]); other criteria, and unnamed ones, use threshold. A grader checks the names against the rubrics it grades and warns about names that match no criterion. TYPE: Mapping[str, float] | None

Example

jev = DecisionModelConfig(model="jev-latest") grader = CriterionGrader( ... judge_model_config=jev, ... escalation=EscalationConfig( ... judges=LLMConfig(model="gemini/gemini-3-flash-preview"), ... threshold=0.72, ... per_criterion={"factuality": 0.95}, ... ), ... )

Validate and store the configuration.

PARAMETER	DESCRIPTION
judges	One LLM judge as a bare LLMConfig (its judge_id is "escalation"), or a non-empty list of JudgeSpec of LLM judges. TYPE: LLMConfig | Sequence[JudgeSpec]
threshold	The confidence threshold, in [0, 1]. TYPE: float
per_criterion	Thresholds by criterion name, each in [0, 1]. Copied. TYPE: Mapping[str, float] | None DEFAULT: None

RAISES	DESCRIPTION
ValueError	If judges is empty, holds anything but JudgeSpec, or names a decision model; if threshold or a per_criterion threshold is not a number in [0, 1]; or if per_criterion is not a mapping from criterion names (strings).

Source code in src/autorubric/graders/criterion_grader.py

def __init__(
    self,
    judges: LLMConfig | Sequence[JudgeSpec],
    threshold: float,
    per_criterion: Mapping[str, float] | None = None,
) -> None:
    """Validate and store the configuration.

    Args:
        judges: One LLM judge as a bare ``LLMConfig`` (its ``judge_id`` is
            ``"escalation"``), or a non-empty list of ``JudgeSpec`` of LLM judges.
        threshold: The confidence threshold, in [0, 1].
        per_criterion: Thresholds by criterion name, each in [0, 1]. Copied.

    Raises:
        ValueError: If ``judges`` is empty, holds anything but ``JudgeSpec``, or names
            a decision model; if ``threshold`` or a ``per_criterion`` threshold is not a
            number in [0, 1]; or if ``per_criterion`` is not a mapping from criterion
            names (strings).
    """
    name = type(self).__name__
    if isinstance(judges, LLMConfig):
        specs = [JudgeSpec(judge_model_config=judges, judge_id="escalation")]
    elif isinstance(judges, DecisionModelConfig):
        raise ValueError(
            f"{name}.judges must be LLM judges, not a decision model "
            "(DecisionModelConfig): they judge the criteria the decision model escalates"
        )
    elif isinstance(judges, (list, tuple)):
        specs = list(judges)
    else:
        raise ValueError(
            f"{name}.judges must be an LLMConfig or a list of JudgeSpec; "
            f"got {type(judges).__name__}"
        )
    if not specs:
        raise ValueError(f"{name}.judges must name at least one judge")
    for i, spec in enumerate(specs):
        if not isinstance(spec, JudgeSpec):
            raise ValueError(
                f"{name}.judges[{i}] must be a JudgeSpec; got {type(spec).__name__} "
                "(a single judge can be passed as a bare LLMConfig, not in a list)"
            )
    ids = [s.judge_id for s in specs if isinstance(s.llm_config, DecisionModelConfig)]
    if ids:
        raise ValueError(
            f"{name}.judges must be LLM judges, not decision models: "
            f"{', '.join(map(repr, ids))} {'is a' if len(ids) == 1 else 'are'} "
            "DecisionModelConfig; escalation judges judge the criteria the decision model "
            "escalates"
        )
    per_criterion = _check_thresholds(
        threshold,
        per_criterion,
        threshold_name=f"{name}.threshold",
        per_criterion_name=f"{name}.per_criterion",
    )
    # Frozen: the generated __setattr__ refuses assignment, so store past it.
    object.__setattr__(self, "judges", specs)
    object.__setattr__(self, "threshold", threshold)
    object.__setattr__(self, "per_criterion", per_criterion)

threshold_for

threshold_for(criterion: Criterion) -> float

The confidence threshold below which criterion is escalated.

Its per_criterion threshold, looked up by the criterion's name, else threshold; an unnamed criterion always gets threshold.

Source code in src/autorubric/graders/criterion_grader.py

def threshold_for(self, criterion: Criterion) -> float:
    """The confidence threshold below which ``criterion`` is escalated.

    Its ``per_criterion`` threshold, looked up by the criterion's name, else
    ``threshold``; an unnamed criterion always gets ``threshold``.
    """
    return _threshold_for(criterion, self.threshold, self.per_criterion)

---

replay_escalation

Build the EvalResult a cascade would have produced from a decision-model run and an LLM run, with no calls.

replay_escalation

replay_escalation(dm_result: EvalResult, llm_result: EvalResult, escalation: EscalationPoint | float, *, per_criterion: Mapping[str, float] | None = None, cannot_assess_config: CannotAssessConfig | None = None, normalize: bool = True, llm_calls: LLMCalls = 'per_criterion') -> EvalResult

Build the EvalResult a confidence cascade would have produced, with no calls.

dm_result is a run of the decision model alone and llm_result a run of the LLM judge(s) alone, over the same items; items are paired by item_idx. For each criterion of each item, the decision model's recorded vote either stands or is superseded, by the rule of a live cascade (CriterionGrader with escalation=): the criterion is escalated when the vote failed (an error), abstained (CANNOT_ASSESS or an NA option), or has a confidence below the criterion's threshold (its per_criterion threshold by name, else threshold).

• A kept criterion gets the decision-model run's report of it: its one vote, and the final verdict aggregated from it.
• An escalated criterion gets the LLM run's report of it, with the decision model's vote in front, superseded=True, and escalated=True. A live cascade aggregates the votes that are not superseded, which are exactly the LLM run's, so the final verdict, final_reason, error and agreement are the ones the LLM run's aggregation recorded; an LLM panel's votes stay aggregated with the LLM run's aggregation, ordinal_aggregation and nominal_aggregation and its judges' weights.
• Replayed votes keep their judge_ids. judge_scores holds the decision model's own score over every criterion (its superseded votes included) and None for each LLM judge, by role. Scores are computed with score_reports under cannot_assess_config and normalize, as the cascade's grader would.

Exactness. The replay equals a live cascade exactly when each run was graded by a CriterionGrader configured as the cascade for everything that stage uses, over the same items and rubrics (guidelines included), and this call's cannot_assess_config and normalize are the cascade's:

• the decision-model run: one judge, the cascade's decision model (the same config, judge_id and weight);
• the LLM run: the cascade's escalation judges (judges=escalation.judges, the same configs, judge_ids and weights; a bare LLMConfig given to EscalationConfig becomes JudgeSpec(config, "escalation")) with every setting they use as the cascade's: its seed (CriterionGrader.seed, generated when not given), llm_calls, shuffle_options, system_prompt, multi_choice_system_prompt, training_data and few_shot_config, and the default response formats (a cascade cannot have others);
• both: the cascade's auto_na_option and aggregation settings (aggregation, ordinal_aggregation, nominal_aggregation).

For CriterionGrader(judge_model_config=dm, escalation=escalation, seed=S) with otherwise default settings, the two runs are graded by CriterionGrader(judge_model_config=dm) and CriterionGrader(judges=escalation.judges, seed=S); for a cascade that also has llm_calls="per_item", the LLM run's grader has it too, and so does this call. Only the criteria are checked (as graded, NA options that auto_na_option adds included): an EvalResult records no grader configuration, so another setting goes unnoticed. With only another seed or judge_id, the LLM prompts differ in the order of a shuffled multi-choice criterion's options and in the few-shot examples, and the replay equals the cascade in distribution only; with another prompt setting (llm_calls, shuffle_options, a system prompt, the few-shot settings) the prompts themselves differ, and the replay is not the cascade's run.

How the LLM run was called. This call's llm_calls says how the LLM run's judges were called, and so which share of its per-item cost and time a live cascade would spend (see Estimates). The replay cannot check it, since an EvalResult records no grader configuration: like the conditions above, it is a declared fact, and for an exact replay it is the cascade's llm_calls too. It changes nothing but the estimates: whatever it says, the replayed verdicts are the LLM run's. A wrong llm_calls therefore biases only the estimated costs and durations, and so only the cost_usd and compute_seconds of escalation_stats, on which EscalationCurve.best never selects.

Estimates. Per-criterion LLM cost and time are not recorded, only per-item totals. An item's LLM cost and time are therefore its LLM run totals times the share of the LLM run's calls for the item that a live cascade would make:

• Called per criterion (the default llm_calls), a judge made a call per criterion, and a live cascade makes one per escalated criterion. An item's completion_cost is its decision-model cost plus its LLM cost pro-rated by the share of its criteria that were escalated, and its duration_seconds its decision-model duration plus its LLM duration pro-rated the same way.
• Called per item (llm_calls="per_item"), a judge made one call about the item's whole rubric, and a live cascade makes that same call for an item with any escalated criterion, and no call otherwise. An item's completion_cost is its decision-model cost plus, when any of its criteria was escalated, its LLM cost in full, and its duration_seconds its decision-model duration plus, likewise, its LLM duration in full.

An item's completion_cost is None when it is not positive, as in a live report. The run's total duration is the decision-model run's total plus the LLM run's, pro-rated by the share of the LLM run's calls (per judge) that a live cascade would make: its escalated pairs over its pairs when called per criterion, its items with an escalated criterion over its items (those with any criterion) when called per item. An item whose decision-model grading raised (it has no votes) makes no call, as a live cascade sends it to no LLM judge, while its calls in the LLM run count among that run's. These are estimates, which escalation_stats sums into cost_usd and compute_seconds. The result says so in its experiment_name, its descriptive field ("replay of decision-model run '<name>' and LLM run '<name>' (estimated cost and time)"), since a replay is not an experiment on disk (experiment_dir is None). It carries no token_usage, whatever llm_calls says. Token usage is recorded per item, not per criterion, so for judges called per criterion an item's cannot be split; for judges called per item it would rest on llm_calls, which the replay cannot check and which changes nothing but the estimates marked as such.

Length penalty. It is not re-applied: compare replayed scores with runs graded without one. Criterion-level metrics are unaffected.

An item whose grading raised in dm_result (it has no votes) stays failed, as it would in a live cascade, whose decision model grades first. An item every decision-model judgment of which failed is replayed like any other: its failed votes escalate, as a live cascade escalates them.

PARAMETER	DESCRIPTION
dm_result	A run of one decision-model judge alone (e.g. CriterionGrader(judge_model_config=DecisionModelConfig(...))), live or loaded with EvalResult.from_experiment. TYPE: EvalResult
llm_result	A run of the LLM judge(s) alone over the same items and criteria, with a vote of every judge on every criterion of every item: whichever criteria a threshold escalates, the run costs a full LLM grading of its items. TYPE: EvalResult
escalation	The threshold, a number in [0, 1], or an EscalationPoint (e.g. one of calibrate_escalation's curve) whose threshold and per_criterion are used. TYPE: EscalationPoint | float
per_criterion	Thresholds for particular criteria, by criterion name, each in [0, 1]; only with a number as escalation. TYPE: Mapping[str, float] | None DEFAULT: None
cannot_assess_config	How abstentions score (score_reports); default CannotAssessConfig(), as CriterionGrader's. TYPE: CannotAssessConfig | None DEFAULT: None
normalize	Whether scores are normalized to [0, 1], as CriterionGrader's. TYPE: bool DEFAULT: True
llm_calls	How the LLM run's judges were called, its grader's llm_calls: "per_criterion" (the default) or "per_item". It sets the unit of the cost and time estimates and nothing else (see How the LLM run was called). TYPE: LLMCalls DEFAULT: 'per_criterion'

RETURNS	DESCRIPTION
EvalResult	An ordinary EvalResult (compute_metrics, escalation_stats and
EvalResult	ItemResult.to_dict work on it), with one item result per paired item, in
EvalResult	item_idx order.

RAISES

DESCRIPTION

ValueError

If the threshold or a per_criterion threshold is not a number in [0, 1]; if per_criterion is given with an EscalationPoint, or the point has no threshold; if llm_calls is neither "per_criterion" nor "per_item"; if the two results do not cover the same items (by item_idx, a result holding an item more than once, or an item whose recorded submissions differ) graded against the same criteria (as graded, NA options included); if dm_result does not come from one decision-model judge (every vote carries probabilities and a confidence, unless its judgment failed); if llm_result has an item whose grading raised (it has no votes; one whose every judgment failed keeps them), a criterion missing one of its judges' votes, a decision model's vote, or judges that differ between items; or if a judge_id names both the decision model and an LLM judge.

WARNS

DESCRIPTION

UserWarning

If a per_criterion name matches no criterion of the replayed items (it sets no threshold), as a live cascade warns. If escalation is an EscalationPoint whose calibration_fingerprint is the fingerprint of the items being replayed: a threshold chosen and evaluated on the same pairs looks better than it will on new items. (An item loaded with EvalResult.from_experiment records no submission, so the check cannot recognize it.)

Source code in src/autorubric/escalation.py

def replay_escalation(
    dm_result: EvalResult,
    llm_result: EvalResult,
    escalation: EscalationPoint | float,
    *,
    per_criterion: Mapping[str, float] | None = None,
    cannot_assess_config: CannotAssessConfig | None = None,
    normalize: bool = True,
    llm_calls: LLMCalls = "per_criterion",
) -> EvalResult:
    """Build the ``EvalResult`` a confidence cascade would have produced, with no calls.

    ``dm_result`` is a run of the decision model alone and ``llm_result`` a run of the LLM
    judge(s) alone, over the same items; items are paired by ``item_idx``. For each
    criterion of each item, the decision model's recorded vote either stands or is
    superseded, by the rule of a live cascade (``CriterionGrader`` with ``escalation=``): the
    criterion is escalated when the vote failed (an ``error``), abstained (``CANNOT_ASSESS``
    or an NA option), or has a ``confidence`` below the criterion's threshold (its
    ``per_criterion`` threshold by name, else ``threshold``).

    - A kept criterion gets the decision-model run's report of it: its one vote, and the
      final verdict aggregated from it.
    - An escalated criterion gets the LLM run's report of it, with the decision model's vote
      in front, ``superseded=True``, and ``escalated=True``. A live cascade aggregates the
      votes that are not superseded, which are exactly the LLM run's, so the final verdict,
      ``final_reason``, ``error`` and ``agreement`` are the ones the LLM run's aggregation
      recorded; an LLM panel's votes stay aggregated with the LLM run's ``aggregation``,
      ``ordinal_aggregation`` and ``nominal_aggregation`` and its judges' weights.
    - Replayed votes keep their ``judge_id``s. ``judge_scores`` holds the decision model's
      own score over every criterion (its superseded votes included) and ``None`` for each
      LLM judge, by role. Scores are computed with ``score_reports`` under
      ``cannot_assess_config`` and ``normalize``, as the cascade's grader would.

    **Exactness.** The replay equals a live cascade exactly when each run was graded by a
    ``CriterionGrader`` configured as the cascade for everything that stage uses, over the
    same items and rubrics (guidelines included), and this call's ``cannot_assess_config``
    and ``normalize`` are the cascade's:

    - the decision-model run: one judge, the cascade's decision model (the same config,
      ``judge_id`` and weight);
    - the LLM run: the cascade's escalation judges (``judges=escalation.judges``, the same
      configs, ``judge_id``s and weights; a bare ``LLMConfig`` given to
      ``EscalationConfig`` becomes ``JudgeSpec(config, "escalation")``) with every setting
      they use as the cascade's: its ``seed`` (``CriterionGrader.seed``, generated when not
      given), ``llm_calls``, ``shuffle_options``, ``system_prompt``,
      ``multi_choice_system_prompt``, ``training_data`` and ``few_shot_config``, and the
      default response formats (a cascade cannot have others);
    - both: the cascade's ``auto_na_option`` and aggregation settings (``aggregation``,
      ``ordinal_aggregation``, ``nominal_aggregation``).

    For ``CriterionGrader(judge_model_config=dm, escalation=escalation, seed=S)`` with
    otherwise default settings, the two runs are graded by
    ``CriterionGrader(judge_model_config=dm)`` and
    ``CriterionGrader(judges=escalation.judges, seed=S)``; for a cascade that also has
    ``llm_calls="per_item"``, the LLM run's grader has it too, and so does this call. Only
    the criteria are checked (as graded, NA options that ``auto_na_option`` adds
    included): an ``EvalResult`` records no grader configuration, so another setting goes
    unnoticed. With only another ``seed`` or ``judge_id``, the LLM prompts differ in the
    order of a shuffled multi-choice criterion's options and in the few-shot examples, and
    the replay equals the cascade in distribution only; with another prompt setting
    (``llm_calls``, ``shuffle_options``, a system prompt, the few-shot settings) the prompts
    themselves differ, and the replay is not the cascade's run.

    **How the LLM run was called.** This call's ``llm_calls`` says how the LLM run's judges
    were called, and so which share of its per-item cost and time a live cascade would
    spend (see *Estimates*). The replay cannot check it, since an ``EvalResult`` records
    no grader configuration: like the conditions above, it is a declared fact, and for an
    exact replay it is the cascade's ``llm_calls`` too. It changes nothing but the
    estimates: whatever it says, the replayed verdicts are the LLM run's. A wrong
    ``llm_calls`` therefore biases only the estimated costs and durations, and so only the
    ``cost_usd`` and ``compute_seconds`` of ``escalation_stats``, on which
    ``EscalationCurve.best`` never selects.

    **Estimates.** Per-criterion LLM cost and time are not recorded, only per-item totals.
    An item's LLM cost and time are therefore its LLM run totals times the share of the
    LLM run's calls for the item that a live cascade would make:

    - Called per criterion (the default ``llm_calls``), a judge made a call per criterion,
      and a live cascade makes one per escalated criterion. An item's ``completion_cost``
      is its decision-model cost plus its LLM cost pro-rated by the share of its criteria
      that were escalated, and its ``duration_seconds`` its decision-model duration plus
      its LLM duration pro-rated the same way.
    - Called per item (``llm_calls="per_item"``), a judge made one call about the item's
      whole rubric, and a live cascade makes that same call for an item with any escalated
      criterion, and no call otherwise. An item's ``completion_cost`` is its decision-model
      cost plus, when any of its criteria was escalated, its LLM cost in full, and its
      ``duration_seconds`` its decision-model duration plus, likewise, its LLM duration in
      full.

    An item's ``completion_cost`` is ``None`` when it is not positive, as in a live
    report. The run's total duration is the decision-model run's total plus the LLM run's,
    pro-rated by the share of the LLM run's calls (per judge) that a live cascade would
    make: its escalated pairs over its pairs when called per criterion, its items with an
    escalated criterion over its items (those with any criterion) when called per item. An
    item whose decision-model grading raised (it has no votes) makes no call, as a live
    cascade sends it to no LLM judge, while its calls in the LLM run count among that
    run's. These are estimates, which ``escalation_stats`` sums into ``cost_usd`` and
    ``compute_seconds``. The result says so in its ``experiment_name``, its descriptive
    field (``"replay of decision-model run '<name>' and LLM run '<name>' (estimated cost
    and time)"``), since a replay is not an experiment on disk (``experiment_dir`` is
    ``None``). It carries no ``token_usage``, whatever ``llm_calls`` says. Token usage is
    recorded per item, not per criterion, so for judges called per criterion an item's
    cannot be split; for judges called per item it would rest on ``llm_calls``, which the
    replay cannot check and which changes nothing but the estimates marked as such.

    **Length penalty.** It is not re-applied: compare replayed scores with runs graded
    without one. Criterion-level metrics are unaffected.

    An item whose grading raised in ``dm_result`` (it has no votes) stays failed, as it
    would in a live cascade, whose decision model grades first. An item every
    decision-model judgment of which failed is replayed like any other: its failed votes
    escalate, as a live cascade escalates them.

    Args:
        dm_result: A run of one decision-model judge alone (e.g.
            ``CriterionGrader(judge_model_config=DecisionModelConfig(...))``), live or
            loaded with ``EvalResult.from_experiment``.
        llm_result: A run of the LLM judge(s) alone over the same items and criteria, with
            a vote of every judge on every criterion of every item: whichever criteria a
            threshold escalates, the run costs a full LLM grading of its items.
        escalation: The threshold, a number in [0, 1], or an ``EscalationPoint`` (e.g. one
            of ``calibrate_escalation``'s curve) whose ``threshold`` and ``per_criterion``
            are used.
        per_criterion: Thresholds for particular criteria, by criterion name, each in
            [0, 1]; only with a number as ``escalation``.
        cannot_assess_config: How abstentions score (``score_reports``); default
            ``CannotAssessConfig()``, as ``CriterionGrader``'s.
        normalize: Whether scores are normalized to [0, 1], as ``CriterionGrader``'s.
        llm_calls: How the LLM run's judges were called, its grader's ``llm_calls``:
            ``"per_criterion"`` (the default) or ``"per_item"``. It sets the unit of the
            cost and time estimates and nothing else (see *How the LLM run was called*).

    Returns:
        An ordinary ``EvalResult`` (``compute_metrics``, ``escalation_stats`` and
        ``ItemResult.to_dict`` work on it), with one item result per paired item, in
        ``item_idx`` order.

    Raises:
        ValueError: If the threshold or a ``per_criterion`` threshold is not a number in
            [0, 1]; if ``per_criterion`` is given with an ``EscalationPoint``, or the point
            has no threshold; if ``llm_calls`` is neither ``"per_criterion"`` nor
            ``"per_item"``; if the two results do not cover the same items (by
            ``item_idx``, a result holding an item more than once, or an item whose
            recorded submissions differ) graded against the same criteria (as graded, NA
            options included); if ``dm_result`` does not come from one decision-model judge
            (every vote carries ``probabilities`` and a ``confidence``, unless its judgment
            failed); if ``llm_result`` has an item whose grading raised (it has no votes;
            one whose every judgment failed keeps them), a criterion missing one of its
            judges' votes, a decision model's vote, or judges that differ between
            items; or if a ``judge_id`` names both the decision model and an LLM judge.

    Warns:
        UserWarning: If a ``per_criterion`` name matches no criterion of the replayed
            items (it sets no threshold), as a live cascade warns. If ``escalation`` is an
            ``EscalationPoint`` whose ``calibration_fingerprint`` is the fingerprint of the
            items being replayed: a threshold chosen and evaluated on the same pairs looks
            better than it will on new items. (An item loaded with
            ``EvalResult.from_experiment`` records no submission, so the check cannot
            recognize it.)
    """
    # The thresholds, checked as EscalationConfig checks them.
    if isinstance(escalation, EscalationPoint):
        if per_criterion is not None:
            raise ValueError(
                "per_criterion goes with a float threshold: an EscalationPoint carries its own "
                "(EscalationPoint.per_criterion)"
            )
        if escalation.threshold is None:
            raise ValueError(
                "This EscalationPoint has no threshold: it was measured on a run "
                "(escalation_stats), which does not record its grader's configuration; pass "
                "a threshold, or a point that has one"
            )
        threshold, per_criterion = escalation.threshold, escalation.per_criterion
        threshold_name = "EscalationPoint.threshold"
        per_criterion_name = "EscalationPoint.per_criterion"
        calibrated_on = escalation.calibration_fingerprint
    else:
        threshold = escalation
        threshold_name, per_criterion_name = "escalation", "per_criterion"
        calibrated_on = None
    per_criterion = _check_thresholds(
        threshold,
        per_criterion,
        threshold_name=threshold_name,
        per_criterion_name=per_criterion_name,
    )
    _check_llm_calls(llm_calls)

    pairs = _paired_items(dm_result, llm_result)
    _warn_unknown_threshold_names(
        per_criterion,
        (_llm_criteria(llm_item) for _, llm_item in pairs),
        source=per_criterion_name,
        where="the replayed items",
    )
    if calibrated_on is not None and calibrated_on == _replayed_fingerprint(pairs):
        warnings.warn(
            "This EscalationPoint was calibrated on the items it replays (its "
            "calibration_fingerprint matches theirs): a threshold chosen and evaluated on the "
            "same pairs looks better than it will on new items; replay held-out items",
            UserWarning,
            stacklevel=_caller_stacklevel(),
        )
    config = cannot_assess_config if cannot_assess_config is not None else CannotAssessConfig()

    def score(reports: list[CriterionReport], normalized: bool) -> float | None:
        return score_reports(reports, config, normalized)

    item_results: list[ItemResult] = []
    # The LLM calls (per judge) a live cascade would make, and those the LLM run made: the
    # escalated pairs and the pairs, or the items with an escalated criterion and the items.
    n_escalated = n_llm_calls = 0
    for dm_item, llm_item in pairs:
        item_result, escalated, judged = _replayed_item(
            dm_item, llm_item, threshold, per_criterion, score, normalize, llm_calls
        )
        item_results.append(item_result)
        n_escalated += escalated
        n_llm_calls += judged
    share = n_escalated / n_llm_calls if n_llm_calls else 0.0

    # Totals as EvalRunner computes them: a failed item's cost counts too.
    errors = [(r.item_idx, r.error) for r in item_results if r.error]
    reports = [r.report for r in item_results]
    total_duration = (
        dm_result.timing_stats.total_duration_seconds
        + llm_result.timing_stats.total_duration_seconds * share
    )
    replayed_at = datetime.now()
    return EvalResult(
        item_results=item_results,
        total_items=len(item_results),
        successful_items=len(item_results) - len(errors),
        failed_items=len(errors),
        total_token_usage=None,
        total_completion_cost=aggregate_completion_cost([r.completion_cost for r in reports]),
        timing_stats=EvalTimingStats.from_durations(
            [r.duration_seconds for r in item_results], total_duration
        ),
        started_at=replayed_at,
        completed_at=replayed_at,
        errors=errors,
        experiment_name=(
            f"replay of {_run_name(dm_result, 'decision-model')} and "
            f"{_run_name(llm_result, 'LLM')} (estimated cost and time)"
        ),
        experiment_dir=None,
    )

---

calibrate_escalation

Replay and measure a cascade at every threshold of a sweep.

calibrate_escalation

calibrate_escalation(dataset: RubricDataset, dm_result: EvalResult, llm_result: EvalResult, *, thresholds: Sequence[float] | None = None, metric: str | Callable[[MetricsResult], float | None] = 'criterion_accuracy', per_criterion: bool = False, min_pairs_per_criterion: int = 30, cannot_assess_config: CannotAssessConfig | None = None, normalize: bool = True, llm_calls: LLMCalls = 'per_criterion') -> EscalationCurve

Measure a confidence cascade at every threshold of a sweep, with no calls.

For each threshold of thresholds the cascade is replayed from the two runs (replay_escalation, scoring as the cascade's grader scores, by cannot_assess_config and normalize) and the replay measured against dataset (escalation_stats, whose metric is read from compute_metrics(replay, dataset)). Each point of the curve carries its thresholds, its escalation_rate, the decision model's accuracy on the pairs it kept and on those it deferred, the fallback's accuracy on the same deferred pairs (see EscalationPoint), metric, the replay's estimated cost_usd and compute_seconds (see replay_escalation; they follow how the LLM run was called, llm_calls), and the calibration_fingerprint of the calibration items: the items of dataset the two runs graded (by item_idx). EscalationCurve.best picks a point, and replay_escalation replays one.

Cost. The sweep makes no calls, but its inputs are two full gradings of the calibration items: the decision model's, and the LLM judges' votes on every criterion of every item (llm_result), whichever criteria a threshold would escalate. That LLM run is the price of calibration; calibrate on a sample of the items the cascade is meant to grade.

Same-data caveat. Every point is measured on the pairs the thresholds are chosen from, so the metric of the point best picks is optimistic: it is the best of many measurements on the same pairs. Measure the chosen point on held-out items graded by the same judges (e.g. the other part of RubricDataset.split_train_test), with replay_escalation or a live cascade. Replaying it on its own calibration items warns, by its calibration_fingerprint, when either run records the items' submissions: runs loaded with EvalResult.from_experiment record none, so replaying the point on them cannot be recognized and does not warn.

Exactness. Each replay is the run a live cascade at the point's thresholds would produce when the two runs meet replay_escalation's exactness conditions (each graded by a CriterionGrader configured as the cascade for everything that stage uses: for the LLM run, judges=escalation.judges with the cascade's seed, llm_calls, shuffle_options, system prompts, few-shot settings, auto_na_option and aggregation settings; e.g. CriterionGrader(judges=escalation.judges, seed=S) for a cascade built with seed=S and otherwise default settings) and when cannot_assess_config and normalize are the cascade's grader's: score-level metrics (score_rmse, score_pearson, ...) read the scores they set. With only another seed or judge_id the LLM prompts differ in the order of a shuffled multi-choice criterion's options and in the few-shot examples, and the curve holds for the cascade in distribution only; with another prompt setting it does not describe the cascade. This call's llm_calls states how the LLM run was called, which the runs do not record: it sets only how each point's cost_usd and compute_seconds are estimated, so a wrong one biases those two alone, on which neither EscalationCurve.best nor the per-criterion fitting below selects.

Per-criterion thresholds (per_criterion=True). A threshold of its own (EscalationConfig.per_criterion) applies to every criterion of a name, so it is fitted on that name's pairs, across the calibration items.

• Which criteria. Each named criterion with at least min_pairs_per_criterion labelled pairs (a pair's item graded, its label not an abstention; see EscalationPoint). The others use the swept global threshold: unnamed criteria, which cannot have a threshold of their own, and named criteria with fewer labelled pairs, which the call warns about.
• The same selection rule, on the run's metric. The fit starts from the global threshold that EscalationCurve.best's rule, at its defaults (no budget, no tolerance), picks on the sweep alone. Each fitted criterion's threshold in turn, in order of first appearance, is then the threshold of the sweep that the same rule picks on the whole run's metric, with the global threshold and the thresholds already fitted held where they are. A criterion's threshold decides whether its own pairs escalate and nothing else, so each fit is made on that criterion's pairs, by what they do to metric, as metric counts them. criterion_accuracy (whose pooling on each of compute_metrics' paths EscalationPoint describes), for one, leaves out an escalated pair whose final verdict abstains or stands in for failed calls, but for a genuine NA verdict it counts as an option (in a rubric without binary criteria, on the per-criterion path); a criterion it does not read (a multi-choice criterion in a rubric with binary criteria, on the per-criterion path) escalates as little as the sweep lets it, while on the pooled path it reads every criterion. No fit lowers the metric, so the point at the start's global threshold measures at least as high with the fits as without them, and so does the point best picks at its defaults. If metric is undefined at every threshold of the sweep, there is no start, and the call fits nothing and warns.
• The points. Each point is one global threshold of the sweep plus the fitted thresholds, the same at every point: its threshold and per_criterion, exactly what replay_escalation(dm_result, llm_result, point) replays and what a live cascade is configured with (EscalationConfig(judges, point.threshold, per_criterion=point.per_criterion)), measured on the whole run like any point. best therefore picks among global thresholds with the fits in place: its budget and tolerance trade on the criteria that follow the global threshold. A point records neither cannot_assess_config and normalize nor llm_calls, so its replay scores, and estimates cost and time, as this call did only when given the same ones.

Fitting replays the sweep once more for each fitted criterion, and once with the fits in place. Fitting on a few dozen pairs is where the same-data caveat bites hardest; min_pairs_per_criterion bounds it.

PARAMETER	DESCRIPTION
dataset	The calibration items with their ground truth: the dataset both runs graded. TYPE: RubricDataset
dm_result	A run of the cascade's decision-model judge alone over the calibration items (as for replay_escalation). TYPE: EvalResult
llm_result	A run of the cascade's escalation judge(s) alone over the same items, with a vote on every criterion (as for replay_escalation). TYPE: EvalResult
thresholds	The global thresholds to sweep, each a number in [0, 1], in the order the curve keeps; default 0.00, 0.02, ..., 1.00. TYPE: Sequence[float] | None DEFAULT: None
metric	The metric of a whole replay, higher being better: the name of a MetricsResult attribute (default "criterion_accuracy") or a function of the MetricsResult (negate a metric where lower is better); a NaN is read as undefined (None). On a dataset whose items have different rubrics, compute_metrics takes its pooled path: MetricsResult.per_criterion is empty and the run-level fields are pooled over rubric points (criterion_accuracy is the exact accuracy over the points of every scale, a point whose prediction abstains left out and one whose label abstains counted as a miss), so a metric of one scale type needs a function that reads pooled_by_scale. TYPE: str | Callable[[MetricsResult], float | None] DEFAULT: 'criterion_accuracy'
per_criterion	Whether named criteria get thresholds of their own, fitted as above. TYPE: bool DEFAULT: False
min_pairs_per_criterion	The labelled pairs a criterion needs for a threshold of its own, a positive int. TYPE: int DEFAULT: 30
cannot_assess_config	How abstentions score in the replays (score_reports), as in replay_escalation: the cascade's grader's; default CannotAssessConfig(). TYPE: CannotAssessConfig | None DEFAULT: None
normalize	Whether the replays' scores are normalized to [0, 1], as the cascade's grader's. TYPE: bool DEFAULT: True
llm_calls	How the LLM run's judges were called, its grader's llm_calls ("per_criterion", the default, or "per_item"), as in replay_escalation: the unit in which the replays estimate cost and time. TYPE: LLMCalls DEFAULT: 'per_criterion'

RETURNS	DESCRIPTION
EscalationCurve	The curve: one point per threshold of thresholds, in their order.

RAISES

DESCRIPTION

ValueError

If thresholds is empty or holds anything but a number in [0, 1]; if per_criterion is not a bool (thresholds by name are replay_escalation's) or min_pairs_per_criterion not a positive int; if llm_calls is neither "per_criterion" nor "per_item". The errors of replay_escalation (the two runs are not a decision model's and its escalation judges' over the same items and criteria) and of escalation_stats (metric gives no number; compute_metrics' own) propagate.

WARNS	DESCRIPTION
UserWarning	With per_criterion=True, naming the named criteria that use the global threshold for want of labelled pairs, or because metric is undefined at every threshold of the sweep.

Source code in src/autorubric/escalation.py

def calibrate_escalation(
    dataset: RubricDataset,
    dm_result: EvalResult,
    llm_result: EvalResult,
    *,
    thresholds: Sequence[float] | None = None,
    metric: str | Callable[[MetricsResult], float | None] = "criterion_accuracy",
    per_criterion: bool = False,
    min_pairs_per_criterion: int = 30,
    cannot_assess_config: CannotAssessConfig | None = None,
    normalize: bool = True,
    llm_calls: LLMCalls = "per_criterion",
) -> EscalationCurve:
    """Measure a confidence cascade at every threshold of a sweep, with no calls.

    For each threshold of ``thresholds`` the cascade is replayed from the two runs
    (``replay_escalation``, scoring as the cascade's grader scores, by
    ``cannot_assess_config`` and ``normalize``) and the replay measured against ``dataset``
    (``escalation_stats``, whose ``metric`` is read from ``compute_metrics(replay,
    dataset)``). Each point of the curve carries its thresholds, its ``escalation_rate``,
    the decision model's accuracy on the pairs it kept and on those it deferred, the
    fallback's accuracy on the same deferred pairs (see ``EscalationPoint``), ``metric``,
    the replay's estimated
    ``cost_usd`` and ``compute_seconds`` (see ``replay_escalation``; they follow how the LLM
    run was called, ``llm_calls``), and the
    ``calibration_fingerprint`` of the calibration items: the items of ``dataset`` the two
    runs graded (by ``item_idx``). ``EscalationCurve.best`` picks a point, and
    ``replay_escalation`` replays one.

    **Cost.** The sweep makes no calls, but its inputs are two full gradings of the
    calibration items: the decision model's, and the LLM judges' votes on every criterion
    of every item (``llm_result``), whichever criteria a threshold would escalate. That LLM
    run is the price of calibration; calibrate on a sample of the items the cascade is
    meant to grade.

    **Same-data caveat.** Every point is measured on the pairs the thresholds are chosen
    from, so the metric of the point ``best`` picks is optimistic: it is the best of many
    measurements on the same pairs. Measure the chosen point on held-out items graded by the
    same judges (e.g. the other part of ``RubricDataset.split_train_test``), with
    ``replay_escalation`` or a live cascade. Replaying it on its own calibration items warns,
    by its ``calibration_fingerprint``, when either run records the items' submissions: runs
    loaded with ``EvalResult.from_experiment`` record none, so replaying the point on them
    cannot be recognized and does not warn.

    **Exactness.** Each replay is the run a live cascade at the point's thresholds would
    produce when the two runs meet ``replay_escalation``'s exactness conditions (each
    graded by a ``CriterionGrader`` configured as the cascade for everything that stage
    uses: for the LLM run, ``judges=escalation.judges`` with the cascade's ``seed``,
    ``llm_calls``, ``shuffle_options``, system prompts, few-shot settings,
    ``auto_na_option`` and aggregation settings; e.g.
    ``CriterionGrader(judges=escalation.judges, seed=S)`` for a cascade built with
    ``seed=S`` and otherwise default settings) and when ``cannot_assess_config`` and
    ``normalize`` are the cascade's grader's: score-level metrics (``score_rmse``,
    ``score_pearson``, ...) read the scores they set. With only another seed or
    ``judge_id`` the LLM prompts differ in the order of a shuffled multi-choice criterion's
    options and in the few-shot examples, and the curve holds for the cascade in
    distribution only; with another prompt setting it does not describe the cascade.
    This call's ``llm_calls`` states how the LLM run was called, which the runs do not
    record: it sets only how each point's ``cost_usd`` and ``compute_seconds`` are
    estimated, so a wrong one biases those two alone, on which neither
    ``EscalationCurve.best`` nor the per-criterion fitting below selects.

    **Per-criterion thresholds** (``per_criterion=True``). A threshold of its own
    (``EscalationConfig.per_criterion``) applies to every criterion of a name, so it is
    fitted on that name's pairs, across the calibration items.

    - *Which criteria.* Each named criterion with at least ``min_pairs_per_criterion``
      labelled pairs (a pair's item graded, its label not an abstention; see
      ``EscalationPoint``). The others use the swept global threshold: unnamed criteria,
      which cannot have a threshold of their own, and named criteria with fewer labelled
      pairs, which the call warns about.
    - *The same selection rule, on the run's metric.* The fit starts from the global
      threshold that ``EscalationCurve.best``'s rule, at its defaults (no budget, no
      tolerance), picks on the sweep alone. Each fitted criterion's threshold in turn, in
      order of first appearance, is then the threshold of the sweep that the same rule picks
      on the whole run's ``metric``, with the global threshold and the thresholds already
      fitted held where they are. A criterion's threshold decides whether its own pairs
      escalate and nothing else, so each fit is made on that criterion's pairs, by what they
      do to ``metric``, as ``metric`` counts them. ``criterion_accuracy`` (whose pooling on
      each of ``compute_metrics``' paths ``EscalationPoint`` describes), for one, leaves out
      an escalated pair whose final verdict abstains or stands in for failed calls, but for
      a genuine NA verdict it counts as an option (in a rubric without binary criteria, on
      the per-criterion path); a criterion it does not read (a multi-choice criterion in a
      rubric with binary criteria, on the per-criterion path) escalates as little as the
      sweep lets it, while on the pooled path it reads every criterion. No fit lowers the
      metric, so the point at the start's global threshold measures at least as high with
      the fits as without them, and so does the point ``best`` picks at its defaults. If
      ``metric`` is undefined at every threshold of the sweep, there is no start, and the
      call fits nothing and warns.
    - *The points.* Each point is one global threshold of the sweep plus the fitted
      thresholds, the same at every point: its ``threshold`` and ``per_criterion``, exactly
      what ``replay_escalation(dm_result, llm_result, point)`` replays and what a live
      cascade is configured with (``EscalationConfig(judges, point.threshold,
      per_criterion=point.per_criterion)``), measured on the whole run like any point.
      ``best`` therefore picks among global thresholds with the fits in place: its budget
      and tolerance trade on the criteria that follow the global threshold. A point records
      neither ``cannot_assess_config`` and ``normalize`` nor ``llm_calls``, so its replay
      scores, and estimates cost and time, as this call did only when given the same ones.

    Fitting replays the sweep once more for each fitted criterion, and once with the fits
    in place.
    Fitting on a few dozen pairs is where the same-data caveat bites hardest;
    ``min_pairs_per_criterion`` bounds it.

    Args:
        dataset: The calibration items with their ground truth: the dataset both runs
            graded.
        dm_result: A run of the cascade's decision-model judge alone over the calibration
            items (as for ``replay_escalation``).
        llm_result: A run of the cascade's escalation judge(s) alone over the same items,
            with a vote on every criterion (as for ``replay_escalation``).
        thresholds: The global thresholds to sweep, each a number in [0, 1], in the order
            the curve keeps; default 0.00, 0.02, ..., 1.00.
        metric: The metric of a whole replay, higher being better: the name of a
            ``MetricsResult`` attribute (default ``"criterion_accuracy"``) or a function of
            the ``MetricsResult`` (negate a metric where lower is better); a NaN is read as
            undefined (``None``). On a dataset whose items have different rubrics,
            ``compute_metrics`` takes its pooled path: ``MetricsResult.per_criterion`` is
            empty and the run-level fields are pooled over rubric points
            (``criterion_accuracy`` is the exact accuracy over the points of every scale,
            a point whose prediction abstains left out and one whose label abstains
            counted as a miss), so a metric of one scale type needs a function that reads
            ``pooled_by_scale``.
        per_criterion: Whether named criteria get thresholds of their own, fitted as above.
        min_pairs_per_criterion: The labelled pairs a criterion needs for a threshold of
            its own, a positive int.
        cannot_assess_config: How abstentions score in the replays (``score_reports``), as
            in ``replay_escalation``: the cascade's grader's; default
            ``CannotAssessConfig()``.
        normalize: Whether the replays' scores are normalized to [0, 1], as the cascade's
            grader's.
        llm_calls: How the LLM run's judges were called, its grader's ``llm_calls``
            (``"per_criterion"``, the default, or ``"per_item"``), as in
            ``replay_escalation``: the unit in which the replays estimate cost and time.

    Returns:
        The curve: one point per threshold of ``thresholds``, in their order.

    Raises:
        ValueError: If ``thresholds`` is empty or holds anything but a number in [0, 1]; if
            ``per_criterion`` is not a bool (thresholds by name are ``replay_escalation``'s)
            or ``min_pairs_per_criterion`` not a positive int; if ``llm_calls`` is neither
            ``"per_criterion"`` nor ``"per_item"``. The errors of
            ``replay_escalation`` (the two runs are not a decision model's and its
            escalation judges' over the same items and criteria) and of
            ``escalation_stats`` (``metric`` gives no number; ``compute_metrics``' own)
            propagate.

    Warns:
        UserWarning: With ``per_criterion=True``, naming the named criteria that use the
            global threshold for want of labelled pairs, or because ``metric`` is undefined
            at every threshold of the sweep.
    """
    sweep = _DEFAULT_THRESHOLDS if thresholds is None else tuple(thresholds)
    if not sweep:
        raise ValueError("thresholds must hold at least one threshold")
    for i, threshold in enumerate(sweep):
        _check_thresholds(
            threshold, None, threshold_name=f"thresholds[{i}]", per_criterion_name="per_criterion"
        )
    if not isinstance(per_criterion, bool):
        raise ValueError(
            "per_criterion must be True or False (whether to fit thresholds per criterion); "
            f"got {per_criterion!r} (thresholds by criterion name go to replay_escalation)"
        )
    if not _is_positive_int(min_pairs_per_criterion):
        raise ValueError(
            f"min_pairs_per_criterion must be a positive int; got {min_pairs_per_criterion!r}"
        )
    _check_llm_calls(llm_calls)

    fingerprint = _calibration_fingerprint(
        (dataset[r.item_idx], dataset.get_item_rubric(r.item_idx).rubric)
        for r in _covered_items(dm_result, dataset)
    )

    def measured(threshold: float, fitted: Mapping[str, float]) -> EscalationPoint:
        """The point of the replay at ``threshold`` with the ``fitted`` thresholds."""
        replay = replay_escalation(
            dm_result,
            llm_result,
            threshold,
            per_criterion=fitted or None,
            cannot_assess_config=cannot_assess_config,
            normalize=normalize,
            llm_calls=llm_calls,
        )
        configuration = {
            "threshold": float(threshold),
            "per_criterion": dict(fitted) or None,
            "calibration_fingerprint": fingerprint,
        }
        return escalation_stats(replay, dataset, metric=metric).model_copy(update=configuration)

    points = [measured(threshold, {}) for threshold in sweep]
    if per_criterion:
        fitted = _fitted_thresholds(
            dataset, dm_result, sweep, points, measured, min_pairs_per_criterion
        )
        if fitted:
            points = [measured(threshold, fitted) for threshold in sweep]
    return EscalationCurve(points=points)

---

escalation_stats

Diagnostics of a live or replayed cascade run.

escalation_stats

escalation_stats(result: EvalResult, dataset: RubricDataset, *, metric: str | Callable[[MetricsResult], float | None] = 'criterion_accuracy') -> EscalationPoint

Diagnostics of a live or replayed cascade run, as an EscalationPoint.

Reads each criterion report's escalated and superseded flags: a kept criterion's one vote is the decision model's, and an escalated criterion's first vote is the decision model's, superseded, while its final verdict is the fallback's. It pools (item, criterion) pairs, so it works on datasets whose items have different rubrics, where compute_metrics has no per-criterion table. The pairs and accuracies are defined in EscalationPoint. Like compute_metrics, it covers the items both in result and in dataset; items whose grading raised have no pairs (an item every judgment of which failed keeps its pairs: its decision model's failures escalated), but their grading time counts in compute_seconds.

The point's threshold and per_criterion are None (an EvalResult does not record its grader's configuration), as is its calibration_fingerprint. cost_usd and compute_seconds are exact sums for a live run and sums of estimates for a replay (replay_escalation).

PARAMETER	DESCRIPTION
result	A cascade run: CriterionGrader(escalation=...) through evaluate, or replay_escalation. A run of its decision model alone reads as a cascade that escalated nothing. TYPE: EvalResult
dataset	The dataset with the ground truth. TYPE: RubricDataset
metric	The metric of the whole run: the name of a MetricsResult attribute (default "criterion_accuracy"), or a function of the MetricsResult, e.g. one that reads pooled_by_scale on a dataset with per-item rubrics. Any real number (a NumPy scalar included; not a bool) is recorded as a float; a NaN is read as undefined: the point's metric is then None. TYPE: str | Callable[[MetricsResult], float | None] DEFAULT: 'criterion_accuracy'

RETURNS	DESCRIPTION
EscalationPoint	The run's EscalationPoint.

RAISES

DESCRIPTION

ValueError

If result is not a cascade run: an item with no judges' votes, a kept criterion with other than one vote, an escalated one whose first vote is not the only superseded one, or a first vote that is not a decision model's (one with probabilities and a confidence, unless its judgment failed). If an item's report and its rubric in dataset differ in their number of criteria, or if metric names no attribute of MetricsResult or gives something other than a number or None. compute_metrics' own errors propagate (e.g. no item with ground truth).

Source code in src/autorubric/escalation.py

def escalation_stats(
    result: EvalResult,
    dataset: RubricDataset,
    *,
    metric: str | Callable[[MetricsResult], float | None] = "criterion_accuracy",
) -> EscalationPoint:
    """Diagnostics of a live or replayed cascade run, as an ``EscalationPoint``.

    Reads each criterion report's ``escalated`` and ``superseded`` flags: a kept criterion's
    one vote is the decision model's, and an escalated criterion's first vote is the
    decision model's, superseded, while its final verdict is the fallback's. It pools
    (item, criterion) pairs, so it works on datasets whose items have different rubrics,
    where ``compute_metrics`` has no per-criterion table. The pairs and accuracies are
    defined in ``EscalationPoint``. Like ``compute_metrics``, it covers the items both in
    ``result`` and in ``dataset``; items whose grading raised have no pairs (an item every
    judgment of which failed keeps its pairs: its decision model's failures escalated),
    but their grading time counts in ``compute_seconds``.

    The point's ``threshold`` and ``per_criterion`` are ``None`` (an ``EvalResult`` does
    not record its grader's configuration), as is its ``calibration_fingerprint``.
    ``cost_usd`` and ``compute_seconds`` are exact sums for a live run and sums of
    estimates for a replay (``replay_escalation``).

    Args:
        result: A cascade run: ``CriterionGrader(escalation=...)`` through ``evaluate``, or
            ``replay_escalation``. A run of its decision model alone reads as a cascade that
            escalated nothing.
        dataset: The dataset with the ground truth.
        metric: The metric of the whole run: the name of a ``MetricsResult`` attribute
            (default ``"criterion_accuracy"``), or a function of the ``MetricsResult``,
            e.g. one that reads ``pooled_by_scale`` on a dataset with per-item rubrics. Any
            real number (a NumPy scalar included; not a bool) is recorded as a float; a
            NaN is read as undefined: the point's ``metric`` is then ``None``.

    Returns:
        The run's ``EscalationPoint``.

    Raises:
        ValueError: If ``result`` is not a cascade run: an item with no judges' votes, a
            kept criterion with other than one vote, an escalated one whose first vote is
            not the only superseded one, or a first vote that is not a decision model's
            (one with ``probabilities`` and a ``confidence``, unless its judgment failed).
            If an item's report and its rubric in ``dataset`` differ in their number of
            criteria, or if ``metric`` names no attribute of ``MetricsResult`` or gives
            something other than a number or ``None``. ``compute_metrics``' own errors
            propagate (e.g. no item with ground truth).
    """
    items = _covered_items(result, dataset)
    n_pairs = n_escalated = 0
    dm_kept: list[bool] = []
    dm_escalated: list[bool] = []
    fallback: list[bool] = []
    for report, dm_vote, label in _cascade_pairs(items, dataset):
        n_pairs += 1
        n_escalated += report.escalated
        # A decision-model failure or abstention escalates at any threshold and is no
        # prediction to compare the fallback's with.
        if label is None or (prediction := _vote_prediction(dm_vote)) is None:
            continue
        if not report.escalated:
            dm_kept.append(prediction == label)
        elif (final := _final_prediction(report)) is not None:
            # A deferred pair the fallback answered: both judges are measured on it.
            dm_escalated.append(prediction == label)
            fallback.append(final == label)

    return EscalationPoint(
        escalation_rate=n_escalated / n_pairs if n_pairs else None,
        dm_accuracy_kept=_accuracy(dm_kept),
        dm_accuracy_escalated=_accuracy(dm_escalated),
        fallback_accuracy_escalated=_accuracy(fallback),
        metric=_metric_value(compute_metrics(result, dataset), metric),
        cost_usd=aggregate_completion_cost([r.report.completion_cost for r in items]),
        compute_seconds=float(sum(r.duration_seconds for r in items)),
    )

---

EscalationCurve

The points of a threshold sweep and the rule that picks one.

EscalationCurve

Bases: BaseModel

The points of a threshold sweep (calibrate_escalation), and the rule that picks one.

Each point is a cascade at one global threshold (plus the fitted per-criterion thresholds, when the calibration fitted any), measured on the calibration items, and carries their calibration_fingerprint. best picks a point by one rule; replay_escalation(dm_result, llm_result, point), given the scoring settings and the llm_calls the calibration was given (a point records neither), rebuilds the point's run on the runs it was calibrated on, or replays its thresholds on held-out runs.

The selection rule (best). Among the points within the escalation budget (escalation_rate <= max_escalation_rate; every point when there is no budget), find the best metric; of the points whose metric is within tolerance of it (inclusively, a gap equal to tolerance up to floating-point rounding included), pick the one that escalates least. A higher metric is better. Ties are broken in order by the higher metric, then the lower threshold (equal here, but on new items a lower threshold never escalates more), then the earlier point. A point without a metric (None, or NaN) is passed over; one without an escalation_rate (a run with no pairs) is outside any budget and ranks after every point that has one; one without a threshold ranks after those with one. tolerance > 0 trades metric for less escalation: best(tolerance=0.005) gives up at most half a point of accuracy. Escalation judges called per criterion make fewer calls as fewer pairs escalate, so the trade is "about the same accuracy for fewer LLM calls". Called per item (llm_calls="per_item"), they make one call per item with any escalated criterion, so escalating fewer pairs can save no call at all: compare the points' cost_usd and compute_seconds before giving up metric for it.

Same-data caveat. Every point was measured on the pairs the sweep chose from, so the metric of the point best picks is optimistic: it is the best of many measurements on those pairs. Measure the chosen point on held-out items graded by the same judges (replay_escalation on their runs, or a live cascade). Replaying it on its own calibration items warns (calibration_fingerprint) when either run records the items' submissions; runs loaded with EvalResult.from_experiment record none, so there the check cannot recognize them.

ATTRIBUTE	DESCRIPTION
points	The operating points, in the order of the sweep. TYPE: list[EscalationPoint]

best

best(max_escalation_rate: float | None = None, tolerance: float = 0.0) -> EscalationPoint

The point the selection rule picks (see the class docstring).

PARAMETER	DESCRIPTION
max_escalation_rate	The escalation budget, a share of pairs in [0, 1] (inclusive), or None for no budget. TYPE: float | None DEFAULT: None
tolerance	How far below the best metric within the budget a point's metric may be, in the metric's units (a number >= 0). TYPE: float DEFAULT: 0.0

RETURNS	DESCRIPTION
EscalationPoint	The least escalating point within tolerance of the best metric within the
EscalationPoint	budget.

RAISES	DESCRIPTION
ValueError	If max_escalation_rate or tolerance is out of range; if the curve has no points, none within the budget, or none within the budget with a metric.

Source code in src/autorubric/escalation.py

def best(
    self, max_escalation_rate: float | None = None, tolerance: float = 0.0
) -> EscalationPoint:
    """The point the selection rule picks (see the class docstring).

    Args:
        max_escalation_rate: The escalation budget, a share of pairs in [0, 1]
            (inclusive), or ``None`` for no budget.
        tolerance: How far below the best metric within the budget a point's metric
            may be, in the metric's units (a number >= 0).

    Returns:
        The least escalating point within ``tolerance`` of the best metric within the
        budget.

    Raises:
        ValueError: If ``max_escalation_rate`` or ``tolerance`` is out of range; if the
            curve has no points, none within the budget, or none within the budget
            with a metric.
    """
    if max_escalation_rate is not None and not (
        _is_real_number(max_escalation_rate) and 0 <= max_escalation_rate <= 1
    ):
        raise ValueError(
            "max_escalation_rate must be a number in [0, 1] or None; "
            f"got {max_escalation_rate!r}"
        )
    if not (_is_real_number(tolerance) and tolerance >= 0):
        raise ValueError(f"tolerance must be a number >= 0; got {tolerance!r}")
    candidates = [(p.escalation_rate, p.metric, p.threshold) for p in self.points]
    return self.points[_selected(candidates, max_escalation_rate, tolerance)]

to_dataframe

to_dataframe() -> DataFrame

The points as a pandas DataFrame: a row per point, a column per field of EscalationPoint (per_criterion holds each point's dict, or None).

pandas is imported here, as MetricsResult.to_dataframe imports it.

Source code in src/autorubric/escalation.py

def to_dataframe(self) -> pd.DataFrame:
    """The points as a pandas ``DataFrame``: a row per point, a column per field of
    ``EscalationPoint`` (``per_criterion`` holds each point's dict, or ``None``).

    pandas is imported here, as ``MetricsResult.to_dataframe`` imports it.
    """
    import pandas as pd

    return pd.DataFrame(
        [point.model_dump() for point in self.points],
        columns=list(EscalationPoint.model_fields),
    )

---

EscalationPoint

One operating point of a cascade: its thresholds and what they achieve.

EscalationPoint

Bases: BaseModel

One operating point of a confidence cascade: its thresholds, and what they achieve.

escalation_stats measures one on a cascade run, live or replayed, and calibrate_escalation one per threshold of a sweep. Given to replay_escalation, a point replays a cascade at its threshold and per_criterion thresholds. A point records neither the scoring settings it was measured with nor how the LLM run's judges were called (llm_calls), so a replay of a calibrated point scores, and estimates cost and time, as the calibration did only when given the same ones.

Pairs and accuracy. A pair is one criterion of one item whose grading did not raise, even one whose every criterion's judgment failed. A pair is labelled when its item has ground truth for the criterion that does not abstain: a binary label other than CANNOT_ASSESS, or a multi-choice label that is not an NA option. An accuracy is the share of a set of labelled pairs, binary and multi-choice pooled, on which a prediction equals the label: the verdict of a binary criterion, the selected option of a multi-choice one. As compute_metrics measures one criterion under its default handling (cannot_assess="exclude", na_mode="exclude": CriterionMetrics.accuracy, exact_accuracy), an abstention (CANNOT_ASSESS, an NA option) is not a prediction, so a pair on which the predictor abstained counts in neither the numerator nor the denominator. (The run-level criterion_accuracy of compute_metrics pools otherwise. On its per-criterion path, for items that share one rubric, it reads the binary pairs alone when there are any, else the multi-choice pairs with an NA prediction or label counted as one more option. On its pooled path, for items with different rubrics, it reads the pairs of every scale, leaves out a pair whose prediction abstains, and counts one whose label abstains as a miss. On either path, a criterion whose judgment failed is no pair.)

• The decision model is measured as compute_metrics(..., per_judge=True) measures a judge: on its own votes, superseded ones included, leaving out a failed judgment (a vote with an error), which is no prediction of the decision model's. A decision-model failure or abstention escalates at any threshold, so it counts in escalation_rate but in no accuracy: it leaves the decision model no prediction to measure, or to compare the fallback's with.
• The fallback's predictions are the escalated pairs' final verdicts, which the escalation judges' votes alone decide, read as compute_metrics reads a run's: a final verdict that abstains is no prediction, and neither is one that stands in for the judges' failed calls (such as the worst case of an unknown failure), which the run's metrics leave out too.
• The deferred pairs are the escalated labelled pairs the decision model answered: it escalated them for a confidence below the threshold. dm_accuracy_escalated and fallback_accuracy_escalated are both measured on the deferred pairs whose final verdict is a prediction, where both judges made one, so they compare the two judges on the same pairs.

ATTRIBUTE	DESCRIPTION
threshold	The confidence threshold below which a criterion is escalated. None when the point was measured on a run (escalation_stats), because an EvalResult does not record its grader's configuration. TYPE: float | None
per_criterion	Thresholds for particular criteria, by criterion name (as EscalationConfig.per_criterion); None when there are none or the point was measured on a run. TYPE: dict[str, float] | None
escalation_rate	The share of the run's pairs, labelled or not, that were escalated. None when the run has no pairs. TYPE: float | None
dm_accuracy_kept	The decision model's accuracy on the pairs it kept. A cascade can pay only if its confidence separates these from the pairs it defers. TYPE: float | None
dm_accuracy_escalated	The decision model's accuracy on the deferred pairs whose final verdict is a prediction (it neither abstains nor stands in for failed calls). TYPE: float | None
fallback_accuracy_escalated	The accuracy of the final verdicts, which the fallback LLM judges decide, on the same pairs. On them, escalating rather than keeping the decision model's verdicts gains right verdicts exactly when this beats dm_accuracy_escalated: the condition for a cascade to pay. The deferred pairs whose final verdict abstains, or stands in for failed calls, are not among them; they leave a metric that leaves them out (such as criterion_accuracy), which rises when the decision model was wrong on them. TYPE: float | None
metric	The chosen metric of the whole run, from compute_metrics(result, dataset). TYPE: float | None
cost_usd	The run's cost: the sum of its items' completion_cost, None when no item has one. Exact for a live run; for a replay, a sum of estimates (see replay_escalation). TYPE: float | None
compute_seconds	The sum of the items' grading times (duration_seconds), not the wall-clock time of the run. Exact for a live run; for a replay, a sum of estimates. TYPE: float
calibration_fingerprint	The fingerprint of the items the point was calibrated on (calibrate_escalation), when it was; None otherwise. replay_escalation warns when it equals the fingerprint of the items being replayed, which it can compute only from runs that record the items' submissions (not from runs loaded with EvalResult.from_experiment). TYPE: str | None

Accuracies are None when their set of labelled pairs is empty.