Skip to content

LLM Infrastructure

LLM client configuration, caching, and generation utilities.

Overview

AutoRubric uses LiteLLM for multi-provider LLM support. The LLMConfig class provides centralized configuration, while LLMClient handles request execution with caching, rate limiting, and retry logic.

For a guided introduction see LLM Judges.

Quick Example

from autorubric import LLMConfig, LLMClient, generate

# Configuration
config = LLMConfig(
    model="openai/gpt-5.6-luna",
    max_tokens=1024,
    cache_enabled=True,
    max_parallel_requests=10,
)

# Direct generation (standalone function)
result = await generate(
    system_prompt="You are a helpful assistant.",
    user_prompt="Explain quantum computing.",
    model="openai/gpt-5.6-luna",
)
print(result)

# Or use the client
client = LLMClient(config)
result = await client.generate(
    system_prompt="You are a helpful assistant.",
    user_prompt="Explain quantum computing.",
)

Provider Configuration

Provider Model Format Environment Variable
OpenAI openai/gpt-4.1, openai/gpt-5.6-luna OPENAI_API_KEY
Anthropic anthropic/claude-sonnet-4-5-20250929 ANTHROPIC_API_KEY
Google gemini/gemini-2.5-flash GEMINI_API_KEY
Azure azure/openai/gpt-4.1 AZURE_API_KEY, AZURE_API_BASE
Groq groq/llama-3.1-70b-versatile GROQ_API_KEY
Ollama ollama/qwen3:14b (local, no key needed)

Temperature

LLMConfig.temperature defaults to None: temperature is left out of the request, so the provider's own default applies. Many current models need or recommend this: reasoning models such as the GPT-5.x family accept only their default temperature, Gemini 3 recommends keeping its default of 1.0, and Anthropic requires temperature 1 when extended thinking is enabled.

An explicit value, including 0.0, is sent unchanged. A temperature= keyword argument to LLMClient.generate() overrides the config for that call (temperature=None omits it for that call). The response cache keys on the temperature actually used for the request (including a per-call override), and an unset temperature has its own key.

LLMConfig(model="openai/gpt-5.2")                        # temperature omitted: provider default
LLMConfig(model="gemini/gemini-3.8-flash", temperature=0.0)  # temperature=0.0 sent

Some models refuse it: OpenAI's Luna models accept neither a custom temperature nor top_p, so leave both unset for them.

A low temperature reduces run-to-run variance, but most providers do not guarantee identical outputs even at 0.0.

Default changed in v1.6.0

Up to and including v1.5.3, temperature defaulted to 0.0 and was always sent. A config that never sets it now samples at the provider's default, so its grades may differ from earlier runs, and responses cached by those runs are not reused (an unset temperature has its own cache key). To keep the old behaviour, set temperature=0.0 explicitly (temperature: 0.0 in YAML). Configs that already set temperature explicitly send the same value and keep their cached responses.

Output Length and Nucleus Sampling

max_tokens and top_p also default to None, which leaves them out of the request so the provider's own limit and default apply. A value you set is sent unchanged, including top_p=0.0. LLMConfig refuses a value no provider can mean with a ValueError: a max_tokens that is not an integer of at least 1, or a top_p that is not a number between 0 and 1 (a bool is not taken for a number).

LLMConfig(model="gemini/gemini-3.8-flash", max_tokens=1024, top_p=0.0)  # both sent
LLMConfig(model="gemini/gemini-3.8-flash", max_tokens=0)                 # ValueError

Changed behaviour

Earlier releases sent max_tokens and top_p only when they were truthy, so top_p=0.0 and max_tokens=0 were silently left out of the request. top_p=0.0 is now sent, and responses cached for it by earlier releases, which were sampled without it, are not reused (it has a new cache key). max_tokens=0, a negative max_tokens and a top_p outside [0, 1] are now refused. Configs that set other values send the same requests and keep their cached responses.

YAML Configuration

# llm_config.yaml
model: openai/gpt-4.1
temperature: 0.0  # optional; omit to use the provider default
max_tokens: 1024
cache_enabled: true
cache_ttl: 3600
config = LLMConfig.from_yaml("llm_config.yaml")
config.to_yaml("llm_config_backup.yaml")

thinking can be a level (thinking: high), a token budget (thinking: 32000), or a ThinkingConfig written as a mapping:

thinking:
  level: high
  budget_tokens: 9000

to_yaml writes a ThinkingLevel enum as its string value, so it loads back as the string "high". That string is equal to ThinkingLevel.HIGH and gives the same thinking settings.

Extended Thinking

Enable step-by-step reasoning for complex evaluations:

# Level-based (cross-provider)
config = LLMConfig(
    model="anthropic/claude-sonnet-4-5-20250929",
    thinking="high",  # "low", "medium", "high", or "none"
)

# Token budget
config = LLMConfig(
    model="anthropic/claude-opus-4-5-20251101",
    thinking=32000,  # Explicit token budget
)

Supported providers: Anthropic, OpenAI (o-series), Gemini (2.5+), DeepSeek.

thinking is checked when the LLMConfig is built. It must be None, a ThinkingLevel or one of the strings "none", "low", "medium", "high", an integer token budget of at least -1, or a ThinkingConfig; anything else raises ValueError there, before any request. That includes an unknown level such as "extreme", True or False (a bool is not a budget), a budget below -1, and a float such as 32000.0. A budget of 0 or -1 is accepted because Gemini 2.5 reads 0 as thinking off and -1 as dynamic thinking. A budget smaller than a provider's minimum passes this check. Anthropic's minimum is 1,024 tokens, and it rejects each request with a smaller budget with a 400 (BadRequestError). A grade does not raise that error as a configuration error, so every call of such a judge abstains (CANNOT_ASSESS, or NA for a multi-choice criterion). Use at least 1,024 for Anthropic models. ThinkingConfig checks its own level and budget_tokens the same way, except that level=None is accepted next to a budget_tokens, which is sent without reading the level. Both check their values only when they are built, so a field assigned afterwards is not checked again.

Response Caching

config = LLMConfig(
    model="openai/gpt-5.6-luna",
    cache_enabled=True,
    cache_dir=".autorubric_cache",
    cache_ttl=3600,  # 1 hour
)

client = LLMClient(config)
client.clear_cache()
stats = client.cache_stats()
# {'size': 1024, 'count': 10, 'directory': '.autorubric_cache'}

Images in Prompts

LLMClient.generate takes the user prompt as Content: a string, an Image, or a list of strings and images. A text prompt is sent as the message's content string, exactly as before. A prompt holding an image is sent as a list of OpenAI-style content parts in the prompt's order, a {"type": "text", "text": ...} part for each run of text and a {"type": "image_url", "image_url": {"url": "data:<media type>;base64,..."}} part for each image. LiteLLM translates these parts for providers with another image format (Anthropic, Gemini). The image is sent as its own bytes, neither resized nor re-encoded, so the provider's own limits on format, size and image count apply.

from autorubric import Image, LLMClient, LLMConfig

client = LLMClient(LLMConfig(model="openai/gpt-5.6-luna"))
reply = await client.generate(
    system_prompt="You describe figures.",
    user_prompt=[Image("figures/fig1.png"), "What does this figure show?"],
)

The response cache keys a prompt holding an image by the SHA-256 of its content, its text and each image's bytes in order, so an identical prompt hits the cache and a changed image misses it. Text prompts keep their existing cache keys. With prompt_caching=True, the system prompt is marked for caching as usual, and the image parts follow it in the user message. When a grader passes shared_prefix (grading on a model that caches prompts; see prompt caching), a prompt holding images is split as text is: the shared part is a user message of its own, any cache marker sits on its last text part, and the rest follows in a second user message. Without few-shot examples the shared part is the item's (the images of the submission, input and reference submission) and the criterion follows it. With examples the shared part is the examples (their images included) and the criterion, which a criterion's calls share across items, and the item, with its images, follows. autorubric.debug = True prints a marker in place of each image, [image: <file name>] for a file and [image: <N> bytes] for bytes.

A provider that refuses an image (a format it does not read, or a model without vision input) usually answers with a bad-request error. A grader routes it like any other failed call, so the criterion abstains (CANNOT_ASSESS, or NA for a multi-choice criterion) and the error is kept on its report. See the Image Inputs recipe for the formats providers accept.

A grader does not leave the model question to the provider. CriterionGrader sends an image to an LLM judge only when the judge is known to accept images, and LLMConfig.images says how that is known:

images Judge accepts images when
None (default) LiteLLM's model map lists the model as accepting images (its entry in litellm.model_cost has supports_vision: true). The map is read once per grader per model, the first time the grader is to send that model an image. The lookup reads only the model map LiteLLM loaded at import, so it sends no request, never waits on a provider, and prints and logs nothing. LiteLLM fetches that map from its repository at import unless LITELLM_LOCAL_MODEL_COST_MAP=True is set (then the copy installed with LiteLLM is used). The model is looked up under its provider-prefixed name and its bare name, and a responses/ route is looked up by its model (openai/responses/gpt-5-mini as openai/gpt-5-mini). A model the map does not hold, such as an Ollama model, is not confirmed.
True Always: your declaration, for a vision model the map does not list (for example ollama/llava).
False Never: your declaration, even for a model the map lists.

Any other value raises ValueError when the config is built (1 and "yes" are not declarations). A judge that is not known to accept images raises ValueError from the grade, naming the judge and the model, before any judge of the grader sends a request. fill_ground_truth raises it too, rather than leaving each item with an image out. LLMClient.generate itself does not check, so a direct call sends what it is given. Text-only grading never reads images or the map, and the field is never sent, so text requests and cache keys are unchanged. An experiment manifest records images for a judge only when it is set.

Prompt Caching (Anthropic)

Reduce latency and cost on repeated calls. Enabled by default, it arranges per-criterion requests so that providers can reuse what an item's calls share:

config = LLMConfig(
    model="anthropic/claude-sonnet-4-5-20250929",
    prompt_caching=True,  # Default
)

LLMConfig

Central configuration class for LLM calls.

LLMConfig dataclass

LLMConfig(model: str, temperature: float | None = None, max_tokens: int | None = None, top_p: float | None = None, timeout: float = 60.0, max_retries: int = 3, retry_min_wait: float = 1.0, retry_max_wait: float = 60.0, max_parallel_requests: int | None = None, cache_enabled: bool = False, cache_dir: str | Path = '.autorubric_cache', cache_ttl: int | None = None, api_key: str | None = None, api_base: str | None = None, thinking: ThinkingParam = None, prompt_caching: bool = True, seed: int | None = None, extra_headers: dict[str, str] = dict(), extra_params: dict[str, Any] = dict(), images: bool | None = None)

Configuration for LLM calls.

ATTRIBUTE DESCRIPTION
model

Model identifier in LiteLLM format (e.g., "openai/gpt-5.2", "anthropic/claude-sonnet-4-5-20250929", "gemini/gemini-3-pro-preview", "ollama/qwen3:14b"). REQUIRED - no default. See LiteLLM docs for full list of supported models.

TYPE: str

temperature

Sampling temperature. None (default) omits it from the request so the provider's own default applies; this is recommended for reasoning models such as GPT-5.x and for Gemini 3, and required for Anthropic extended thinking. An explicit value, including 0.0, is sent unchanged. Temperature 0.0 reduces but does not guarantee determinism on most providers. Note: the default changed from 0.0 to None in v1.6.0. To keep the previous behavior, pass temperature=0.0 explicitly (it keeps the same cache keys).

TYPE: float | None

max_tokens

Maximum tokens in the response: None (default) omits it from the request, so the provider's limit applies; otherwise an integer >= 1, sent unchanged.

TYPE: int | None

top_p

Nucleus sampling parameter: None (default) omits it from the request, so the provider's default applies; otherwise a number in [0, 1], sent unchanged (including 0).

TYPE: float | None

timeout

Request timeout in seconds.

TYPE: float

max_retries

Maximum retry attempts for transient failures.

TYPE: int

retry_min_wait

Minimum wait between retries (seconds).

TYPE: float

retry_max_wait

Maximum wait between retries (seconds).

TYPE: float

max_parallel_requests

Maximum concurrent requests to this model's provider. When set, a global per-provider semaphore limits parallel requests. The limit applies within one event loop; each asyncio.run call has its own. None (default) means unlimited parallel requests.

TYPE: int | None

cache_enabled

Default caching behavior (can be overridden per-request).

TYPE: bool

cache_dir

Directory for response cache.

TYPE: str | Path

cache_ttl

Cache time-to-live in seconds (None = no expiration).

TYPE: int | None

api_key

Optional API key override (otherwise uses environment variables).

TYPE: str | None

api_base

Optional API base URL override.

TYPE: str | None

thinking

Enable thinking/reasoning mode (unified across providers). Accepts multiple formats: - ThinkingLevel enum: ThinkingLevel.HIGH, ThinkingLevel.MEDIUM, etc. - String: "low", "medium", "high", "none" - Int: Direct token budget (e.g., 32000), an integer >= -1 (see ThinkingConfig.budget_tokens); a bool is not a budget - ThinkingConfig: Full configuration with level and/or budget_tokens - None: Disable thinking (default)

Any other value raises ValueError when the config is built.

Provider support: - Anthropic: Extended thinking (claude-sonnet-4-5, claude-opus-4-5+) - OpenAI: Reasoning for o-series and GPT-5 models - Gemini: Thinking mode (2.5+, 3.0+ models) - DeepSeek: Reasoning content

TYPE: ThinkingParam

prompt_caching

Arrange requests for the provider's prompt cache (default: True). A per-criterion call sends what an item's calls share as a user message of its own when LiteLLM's model map says the model caches prompts, and CriterionGrader lets one call finish before the rest when that prefix is long enough to cache. Model ids that start with anthropic/, claude or openrouter/anthropic/ also get cache_control: {"type": "ephemeral"} on the system message and that shared message, OpenAI models from GPT-5.6 on get explicit cache breakpoints there, and Gemini 2.5 and 3 models a five-minute context cache when the prefix reaches 4,096 tokens. Prompt caching is generally available, so no beta header is added, and an anthropic-beta value in extra_headers is sent unchanged. Claude models served through Bedrock or Vertex AI are sent one user message with no marker. With False every request is one user message; providers that cache long prompts on their own (OpenAI, DeepSeek) still do so, at message endings.

TYPE: bool

seed

Random seed for reproducible outputs (OpenAI, some other providers).

TYPE: int | None

extra_headers

Additional HTTP headers for provider-specific features.

TYPE: dict[str, str]

extra_params

Additional provider-specific parameters passed to LiteLLM.

TYPE: dict[str, Any]

images

Whether the model accepts images in its prompts. None (default) looks the model up in LiteLLM's model map (the one it loaded at import; no request is sent) the first time a grader is to send this judge an image; True or False is your declaration and wins over the map. A judge whose model neither LiteLLM lists nor you declare as accepting images is never sent one: grading an item holding images refuses it before any request (CriterionGrader). Set images=True for a vision model LiteLLM does not map (e.g. a LLaVA or Qwen-VL model served through Ollama, vLLM or OpenRouter). Text-only grading never reads it, so it changes no request. Only None, True and False are accepted.

TYPE: bool | None

Examples:

Basic usage without thinking (provider-default temperature)

config = LLMConfig(model="openai/gpt-5.2")

Explicit temperature

config = LLMConfig(model="gemini/gemini-3.8-flash", temperature=0.0) # sent as-is

Enable thinking with a level

config = LLMConfig(model="anthropic/claude-sonnet-4-5-20250929", thinking="high") config = LLMConfig(model="openai/responses/gpt-5-mini", thinking=ThinkingLevel.HIGH)

Enable thinking with explicit token budget

config = LLMConfig(model="anthropic/claude-opus-4-5-20251101", thinking=32000)

Full control with ThinkingConfig

config = LLMConfig( model="gemini/gemini-2.5-pro", thinking=ThinkingConfig(level=ThinkingLevel.HIGH, budget_tokens=50000) )

get_thinking_config

get_thinking_config() -> ThinkingConfig | None

Get normalized thinking configuration.

RAISES DESCRIPTION
ValueError

If thinking is not a ThinkingParam form. __post_init__ refuses such a value, so only one assigned to thinking after construction can raise (fields assigned later are not checked again).

Source code in src/autorubric/llm.py
def get_thinking_config(self) -> ThinkingConfig | None:
    """Get normalized thinking configuration.

    Raises:
        ValueError: If ``thinking`` is not a ``ThinkingParam`` form. ``__post_init__``
            refuses such a value, so only one assigned to ``thinking`` after
            construction can raise (fields assigned later are not checked again).
    """
    return _normalize_thinking_param(self.thinking, f"{type(self).__name__}.thinking")

from_yaml classmethod

from_yaml(path: str | Path) -> LLMConfig

Load LLMConfig from a YAML file.

PARAMETER DESCRIPTION
path

Path to YAML configuration file.

TYPE: str | Path

RETURNS DESCRIPTION
LLMConfig

LLMConfig instance with values from the YAML file.

RAISES DESCRIPTION
FileNotFoundError

If the file doesn't exist.

ValueError

If required fields are missing or invalid.

Example YAML file (llm_config.yaml): model: openai/gpt-5.2 # temperature omitted: the provider's default applies max_tokens: 1024 cache_enabled: true cache_ttl: 3600

thinking accepts a level string (high), an int token budget (32000), or a mapping of ThinkingConfig fields ({level: high, budget_tokens: 9000}), which is loaded as a ThinkingConfig.

Source code in src/autorubric/llm.py
@classmethod
def from_yaml(cls, path: str | Path) -> LLMConfig:
    """Load LLMConfig from a YAML file.

    Args:
        path: Path to YAML configuration file.

    Returns:
        LLMConfig instance with values from the YAML file.

    Raises:
        FileNotFoundError: If the file doesn't exist.
        ValueError: If required fields are missing or invalid.

    Example YAML file (llm_config.yaml):
        model: openai/gpt-5.2
        # temperature omitted: the provider's default applies
        max_tokens: 1024
        cache_enabled: true
        cache_ttl: 3600

    ``thinking`` accepts a level string (``high``), an int token budget (``32000``),
    or a mapping of ``ThinkingConfig`` fields (``{level: high, budget_tokens: 9000}``),
    which is loaded as a ``ThinkingConfig``.
    """
    path = Path(path)
    if not path.exists():
        raise FileNotFoundError(f"LLM config file not found: {path}")

    with open(path, encoding="utf-8") as f:
        data = yaml.safe_load(f)

    if not isinstance(data, dict):
        raise ValueError(f"Invalid YAML config: expected dict, got {type(data).__name__}")

    if "model" not in data:
        raise ValueError("LLM config YAML must specify 'model' field")

    # Every dataclass field is a known key (derived, so new fields can't be missed);
    # anything else is a provider-specific param and goes to extra_params.
    known_fields = {f.name for f in fields(cls)}
    extra = {k: v for k, v in data.items() if k not in known_fields}
    if extra:
        data.setdefault("extra_params", {}).update(extra)
        for k in extra:
            del data[k]

    # to_yaml writes a ThinkingConfig as a mapping of its fields; rebuild it. Level
    # strings and int budgets are already valid ThinkingParam values.
    thinking = data.get("thinking")
    if isinstance(thinking, dict):
        try:
            data["thinking"] = ThinkingConfig(**thinking)
        except TypeError as e:
            raise ValueError(f"Invalid 'thinking' mapping in LLM config YAML: {e}") from e

    return cls(**data)

to_yaml

to_yaml(path: str | Path) -> None

Save LLMConfig to a YAML file.

The file holds only plain YAML (yaml.safe_load reads it): enums are written as their values and paths as strings (so a Path cache_dir loads back as a str). thinking is written so that from_yaml restores an equal value with the same get_thinking_config():

  • A level string or an int budget is written unchanged.
  • A ThinkingConfig is written as a mapping of its fields ({level: high, budget_tokens: 9000}) and loads back as a ThinkingConfig.
  • A ThinkingLevel member is written as its string value (high) and loads back as that plain string, not the enum. Restoring the enum would need a custom YAML tag, which yaml.safe_load and hand-written files do not use, and it would add nothing: ThinkingLevel is a str enum, so the string compares equal to the member and normalizes to the same ThinkingConfig.
PARAMETER DESCRIPTION
path

Path to write YAML configuration file.

TYPE: str | Path

Source code in src/autorubric/llm.py
def to_yaml(self, path: str | Path) -> None:
    """Save LLMConfig to a YAML file.

    The file holds only plain YAML (``yaml.safe_load`` reads it): enums are written as
    their values and paths as strings (so a ``Path`` ``cache_dir`` loads back as a
    ``str``). ``thinking`` is written so that ``from_yaml`` restores an equal value
    with the same ``get_thinking_config()``:

    - A level string or an int budget is written unchanged.
    - A ``ThinkingConfig`` is written as a mapping of its fields
      (``{level: high, budget_tokens: 9000}``) and loads back as a ``ThinkingConfig``.
    - A ``ThinkingLevel`` member is written as its string value (``high``) and loads
      back as that plain string, not the enum. Restoring the enum would need a custom
      YAML tag, which ``yaml.safe_load`` and hand-written files do not use, and it
      would add nothing: ``ThinkingLevel`` is a ``str`` enum, so the string compares
      equal to the member and normalizes to the same ``ThinkingConfig``.

    Args:
        path: Path to write YAML configuration file.
    """
    path = Path(path)
    data = asdict(self)

    # Remove None values and empty dicts for cleaner YAML
    data = {k: v for k, v in data.items() if v is not None and v != {}}

    with open(path, "w", encoding="utf-8") as f:
        yaml.dump(data, f, Dumper=_LLMConfigDumper, default_flow_style=False, sort_keys=False)

LLMClient

Async client for LLM generation with caching and rate limiting.

LLMClient

LLMClient(config: LLMConfig, *, cache_namespace: str | None = None)

Unified LLM client with retries, caching, and structured output support.

Uses diskcache for efficient, thread-safe response caching.

Initialize LLM client.

PARAMETER DESCRIPTION
config

LLMConfig instance. The model field is required.

TYPE: LLMConfig

cache_namespace

Keeps this client's response-cache entries apart from those of other clients that send identical requests, such as several judges polling one model, whose answers are independent samples. It becomes part of the cache key; None (the default) leaves the key exactly as without it.

TYPE: str | None DEFAULT: None

RAISES DESCRIPTION
ValueError

If config.model is not specified.

Source code in src/autorubric/llm.py
def __init__(self, config: LLMConfig, *, cache_namespace: str | None = None):
    """Initialize LLM client.

    Args:
        config: LLMConfig instance. The model field is required.
        cache_namespace: Keeps this client's response-cache entries apart from those of
            other clients that send identical requests, such as several judges polling
            one model, whose answers are independent samples. It becomes part of the
            cache key; ``None`` (the default) leaves the key exactly as without it.

    Raises:
        ValueError: If config.model is not specified.
    """
    if not config.model:
        raise ValueError("LLMConfig.model is required and cannot be empty")

    self.config = config
    self._cache_namespace = cache_namespace
    self._cache: diskcache.Cache | None = None

    if self.config.cache_enabled:
        self._init_cache()

generate async

generate(system_prompt: str, user_prompt: Content, response_format: type[T] | None = None, use_cache: bool | None = None, return_thinking: bool = False, return_result: bool = False, shared_prefix: Content | None = None, pre_1_7_user_prompt: str | None = None, pre_1_7_system_prompt: str | None = None, **kwargs: Any) -> str | T | GenerateResult

Generate LLM response with optional structured output.

PARAMETER DESCRIPTION
system_prompt

System message for the LLM.

TYPE: str

user_prompt

User message for the LLM: text, or a document holding images, sent as content parts (see _user_message). Its images are expected to hold their bytes already (Image._loaded()); one that does not is read from its file here.

TYPE: Content

response_format

Optional Pydantic model class for structured output. When provided, LiteLLM uses the model's JSON schema to constrain the LLM output and returns a validated Pydantic instance.

TYPE: type[T] | None DEFAULT: None

use_cache

Whether to use caching for this request. - None (default): Use config.cache_enabled setting - True: Force cache usage (initializes cache if needed) - False: Skip cache for this request

TYPE: bool | None DEFAULT: None

return_thinking

If True and thinking is enabled, return a GenerateResult with both content and thinking. If False (default), only return content. Note: When response_format is provided, thinking is injected into the 'reasoning' field if it exists, regardless of this setting.

TYPE: bool DEFAULT: False

return_result

If True, always return a GenerateResult with full details including usage statistics and completion cost. This is useful when you need to track token usage. When True, takes precedence over the default return behavior.

TYPE: bool DEFAULT: False

shared_prefix

The start of user_prompt shared with other calls, sent as a message of its own where that helps a provider's prompt cache. It does not change the response-cache key, which hashes user_prompt. For a document it is content too, its images in the message of its own with the text around them (_content_removeprefix checks that it starts user_prompt), the cache marker on its last text part (_marked).

TYPE: Content | None DEFAULT: None

pre_1_7_user_prompt

The text this user_prompt had before 1.7.0. On a response-cache miss, the key of that text is read too, so replies cached by earlier releases are reused; nothing is written under it. Kept for backward compatibility and planned for removal in 2.0.0.

TYPE: str | None DEFAULT: None

pre_1_7_system_prompt

The text system_prompt had before 1.7.0 for that lookup, when it differs.

TYPE: str | None DEFAULT: None

**kwargs

Override any LLMConfig parameters for this call. In particular, temperature accepts a float to send for this call, or None to omit it so the provider's default applies (regardless of config.temperature).

TYPE: Any DEFAULT: {}

RETURNS DESCRIPTION
str | T | GenerateResult

If return_result=True or return_thinking=True: GenerateResult with content, thinking, usage, cost, and parsed (if response_format was provided).

str | T | GenerateResult

If response_format is None: String response from the LLM.

str | T | GenerateResult

If response_format is provided: Validated Pydantic model instance. If thinking is enabled and the response_format has a 'reasoning' field, it will be populated with the model's thinking trace (a trace that is not a string is dropped).

RAISES DESCRIPTION
APIError

If all retries fail

ValueError

If the response has no choices, or the reply has no content (e.g. a refusal) or content that is not a string (e.g. a list of content parts), with or without response_format; or if response_format is provided and the reply is not JSON (json.JSONDecodeError). Such a reply is not cached.

ValidationError

If response doesn't match schema, including a JSON reply that is not an object (with or without a thinking trace).

Source code in src/autorubric/llm.py
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620
1621
1622
1623
1624
1625
1626
1627
1628
1629
1630
1631
1632
1633
1634
1635
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676
1677
1678
1679
1680
1681
1682
1683
1684
1685
1686
1687
1688
1689
1690
1691
1692
1693
async def generate(
    self,
    system_prompt: str,
    user_prompt: Content,
    response_format: type[T] | None = None,
    use_cache: bool | None = None,
    return_thinking: bool = False,
    return_result: bool = False,
    shared_prefix: Content | None = None,
    pre_1_7_user_prompt: str | None = None,
    pre_1_7_system_prompt: str | None = None,
    **kwargs: Any,
) -> str | T | GenerateResult:
    """Generate LLM response with optional structured output.

    Args:
        system_prompt: System message for the LLM.
        user_prompt: User message for the LLM: text, or a document holding images,
            sent as content parts (see ``_user_message``). Its images are expected
            to hold their bytes already (``Image._loaded()``); one that does not is
            read from its file here.
        response_format: Optional Pydantic model class for structured output.
            When provided, LiteLLM uses the model's JSON schema to constrain
            the LLM output and returns a validated Pydantic instance.
        use_cache: Whether to use caching for this request.
            - None (default): Use config.cache_enabled setting
            - True: Force cache usage (initializes cache if needed)
            - False: Skip cache for this request
        return_thinking: If True and thinking is enabled, return a GenerateResult
            with both content and thinking. If False (default), only return content.
            Note: When response_format is provided, thinking is injected into the
            'reasoning' field if it exists, regardless of this setting.
        return_result: If True, always return a GenerateResult with full details
            including usage statistics and completion cost. This is useful when
            you need to track token usage. When True, takes precedence over the
            default return behavior.
        shared_prefix: The start of ``user_prompt`` shared with other calls, sent as a
            message of its own where that helps a provider's prompt cache. It does not
            change the response-cache key, which hashes ``user_prompt``. For a document
            it is content too, its images in the message of its own with the text
            around them (``_content_removeprefix`` checks that it starts
            ``user_prompt``), the cache marker on its last text part (``_marked``).
        pre_1_7_user_prompt: The text this ``user_prompt`` had before 1.7.0. On a
            response-cache miss, the key of that text is read too, so replies cached by
            earlier releases are reused; nothing is written under it. Kept for backward
            compatibility and planned for removal in 2.0.0.
        pre_1_7_system_prompt: The text ``system_prompt`` had before 1.7.0 for that lookup,
            when it differs.
        **kwargs: Override any LLMConfig parameters for this call. In particular,
            ``temperature`` accepts a float to send for this call, or None to omit it
            so the provider's default applies (regardless of ``config.temperature``).

    Returns:
        If return_result=True or return_thinking=True: GenerateResult with content,
            thinking, usage, cost, and parsed (if response_format was provided).
        If response_format is None: String response from the LLM.
        If response_format is provided: Validated Pydantic model instance.
            If thinking is enabled and the response_format has a 'reasoning'
            field, it will be populated with the model's thinking trace (a trace
            that is not a string is dropped).

    Raises:
        litellm.APIError: If all retries fail
        ValueError: If the response has no choices, or the reply has no content (e.g.
            a refusal) or content that is not a string (e.g. a list of content parts),
            with or without response_format; or if response_format is provided and the
            reply is not JSON (``json.JSONDecodeError``). Such a reply is not cached.
        pydantic.ValidationError: If response doesn't match schema, including a JSON
            reply that is not an object (with or without a thinking trace).
    """
    user = _normalize_content(user_prompt)

    # Determine caching behavior for this request
    should_cache = use_cache if use_cache is not None else self.config.cache_enabled

    # Effective temperature: a per-call override wins; None means "omit from the request".
    temperature: float | None = kwargs.get("temperature", self.config.temperature)

    # Check cache first
    cache_key: str | None = None
    if should_cache:
        cache = self._ensure_cache()
        cache_key = self._cache_key(
            self.config.model,
            system_prompt,
            user,
            response_format,
            kwargs,
        )
        cached = cache.get(cache_key)
        if cached is None and pre_1_7_user_prompt is not None:
            # Backward compatibility: read a reply cached before 1.7.0, whose key hashed
            # the old layout. Nothing is written under that key, and 2.0.0 is planned to
            # drop this lookup.
            cached = cache.get(
                self._cache_key(
                    self.config.model,
                    pre_1_7_system_prompt or system_prompt,
                    pre_1_7_user_prompt,
                    response_format,
                    kwargs,
                )
            )
        if cached is not None:
            logger.debug(f"Cache hit for {cache_key[:8]}...")
            return cached  # type: ignore[return-value]

    # Build request parameters
    model = kwargs.get("model", self.config.model)

    gemini = _GEMINI_CONTEXT_CACHE_MODELS.match(model) is not None
    # What follows the shared prefix, or None when there is none or it does not start
    # the prompt.
    after_prefix = (
        _content_removeprefix(user, shared_prefix) if shared_prefix is not None else None
    )
    split = (
        after_prefix is not None
        and shared_prefix is not None
        and self.config.prompt_caching
        # A server of the caller's own is not the provider the model map describes.
        and self.config.api_base is None
        and _caches_prompts(model)
        # A context cache is created per prefix, so one that is too short is not asked for.
        and (not gemini or _prefix_reaches_cache_minimum(model, system_prompt, shared_prefix))
    )
    # Anthropic and Gemini cache at cache_control markers on message content. The other
    # providers cache automatically, at message endings, which is why the shared prefix is
    # a message of its own: a prefix ending inside a message is not reused.
    explicit = split and _EXPLICIT_BREAKPOINT_MODELS.match(model) is not None
    mark = (
        _breakpoint_marked
        if explicit
        else _context_cache_marked
        if split and gemini
        else _cache_marked
        if self.config.prompt_caching and _is_anthropic_model(model)
        else None
    )
    system_message: dict[str, Any] = {
        "role": "system",
        "content": mark(system_prompt) if mark else system_prompt,
    }
    if split:
        assert shared_prefix is not None and after_prefix is not None
        tail = _content_removeprefix(after_prefix, "\n\n")
        if tail is None:
            tail = after_prefix
        messages = [
            system_message,
            {
                "role": "user",
                "content": mark(shared_prefix)
                if mark
                else _user_message(shared_prefix)["content"],
            },
            _user_message(tail),
        ]
    else:
        messages = [system_message, _user_message(user)]

    params: dict[str, Any] = {
        "model": model,
        "messages": messages,
        # Only send temperature when set, so the provider default applies otherwise
        # (0.0 is a real value and is sent).
        **({"temperature": temperature} if temperature is not None else {}),
        "timeout": kwargs.get("timeout", self.config.timeout),
        **self.config.extra_params,
    }

    if explicit:
        params["extra_body"] = {
            **params.get("extra_body", {}),
            "prompt_cache_options": {"mode": "explicit"},
        }

    # Sent whenever set: LLMConfig refuses values no provider can mean, and a top_p of
    # 0 is a real setting.
    if self.config.max_tokens is not None:
        params["max_tokens"] = kwargs.get("max_tokens", self.config.max_tokens)
    if self.config.top_p is not None:
        params["top_p"] = kwargs.get("top_p", self.config.top_p)
    if self.config.api_key:
        params["api_key"] = self.config.api_key
    if self.config.api_base:
        params["api_base"] = self.config.api_base

    # Thinking/Reasoning configuration (unified across providers)
    thinking_config = self.config.get_thinking_config()
    if thinking_config is not None:
        # Determine whether to use reasoning_effort or explicit thinking dict
        # Use explicit budget_tokens when specified for fine-grained control
        # Otherwise use reasoning_effort for better cross-provider compatibility
        if thinking_config.budget_tokens is not None:
            # Explicit token budget - use thinking dict (Anthropic/Gemini style)
            params["thinking"] = {
                "type": "enabled",
                "budget_tokens": thinking_config.budget_tokens,
            }
        else:
            # Level-based - use reasoning_effort for cross-provider support
            # LiteLLM translates this to provider-specific parameters
            params["reasoning_effort"] = thinking_config.get_reasoning_effort()

    # Extra headers configuration
    extra_headers = dict(self.config.extra_headers)
    if extra_headers:
        params["extra_headers"] = extra_headers

    if self.config.seed is not None:
        params["seed"] = self.config.seed

    # Enable structured output if Pydantic model provided. The schema sent to the
    # provider drops the post-hoc ``reasoning`` slot so strict-mode backends don't force
    # non-OpenAI models (e.g. Llama/Qwen on Groq) to emit it; see
    # _provider_response_format. Parsing below still uses the full Pydantic model.
    if response_format is not None:
        params["response_format"] = _provider_response_format(response_format)

    # Print debug prompt if debug mode is enabled
    import autorubric

    if autorubric.debug:
        _print_debug_prompt(system_prompt, user, model)

    # Make request with retries
    context_cached = split and gemini
    retry_decorator = self._get_retry_decorator()
    thinking_content: str | None = None
    refusal: Any = None
    raw_response: Any = None

    @retry_decorator
    async def _call() -> str:
        nonlocal thinking_content, refusal, raw_response, context_cached
        try:
            response = await litellm.acompletion(**params)
        except openai.APIError:
            if not context_cached:
                raise
            # LiteLLM creates the context cache inside the call, so a refused creation
            # (a prefix under the minimum, no cache quota) fails it. The cache is only a
            # discount: send the prompt without one from here on, its images (if it holds
            # any) as content parts in the one user message.
            logger.warning("Gemini context cache failed; sending %s without one", model)
            context_cached = False
            params["messages"] = [
                {"role": "system", "content": system_prompt},
                _user_message(user),
            ]
            response = await litellm.acompletion(**params)
        raw_response = response

        # A response with no choices (empty, ``None`` or missing) carries no reply at all;
        # like a reply with nothing to parse, it raises ``ValueError`` (``parse``), not the
        # ``IndexError`` of ``choices[0]``. ``ValueError`` is not retried.
        choices = getattr(response, "choices", None)
        if not choices:
            raise ValueError("LLM reply has no choices")
        message = choices[0].message

        # Extract thinking/reasoning content (standardized across providers)
        # LiteLLM provides unified `reasoning_content` field
        thinking_content = _extract_thinking_content(message)
        # OpenAI-style structured-output refusals come with no content.
        refusal = getattr(message, "refusal", None)

        return message.content  # type: ignore[return-value]

    # Apply rate limiting if configured
    semaphore = await RateLimitPool.get_instance().get_semaphore(
        model, self.config.max_parallel_requests
    )
    if semaphore is not None:
        async with semaphore:
            response_content = await _call()
    else:
        response_content = await _call()

    # Extract usage and cost from the raw response
    usage = _extract_usage_from_response(raw_response)
    cost = _calculate_completion_cost(raw_response)

    # A reply with no content (e.g. a refusal), or whose content is not a string, is no
    # answer, structured or not: there is no JSON text to parse, and no text for a
    # plain-text call to return (it returns a ``str``, which callers treat as text). Like
    # a reply that is not JSON, it raises ``ValueError`` (``parse``), before anything is
    # cached. LiteLLM types the content as a string; anything else (e.g. a list of
    # content parts a provider transform left in place) would make ``json.loads`` raise
    # ``TypeError`` (``unknown``), or a caller's string operations fail.
    if response_content is None:
        detail = f" (refusal: {refusal})" if isinstance(refusal, str) and refusal else ""
        raise ValueError(f"LLM reply has no content{detail}")
    if not isinstance(response_content, str):
        raise ValueError(
            f"LLM reply content is not a string (got {type(response_content).__name__})"
        )

    # Parse structured output if requested
    parsed_response: T | None = None
    if response_format is not None:
        # LiteLLM returns JSON string when response_format is set
        # Parse it into the Pydantic model.
        data = json.loads(response_content)

        # Inject thinking content into the reasoning field if available. Only an object
        # can take it: any other JSON value is left as it is, to fail validation (#37).
        if (
            thinking_content
            and "reasoning" in response_format.model_fields
            and isinstance(data, dict)
        ):
            data["reasoning"] = thinking_content

        parsed_response = response_format.model_validate(data)

    # Determine what to return
    result: str | T | GenerateResult
    if return_result or return_thinking:
        # Return full GenerateResult with all details
        result = GenerateResult(
            content=response_content,
            thinking=thinking_content,
            raw_response=raw_response,
            usage=usage,
            cost=cost,
            parsed=parsed_response,
        )
    elif response_format is not None:
        # Return just the parsed Pydantic model
        result = parsed_response  # type: ignore[assignment]
    else:
        # Return just the string content
        result = response_content

    # Cache the response (cache the parsed object for structured outputs)
    if should_cache and cache_key:
        cache = self._ensure_cache()
        cache.set(
            cache_key,
            result,
            expire=self.config.cache_ttl,
        )
        logger.debug(f"Cached response for {cache_key[:8]}...")

    return result

clear_cache

clear_cache() -> int

Clear all cached responses.

RETURNS DESCRIPTION
int

Number of entries cleared.

Source code in src/autorubric/llm.py
def clear_cache(self) -> int:
    """Clear all cached responses.

    Returns:
        Number of entries cleared.
    """
    if self._cache is None:
        return 0
    count = len(self._cache)
    self._cache.clear()
    return count

cache_stats

cache_stats() -> dict[str, Any]

Get cache statistics.

RETURNS DESCRIPTION
dict[str, Any]

Dict with 'size', 'count', and 'directory' keys.

Source code in src/autorubric/llm.py
def cache_stats(self) -> dict[str, Any]:
    """Get cache statistics.

    Returns:
        Dict with 'size', 'count', and 'directory' keys.
    """
    if self._cache is None:
        return {"size": 0, "count": 0, "directory": None}
    return {
        "size": self._cache.volume(),
        "count": len(self._cache),
        "directory": str(self._cache.directory),
    }

close

close() -> None

Close the on-disk cache, releasing its file handles.

diskcache keeps the underlying SQLite database open for the lifetime of the cache object. Call this when done with the client so the cache directory can be removed (notably on Windows, which refuses to delete files that are still open). Safe to call when no cache was initialized.

Source code in src/autorubric/llm.py
def close(self) -> None:
    """Close the on-disk cache, releasing its file handles.

    diskcache keeps the underlying SQLite database open for the lifetime of
    the cache object. Call this when done with the client so the cache
    directory can be removed (notably on Windows, which refuses to delete
    files that are still open). Safe to call when no cache was initialized.
    """
    if self._cache is not None:
        self._cache.close()
        self._cache = None

generate

Convenience function for one-off LLM generation.

generate async

generate(system_prompt: str, user_prompt: Content, model: str, response_format: type[T] | None = None, **kwargs: Any) -> str | T

Simple one-shot generation function.

For repeated calls, prefer creating an LLMClient instance.

PARAMETER DESCRIPTION
system_prompt

System message for the LLM.

TYPE: str

user_prompt

User message for the LLM: text, or a document holding images.

TYPE: Content

model

Model identifier (REQUIRED).

TYPE: str

response_format

Optional Pydantic model for structured output.

TYPE: type[T] | None DEFAULT: None

**kwargs

Additional LLMConfig parameters.

TYPE: Any DEFAULT: {}

Example

Simple string response

response = await generate( "You are a helpful assistant.", "What is 2+2?", model="openai/gpt-5.2-mini" )

Structured output

from pydantic import BaseModel

class MathAnswer(BaseModel): result: int explanation: str

answer = await generate( "You are a math tutor.", "What is 2+2?", model="openai/gpt-5.2-mini", response_format=MathAnswer ) print(answer.result) # 4

Source code in src/autorubric/llm.py
async def generate(
    system_prompt: str,
    user_prompt: Content,
    model: str,
    response_format: type[T] | None = None,
    **kwargs: Any,
) -> str | T:
    """Simple one-shot generation function.

    For repeated calls, prefer creating an LLMClient instance.

    Args:
        system_prompt: System message for the LLM.
        user_prompt: User message for the LLM: text, or a document holding images.
        model: Model identifier (REQUIRED).
        response_format: Optional Pydantic model for structured output.
        **kwargs: Additional LLMConfig parameters.

    Example:
        # Simple string response
        response = await generate(
            "You are a helpful assistant.",
            "What is 2+2?",
            model="openai/gpt-5.2-mini"
        )

        # Structured output
        from pydantic import BaseModel

        class MathAnswer(BaseModel):
            result: int
            explanation: str

        answer = await generate(
            "You are a math tutor.",
            "What is 2+2?",
            model="openai/gpt-5.2-mini",
            response_format=MathAnswer
        )
        print(answer.result)  # 4
    """
    config = LLMConfig(model=model, **kwargs)
    client = LLMClient(config)
    return await client.generate(system_prompt, user_prompt, response_format=response_format)

GenerateResult

Result from LLM generation.

GenerateResult dataclass

GenerateResult(content: str, thinking: str | None = None, raw_response: Any = None, usage: TokenUsage | None = None, cost: float | None = None, parsed: Any = None)

Result from LLM generation including content, thinking, and usage statistics.

ATTRIBUTE DESCRIPTION
content

The main response content from the LLM (raw string).

TYPE: str

thinking

The thinking/reasoning trace if thinking was enabled. None if thinking was not enabled or the provider doesn't support it.

TYPE: str | None

raw_response

The raw LiteLLM response object for advanced use cases.

TYPE: Any

usage

Token usage statistics from this LLM call.

TYPE: TokenUsage | None

cost

Completion cost in USD for this LLM call, calculated using LiteLLM's completion_cost() function. None if cost calculation fails.

TYPE: float | None

parsed

The parsed Pydantic model instance when response_format was provided. None if no response_format was used or parsing failed.

TYPE: Any


ThinkingConfig

Configuration for extended thinking/reasoning.

ThinkingConfig dataclass

ThinkingConfig(level: ThinkingLevel | ThinkingLevelLiteral = MEDIUM, budget_tokens: int | None = None)

Detailed configuration for LLM thinking/reasoning.

Provides a uniform interface across providers: - Anthropic: Extended thinking (claude-sonnet-4-5, claude-opus-4-5+) - OpenAI: Reasoning (o-series, GPT-5 models via openai/responses/ prefix) - Gemini: Thinking mode (2.5+, 3.0+ models) - DeepSeek: Reasoning content

ATTRIBUTE DESCRIPTION
level

High-level thinking effort. Used when budget_tokens is not set. Defaults to MEDIUM for a good balance of quality and latency.

TYPE: ThinkingLevel | ThinkingLevelLiteral

budget_tokens

Explicit token budget for thinking (provider-specific): None, or an integer >= -1, sent as thinking={"type": "enabled", "budget_tokens": ...}. When set, overrides level. Recommended: 10000-50000 for complex tasks. Providers that don't support explicit budgets will map this to the nearest level. Gemini 2.5 reads 0 as thinking off and -1 as a budget the model sets itself (dynamic thinking); a provider with a minimum budget (Anthropic's is 1024) rejects a smaller one.

TYPE: int | None

Both fields are checked when the config is built: level must be a ThinkingLevel or its string value, stored as the member, or None alongside a budget_tokens (a budget is sent without reading level), and budget_tokens None or an integer

= -1 (numpy integers too, stored as int; never a bool). Anything else raises ValueError.

Examples:

Simple: use a thinking level

ThinkingConfig(level=ThinkingLevel.HIGH)

Fine-grained: specify exact token budget

ThinkingConfig(budget_tokens=32000) # For complex reasoning tasks

Disable thinking

ThinkingConfig(level=ThinkingLevel.NONE)

get_effective_budget

get_effective_budget() -> int

Get the effective token budget based on level or explicit budget.

Source code in src/autorubric/llm.py
def get_effective_budget(self) -> int:
    """Get the effective token budget based on level or explicit budget."""
    if self.budget_tokens is not None:
        return self.budget_tokens
    return THINKING_LEVEL_BUDGETS.get(self.level.value, 2048)

get_reasoning_effort

get_reasoning_effort() -> str

Get the reasoning_effort string for LiteLLM.

Source code in src/autorubric/llm.py
def get_reasoning_effort(self) -> str:
    """Get the reasoning_effort string for LiteLLM."""
    return self.level.value

ThinkingLevel

Enum for thinking level presets.

ThinkingLevel

Bases: str, Enum

Standardized thinking/reasoning effort levels across LLM providers.

LiteLLM translates these to provider-specific parameters: - Anthropic: thinking={type, budget_tokens} with low→1024, medium→2048, high→4096 - OpenAI: reasoning_effort parameter for o-series and GPT-5 models - Gemini: thinking configuration with similar token budgets - DeepSeek: Standardized via LiteLLM's reasoning_effort

Higher levels mean more "thinking" tokens/steps, trading latency for quality.


ThinkingLevelLiteral

Type alias for thinking level strings.

ThinkingLevelLiteral module-attribute

ThinkingLevelLiteral = Literal['none', 'low', 'medium', 'high']

ThinkingParam

Type alias for thinking parameter (level or budget).

ThinkingParam module-attribute

ThinkingParam = ThinkingConfig | ThinkingLevel | ThinkingLevelLiteral | int | None

Type for the thinking parameter in LLMConfig.

Accepts: - ThinkingConfig: Full configuration object - ThinkingLevel: Enum value (e.g., ThinkingLevel.HIGH) - str: Level as string ("low", "medium", "high", "none") - int: Direct token budget (e.g., 32000), an integer >= -1 (never a bool) - None: Disable thinking


classify_grading_error

Classify an exception raised while grading a single criterion into an ErrorCategory. The grading pipeline uses the result to route a failed judge call: infrastructure and parse errors are treated as abstentions (mapped to CANNOT_ASSESS / na=True and excluded from scoring under the default SKIP strategy), while unknown errors fall back to a conservative worst-case verdict. These verdicts only stand in for failed calls: an item whose every criterion's judgment failed has no score, and its report's error begins Every criterion's judgment failed:. Configuration errors (LiteLLM's UnsupportedParamsError, 401, 402, 403 and 404 errors from the provider or from TypeSafe (a 402 Payment Required says the account has no credits left), and the OpenAI SDK's refusal to build its client, e.g. with no OpenAI or Azure API key, which LiteLLM reports as an InternalServerError, or an APIError for Azure) still classify as infrastructure, but the grader never routes them: they fail every call the judge makes, so the grade raises a ValueError naming the judge, chained from the original exception. Item-specific 400s such as ContextWindowExceededError still abstain.

classify_grading_error

classify_grading_error(exc: BaseException) -> ErrorCategory

Classify an exception raised while grading a criterion.

Reuses the underlying OpenAI exception taxonomy that LiteLLM builds on: every transient API failure (Timeout, APIConnectionError, RateLimitError, ServiceUnavailableError, InternalServerError, status errors, etc.) subclasses openai.APIError (including litellm.APIError itself), while parse/validation failures (json.JSONDecodeError -> ValueError, pydantic.ValidationError) do not. LLMClient.generate raises one of these for every structured reply it cannot use: one that is not JSON, not a JSON object (with or without a thinking trace), or that does not match the schema is a ValidationError or JSONDecodeError. A reply with no content (e.g. a refusal) or whose content is not a string (e.g. a list of content parts) is a ValueError, as is a response with no choices, structured or plain-text alike.

Decision-model failures raised by the optional TypeSafe SDK (typesafe-sdk) are routed by the SDK's own exception types, ahead of the ValueError rule:

  • TypeSafeAPIResponseValidationError (the endpoint's answer did not validate), TypeSafeBadRequestError (400) and TypeSafeUnprocessableEntityError (422, e.g. a request over the model's input limit) are "parse": the endpoint answered, but no usable answer came back for the request that was built.
  • Every other TypeSafeError is "infrastructure": authentication, permission, not found, rate limit, server errors, and connection failures and timeouts (which subclass TypeSafeError directly, not TypeSafeAPIError).

Configuration errors are classified like any other API error ("infrastructure"), but CriterionGrader never routes one: an exception that says the judge is misconfigured (LiteLLM's UnsupportedParamsError, a 401, 402 (no credits left), 403 or 404 from the provider (but OpenRouter's 403, a moderation flag on the input) or from TypeSafe, or the OpenAI SDK's refusal to build its client, e.g. with no OpenAI or Azure API key, which LiteLLM reports as an InternalServerError or, for Azure, an APIError) fails every call the judge makes, so the grade raises a ValueError naming the judge, chained from it, instead of abstaining (#40). Item-specific 400s (ContextWindowExceededError, ContentPolicyViolationError) and a plain BadRequestError still abstain.

PARAMETER DESCRIPTION
exc

The exception raised during a judge call.

TYPE: BaseException

RETURNS DESCRIPTION
ErrorCategory

"infrastructure" for API/network errors, "parse" for JSON/validation

ErrorCategory

errors, and "unknown" for anything else.

Source code in src/autorubric/llm.py
def classify_grading_error(exc: BaseException) -> ErrorCategory:
    """Classify an exception raised while grading a criterion.

    Reuses the underlying OpenAI exception taxonomy that LiteLLM builds on: every
    transient API failure (``Timeout``, ``APIConnectionError``, ``RateLimitError``,
    ``ServiceUnavailableError``, ``InternalServerError``, status errors, etc.) subclasses
    ``openai.APIError`` (including ``litellm.APIError`` itself), while parse/validation
    failures (``json.JSONDecodeError`` -> ``ValueError``, ``pydantic.ValidationError``)
    do not. ``LLMClient.generate`` raises one of these for every structured reply it
    cannot use: one that is not JSON, not a JSON object (with or without a thinking
    trace), or that does not match the schema is a ``ValidationError`` or
    ``JSONDecodeError``. A reply with no content (e.g. a refusal) or whose content is not
    a string (e.g. a list of content parts) is a ``ValueError``, as is a response with no
    choices, structured or plain-text alike.

    Decision-model failures raised by the optional TypeSafe SDK (``typesafe-sdk``) are
    routed by the SDK's own exception types, ahead of the ``ValueError`` rule:

    - ``TypeSafeAPIResponseValidationError`` (the endpoint's answer did not validate),
      ``TypeSafeBadRequestError`` (400) and ``TypeSafeUnprocessableEntityError`` (422, e.g.
      a request over the model's input limit) are ``"parse"``: the endpoint answered, but
      no usable answer came back for the request that was built.
    - Every other ``TypeSafeError`` is ``"infrastructure"``: authentication, permission,
      not found, rate limit, server errors, and connection failures and timeouts (which
      subclass ``TypeSafeError`` directly, not ``TypeSafeAPIError``).

    Configuration errors are classified like any other API error (``"infrastructure"``),
    but ``CriterionGrader`` never routes one: an exception that says the judge is
    misconfigured (LiteLLM's ``UnsupportedParamsError``, a 401, 402 (no credits left), 403
    or 404 from the provider (but OpenRouter's 403, a moderation flag on the input) or from
    TypeSafe, or the OpenAI SDK's refusal to build its client, e.g. with no OpenAI or Azure
    API key, which LiteLLM reports as an ``InternalServerError`` or, for Azure, an ``APIError``)
    fails every call the judge makes, so the grade raises a ``ValueError`` naming the
    judge, chained from it, instead of abstaining (#40). Item-specific 400s
    (``ContextWindowExceededError``, ``ContentPolicyViolationError``) and a plain
    ``BadRequestError`` still abstain.

    Args:
        exc: The exception raised during a judge call.

    Returns:
        ``"infrastructure"`` for API/network errors, ``"parse"`` for JSON/validation
        errors, and ``"unknown"`` for anything else.
    """
    if isinstance(exc, openai.APIError):
        return "infrastructure"
    typesafe_category = _classify_typesafe_error(exc)
    if typesafe_category is not None:
        return typesafe_category
    if isinstance(exc, (ValidationError, ValueError)):
        return "parse"
    return "unknown"

ErrorCategory

Type alias for the category of a grading failure: Literal["infrastructure", "parse", "unknown"].

  • "infrastructure": API/network failure (timeout, connection, rate limit, server error). Not the submission's fault. (A configuration error, e.g. an invalid API key, is not routed at all: CriterionGrader raises it.)
  • "parse": the judge responded but its output could not be parsed/validated into the expected schema: a reply that is not JSON, is not a JSON object, has no content (e.g. a refusal) or content that is not a string (e.g. a list of content parts), with or without a thinking trace, and a response with no choices at all. Also not the submission's fault. LLMClient.generate raises the same ValueError for a plain-text call (no response_format) whose reply has no content or content that is not a string, rather than returning it, and does not cache it.
  • "unknown": an unexpected error that does not fit the above categories.

ErrorCategory module-attribute

ErrorCategory = Literal['infrastructure', 'parse', 'unknown']

Category of a failure encountered while grading a single criterion.

  • infrastructure: API/network failure (timeout, connection, rate limit, server error). Not the submission's fault; the judge never produced a usable response. (A configuration error, e.g. an invalid API key, is not routed at all: CriterionGrader raises it.)
  • parse: the judge responded but its output could not be parsed/validated into the expected schema. Also not the submission's fault.
  • unknown: an unexpected error that does not fit the above categories.