LLM Infrastructure¶
LLM client configuration, caching, and generation utilities.
Overview¶
AutoRubric uses LiteLLM for multi-provider LLM support. The LLMConfig class provides centralized configuration, while LLMClient handles request execution with caching, rate limiting, and retry logic.
For a guided introduction see LLM Judges.
Quick Example¶
from autorubric import LLMConfig, LLMClient, generate
# Configuration
config = LLMConfig(
model="openai/gpt-5.6-luna",
max_tokens=1024,
cache_enabled=True,
max_parallel_requests=10,
)
# Direct generation (standalone function)
result = await generate(
system_prompt="You are a helpful assistant.",
user_prompt="Explain quantum computing.",
model="openai/gpt-5.6-luna",
)
print(result)
# Or use the client
client = LLMClient(config)
result = await client.generate(
system_prompt="You are a helpful assistant.",
user_prompt="Explain quantum computing.",
)
Provider Configuration¶
| Provider | Model Format | Environment Variable |
|---|---|---|
| OpenAI | openai/gpt-4.1, openai/gpt-5.6-luna |
OPENAI_API_KEY |
| Anthropic | anthropic/claude-sonnet-4-5-20250929 |
ANTHROPIC_API_KEY |
gemini/gemini-2.5-flash |
GEMINI_API_KEY |
|
| Azure | azure/openai/gpt-4.1 |
AZURE_API_KEY, AZURE_API_BASE |
| Groq | groq/llama-3.1-70b-versatile |
GROQ_API_KEY |
| Ollama | ollama/qwen3:14b |
(local, no key needed) |
Temperature¶
LLMConfig.temperature defaults to None: temperature is left out of the request, so the provider's own default applies. Many current models need or recommend this: reasoning models such as the GPT-5.x family accept only their default temperature, Gemini 3 recommends keeping its default of 1.0, and Anthropic requires temperature 1 when extended thinking is enabled.
An explicit value, including 0.0, is sent unchanged. A temperature= keyword argument to LLMClient.generate() overrides the config for that call (temperature=None omits it for that call). The response cache keys on the temperature actually used for the request (including a per-call override), and an unset temperature has its own key.
LLMConfig(model="openai/gpt-5.2") # temperature omitted: provider default
LLMConfig(model="gemini/gemini-3.8-flash", temperature=0.0) # temperature=0.0 sent
Some models refuse it: OpenAI's Luna models accept neither a custom temperature nor top_p, so leave both unset for them.
A low temperature reduces run-to-run variance, but most providers do not guarantee identical outputs even at 0.0.
Default changed in v1.6.0
Up to and including v1.5.3, temperature defaulted to 0.0 and was always sent. A config that never sets it now samples at the provider's default, so its grades may differ from earlier runs, and responses cached by those runs are not reused (an unset temperature has its own cache key). To keep the old behaviour, set temperature=0.0 explicitly (temperature: 0.0 in YAML). Configs that already set temperature explicitly send the same value and keep their cached responses.
Output Length and Nucleus Sampling¶
max_tokens and top_p also default to None, which leaves them out of the request so the provider's own limit and default apply. A value you set is sent unchanged, including top_p=0.0. LLMConfig refuses a value no provider can mean with a ValueError: a max_tokens that is not an integer of at least 1, or a top_p that is not a number between 0 and 1 (a bool is not taken for a number).
LLMConfig(model="gemini/gemini-3.8-flash", max_tokens=1024, top_p=0.0) # both sent
LLMConfig(model="gemini/gemini-3.8-flash", max_tokens=0) # ValueError
Changed behaviour
Earlier releases sent max_tokens and top_p only when they were truthy, so top_p=0.0 and max_tokens=0 were silently left out of the request. top_p=0.0 is now sent, and responses cached for it by earlier releases, which were sampled without it, are not reused (it has a new cache key). max_tokens=0, a negative max_tokens and a top_p outside [0, 1] are now refused. Configs that set other values send the same requests and keep their cached responses.
YAML Configuration¶
# llm_config.yaml
model: openai/gpt-4.1
temperature: 0.0 # optional; omit to use the provider default
max_tokens: 1024
cache_enabled: true
cache_ttl: 3600
thinking can be a level (thinking: high), a token budget (thinking: 32000), or a ThinkingConfig written as a mapping:
to_yaml writes a ThinkingLevel enum as its string value, so it loads back as the string "high". That string is equal to ThinkingLevel.HIGH and gives the same thinking settings.
Extended Thinking¶
Enable step-by-step reasoning for complex evaluations:
# Level-based (cross-provider)
config = LLMConfig(
model="anthropic/claude-sonnet-4-5-20250929",
thinking="high", # "low", "medium", "high", or "none"
)
# Token budget
config = LLMConfig(
model="anthropic/claude-opus-4-5-20251101",
thinking=32000, # Explicit token budget
)
Supported providers: Anthropic, OpenAI (o-series), Gemini (2.5+), DeepSeek.
thinking is checked when the LLMConfig is built. It must be None, a ThinkingLevel or one of the strings "none", "low", "medium", "high", an integer token budget of at least -1, or a ThinkingConfig; anything else raises ValueError there, before any request. That includes an unknown level such as "extreme", True or False (a bool is not a budget), a budget below -1, and a float such as 32000.0. A budget of 0 or -1 is accepted because Gemini 2.5 reads 0 as thinking off and -1 as dynamic thinking. A budget smaller than a provider's minimum passes this check. Anthropic's minimum is 1,024 tokens, and it rejects each request with a smaller budget with a 400 (BadRequestError). A grade does not raise that error as a configuration error, so every call of such a judge abstains (CANNOT_ASSESS, or NA for a multi-choice criterion). Use at least 1,024 for Anthropic models. ThinkingConfig checks its own level and budget_tokens the same way, except that level=None is accepted next to a budget_tokens, which is sent without reading the level. Both check their values only when they are built, so a field assigned afterwards is not checked again.
Response Caching¶
config = LLMConfig(
model="openai/gpt-5.6-luna",
cache_enabled=True,
cache_dir=".autorubric_cache",
cache_ttl=3600, # 1 hour
)
client = LLMClient(config)
client.clear_cache()
stats = client.cache_stats()
# {'size': 1024, 'count': 10, 'directory': '.autorubric_cache'}
Images in Prompts¶
LLMClient.generate takes the user prompt as Content: a string, an Image, or a list of strings and images. A text prompt is sent as the message's content string, exactly as before. A prompt holding an image is sent as a list of OpenAI-style content parts in the prompt's order, a {"type": "text", "text": ...} part for each run of text and a {"type": "image_url", "image_url": {"url": "data:<media type>;base64,..."}} part for each image. LiteLLM translates these parts for providers with another image format (Anthropic, Gemini). The image is sent as its own bytes, neither resized nor re-encoded, so the provider's own limits on format, size and image count apply.
from autorubric import Image, LLMClient, LLMConfig
client = LLMClient(LLMConfig(model="openai/gpt-5.6-luna"))
reply = await client.generate(
system_prompt="You describe figures.",
user_prompt=[Image("figures/fig1.png"), "What does this figure show?"],
)
The response cache keys a prompt holding an image by the SHA-256 of its content, its text and each image's bytes in order, so an identical prompt hits the cache and a changed image misses it. Text prompts keep their existing cache keys. With prompt_caching=True, the system prompt is marked for caching as usual, and the image parts follow it in the user message. When a grader passes shared_prefix (grading on a model that caches prompts; see prompt caching), a prompt holding images is split as text is: the shared part is a user message of its own, any cache marker sits on its last text part, and the rest follows in a second user message. Without few-shot examples the shared part is the item's (the images of the submission, input and reference submission) and the criterion follows it. With examples the shared part is the examples (their images included) and the criterion, which a criterion's calls share across items, and the item, with its images, follows. autorubric.debug = True prints a marker in place of each image, [image: <file name>] for a file and [image: <N> bytes] for bytes.
A provider that refuses an image (a format it does not read, or a model without vision input) usually answers with a bad-request error. A grader routes it like any other failed call, so the criterion abstains (CANNOT_ASSESS, or NA for a multi-choice criterion) and the error is kept on its report. See the Image Inputs recipe for the formats providers accept.
A grader does not leave the model question to the provider. CriterionGrader sends an image to an LLM judge only when the judge is known to accept images, and LLMConfig.images says how that is known:
images |
Judge accepts images when |
|---|---|
None (default) |
LiteLLM's model map lists the model as accepting images (its entry in litellm.model_cost has supports_vision: true). The map is read once per grader per model, the first time the grader is to send that model an image. The lookup reads only the model map LiteLLM loaded at import, so it sends no request, never waits on a provider, and prints and logs nothing. LiteLLM fetches that map from its repository at import unless LITELLM_LOCAL_MODEL_COST_MAP=True is set (then the copy installed with LiteLLM is used). The model is looked up under its provider-prefixed name and its bare name, and a responses/ route is looked up by its model (openai/responses/gpt-5-mini as openai/gpt-5-mini). A model the map does not hold, such as an Ollama model, is not confirmed. |
True |
Always: your declaration, for a vision model the map does not list (for example ollama/llava). |
False |
Never: your declaration, even for a model the map lists. |
Any other value raises ValueError when the config is built (1 and "yes" are not declarations). A judge that is not known to accept images raises ValueError from the grade, naming the judge and the model, before any judge of the grader sends a request. fill_ground_truth raises it too, rather than leaving each item with an image out. LLMClient.generate itself does not check, so a direct call sends what it is given. Text-only grading never reads images or the map, and the field is never sent, so text requests and cache keys are unchanged. An experiment manifest records images for a judge only when it is set.
Prompt Caching (Anthropic)¶
Reduce latency and cost on repeated calls. Enabled by default, it arranges per-criterion requests so that providers can reuse what an item's calls share:
LLMConfig¶
Central configuration class for LLM calls.
LLMConfig
dataclass
¶
LLMConfig(model: str, temperature: float | None = None, max_tokens: int | None = None, top_p: float | None = None, timeout: float = 60.0, max_retries: int = 3, retry_min_wait: float = 1.0, retry_max_wait: float = 60.0, max_parallel_requests: int | None = None, cache_enabled: bool = False, cache_dir: str | Path = '.autorubric_cache', cache_ttl: int | None = None, api_key: str | None = None, api_base: str | None = None, thinking: ThinkingParam = None, prompt_caching: bool = True, seed: int | None = None, extra_headers: dict[str, str] = dict(), extra_params: dict[str, Any] = dict(), images: bool | None = None)
Configuration for LLM calls.
| ATTRIBUTE | DESCRIPTION |
|---|---|
model |
Model identifier in LiteLLM format (e.g., "openai/gpt-5.2", "anthropic/claude-sonnet-4-5-20250929", "gemini/gemini-3-pro-preview", "ollama/qwen3:14b"). REQUIRED - no default. See LiteLLM docs for full list of supported models.
TYPE:
|
temperature |
Sampling temperature. None (default) omits it from the request so
the provider's own default applies; this is recommended for reasoning models
such as GPT-5.x and for Gemini 3, and required for Anthropic extended
thinking. An explicit value, including 0.0, is sent unchanged. Temperature
0.0 reduces but does not guarantee determinism on most providers.
Note: the default changed from 0.0 to None in v1.6.0. To keep the previous
behavior, pass
TYPE:
|
max_tokens |
Maximum tokens in the response: None (default) omits it from the request, so the provider's limit applies; otherwise an integer >= 1, sent unchanged.
TYPE:
|
top_p |
Nucleus sampling parameter: None (default) omits it from the request, so the provider's default applies; otherwise a number in [0, 1], sent unchanged (including 0).
TYPE:
|
timeout |
Request timeout in seconds.
TYPE:
|
max_retries |
Maximum retry attempts for transient failures.
TYPE:
|
retry_min_wait |
Minimum wait between retries (seconds).
TYPE:
|
retry_max_wait |
Maximum wait between retries (seconds).
TYPE:
|
max_parallel_requests |
Maximum concurrent requests to this model's provider.
When set, a global per-provider semaphore limits parallel requests. The limit
applies within one event loop; each
TYPE:
|
cache_enabled |
Default caching behavior (can be overridden per-request).
TYPE:
|
cache_dir |
Directory for response cache.
TYPE:
|
cache_ttl |
Cache time-to-live in seconds (None = no expiration).
TYPE:
|
api_key |
Optional API key override (otherwise uses environment variables).
TYPE:
|
api_base |
Optional API base URL override.
TYPE:
|
thinking |
Enable thinking/reasoning mode (unified across providers). Accepts
multiple formats:
- ThinkingLevel enum: ThinkingLevel.HIGH, ThinkingLevel.MEDIUM, etc.
- String: "low", "medium", "high", "none"
- Int: Direct token budget (e.g., 32000), an integer >= -1 (see
Any other value raises Provider support: - Anthropic: Extended thinking (claude-sonnet-4-5, claude-opus-4-5+) - OpenAI: Reasoning for o-series and GPT-5 models - Gemini: Thinking mode (2.5+, 3.0+ models) - DeepSeek: Reasoning content
TYPE:
|
prompt_caching |
Arrange requests for the provider's prompt cache (default: True).
A per-criterion call sends what an item's calls share as a user message of its
own when LiteLLM's model map says the model caches prompts, and
TYPE:
|
seed |
Random seed for reproducible outputs (OpenAI, some other providers).
TYPE:
|
extra_headers |
Additional HTTP headers for provider-specific features.
TYPE:
|
extra_params |
Additional provider-specific parameters passed to LiteLLM.
TYPE:
|
images |
Whether the model accepts images in its prompts. None (default) looks
the model up in LiteLLM's model map (the one it loaded at import; no request
is sent) the first time a grader is to send this judge an image;
TYPE:
|
Examples:
Basic usage without thinking (provider-default temperature)¶
config = LLMConfig(model="openai/gpt-5.2")
Explicit temperature¶
config = LLMConfig(model="gemini/gemini-3.8-flash", temperature=0.0) # sent as-is
Enable thinking with a level¶
config = LLMConfig(model="anthropic/claude-sonnet-4-5-20250929", thinking="high") config = LLMConfig(model="openai/responses/gpt-5-mini", thinking=ThinkingLevel.HIGH)
Enable thinking with explicit token budget¶
config = LLMConfig(model="anthropic/claude-opus-4-5-20251101", thinking=32000)
Full control with ThinkingConfig¶
config = LLMConfig( model="gemini/gemini-2.5-pro", thinking=ThinkingConfig(level=ThinkingLevel.HIGH, budget_tokens=50000) )
get_thinking_config
¶
get_thinking_config() -> ThinkingConfig | None
Get normalized thinking configuration.
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If |
Source code in src/autorubric/llm.py
from_yaml
classmethod
¶
from_yaml(path: str | Path) -> LLMConfig
Load LLMConfig from a YAML file.
| PARAMETER | DESCRIPTION |
|---|---|
path
|
Path to YAML configuration file.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
LLMConfig
|
LLMConfig instance with values from the YAML file. |
| RAISES | DESCRIPTION |
|---|---|
FileNotFoundError
|
If the file doesn't exist. |
ValueError
|
If required fields are missing or invalid. |
Example YAML file (llm_config.yaml): model: openai/gpt-5.2 # temperature omitted: the provider's default applies max_tokens: 1024 cache_enabled: true cache_ttl: 3600
thinking accepts a level string (high), an int token budget (32000),
or a mapping of ThinkingConfig fields ({level: high, budget_tokens: 9000}),
which is loaded as a ThinkingConfig.
Source code in src/autorubric/llm.py
to_yaml
¶
Save LLMConfig to a YAML file.
The file holds only plain YAML (yaml.safe_load reads it): enums are written as
their values and paths as strings (so a Path cache_dir loads back as a
str). thinking is written so that from_yaml restores an equal value
with the same get_thinking_config():
- A level string or an int budget is written unchanged.
- A
ThinkingConfigis written as a mapping of its fields ({level: high, budget_tokens: 9000}) and loads back as aThinkingConfig. - A
ThinkingLevelmember is written as its string value (high) and loads back as that plain string, not the enum. Restoring the enum would need a custom YAML tag, whichyaml.safe_loadand hand-written files do not use, and it would add nothing:ThinkingLevelis astrenum, so the string compares equal to the member and normalizes to the sameThinkingConfig.
| PARAMETER | DESCRIPTION |
|---|---|
path
|
Path to write YAML configuration file.
TYPE:
|
Source code in src/autorubric/llm.py
LLMClient¶
Async client for LLM generation with caching and rate limiting.
LLMClient
¶
LLMClient(config: LLMConfig, *, cache_namespace: str | None = None)
Unified LLM client with retries, caching, and structured output support.
Uses diskcache for efficient, thread-safe response caching.
Initialize LLM client.
| PARAMETER | DESCRIPTION |
|---|---|
config
|
LLMConfig instance. The model field is required.
TYPE:
|
cache_namespace
|
Keeps this client's response-cache entries apart from those of
other clients that send identical requests, such as several judges polling
one model, whose answers are independent samples. It becomes part of the
cache key;
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If config.model is not specified. |
Source code in src/autorubric/llm.py
generate
async
¶
generate(system_prompt: str, user_prompt: Content, response_format: type[T] | None = None, use_cache: bool | None = None, return_thinking: bool = False, return_result: bool = False, shared_prefix: Content | None = None, pre_1_7_user_prompt: str | None = None, pre_1_7_system_prompt: str | None = None, **kwargs: Any) -> str | T | GenerateResult
Generate LLM response with optional structured output.
| PARAMETER | DESCRIPTION |
|---|---|
system_prompt
|
System message for the LLM.
TYPE:
|
user_prompt
|
User message for the LLM: text, or a document holding images,
sent as content parts (see
TYPE:
|
response_format
|
Optional Pydantic model class for structured output. When provided, LiteLLM uses the model's JSON schema to constrain the LLM output and returns a validated Pydantic instance.
TYPE:
|
use_cache
|
Whether to use caching for this request. - None (default): Use config.cache_enabled setting - True: Force cache usage (initializes cache if needed) - False: Skip cache for this request
TYPE:
|
return_thinking
|
If True and thinking is enabled, return a GenerateResult with both content and thinking. If False (default), only return content. Note: When response_format is provided, thinking is injected into the 'reasoning' field if it exists, regardless of this setting.
TYPE:
|
return_result
|
If True, always return a GenerateResult with full details including usage statistics and completion cost. This is useful when you need to track token usage. When True, takes precedence over the default return behavior.
TYPE:
|
shared_prefix
|
The start of
TYPE:
|
pre_1_7_user_prompt
|
The text this
TYPE:
|
pre_1_7_system_prompt
|
The text
TYPE:
|
**kwargs
|
Override any LLMConfig parameters for this call. In particular,
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
str | T | GenerateResult
|
If return_result=True or return_thinking=True: GenerateResult with content, thinking, usage, cost, and parsed (if response_format was provided). |
str | T | GenerateResult
|
If response_format is None: String response from the LLM. |
str | T | GenerateResult
|
If response_format is provided: Validated Pydantic model instance. If thinking is enabled and the response_format has a 'reasoning' field, it will be populated with the model's thinking trace (a trace that is not a string is dropped). |
| RAISES | DESCRIPTION |
|---|---|
APIError
|
If all retries fail |
ValueError
|
If the response has no choices, or the reply has no content (e.g.
a refusal) or content that is not a string (e.g. a list of content parts),
with or without response_format; or if response_format is provided and the
reply is not JSON ( |
ValidationError
|
If response doesn't match schema, including a JSON reply that is not an object (with or without a thinking trace). |
Source code in src/autorubric/llm.py
1347 1348 1349 1350 1351 1352 1353 1354 1355 1356 1357 1358 1359 1360 1361 1362 1363 1364 1365 1366 1367 1368 1369 1370 1371 1372 1373 1374 1375 1376 1377 1378 1379 1380 1381 1382 1383 1384 1385 1386 1387 1388 1389 1390 1391 1392 1393 1394 1395 1396 1397 1398 1399 1400 1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 1423 1424 1425 1426 1427 1428 1429 1430 1431 1432 1433 1434 1435 1436 1437 1438 1439 1440 1441 1442 1443 1444 1445 1446 1447 1448 1449 1450 1451 1452 1453 1454 1455 1456 1457 1458 1459 1460 1461 1462 1463 1464 1465 1466 1467 1468 1469 1470 1471 1472 1473 1474 1475 1476 1477 1478 1479 1480 1481 1482 1483 1484 1485 1486 1487 1488 1489 1490 1491 1492 1493 1494 1495 1496 1497 1498 1499 1500 1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 1523 1524 1525 1526 1527 1528 1529 1530 1531 1532 1533 1534 1535 1536 1537 1538 1539 1540 1541 1542 1543 1544 1545 1546 1547 1548 1549 1550 1551 1552 1553 1554 1555 1556 1557 1558 1559 1560 1561 1562 1563 1564 1565 1566 1567 1568 1569 1570 1571 1572 1573 1574 1575 1576 1577 1578 1579 1580 1581 1582 1583 1584 1585 1586 1587 1588 1589 1590 1591 1592 1593 1594 1595 1596 1597 1598 1599 1600 1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 1620 1621 1622 1623 1624 1625 1626 1627 1628 1629 1630 1631 1632 1633 1634 1635 1636 1637 1638 1639 1640 1641 1642 1643 1644 1645 1646 1647 1648 1649 1650 1651 1652 1653 1654 1655 1656 1657 1658 1659 1660 1661 1662 1663 1664 1665 1666 1667 1668 1669 1670 1671 1672 1673 1674 1675 1676 1677 1678 1679 1680 1681 1682 1683 1684 1685 1686 1687 1688 1689 1690 1691 1692 1693 | |
clear_cache
¶
Clear all cached responses.
| RETURNS | DESCRIPTION |
|---|---|
int
|
Number of entries cleared. |
cache_stats
¶
Get cache statistics.
| RETURNS | DESCRIPTION |
|---|---|
dict[str, Any]
|
Dict with 'size', 'count', and 'directory' keys. |
Source code in src/autorubric/llm.py
close
¶
Close the on-disk cache, releasing its file handles.
diskcache keeps the underlying SQLite database open for the lifetime of the cache object. Call this when done with the client so the cache directory can be removed (notably on Windows, which refuses to delete files that are still open). Safe to call when no cache was initialized.
Source code in src/autorubric/llm.py
generate¶
Convenience function for one-off LLM generation.
generate
async
¶
generate(system_prompt: str, user_prompt: Content, model: str, response_format: type[T] | None = None, **kwargs: Any) -> str | T
Simple one-shot generation function.
For repeated calls, prefer creating an LLMClient instance.
| PARAMETER | DESCRIPTION |
|---|---|
system_prompt
|
System message for the LLM.
TYPE:
|
user_prompt
|
User message for the LLM: text, or a document holding images.
TYPE:
|
model
|
Model identifier (REQUIRED).
TYPE:
|
response_format
|
Optional Pydantic model for structured output.
TYPE:
|
**kwargs
|
Additional LLMConfig parameters.
TYPE:
|
Example
Simple string response¶
response = await generate( "You are a helpful assistant.", "What is 2+2?", model="openai/gpt-5.2-mini" )
Structured output¶
from pydantic import BaseModel
class MathAnswer(BaseModel): result: int explanation: str
answer = await generate( "You are a math tutor.", "What is 2+2?", model="openai/gpt-5.2-mini", response_format=MathAnswer ) print(answer.result) # 4
Source code in src/autorubric/llm.py
GenerateResult¶
Result from LLM generation.
GenerateResult
dataclass
¶
GenerateResult(content: str, thinking: str | None = None, raw_response: Any = None, usage: TokenUsage | None = None, cost: float | None = None, parsed: Any = None)
Result from LLM generation including content, thinking, and usage statistics.
| ATTRIBUTE | DESCRIPTION |
|---|---|
content |
The main response content from the LLM (raw string).
TYPE:
|
thinking |
The thinking/reasoning trace if thinking was enabled. None if thinking was not enabled or the provider doesn't support it.
TYPE:
|
raw_response |
The raw LiteLLM response object for advanced use cases.
TYPE:
|
usage |
Token usage statistics from this LLM call.
TYPE:
|
cost |
Completion cost in USD for this LLM call, calculated using LiteLLM's completion_cost() function. None if cost calculation fails.
TYPE:
|
parsed |
The parsed Pydantic model instance when response_format was provided. None if no response_format was used or parsing failed.
TYPE:
|
ThinkingConfig¶
Configuration for extended thinking/reasoning.
ThinkingConfig
dataclass
¶
ThinkingConfig(level: ThinkingLevel | ThinkingLevelLiteral = MEDIUM, budget_tokens: int | None = None)
Detailed configuration for LLM thinking/reasoning.
Provides a uniform interface across providers: - Anthropic: Extended thinking (claude-sonnet-4-5, claude-opus-4-5+) - OpenAI: Reasoning (o-series, GPT-5 models via openai/responses/ prefix) - Gemini: Thinking mode (2.5+, 3.0+ models) - DeepSeek: Reasoning content
| ATTRIBUTE | DESCRIPTION |
|---|---|
level |
High-level thinking effort. Used when budget_tokens is not set. Defaults to MEDIUM for a good balance of quality and latency.
TYPE:
|
budget_tokens |
Explicit token budget for thinking (provider-specific): None, or
an integer >= -1, sent as
TYPE:
|
Both fields are checked when the config is built: level must be a ThinkingLevel
or its string value, stored as the member, or None alongside a budget_tokens (a
budget is sent without reading level), and budget_tokens None or an integer
= -1 (numpy integers too, stored as
int; never abool). Anything else raisesValueError.
Examples:
Simple: use a thinking level¶
ThinkingConfig(level=ThinkingLevel.HIGH)
Fine-grained: specify exact token budget¶
ThinkingConfig(budget_tokens=32000) # For complex reasoning tasks
Disable thinking¶
ThinkingConfig(level=ThinkingLevel.NONE)
get_effective_budget
¶
Get the effective token budget based on level or explicit budget.
ThinkingLevel¶
Enum for thinking level presets.
ThinkingLevel
¶
Bases: str, Enum
Standardized thinking/reasoning effort levels across LLM providers.
LiteLLM translates these to provider-specific parameters: - Anthropic: thinking={type, budget_tokens} with low→1024, medium→2048, high→4096 - OpenAI: reasoning_effort parameter for o-series and GPT-5 models - Gemini: thinking configuration with similar token budgets - DeepSeek: Standardized via LiteLLM's reasoning_effort
Higher levels mean more "thinking" tokens/steps, trading latency for quality.
ThinkingLevelLiteral¶
Type alias for thinking level strings.
ThinkingLevelLiteral
module-attribute
¶
ThinkingParam¶
Type alias for thinking parameter (level or budget).
ThinkingParam
module-attribute
¶
ThinkingParam = ThinkingConfig | ThinkingLevel | ThinkingLevelLiteral | int | None
Type for the thinking parameter in LLMConfig.
Accepts: - ThinkingConfig: Full configuration object - ThinkingLevel: Enum value (e.g., ThinkingLevel.HIGH) - str: Level as string ("low", "medium", "high", "none") - int: Direct token budget (e.g., 32000), an integer >= -1 (never a bool) - None: Disable thinking
classify_grading_error¶
Classify an exception raised while grading a single criterion into an ErrorCategory. The grading pipeline uses the result to route a failed judge call: infrastructure and parse errors are treated as abstentions (mapped to CANNOT_ASSESS / na=True and excluded from scoring under the default SKIP strategy), while unknown errors fall back to a conservative worst-case verdict. These verdicts only stand in for failed calls: an item whose every criterion's judgment failed has no score, and its report's error begins Every criterion's judgment failed:. Configuration errors (LiteLLM's UnsupportedParamsError, 401, 402, 403 and 404 errors from the provider or from TypeSafe (a 402 Payment Required says the account has no credits left), and the OpenAI SDK's refusal to build its client, e.g. with no OpenAI or Azure API key, which LiteLLM reports as an InternalServerError, or an APIError for Azure) still classify as infrastructure, but the grader never routes them: they fail every call the judge makes, so the grade raises a ValueError naming the judge, chained from the original exception. Item-specific 400s such as ContextWindowExceededError still abstain.
classify_grading_error
¶
classify_grading_error(exc: BaseException) -> ErrorCategory
Classify an exception raised while grading a criterion.
Reuses the underlying OpenAI exception taxonomy that LiteLLM builds on: every
transient API failure (Timeout, APIConnectionError, RateLimitError,
ServiceUnavailableError, InternalServerError, status errors, etc.) subclasses
openai.APIError (including litellm.APIError itself), while parse/validation
failures (json.JSONDecodeError -> ValueError, pydantic.ValidationError)
do not. LLMClient.generate raises one of these for every structured reply it
cannot use: one that is not JSON, not a JSON object (with or without a thinking
trace), or that does not match the schema is a ValidationError or
JSONDecodeError. A reply with no content (e.g. a refusal) or whose content is not
a string (e.g. a list of content parts) is a ValueError, as is a response with no
choices, structured or plain-text alike.
Decision-model failures raised by the optional TypeSafe SDK (typesafe-sdk) are
routed by the SDK's own exception types, ahead of the ValueError rule:
TypeSafeAPIResponseValidationError(the endpoint's answer did not validate),TypeSafeBadRequestError(400) andTypeSafeUnprocessableEntityError(422, e.g. a request over the model's input limit) are"parse": the endpoint answered, but no usable answer came back for the request that was built.- Every other
TypeSafeErroris"infrastructure": authentication, permission, not found, rate limit, server errors, and connection failures and timeouts (which subclassTypeSafeErrordirectly, notTypeSafeAPIError).
Configuration errors are classified like any other API error ("infrastructure"),
but CriterionGrader never routes one: an exception that says the judge is
misconfigured (LiteLLM's UnsupportedParamsError, a 401, 402 (no credits left), 403
or 404 from the provider (but OpenRouter's 403, a moderation flag on the input) or from
TypeSafe, or the OpenAI SDK's refusal to build its client, e.g. with no OpenAI or Azure
API key, which LiteLLM reports as an InternalServerError or, for Azure, an APIError)
fails every call the judge makes, so the grade raises a ValueError naming the
judge, chained from it, instead of abstaining (#40). Item-specific 400s
(ContextWindowExceededError, ContentPolicyViolationError) and a plain
BadRequestError still abstain.
| PARAMETER | DESCRIPTION |
|---|---|
exc
|
The exception raised during a judge call.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
ErrorCategory
|
|
ErrorCategory
|
errors, and |
Source code in src/autorubric/llm.py
ErrorCategory¶
Type alias for the category of a grading failure: Literal["infrastructure", "parse", "unknown"].
"infrastructure": API/network failure (timeout, connection, rate limit, server error). Not the submission's fault. (A configuration error, e.g. an invalid API key, is not routed at all:CriterionGraderraises it.)"parse": the judge responded but its output could not be parsed/validated into the expected schema: a reply that is not JSON, is not a JSON object, has no content (e.g. a refusal) or content that is not a string (e.g. a list of content parts), with or without a thinking trace, and a response with no choices at all. Also not the submission's fault.LLMClient.generateraises the sameValueErrorfor a plain-text call (noresponse_format) whose reply has no content or content that is not a string, rather than returning it, and does not cache it."unknown": an unexpected error that does not fit the above categories.
ErrorCategory
module-attribute
¶
Category of a failure encountered while grading a single criterion.
- infrastructure: API/network failure (timeout, connection, rate limit, server error).
Not the submission's fault; the judge never produced a usable response. (A configuration
error, e.g. an invalid API key, is not routed at all:
CriterionGraderraises it.) - parse: the judge responded but its output could not be parsed/validated into the expected schema. Also not the submission's fault.
- unknown: an unexpected error that does not fit the above categories.