Skip to content

Dataset

Dataset management classes for organizing evaluation data with optional ground truth labels.

Overview

The RubricDataset class provides structured storage for evaluation datasets, including submissions, optional ground truth verdicts, per-item rubrics, and reference submissions. Datasets serialize to JSON (to_json/to_file, and back with from_json/from_file) for sharing and reproducibility.

Quick Example

from autorubric import Rubric, Criterion, CriterionVerdict, DataItem, RubricDataset

# Create a rubric
rubric = Rubric([
    Criterion(name="accuracy", weight=10.0, requirement="Factually correct"),
    Criterion(name="clarity", weight=5.0, requirement="Clear and concise"),
])

# Create a dataset (prompt is now optional)
dataset = RubricDataset(
    name="photosynthesis-eval",
    prompt="Explain photosynthesis",  # Optional global prompt
    rubric=rubric,
)

# Add items with ground truth
dataset.add_item(
    submission="Photosynthesis is the process by which plants convert sunlight...",
    description="Good response",
    ground_truth=[CriterionVerdict.MET, CriterionVerdict.MET]
)

# Serialize
dataset.to_file("dataset.json")

# Load
loaded = RubricDataset.from_file("dataset.json")

A multi-choice criterion's ground truth is an option label. from_json and from_file check each label against the criterion's options. For a criterion without an NA option of its own, the label of the NA option the grader adds ("Cannot assess / not applicable", CANONICAL_NA_OPTION) is accepted too, because fill_ground_truth records it when a judge chooses that option. It is read as an NA label everywhere (see CANNOT_ASSESS handling).

Per-Item Rubrics

For datasets where each item requires a unique rubric (e.g., question-specific evaluation):

item = DataItem(
    submission="Answer to question 1...",
    description="Q1",
    rubric=Rubric([
        Criterion(weight=1.0, requirement="Correct answer for Q1"),
    ])
)

dataset = RubricDataset(
    prompt="Answer the question",
    rubric=None,  # No global rubric
    items=[item],
)

# Get effective rubric for an item
rubric = dataset.get_item_rubric(0)  # Returns item's rubric

Per-Item Prompts

Different items can have different prompts:

# Create items with individual prompts
item1 = DataItem(
    submission="The answer is 42.",
    description="Math problem",
    prompt="Evaluate this answer to: What is 6 x 7?"
)

item2 = DataItem(
    submission="Analysis of Hamlet's soliloquy...",
    description="Literary analysis"
    # No prompt specified, will use global prompt
)

# Create dataset with optional global prompt
dataset = RubricDataset(
    prompt="Evaluate this response",  # Optional global prompt (fallback)
    rubric=rubric,
    items=[item1, item2],
)

# Get effective prompt for an item
prompt = dataset.get_item_prompt(0)  # Returns item1's prompt
prompt = dataset.get_item_prompt(1)  # Returns global prompt (item2 has none)

# Raises ValueError if neither item nor dataset has a prompt
try:
    prompt = dataset.get_item_prompt(0)
except ValueError:
    print("No prompt available for this item")

Use in grading:

for i, item in enumerate(dataset):
    result = await rubric.grade(
        to_grade=item.submission,
        grader=grader,
        query=dataset.get_item_prompt(i),
    )

Reference Submissions

Provide exemplar responses for judge calibration:

# Global reference for all items
dataset = RubricDataset(
    prompt="Explain photosynthesis",
    rubric=rubric,
    reference_submission="Detailed explanation of photosynthesis...",
)

# Per-item reference (overrides global)
dataset.add_item(
    submission="Student answer...",
    description="Q1",
    reference_submission="Custom reference for this item",
)

# Get effective reference
ref = dataset.get_item_reference_submission(0)

Images in Dataset Files

A submission, prompt or reference submission can hold images, at the item level and at the dataset level. In Python it is Content, which is a string, an Image, or a list of strings and images read in order (a document, such as a figure followed by its caption):

from autorubric import Image

dataset.add_item(
    submission=[Image("figures/fig1.png"), "Figure 1: accuracy by model size."],
    description="Figure with caption",
)

A dataset file holds the same three shapes. A string is text. An image is an object whose only key is "image", with the image's path as its value. A document is an array of strings and image objects:

{
  "submission": [{"image": "figures/fig1.png"}, "Figure 1: accuracy by model size."],
  "description": "Figure with caption"
}

Paths are written with / on every platform. from_file resolves a relative path against the directory of the dataset file, so the file and its figures can be moved together. from_json(text, base_dir=...) resolves it against base_dir, or against the current working directory when base_dir is not given. An absolute path is kept as it is. Loading opens no image file, so a dataset whose images are missing still loads, and the missing file is reported when its item is graded. Any other JSON value loads unchanged, as it did before images existed, and is graded as it was then. A JSON object submission is read as the {"thinking": ..., "output": ...} form, so an object with neither key, such as a mistyped image object like {"img": "fig.png"}, grades as an empty submission and raises no error. A number raises AttributeError when its item is graded. An array holding a number or an object raises TypeError, as does a prompt or reference submission that is not content. EvalRunner records each such failure against its own item and grades the others.

When writing, to_file stores each image path relative to the file's directory when the image is inside it, and as an absolute path otherwise. to_json(base_dir=...) does the same relative to base_dir and writes absolute paths when base_dir is not given. An image built from bytes (Image(data)) has no path, so to_json refuses it with a ValueError that names the item and field. to_file saves such an image beside the dataset file as <stem>_images/<sha256>.<ext> (for results.json, the directory results_images/), writes the file only if it is not there yet, and refers to it by that relative path. A dataset without images is written exactly as before.

Train/Test Split

train_data, test_data = dataset.split_train_test(
    n_train=100,
    stratify=True,  # Balance by ground truth verdicts
    seed=42,
)

DataItem

A single item in an evaluation dataset.

DataItem dataclass

DataItem(submission: Content, description: str, ground_truth: list[CriterionVerdict | str] | None = None, rubric: Rubric | None = None, reference_submission: Content | None = None, prompt: Content | None = None, ground_truth_reasons: list[str | None] | None = None)

A single item to be graded, optionally with ground truth verdicts.

ATTRIBUTE DESCRIPTION
submission

The content to be evaluated (Content): plain text (or a JSON-serialized string for structured data, e.g. dialogues), an Image, or a document, a list of text and images read in order.

TYPE: Content

description

A brief description of this item (e.g., "High quality response").

TYPE: str

ground_truth

Optional list of ground truth values, one per criterion. - For binary criteria: CriterionVerdict (MET, UNMET, CANNOT_ASSESS) - For multi-choice criteria: str (option label) Used for computing evaluation metrics against LLM predictions.

TYPE: list[CriterionVerdict | str] | None

rubric

Optional per-item rubric. If provided, this rubric is used for grading instead of the dataset-level rubric. Useful for datasets where each item has unique evaluation criteria (e.g., ResearcherBench). It replaces the dataset-level rubric entirely, guidelines included: this rubric carries its own guidelines (possibly none) and never inherits the dataset-level ones.

TYPE: Rubric | None

reference_submission

Optional exemplar response for grading context (Content, as submission). When present, helps calibrate the grader's expectations. Item-level takes precedence over dataset-level reference.

TYPE: Content | None

prompt

Optional per-item prompt (Content, as submission). If provided, overrides the dataset-level prompt for this item.

TYPE: Content | None

ground_truth_reasons

Optional written reasons for the ground truth, one per criterion and aligned index-for-index with ground_truth, which must then be set. A None entry means that criterion's ground truth has no written reason. A few-shot example drawn from this item shows its reason for the criterion when FewShotConfig.include_reason is True.

TYPE: list[str | None] | None

Example

Binary criteria only

item = DataItem( ... submission="The Industrial Revolution began in Britain around 1760...", ... description="Excellent essay covering all criteria", ... ground_truth=[CriterionVerdict.MET, CriterionVerdict.MET, CriterionVerdict.UNMET] ... )

Mixed binary and multi-choice

item = DataItem( ... submission="The assistant responded helpfully...", ... description="Good dialogue", ... ground_truth=[CriterionVerdict.MET, "Very satisfied", "Yes - reasonable"] ... )

With written reasons for the ground truth (None where there is none)

item = DataItem( ... submission="The Industrial Revolution began in Britain around 1760...", ... description="Excellent essay", ... ground_truth=[CriterionVerdict.MET, CriterionVerdict.UNMET], ... ground_truth_reasons=["Dates the start and names Britain.", None], ... )

With per-item rubric

from autorubric import Rubric, Criterion item = DataItem( ... submission="Response to a specific question...", ... description="Question-specific grading", ... rubric=Rubric([Criterion(name="Relevance", weight=1.0, requirement="...")]) ... )


RubricDataset

Container for evaluation datasets with optional ground truth.

RubricDataset dataclass

RubricDataset(prompt: Content | None = None, rubric: Rubric | None = None, items: list[DataItem] = list(), name: str | None = None, reference_submission: Content | None = None)

A collection of DataItems tied to a specific prompt and rubric.

The RubricDataset encapsulates: - The prompt that generated the responses - The rubric used for evaluation (global or per-item) - A collection of DataItems with optional ground truth labels

This is useful for: - Evaluating LLM grader accuracy against human judgments - Training reward models with labeled data - Benchmarking different grading strategies

ATTRIBUTE DESCRIPTION
prompt

Optional global prompt/question that items are responses to (Content: text, an Image, or a document). Can be None if all items have their own prompts.

TYPE: Content | None

rubric

Optional global Rubric used to evaluate items. Can be None if all items have their own rubrics.

TYPE: Rubric | None

items

List of DataItem instances to evaluate.

TYPE: list[DataItem]

name

Optional name for the dataset (e.g., "essay-grading-v1").

TYPE: str | None

reference_submission

Optional global exemplar response for grading context (Content). When present, provides calibration for the grader. Item-level reference takes precedence over this dataset-level reference.

TYPE: Content | None

Example

from autorubric import Rubric, Criterion, CriterionVerdict rubric = Rubric([ ... Criterion(name="Accuracy", weight=10.0, requirement="Factually correct"), ... Criterion(name="Clarity", weight=5.0, requirement="Clear and concise"), ... ]) dataset = RubricDataset( ... prompt="Explain photosynthesis", ... rubric=rubric, ... ) dataset.add_item( ... submission="Photosynthesis is the process...", ... description="Good response", ... ground_truth=[CriterionVerdict.MET, CriterionVerdict.MET] ... )

criterion_names property

criterion_names: list[str]

Get criterion names from global rubric.

RAISES DESCRIPTION
ValueError

If no global rubric is set.

num_criteria property

num_criteria: int

Number of criteria in the global rubric.

RAISES DESCRIPTION
ValueError

If no global rubric is set.

total_positive_weight property

total_positive_weight: float

Sum of all positive criterion weights in global rubric.

RAISES DESCRIPTION
ValueError

If no global rubric is set.

get_item_rubric

get_item_rubric(idx: int) -> Rubric

Get the effective rubric for an item (per-item or global fallback).

PARAMETER DESCRIPTION
idx

Index of the item.

TYPE: int

RETURNS DESCRIPTION
Rubric

The item's rubric if set, otherwise the dataset's global rubric.

RAISES DESCRIPTION
ValueError

If neither item nor dataset has a rubric.

Source code in src/autorubric/dataset.py
def get_item_rubric(self, idx: int) -> Rubric:
    """Get the effective rubric for an item (per-item or global fallback).

    Args:
        idx: Index of the item.

    Returns:
        The item's rubric if set, otherwise the dataset's global rubric.

    Raises:
        ValueError: If neither item nor dataset has a rubric.
    """
    item = self.items[idx]
    if item.rubric is not None:
        return item.rubric
    if self.rubric is not None:
        return self.rubric
    raise ValueError(f"Item {idx} has no rubric and dataset has no global rubric")

get_item_reference_submission

get_item_reference_submission(idx: int) -> Content | None

Get the effective reference submission for an item.

Item-level reference takes precedence over dataset-level reference.

PARAMETER DESCRIPTION
idx

Index of the item.

TYPE: int

RETURNS DESCRIPTION
Content | None

The item's reference_submission if set, otherwise the dataset's

Content | None

global reference_submission. May be None if neither is set.

Source code in src/autorubric/dataset.py
def get_item_reference_submission(self, idx: int) -> Content | None:
    """Get the effective reference submission for an item.

    Item-level reference takes precedence over dataset-level reference.

    Args:
        idx: Index of the item.

    Returns:
        The item's reference_submission if set, otherwise the dataset's
        global reference_submission. May be None if neither is set.
    """
    item = self.items[idx]
    if item.reference_submission is not None:
        return item.reference_submission
    return self.reference_submission

get_item_prompt

get_item_prompt(idx: int) -> Content

Get the effective prompt for an item (per-item or global fallback).

PARAMETER DESCRIPTION
idx

Index of the item.

TYPE: int

RETURNS DESCRIPTION
Content

The item's prompt if set, otherwise the dataset's global prompt.

RAISES DESCRIPTION
ValueError

If neither item nor dataset has a prompt.

Source code in src/autorubric/dataset.py
def get_item_prompt(self, idx: int) -> Content:
    """Get the effective prompt for an item (per-item or global fallback).

    Args:
        idx: Index of the item.

    Returns:
        The item's prompt if set, otherwise the dataset's global prompt.

    Raises:
        ValueError: If neither item nor dataset has a prompt.
    """
    item = self.items[idx]
    if item.prompt is not None:
        return item.prompt
    if self.prompt is not None:
        return self.prompt
    raise ValueError(f"Item {idx} has no prompt and dataset has no global prompt")

compute_weighted_score

compute_weighted_score(verdicts: list[CriterionVerdict | str], normalize: bool = True, rubric: Rubric | None = None) -> float | None

Compute weighted score from verdicts (binary or multi-choice).

PARAMETER DESCRIPTION
verdicts

List of verdict values, one per criterion. - For binary criteria: CriterionVerdict (MET=1.0, UNMET=0.0) - For multi-choice criteria: str (option label, resolved to value)

TYPE: list[CriterionVerdict | str]

normalize

If True, normalize score to [0, 1]. If False, return raw sum.

TYPE: bool DEFAULT: True

rubric

Optional rubric to use for scoring. If None, uses global rubric.

TYPE: Rubric | None DEFAULT: None

RETURNS DESCRIPTION
float | None

Weighted score based on criterion weights and verdicts, or None when

float | None

every verdict abstains (CANNOT_ASSESS or an NA option), leaving nothing to

float | None

score.

RAISES DESCRIPTION
ValueError

If a multi-choice label doesn't match any option, or if rubric is None and no global rubric is set.

Source code in src/autorubric/dataset.py
def compute_weighted_score(
    self,
    verdicts: list[CriterionVerdict | str],
    normalize: bool = True,
    rubric: Rubric | None = None,
) -> float | None:
    """Compute weighted score from verdicts (binary or multi-choice).

    Args:
        verdicts: List of verdict values, one per criterion.
            - For binary criteria: CriterionVerdict (MET=1.0, UNMET=0.0)
            - For multi-choice criteria: str (option label, resolved to value)
        normalize: If True, normalize score to [0, 1]. If False, return raw sum.
        rubric: Optional rubric to use for scoring. If None, uses global rubric.

    Returns:
        Weighted score based on criterion weights and verdicts, or ``None`` when
        every verdict abstains (CANNOT_ASSESS or an NA option), leaving nothing to
        score.

    Raises:
        ValueError: If a multi-choice label doesn't match any option, or if
            rubric is None and no global rubric is set.
    """
    effective_rubric = rubric if rubric is not None else self.rubric
    if effective_rubric is None:
        raise ValueError("Cannot compute score: no rubric provided and no global rubric set")
    # Delegate to the rubric's unified scorer. The default
    # cannot_assess_strategy=SKIP matches this method's historical behavior
    # (CANNOT_ASSESS / NA excluded from numerator and denominator), and routes
    # through the shared score_reports core so all scoring paths agree.
    return effective_rubric.compute_score(verdicts, normalize=normalize)

add_item

add_item(submission: Content, description: str, ground_truth: list[CriterionVerdict | str] | None = None, rubric: Rubric | None = None, reference_submission: Content | None = None, prompt: Content | None = None, *, ground_truth_reasons: list[str | None] | None = None) -> None

Add a new item to the dataset.

PARAMETER DESCRIPTION
submission

The content to be evaluated.

TYPE: Content

description

A brief description of this item.

TYPE: str

ground_truth

Optional list of ground truth values. - For binary criteria: CriterionVerdict (MET, UNMET, CANNOT_ASSESS) - For multi-choice criteria: str (option label)

TYPE: list[CriterionVerdict | str] | None DEFAULT: None

rubric

Optional per-item rubric. If None, uses global rubric.

TYPE: Rubric | None DEFAULT: None

reference_submission

Optional exemplar response for grading context.

TYPE: Content | None DEFAULT: None

prompt

Optional per-item prompt. If None, uses global prompt.

TYPE: Content | None DEFAULT: None

ground_truth_reasons

Optional written reasons for the ground truth, one per criterion and aligned with ground_truth (which must then be given); a None entry means that criterion has no written reason. Few-shot examples show them when FewShotConfig.include_reason is True.

TYPE: list[str | None] | None DEFAULT: None

RAISES DESCRIPTION
ValueError

If ground_truth length doesn't match effective rubric criteria count, or if neither per-item nor global rubric is available, or if neither per-item nor global prompt is available, or if ground_truth_reasons is not a list of str or None entries aligned with ground_truth.

Source code in src/autorubric/dataset.py
def add_item(
    self,
    submission: Content,
    description: str,
    ground_truth: list[CriterionVerdict | str] | None = None,
    rubric: Rubric | None = None,
    reference_submission: Content | None = None,
    prompt: Content | None = None,
    *,
    ground_truth_reasons: list[str | None] | None = None,
) -> None:
    """Add a new item to the dataset.

    Args:
        submission: The content to be evaluated.
        description: A brief description of this item.
        ground_truth: Optional list of ground truth values.
            - For binary criteria: CriterionVerdict (MET, UNMET, CANNOT_ASSESS)
            - For multi-choice criteria: str (option label)
        rubric: Optional per-item rubric. If None, uses global rubric.
        reference_submission: Optional exemplar response for grading context.
        prompt: Optional per-item prompt. If None, uses global prompt.
        ground_truth_reasons: Optional written reasons for the ground truth, one per
            criterion and aligned with ``ground_truth`` (which must then be given); a
            ``None`` entry means that criterion has no written reason. Few-shot
            examples show them when ``FewShotConfig.include_reason`` is True.

    Raises:
        ValueError: If ground_truth length doesn't match effective rubric criteria count,
            or if neither per-item nor global rubric is available, or if neither
            per-item nor global prompt is available, or if ground_truth_reasons is
            not a list of ``str`` or ``None`` entries aligned with ground_truth.
    """
    if prompt is None and self.prompt is None:
        raise ValueError(
            "Cannot add item: no per-item prompt provided and no global prompt set"
        )
    item = DataItem(
        submission=submission,
        description=description,
        ground_truth=ground_truth,
        rubric=rubric,
        reference_submission=reference_submission,
        prompt=prompt,
        ground_truth_reasons=ground_truth_reasons,
    )
    effective_rubric = item.rubric if item.rubric is not None else self.rubric
    if effective_rubric is None:
        raise ValueError(
            "Cannot add item: no per-item rubric provided and no global rubric set"
        )
    if item.ground_truth is not None and len(item.ground_truth) != len(effective_rubric.rubric):
        raise ValueError(
            f"Ground truth has {len(item.ground_truth)} values, "
            f"but rubric has {len(effective_rubric.rubric)} criteria"
        )
    self.items.append(item)

to_json

to_json(indent: int | None = 2, *, base_dir: str | Path | None = None) -> str

Serialize the dataset to a JSON string.

A submission, prompt or reference submission is written as from_json reads it: text as a string, an Image as {"image": "<path>"} and a document as an array of both. An image's path is written with /, relative to base_dir when its file is inside that directory (compared as written: symlinks and .. are not resolved), else absolute; with no base_dir it is always absolute. No image file is read. A dataset without images is written exactly as before images existed.

PARAMETER DESCRIPTION
indent

Number of spaces for indentation. None for compact output.

TYPE: int | None DEFAULT: 2

base_dir

The directory image paths are written relative to, normally that of the file the JSON is saved in (to_file passes it).

TYPE: str | Path | None DEFAULT: None

RETURNS DESCRIPTION
str

JSON string representation of the dataset.

RAISES DESCRIPTION
ValueError

If an image is held as bytes, which has no path to write; the message names the item and field. to_file saves such images.

Source code in src/autorubric/dataset.py
def to_json(self, indent: int | None = 2, *, base_dir: str | Path | None = None) -> str:
    """Serialize the dataset to a JSON string.

    A submission, prompt or reference submission is written as ``from_json`` reads it:
    text as a string, an ``Image`` as ``{"image": "<path>"}`` and a document as an
    array of both. An image's path is written with ``/``, relative to ``base_dir`` when
    its file is inside that directory (compared as written: symlinks and ``..`` are not
    resolved), else absolute; with no ``base_dir`` it is always absolute. No image file
    is read. A dataset without images is written exactly as before images existed.

    Args:
        indent: Number of spaces for indentation. None for compact output.
        base_dir: The directory image paths are written relative to, normally that of
            the file the JSON is saved in (``to_file`` passes it).

    Returns:
        JSON string representation of the dataset.

    Raises:
        ValueError: If an image is held as bytes, which has no path to write; the
            message names the item and field. ``to_file`` saves such images.
    """
    return self._to_json(indent, _image_path_writer(base_dir))

to_file

to_file(path: str | Path) -> None

Save dataset to a JSON file.

Image paths are written relative to the file's directory when inside it, else absolute (to_json with base_dir set to that directory), so from_file still finds the images inside the directory after the directory is moved. An image held as bytes is saved beside the file as <stem>_images/<digest>.<ext> (<stem> the file name without its suffix, <digest> the image's SHA-256, <ext> one of png, jpg, gif and webp), written only when no such file exists yet, and referred to by that relative path. The JSON file is written last. A dataset without images is written by self.to_json(), as before images existed, so a subclass's to_json still writes it.

PARAMETER DESCRIPTION
path

Path to write the JSON file.

TYPE: str | Path

Source code in src/autorubric/dataset.py
def to_file(self, path: str | Path) -> None:
    """Save dataset to a JSON file.

    Image paths are written relative to the file's directory when inside it, else
    absolute (``to_json`` with ``base_dir`` set to that directory), so ``from_file``
    still finds the images inside the directory after the directory is moved. An
    image held as bytes is saved beside the file as ``<stem>_images/<digest>.<ext>``
    (``<stem>`` the file name without its suffix, ``<digest>`` the image's SHA-256,
    ``<ext>`` one of ``png``, ``jpg``, ``gif`` and ``webp``), written only when no such
    file exists yet, and referred to by that relative path. The JSON file is written
    last. A dataset without images is written by ``self.to_json()``, as before images
    existed, so a subclass's ``to_json`` still writes it.

    Args:
        path: Path to write the JSON file.
    """
    path = Path(path)
    if not _dataset_has_images(self):
        path.write_text(self.to_json(), encoding="utf-8")
        return
    base_dir = path.parent
    write_path = _image_path_writer(base_dir)
    images_dir = f"{path.stem}_images"
    pending: dict[Path, bytes] = {}

    def write_image(image: Image, where: str) -> str:
        if image.path is not None:
            return write_path(image, where)
        name = f"{images_dir}/{image.digest}.{_IMAGE_EXTENSIONS[image.media_type]}"
        pending[base_dir / name] = image.read_bytes()
        return name

    text = self._to_json(2, write_image)
    for image_path, data in pending.items():
        if not image_path.exists():
            image_path.parent.mkdir(parents=True, exist_ok=True)
            image_path.write_bytes(data)
    path.write_text(text, encoding="utf-8")

from_json classmethod

from_json(json_string: str, *, base_dir: str | Path | None = None) -> RubricDataset

Deserialize a dataset from a JSON string.

A submission, prompt or reference submission (of an item or of the dataset) is read as content: a string is text; an image object, {"image": "<path>"} with "image" its only key and a string value, is an Image; an array whose every element is a string or an image object is a document, read as a list of its parts. Any other value is kept as it is, so every file that loaded before images existed still loads, and is graded as it was then (Grader.grade): an object is read as the {thinking, output} form, so one with neither key grades as an empty submission, while a number, or an array holding a part that is not content, raises when its item is graded. An image path is written with /; a relative one is resolved against base_dir, or the current working directory at the time of the call when base_dir is None, and an absolute one is kept; the working directory is consulted only for such a relative path (a relative base_dir included). No image file is opened: a missing file fails when its item is graded.

PARAMETER DESCRIPTION
json_string

JSON string representation of the dataset.

TYPE: str

base_dir

The directory relative image paths are resolved against, normally that of the file the JSON was read from (from_file passes it).

TYPE: str | Path | None DEFAULT: None

RETURNS DESCRIPTION
RubricDataset

RubricDataset instance.

RAISES DESCRIPTION
ValueError

If the JSON is invalid, missing required fields, or if an item has no rubric when no global rubric is set, or if an item's ground_truth_reasons do not align with its ground_truth, or if an image object's path is empty.

Source code in src/autorubric/dataset.py
@classmethod
def from_json(cls, json_string: str, *, base_dir: str | Path | None = None) -> RubricDataset:
    """Deserialize a dataset from a JSON string.

    A submission, prompt or reference submission (of an item or of the dataset) is read
    as content: a string is text; an image object, ``{"image": "<path>"}`` with
    ``"image"`` its only key and a string value, is an ``Image``; an array whose every
    element is a string or an image object is a document, read as a list of its parts.
    Any other value is kept as it is, so every file that loaded before images existed
    still loads, and is graded as it was then (``Grader.grade``): an object is read as
    the ``{thinking, output}`` form, so one with neither key grades as an empty
    submission, while a number, or an array holding a part that is not content, raises
    when its item is graded. An image path is written with ``/``; a relative one is
    resolved against ``base_dir``, or the current working directory at the time of the
    call when ``base_dir`` is None, and an absolute one is kept; the working directory is
    consulted only for such a relative path (a relative ``base_dir`` included). No
    image file is opened: a missing file fails when its item is graded.

    Args:
        json_string: JSON string representation of the dataset.
        base_dir: The directory relative image paths are resolved against, normally
            that of the file the JSON was read from (``from_file`` passes it).

    Returns:
        RubricDataset instance.

    Raises:
        ValueError: If the JSON is invalid, missing required fields, or if
            an item has no rubric when no global rubric is set, or if an item's
            ``ground_truth_reasons`` do not align with its ``ground_truth``, or if an
            image object's path is empty.
    """
    try:
        data = json.loads(json_string)
    except json.JSONDecodeError as e:
        raise ValueError(f"Failed to parse JSON: {e}") from e

    if not isinstance(data, dict):
        raise ValueError(f"Expected JSON object, got {type(data).__name__}")

    # Validate required fields
    if "rubric" not in data:
        raise ValueError("Missing required field: 'rubric'")

    # Parse global rubric (can be None/null)
    rubric_data = data["rubric"]
    rubric: Rubric | None = None
    if rubric_data is not None:
        rubric = Rubric.from_dict(rubric_data)

    # Parse items
    items: list[DataItem] = []
    for i, item_data in enumerate(data.get("items", [])):
        if not isinstance(item_data, dict):
            raise ValueError(f"Item {i} must be a dict, got {type(item_data).__name__}")

        submission = _content_from_json(
            item_data.get("submission"), base_dir, f"Item {i} submission"
        )
        description = item_data.get("description")

        if submission is None:
            raise ValueError(f"Item {i} missing required field: 'submission'")
        if description is None:
            raise ValueError(f"Item {i} missing required field: 'description'")

        # Parse per-item rubric if present
        item_rubric_data = item_data.get("rubric")
        item_rubric: Rubric | None = None
        if item_rubric_data is not None:
            item_rubric = Rubric.from_dict(item_rubric_data)

        # Validate that item has access to a rubric
        effective_rubric = item_rubric if item_rubric is not None else rubric
        if effective_rubric is None:
            raise ValueError(f"Item {i} has no rubric and dataset has no global rubric")

        # Parse ground truth against the effective rubric
        ground_truth_raw = item_data.get("ground_truth")
        ground_truth: list[CriterionVerdict | str] | None = None
        if ground_truth_raw is not None:
            ground_truth = []
            for j, v in enumerate(ground_truth_raw):
                criterion = (
                    effective_rubric.rubric[j] if j < len(effective_rubric.rubric) else None
                )

                if criterion is not None and criterion.is_multi_choice:
                    # Multi-choice: keep as string (option label)
                    if not isinstance(v, str):
                        raise ValueError(
                            f"Item {i}, ground_truth[{j}]: multi-choice criterion "
                            f"expects option label string, got {type(v).__name__}"
                        )
                    # Validate that the label names an option, among the options a
                    # judge is shown: the label of the NA option the grader adds to a
                    # criterion without one (CANONICAL_NA_OPTION, which
                    # fill_ground_truth records when a judge chose it) is one too
                    try:
                        criterion.with_guaranteed_na_option().find_option_by_label(v)
                    except ValueError as e:
                        raise ValueError(f"Item {i}, ground_truth[{j}]: {e}") from None
                    ground_truth.append(v)
                else:
                    # Binary: parse as CriterionVerdict
                    try:
                        ground_truth.append(CriterionVerdict(v))
                    except ValueError:
                        raise ValueError(
                            f"Item {i}, ground_truth[{j}]: invalid verdict '{v}'. "
                            f"Must be 'MET', 'UNMET', or 'CANNOT_ASSESS'."
                        ) from None

        # Parse ground truth reasons if present (DataItem checks them against the
        # ground truth)
        ground_truth_reasons = item_data.get("ground_truth_reasons")
        # Parse per-item reference_submission if present
        item_reference = _content_from_json(
            item_data.get("reference_submission"), base_dir, f"Item {i} reference_submission"
        )
        # Parse per-item prompt if present
        item_prompt = _content_from_json(item_data.get("prompt"), base_dir, f"Item {i} prompt")

        items.append(
            DataItem(
                submission=submission,
                description=description,
                ground_truth=ground_truth,
                rubric=item_rubric,
                reference_submission=item_reference,
                prompt=item_prompt,
                ground_truth_reasons=ground_truth_reasons,
            )
        )

    return cls(
        prompt=_content_from_json(data.get("prompt"), base_dir, "The dataset prompt"),
        rubric=rubric,
        items=items,
        name=data.get("name"),
        reference_submission=_content_from_json(
            data.get("reference_submission"), base_dir, "The dataset reference_submission"
        ),
    )

from_file classmethod

from_file(path: str | Path) -> RubricDataset

Load dataset from a JSON file.

Relative image paths are resolved against the file's directory, not the current working directory (from_json with base_dir set to it). A subclass whose from_json takes no base_dir (one written before images existed) is called as before, from_json(text), so its relative image paths resolve against the working directory.

PARAMETER DESCRIPTION
path

Path to the JSON file.

TYPE: str | Path

RETURNS DESCRIPTION
RubricDataset

RubricDataset instance.

RAISES DESCRIPTION
FileNotFoundError

If the file doesn't exist.

ValueError

If the JSON is invalid.

Source code in src/autorubric/dataset.py
@classmethod
def from_file(cls, path: str | Path) -> RubricDataset:
    """Load dataset from a JSON file.

    Relative image paths are resolved against the file's directory, not the current
    working directory (``from_json`` with ``base_dir`` set to it). A subclass whose
    ``from_json`` takes no ``base_dir`` (one written before images existed) is called
    as before, ``from_json(text)``, so its relative image paths resolve against the
    working directory.

    Args:
        path: Path to the JSON file.

    Returns:
        RubricDataset instance.

    Raises:
        FileNotFoundError: If the file doesn't exist.
        ValueError: If the JSON is invalid.
    """
    path = Path(path)
    if not path.exists():
        raise FileNotFoundError(f"Dataset file not found: {path}")

    text = path.read_text(encoding="utf-8")
    if not _takes_base_dir(cls.from_json):
        return cls.from_json(text)
    return cls.from_json(text, base_dir=path.parent)

split_train_test

split_train_test(n_train: int, *, stratify: bool = True, seed: int | None = None) -> tuple[RubricDataset, RubricDataset]

Split dataset into training and test sets.

The training set can be used to provide few-shot examples for grading, while the test set is used for evaluation.

PARAMETER DESCRIPTION
n_train

Exact number of items for training set.

TYPE: int

stratify

If True, stratify by per-criterion verdict distribution. This ensures each split has similar proportion of MET/UNMET/CANNOT_ASSESS for each criterion position. Requires all items to have ground_truth.

TYPE: bool DEFAULT: True

seed

Random seed for reproducible splits.

TYPE: int | None DEFAULT: None

RETURNS DESCRIPTION
tuple[RubricDataset, RubricDataset]

Tuple of (train_dataset, test_dataset).

RAISES DESCRIPTION
ValueError

If n_train is invalid or stratify=True but items lack ground_truth.

Example

dataset = RubricDataset.from_file("data.json") train, test = dataset.split_train_test(n_train=100, stratify=True, seed=42) print(f"Train: {len(train)}, Test: {len(test)}")

Source code in src/autorubric/dataset.py
def split_train_test(
    self,
    n_train: int,
    *,
    stratify: bool = True,
    seed: int | None = None,
) -> tuple[RubricDataset, RubricDataset]:
    """Split dataset into training and test sets.

    The training set can be used to provide few-shot examples for grading,
    while the test set is used for evaluation.

    Args:
        n_train: Exact number of items for training set.
        stratify: If True, stratify by per-criterion verdict distribution.
            This ensures each split has similar proportion of MET/UNMET/CANNOT_ASSESS
            for each criterion position. Requires all items to have ground_truth.
        seed: Random seed for reproducible splits.

    Returns:
        Tuple of (train_dataset, test_dataset).

    Raises:
        ValueError: If n_train is invalid or stratify=True but items lack ground_truth.

    Example:
        >>> dataset = RubricDataset.from_file("data.json")
        >>> train, test = dataset.split_train_test(n_train=100, stratify=True, seed=42)
        >>> print(f"Train: {len(train)}, Test: {len(test)}")
    """
    if n_train < 0:
        raise ValueError(f"n_train must be non-negative, got {n_train}")
    if n_train > len(self.items):
        raise ValueError(f"n_train ({n_train}) exceeds dataset size ({len(self.items)})")

    rng = random.Random(seed)

    if stratify:
        train_items, test_items = self._stratified_split(n_train, rng)
    else:
        indices = list(range(len(self.items)))
        rng.shuffle(indices)
        train_items = [self.items[i] for i in indices[:n_train]]
        test_items = [self.items[i] for i in indices[n_train:]]

    train_dataset = RubricDataset(
        prompt=self.prompt,
        rubric=self.rubric,
        items=train_items,
        name=self.name,
        reference_submission=self.reference_submission,
    )
    test_dataset = RubricDataset(
        prompt=self.prompt,
        rubric=self.rubric,
        items=test_items,
        reference_submission=self.reference_submission,
        name=self.name,
    )

    return train_dataset, test_dataset