Dataset¶
Dataset management classes for organizing evaluation data with optional ground truth labels.
Overview¶
The RubricDataset class provides structured storage for evaluation datasets, including submissions, optional ground truth verdicts, per-item rubrics, and reference submissions. Datasets serialize to JSON (to_json/to_file, and back with from_json/from_file) for sharing and reproducibility.
Quick Example¶
from autorubric import Rubric, Criterion, CriterionVerdict, DataItem, RubricDataset
# Create a rubric
rubric = Rubric([
Criterion(name="accuracy", weight=10.0, requirement="Factually correct"),
Criterion(name="clarity", weight=5.0, requirement="Clear and concise"),
])
# Create a dataset (prompt is now optional)
dataset = RubricDataset(
name="photosynthesis-eval",
prompt="Explain photosynthesis", # Optional global prompt
rubric=rubric,
)
# Add items with ground truth
dataset.add_item(
submission="Photosynthesis is the process by which plants convert sunlight...",
description="Good response",
ground_truth=[CriterionVerdict.MET, CriterionVerdict.MET]
)
# Serialize
dataset.to_file("dataset.json")
# Load
loaded = RubricDataset.from_file("dataset.json")
A multi-choice criterion's ground truth is an option label. from_json and from_file check
each label against the criterion's options. For a criterion without an NA option of its own,
the label of the NA option the grader adds ("Cannot assess / not applicable",
CANONICAL_NA_OPTION) is accepted too, because fill_ground_truth records it when a judge
chooses that option. It is read as an NA label everywhere (see
CANNOT_ASSESS handling).
Per-Item Rubrics¶
For datasets where each item requires a unique rubric (e.g., question-specific evaluation):
item = DataItem(
submission="Answer to question 1...",
description="Q1",
rubric=Rubric([
Criterion(weight=1.0, requirement="Correct answer for Q1"),
])
)
dataset = RubricDataset(
prompt="Answer the question",
rubric=None, # No global rubric
items=[item],
)
# Get effective rubric for an item
rubric = dataset.get_item_rubric(0) # Returns item's rubric
Per-Item Prompts¶
Different items can have different prompts:
# Create items with individual prompts
item1 = DataItem(
submission="The answer is 42.",
description="Math problem",
prompt="Evaluate this answer to: What is 6 x 7?"
)
item2 = DataItem(
submission="Analysis of Hamlet's soliloquy...",
description="Literary analysis"
# No prompt specified, will use global prompt
)
# Create dataset with optional global prompt
dataset = RubricDataset(
prompt="Evaluate this response", # Optional global prompt (fallback)
rubric=rubric,
items=[item1, item2],
)
# Get effective prompt for an item
prompt = dataset.get_item_prompt(0) # Returns item1's prompt
prompt = dataset.get_item_prompt(1) # Returns global prompt (item2 has none)
# Raises ValueError if neither item nor dataset has a prompt
try:
prompt = dataset.get_item_prompt(0)
except ValueError:
print("No prompt available for this item")
Use in grading:
for i, item in enumerate(dataset):
result = await rubric.grade(
to_grade=item.submission,
grader=grader,
query=dataset.get_item_prompt(i),
)
Reference Submissions¶
Provide exemplar responses for judge calibration:
# Global reference for all items
dataset = RubricDataset(
prompt="Explain photosynthesis",
rubric=rubric,
reference_submission="Detailed explanation of photosynthesis...",
)
# Per-item reference (overrides global)
dataset.add_item(
submission="Student answer...",
description="Q1",
reference_submission="Custom reference for this item",
)
# Get effective reference
ref = dataset.get_item_reference_submission(0)
Images in Dataset Files¶
A submission, prompt or reference submission can hold images, at the item level and at the
dataset level. In Python it is Content, which is a string, an Image, or a list of strings
and images read in order (a document, such as a figure followed by its caption):
from autorubric import Image
dataset.add_item(
submission=[Image("figures/fig1.png"), "Figure 1: accuracy by model size."],
description="Figure with caption",
)
A dataset file holds the same three shapes. A string is text. An image is an object whose only
key is "image", with the image's path as its value. A document is an array of strings and
image objects:
{
"submission": [{"image": "figures/fig1.png"}, "Figure 1: accuracy by model size."],
"description": "Figure with caption"
}
Paths are written with / on every platform. from_file resolves a relative path against the
directory of the dataset file, so the file and its figures can be moved together.
from_json(text, base_dir=...) resolves it against base_dir, or against the current working
directory when base_dir is not given. An absolute path is kept as it is. Loading opens no
image file, so a dataset whose images are missing still loads, and the missing file is
reported when its item is graded. Any other JSON value loads unchanged, as it did before
images existed, and is graded as it was then. A JSON object submission is read as the
{"thinking": ..., "output": ...} form, so an object with neither key, such as a mistyped
image object like {"img": "fig.png"}, grades as an empty submission and raises no error.
A number raises AttributeError when its item is graded. An array holding a number or an
object raises TypeError, as does a prompt or reference submission that is not content.
EvalRunner records each such failure against its own item and grades the others.
When writing, to_file stores each image path relative to the file's directory when the image
is inside it, and as an absolute path otherwise. to_json(base_dir=...) does the same relative
to base_dir and writes absolute paths when base_dir is not given. An image built from bytes
(Image(data)) has no path, so to_json refuses it with a ValueError that names the item and
field. to_file saves such an image beside the dataset file as
<stem>_images/<sha256>.<ext> (for results.json, the directory results_images/), writes the
file only if it is not there yet, and refers to it by that relative path. A dataset without
images is written exactly as before.
Train/Test Split¶
train_data, test_data = dataset.split_train_test(
n_train=100,
stratify=True, # Balance by ground truth verdicts
seed=42,
)
DataItem¶
A single item in an evaluation dataset.
DataItem
dataclass
¶
DataItem(submission: Content, description: str, ground_truth: list[CriterionVerdict | str] | None = None, rubric: Rubric | None = None, reference_submission: Content | None = None, prompt: Content | None = None, ground_truth_reasons: list[str | None] | None = None)
A single item to be graded, optionally with ground truth verdicts.
| ATTRIBUTE | DESCRIPTION |
|---|---|
submission |
The content to be evaluated (
TYPE:
|
description |
A brief description of this item (e.g., "High quality response").
TYPE:
|
ground_truth |
Optional list of ground truth values, one per criterion. - For binary criteria: CriterionVerdict (MET, UNMET, CANNOT_ASSESS) - For multi-choice criteria: str (option label) Used for computing evaluation metrics against LLM predictions.
TYPE:
|
rubric |
Optional per-item rubric. If provided, this rubric is used for grading
instead of the dataset-level rubric. Useful for datasets where each item
has unique evaluation criteria (e.g., ResearcherBench). It replaces the
dataset-level rubric entirely, guidelines included: this rubric carries its
own
TYPE:
|
reference_submission |
Optional exemplar response for grading context
(
TYPE:
|
prompt |
Optional per-item prompt (
TYPE:
|
ground_truth_reasons |
Optional written reasons for the ground truth, one per
criterion and aligned index-for-index with
TYPE:
|
Example
Binary criteria only¶
item = DataItem( ... submission="The Industrial Revolution began in Britain around 1760...", ... description="Excellent essay covering all criteria", ... ground_truth=[CriterionVerdict.MET, CriterionVerdict.MET, CriterionVerdict.UNMET] ... )
Mixed binary and multi-choice¶
item = DataItem( ... submission="The assistant responded helpfully...", ... description="Good dialogue", ... ground_truth=[CriterionVerdict.MET, "Very satisfied", "Yes - reasonable"] ... )
With written reasons for the ground truth (None where there is none)¶
item = DataItem( ... submission="The Industrial Revolution began in Britain around 1760...", ... description="Excellent essay", ... ground_truth=[CriterionVerdict.MET, CriterionVerdict.UNMET], ... ground_truth_reasons=["Dates the start and names Britain.", None], ... )
With per-item rubric¶
from autorubric import Rubric, Criterion item = DataItem( ... submission="Response to a specific question...", ... description="Question-specific grading", ... rubric=Rubric([Criterion(name="Relevance", weight=1.0, requirement="...")]) ... )
RubricDataset¶
Container for evaluation datasets with optional ground truth.
RubricDataset
dataclass
¶
RubricDataset(prompt: Content | None = None, rubric: Rubric | None = None, items: list[DataItem] = list(), name: str | None = None, reference_submission: Content | None = None)
A collection of DataItems tied to a specific prompt and rubric.
The RubricDataset encapsulates: - The prompt that generated the responses - The rubric used for evaluation (global or per-item) - A collection of DataItems with optional ground truth labels
This is useful for: - Evaluating LLM grader accuracy against human judgments - Training reward models with labeled data - Benchmarking different grading strategies
| ATTRIBUTE | DESCRIPTION |
|---|---|
prompt |
Optional global prompt/question that items are responses to
(
TYPE:
|
rubric |
Optional global Rubric used to evaluate items. Can be None if all items have their own rubrics.
TYPE:
|
items |
List of DataItem instances to evaluate.
TYPE:
|
name |
Optional name for the dataset (e.g., "essay-grading-v1").
TYPE:
|
reference_submission |
Optional global exemplar response for grading context
(
TYPE:
|
Example
from autorubric import Rubric, Criterion, CriterionVerdict rubric = Rubric([ ... Criterion(name="Accuracy", weight=10.0, requirement="Factually correct"), ... Criterion(name="Clarity", weight=5.0, requirement="Clear and concise"), ... ]) dataset = RubricDataset( ... prompt="Explain photosynthesis", ... rubric=rubric, ... ) dataset.add_item( ... submission="Photosynthesis is the process...", ... description="Good response", ... ground_truth=[CriterionVerdict.MET, CriterionVerdict.MET] ... )
criterion_names
property
¶
Get criterion names from global rubric.
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If no global rubric is set. |
num_criteria
property
¶
Number of criteria in the global rubric.
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If no global rubric is set. |
total_positive_weight
property
¶
Sum of all positive criterion weights in global rubric.
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If no global rubric is set. |
get_item_rubric
¶
get_item_rubric(idx: int) -> Rubric
Get the effective rubric for an item (per-item or global fallback).
| PARAMETER | DESCRIPTION |
|---|---|
idx
|
Index of the item.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Rubric
|
The item's rubric if set, otherwise the dataset's global rubric. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If neither item nor dataset has a rubric. |
Source code in src/autorubric/dataset.py
get_item_reference_submission
¶
get_item_reference_submission(idx: int) -> Content | None
Get the effective reference submission for an item.
Item-level reference takes precedence over dataset-level reference.
| PARAMETER | DESCRIPTION |
|---|---|
idx
|
Index of the item.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Content | None
|
The item's reference_submission if set, otherwise the dataset's |
Content | None
|
global reference_submission. May be None if neither is set. |
Source code in src/autorubric/dataset.py
get_item_prompt
¶
get_item_prompt(idx: int) -> Content
Get the effective prompt for an item (per-item or global fallback).
| PARAMETER | DESCRIPTION |
|---|---|
idx
|
Index of the item.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
Content
|
The item's prompt if set, otherwise the dataset's global prompt. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If neither item nor dataset has a prompt. |
Source code in src/autorubric/dataset.py
compute_weighted_score
¶
compute_weighted_score(verdicts: list[CriterionVerdict | str], normalize: bool = True, rubric: Rubric | None = None) -> float | None
Compute weighted score from verdicts (binary or multi-choice).
| PARAMETER | DESCRIPTION |
|---|---|
verdicts
|
List of verdict values, one per criterion. - For binary criteria: CriterionVerdict (MET=1.0, UNMET=0.0) - For multi-choice criteria: str (option label, resolved to value)
TYPE:
|
normalize
|
If True, normalize score to [0, 1]. If False, return raw sum.
TYPE:
|
rubric
|
Optional rubric to use for scoring. If None, uses global rubric.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
float | None
|
Weighted score based on criterion weights and verdicts, or |
float | None
|
every verdict abstains (CANNOT_ASSESS or an NA option), leaving nothing to |
float | None
|
score. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If a multi-choice label doesn't match any option, or if rubric is None and no global rubric is set. |
Source code in src/autorubric/dataset.py
add_item
¶
add_item(submission: Content, description: str, ground_truth: list[CriterionVerdict | str] | None = None, rubric: Rubric | None = None, reference_submission: Content | None = None, prompt: Content | None = None, *, ground_truth_reasons: list[str | None] | None = None) -> None
Add a new item to the dataset.
| PARAMETER | DESCRIPTION |
|---|---|
submission
|
The content to be evaluated.
TYPE:
|
description
|
A brief description of this item.
TYPE:
|
ground_truth
|
Optional list of ground truth values. - For binary criteria: CriterionVerdict (MET, UNMET, CANNOT_ASSESS) - For multi-choice criteria: str (option label)
TYPE:
|
rubric
|
Optional per-item rubric. If None, uses global rubric.
TYPE:
|
reference_submission
|
Optional exemplar response for grading context.
TYPE:
|
prompt
|
Optional per-item prompt. If None, uses global prompt.
TYPE:
|
ground_truth_reasons
|
Optional written reasons for the ground truth, one per
criterion and aligned with
TYPE:
|
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If ground_truth length doesn't match effective rubric criteria count,
or if neither per-item nor global rubric is available, or if neither
per-item nor global prompt is available, or if ground_truth_reasons is
not a list of |
Source code in src/autorubric/dataset.py
to_json
¶
Serialize the dataset to a JSON string.
A submission, prompt or reference submission is written as from_json reads it:
text as a string, an Image as {"image": "<path>"} and a document as an
array of both. An image's path is written with /, relative to base_dir when
its file is inside that directory (compared as written: symlinks and .. are not
resolved), else absolute; with no base_dir it is always absolute. No image file
is read. A dataset without images is written exactly as before images existed.
| PARAMETER | DESCRIPTION |
|---|---|
indent
|
Number of spaces for indentation. None for compact output.
TYPE:
|
base_dir
|
The directory image paths are written relative to, normally that of
the file the JSON is saved in (
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
str
|
JSON string representation of the dataset. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If an image is held as bytes, which has no path to write; the
message names the item and field. |
Source code in src/autorubric/dataset.py
to_file
¶
Save dataset to a JSON file.
Image paths are written relative to the file's directory when inside it, else
absolute (to_json with base_dir set to that directory), so from_file
still finds the images inside the directory after the directory is moved. An
image held as bytes is saved beside the file as <stem>_images/<digest>.<ext>
(<stem> the file name without its suffix, <digest> the image's SHA-256,
<ext> one of png, jpg, gif and webp), written only when no such
file exists yet, and referred to by that relative path. The JSON file is written
last. A dataset without images is written by self.to_json(), as before images
existed, so a subclass's to_json still writes it.
| PARAMETER | DESCRIPTION |
|---|---|
path
|
Path to write the JSON file.
TYPE:
|
Source code in src/autorubric/dataset.py
from_json
classmethod
¶
from_json(json_string: str, *, base_dir: str | Path | None = None) -> RubricDataset
Deserialize a dataset from a JSON string.
A submission, prompt or reference submission (of an item or of the dataset) is read
as content: a string is text; an image object, {"image": "<path>"} with
"image" its only key and a string value, is an Image; an array whose every
element is a string or an image object is a document, read as a list of its parts.
Any other value is kept as it is, so every file that loaded before images existed
still loads, and is graded as it was then (Grader.grade): an object is read as
the {thinking, output} form, so one with neither key grades as an empty
submission, while a number, or an array holding a part that is not content, raises
when its item is graded. An image path is written with /; a relative one is
resolved against base_dir, or the current working directory at the time of the
call when base_dir is None, and an absolute one is kept; the working directory is
consulted only for such a relative path (a relative base_dir included). No
image file is opened: a missing file fails when its item is graded.
| PARAMETER | DESCRIPTION |
|---|---|
json_string
|
JSON string representation of the dataset.
TYPE:
|
base_dir
|
The directory relative image paths are resolved against, normally
that of the file the JSON was read from (
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
RubricDataset
|
RubricDataset instance. |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If the JSON is invalid, missing required fields, or if
an item has no rubric when no global rubric is set, or if an item's
|
Source code in src/autorubric/dataset.py
694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 835 836 837 | |
from_file
classmethod
¶
from_file(path: str | Path) -> RubricDataset
Load dataset from a JSON file.
Relative image paths are resolved against the file's directory, not the current
working directory (from_json with base_dir set to it). A subclass whose
from_json takes no base_dir (one written before images existed) is called
as before, from_json(text), so its relative image paths resolve against the
working directory.
| PARAMETER | DESCRIPTION |
|---|---|
path
|
Path to the JSON file.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
RubricDataset
|
RubricDataset instance. |
| RAISES | DESCRIPTION |
|---|---|
FileNotFoundError
|
If the file doesn't exist. |
ValueError
|
If the JSON is invalid. |
Source code in src/autorubric/dataset.py
split_train_test
¶
split_train_test(n_train: int, *, stratify: bool = True, seed: int | None = None) -> tuple[RubricDataset, RubricDataset]
Split dataset into training and test sets.
The training set can be used to provide few-shot examples for grading, while the test set is used for evaluation.
| PARAMETER | DESCRIPTION |
|---|---|
n_train
|
Exact number of items for training set.
TYPE:
|
stratify
|
If True, stratify by per-criterion verdict distribution. This ensures each split has similar proportion of MET/UNMET/CANNOT_ASSESS for each criterion position. Requires all items to have ground_truth.
TYPE:
|
seed
|
Random seed for reproducible splits.
TYPE:
|
| RETURNS | DESCRIPTION |
|---|---|
tuple[RubricDataset, RubricDataset]
|
Tuple of (train_dataset, test_dataset). |
| RAISES | DESCRIPTION |
|---|---|
ValueError
|
If n_train is invalid or stratify=True but items lack ground_truth. |
Example
dataset = RubricDataset.from_file("data.json") train, test = dataset.split_train_test(n_train=100, stratify=True, seed=42) print(f"Train: {len(train)}, Test: {len(test)}")