Structured evaluation for LLM-generated content. Automated grading for human-authored content. Structured evaluation for non-verifiable domains.

AutoRubric brings together best practices from rubric science and LLM-as-a-Judge research, and grades with AI judges: LLMs, decision models, or both. Define criteria, validate against humans, and iterate on your rubrics with built-in meta-evaluation.

Get Started Cookbook Research

import asyncio
from autorubric import Rubric, LLMConfig, DecisionModelConfig
from autorubric.graders import CriterionGrader

async def main():
    grader = CriterionGrader(judge_model_config=LLMConfig(model="openai/gpt-5.6-luna"))
    # Or let a decision model grade the whole rubric in one request (pip install 'autorubric[typesafe]'):
    # grader = CriterionGrader(judge_model_config=DecisionModelConfig(model="jev-latest"))

    rubric = Rubric.from_dict([
        {"weight": 10.0, "requirement": "States NMC cell-level energy density in the 250-300 Wh/kg range"},
        {"weight": 8.0, "requirement": "Identifies LFP thermal runaway threshold (~270°C) as higher than NMC (~210°C)"},
        {"weight": 6.0, "requirement": "States LFP cycle life advantage (2000-5000 cycles vs 1000-2000 for NMC)"},
        {"weight": -15.0, "requirement": "Incorrectly claims LFP has higher gravimetric energy density than NMC"}
    ])

    result = await rubric.grade(
        to_grade="""NMC cathodes (LiNixMnyCozO2) achieve 250-280 Wh/kg at the cell level,
        while LFP (LiFePO4) typically reaches 150-205 Wh/kg. However, LFP offers superior
        thermal stability with decomposition onset at ~270°C compared to ~210°C for NMC,
        and delivers 2000-5000 charge cycles versus 1000-2000 for NMC.""",
        grader=grader,
        query="Compare NMC and LFP cathode materials for EV battery applications.",
    )

    print(f"Score: {result.score:.2f}")
    for criterion in result.report:
        print(f"  [{criterion.final_verdict}] {criterion.criterion.requirement}")

asyncio.run(main())

Why AutoRubric?

LLM judges are increasingly used to evaluate LLM outputs, but ad-hoc prompting leads to inconsistent, biased results. Research shows that LLM judges suffer from position bias, self-preference, and verbosity preference. Vague scoring criteria produce unreliable assessments.

AutoRubric codifies findings from educational measurement theory and LLM evaluation research: analytic rubrics with atomic criteria, multi-judge panels, explicit uncertainty handling, and validation against human judgments. It also provides meta-rubric evaluation to assess and improve your rubrics themselves.

AutoRubric grades with two kinds of AI judges. LLM judges take one criterion per call and explain each verdict. Decision models, such as TypeSafe's Jev, grade the whole rubric in one request and attach a probability to every verdict. Use either alone, mix them in one panel, or let a decision model take the first pass and escalate only its uncertain criteria to an LLM.

What AutoRubric Offers

Core

Weighted Criteria

Define rubrics with positive and negative weights. Binary verdicts or multi-choice scales (ordinal, nominal). Per-criterion explanations or probabilities.

Core

Decision-Model Judges

Grade a whole rubric in one request with TypeSafe's Jev or any System One endpoint. A probability and a confidence for every verdict.

Robustness

Ensemble Judging

Combine LLM and decision-model judges with voting strategies. Mitigate self-preference bias with cross-provider panels.

Robustness

Few-Shot Calibration

Calibrate LLM judges with labeled examples. Balanced sampling prevents verdict bias. Works with single or ensemble judges.

Robustness

Bias Mitigation

Position bias mitigation via option shuffling. CANNOT_ASSESS verdict for explicit uncertainty. Length penalties for verbosity.

Validation

Agreement Metrics

Cohen's kappa, accuracy, precision, recall, F1. Spearman, Kendall, and Pearson correlations. Bootstrap confidence intervals.

Validation

Distribution Analysis

Earth Mover's Distance and KS tests for score distributions. Systematic bias detection. Per-judge breakdowns.

Validation

Cascade Calibration

Sweep escalation thresholds offline from one decision-model run and one LLM run, with no extra calls. Per-criterion thresholds and held-out replay.

Meta-Evaluation

Meta-Rubric Feedback

Evaluate your rubrics for clarity, structure, and LLM-friendliness. Get actionable suggestions to improve criteria.

Operations

Batch Evaluation

High-throughput processing with rate limiting. Checkpointing and resumption. Timing stats and cost tracking.

Operations

Confidence Cascades

A decision model grades every criterion first. Only failed, abstained, or low-confidence criteria go to LLM judges.

Operations

100+ LLM Providers

OpenAI, Anthropic, Google, Azure, Groq, Ollama via LiteLLM. Extended thinking support. Response caching.

This research was developed with funding from the Defense Advanced Research Projects Agency's (DARPA) SciFy program (Agreement No. HR00112520300). The views expressed are those of the author and do not reflect the official policy or position of the Department of Defense or the U.S. Government.