AutoRubric¶

A Python library for evaluating text outputs against weighted criteria using AI judges such as LLMs and decision models.

What is AutoRubric?¶

AutoRubric provides a structured, research-backed approach to evaluating LLM outputs using rubric-based grading with AI judges. Instead of relying on vague quality assessments, AutoRubric enables you to define explicit, weighted criteria and receive detailed per-criterion verdicts.

Judges come in two kinds. LLM judges take one criterion per call by default and explain each verdict — or, with llm_calls="per_item", the whole rubric in one call. Decision models, such as TypeSafe's Jev, always grade a whole rubric in one request and return probabilities instead of explanations. Use either alone, mix them in an ensemble, or let a decision model take the first pass and hand its uncertain criteria to an LLM.

Key Features¶

Rubric-based evaluation: Define weighted criteria with explicit requirements
LLM judges: Works with OpenAI, Anthropic, Google, Azure, Groq, Ollama, and 100+ providers via LiteLLM, with a written explanation for each verdict
Decision-model judges: Grade a whole rubric in one request with a probabilistic decision model such as TypeSafe's Jev, alone, in an ensemble, or as a cheap first pass with an LLM fallback
Ensemble judging: Combine multiple judges, LLMs and decision models alike, to reduce bias and improve robustness
Few-shot learning: Calibrate LLM judges with labeled examples
Multi-choice criteria: Support for ordinal and nominal scales beyond binary verdicts
Structured outputs: Type-safe responses with detailed per-criterion reports
Batch evaluation: High-throughput processing with checkpointing and resumption
Metrics & validation: Comprehensive agreement metrics and bootstrap confidence intervals

Installation¶

uvpip

uv add autorubric

pip install autorubric

Quick Example¶

import asyncio
from autorubric import Rubric, LLMConfig
from autorubric.graders import CriterionGrader

async def main():
    # Configure LLM judge
    grader = CriterionGrader(judge_model_config=LLMConfig(model="openai/gpt-4.1-mini"))

    # Define evaluation rubric
    rubric = Rubric.from_dict([
        {"weight": 10.0, "requirement": "States the correct answer"},
        {"weight": 5.0, "requirement": "Provides clear explanation"},
        {"weight": -10.0, "requirement": "Contains factual errors"}
    ])

    # Grade a response
    result = await rubric.grade(
        to_grade="The answer is 42. This is derived from...",
        grader=grader,
        query="What is the answer to life, the universe, and everything?",
    )

    # result.score is `float | None` (None if the grade failed or no criterion was left
    # to score); guard before formatting.
    print(f"Score: {result.score:.2f}" if result.score is not None else "Score: n/a")
    for criterion in result.report:
        print(f"  [{criterion.final_verdict}] {criterion.criterion.requirement}")

asyncio.run(main())

Why AutoRubric?¶

AutoRubric is built on research findings about effective LLM-as-a-judge evaluation:

Analytic rubrics: Multiple explicit criteria increase interpretability and help diagnose failure modes (Casabianca et al., 2025; Ye et al., 2023)
Ensemble judging: Multi-LLM panels reduce systematic bias and self-preference (Verga et al., 2024; He et al., 2025)
Structured evaluation: Form-filling with per-criterion verdicts improves repeatability (Kim et al., 2024)
Position bias mitigation: Randomized option order reduces positional effects (Wang et al., 2023)
CANNOT_ASSESS handling: Explicit uncertainty option prevents low-confidence guessing (Min et al., 2023)

Documentation¶

Guide	Description
Quickstart	Get up and running with installation, configuration, and your first evaluation
LLM Judges	Judges that explain each verdict, one call per criterion by default or the whole rubric in one call, from 100+ providers
Decision-Model Judges	Probabilistic judges that grade a whole rubric in one request, and confidence cascades
Cookbook	Practical examples and recipes for common evaluation scenarios
API Reference	Complete API documentation with all classes, functions, and types

Requirements¶

Python 3.11+
LiteLLM for multi-provider LLM support
Pydantic for structured outputs
Optional: the TypeSafe SDK for decision-model judges (pip install 'autorubric[typesafe]')

References¶

Casabianca, J., McCaffrey, D. F., Johnson, M. S., Alper, N., and Zubenko, V. (2025). Validity Arguments For Constructed Response Scoring Using Generative Artificial Intelligence Applications. arXiv:2501.02334.

He, J., Shi, J., Zhuo, T. Y., Treude, C., Sun, J., Xing, Z., Du, X., and Lo, D. (2025). LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead. arXiv:2510.24367.

Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., and Seo, M. (2024). Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4334–4353.

Min, S., Krishna, K., Lyu, X., Lewis, M., Yih, W., Koh, P. W., Iyyer, M., Zettlemoyer, L., and Hajishirzi, H. (2023). FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 12076–12100.

Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., and Lewis, P. (2024). Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv:2404.18796.

Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., and Sui, Z. (2023). Large Language Models are not Fair Evaluators. arXiv:2305.17926.

Ye, S., Kim, D., Kim, S., Hwang, H., Kim, S., Jo, Y., Thorne, J., Kim, J., and Seo, M. (2023). FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets. In Proceedings of the 2024 International Conference on Learning Representations (ICLR).

License¶

MIT License - see LICENSE for details.