Cookbook

Practical recipes for evaluating text outputs with AutoRubric. Each recipe solves a specific real-world scenario with focused code snippets and complete runnable examples.

AutoRubric grades with two kinds of AI judges. LLM judges take one criterion per call by default and explain each verdict — or, with llm_calls="per_item", the whole rubric in one call. Decision models, such as TypeSafe's Jev, always grade a whole rubric in one request and return probabilities instead of explanations. Both plug into the same CriterionGrader, and you can use either alone, mix them in an ensemble, or let a decision model take the first pass and hand its uncertain criteria to an LLM.

Recipe Index

Tier 1: Foundation

Start here if you're new to AutoRubric. LLM Judges and Decision-Model Judges introduce the two kinds of judges, Cheap First-Pass Grading combines them, and Grading a Whole Rubric in One Call cuts an LLM judge to one call per item.

Recipe	Domain	What You'll Learn
Your First Evaluation	Tech Support	Basic rubric creation and grading
LLM Judges	—	Configuring, steering, and combining LLM judges; choosing a judge kind
Decision-Model Judges	—	One-request judges with probabilities, mixed ensembles, and confidence cascades
Managing Datasets	Medical Triage	Loading, saving, and splitting datasets
Working with Explanations	Essay Feedback	Accessing and formatting per-criterion reasons
Cheap First-Pass Grading with Jev and an LLM Fallback	Short-Answer Grading	Calibrating and deploying a decision-model cascade
Grading a Whole Rubric in One Call	Long-Answer Grading	Switching an LLM judge to one call per item, and validating the switch on labelled data

Tier 2: Reliability

Improve grading consistency and accuracy.

Recipe	Domain	What You'll Learn
Few-Shot Calibration	Legal Contracts	Calibrating judges with labeled examples
Ensemble Judging	Job Applications	Multi-judge voting for high-stakes decisions
Handling CANNOT_ASSESS	RAG Responses	Strategies for uncertain verdicts
Fixing Seeds	Product Reviews	Pinning shuffles and few-shot selection for reproducibility

Tier 3: Advanced Evaluation

Sophisticated evaluation techniques.

Recipe	Domain	What You'll Learn
Multi-Choice Rubrics	Restaurant Reviews	Ordinal/nominal scales with Likert ratings
Extended Thinking	Security Assessments	Deep reasoning for complex evaluations
Length Penalty	Executive Summaries	Penalizing verbose responses

Tier 4: Validation & Production

Deploy with confidence.

Recipe	Domain	What You'll Learn
Evaluating Rubric Quality	Peer Review	Meta-rubrics to validate and improve rubrics
Automated Rubric Improvement	EV Analysis	LLM-driven iterative refinement of rubrics
Held-Out Rubric Improvement	Peer Review	Data-driven criterion refinement using grading errors
Judge Validation	Content Moderation	Measuring agreement with human labels
Synthetic Ground Truth	Product Descriptions	Bootstrapping labels from strong models
Batch Evaluation	Customer Feedback	Checkpointing, resumption, and cost tracking

Tier 5: Specialized

Advanced patterns for specific needs.

Recipe	Domain	What You'll Learn
Per-Item Rubrics	Coding Interviews	Different rubrics for different items
Cost Optimization	News Fact-Checking	Caching and model selection strategies
Configuration Management	Academic Papers	Sharing reproducible configs across teams
Evaluating Agent Skills	Peer Review	Skill evaluation with with/without-skill comparison
Improving Agent Skills	Peer Review	Automated skill refinement using rubric feedback

Quick Start

If you haven't installed AutoRubric yet:

pip install autorubric

Set up your API key for your preferred LLM provider:

export OPENAI_API_KEY=your_key_here
# or
export ANTHROPIC_API_KEY=your_key_here
# or
export GEMINI_API_KEY=your_key_here

For decision-model judges, install the TypeSafe extra and set its key:

pip install 'autorubric[typesafe]'
export TYPESAFE_API_KEY=your_key_here

Then jump into Your First Evaluation to get started.

Recipe Format

Each recipe follows a consistent structure:

1. The Scenario - A realistic problem you might face
2. What You'll Learn - Key features and concepts covered
3. The Solution - Step-by-step implementation with focused code snippets
4. Key Takeaways - Summary of important points
5. Appendix: Complete Code - Full runnable script you can copy-paste

The two judge chapters in Tier 1, LLM Judges and Decision-Model Judges, are guides rather than recipes: they explain how each kind of judge works and how to configure it.

Prerequisites

All recipes assume:

• Python 3.11+
• AutoRubric installed (pip install autorubric)
• An API key for at least one supported LLM provider (recipes that use a decision model also need a TypeSafe API key)
• Basic familiarity with async/await (recipes use asyncio.run() for simplicity)