Lab 03: Calibrated LLM-as-a-Judge

Series: AI Eval Labs · Phase: Foundations · Depends on: Lab 01, Lab 02

Core scope

• [ ] Human-labeled anchor set of ~100 items
• [ ] Measure verbosity, position, and self-preference bias
• [ ] Document judge agreement vs. human labels (Cohen's kappa)

Stretch

• [ ] Grow anchor set to 500 items

Links

• Code: series/ai-eval/03-llm-judge
• Demo: coming soon
• Write-up: coming soon on markrox.ai