AI Eval Labs
Hands-on builds in AI evaluation: grading, statistics, judges, robustness, and production eval pipelines.
| Lab | Title | Phase | Status |
|---|---|---|---|
| 00 | Setup and Reference Agent | Setup | Not started |
| 01 | Trajectory Grading Engine | Foundations | Not started |
| 02 | Statistical Significance Engine | Foundations | Not started |
| 03 | Calibrated LLM-as-a-Judge | Foundations | Not started |
| 04 | Dataset Contamination Checker | Foundations | Not started |
| 05 | RAG Adversarial Harness | Robustness and Security | Not started |
| 06 | Synthetic Edge-Case Generator | Robustness and Security | Not started |
| 07 | Agent Red-Team Fuzzer | Robustness and Security | Not started |
| 08 | Context Window Eviction Tester | Robustness and Security | Not started |
| 09 | CI/CD Regression Gate | Production Pipelines | Not started |
| 10 | Counterfactual Replay Debugger | Production Pipelines | Not started |
| 11 | Shadow Routing Comparator | Production Pipelines | Not started |
| 12 | Production Drift Monitor | Production Pipelines | Not started |
| 13 | Cost-Quality Pareto Dashboard | Production Pipelines | Not started |
| 14 | Automated DPO Flywheel | Production Pipelines | Not started |
| 15 | Public Eval Methodology Teardown | Publish | Not started |