Lab 10: Counterfactual Replay Debugger

Series: AI Eval Labs · Phase: Production Pipelines · Depends on: Lab 01

Core scope

• [ ] Record every LLM hop and tool output
• [ ] Swap one node and replay the rest of the trajectory
• [ ] Isolate the step that introduced a hallucination

Stretch

• [ ] Automated bisection across nodes

Links

• Code: series/ai-eval/10-replay-debugger
• Demo: coming soon
• Write-up: coming soon on markrox.ai