Lab 03: Decision Model Bake-off
Series: System One · Phase: Model Evaluation · Depends on: Lab 00
Core scope
- [ ] Evaluate Nimble and at least one other Jev-compatible model on the same labeled tasks
- [ ] Measure decision quality, probability calibration, review coverage, latency, and memory
- [ ] Validate returned probabilities against held-out labels; do not treat choice confidence as probability of correctness
- [ ] Record hardware and model versions for reproducible comparisons
Stretch
- [ ] Compare multimodal models on public or synthetic image and audio tasks
Links
- Code:
series/system-one/03-decision-model-bakeoff - Demo: coming soon
- Write-up: coming soon on markrox.ai