Skip to content

Lab 03: Decision Model Bake-off

Series: System One · Phase: Model Evaluation · Depends on: Lab 00

Core scope

  • [ ] Evaluate Nimble and at least one other Jev-compatible model on the same labeled tasks
  • [ ] Measure decision quality, probability calibration, review coverage, latency, and memory
  • [ ] Validate returned probabilities against held-out labels; do not treat choice confidence as probability of correctness
  • [ ] Record hardware and model versions for reproducible comparisons

Stretch

  • [ ] Compare multimodal models on public or synthetic image and audio tasks