Skip to content

Lab 00: Setup and Reference Agent

Series: AI Eval Labs · Phase: Setup · Depends on: None

Core scope

  • [ ] Repo, issues, labs.markrox.ai, and #dsi-ai-labs (done)
  • [ ] Model registry (models.yaml) with mock, local, and hosted entries behind one client
  • [ ] Research corpus of ~40 public-domain US federal AI documents, chunked and indexed
  • [ ] Research assistant agent with 4 tools: search_corpus, read_document, list_sections, submit_answer
  • [ ] Task suite of ~30 tasks with expected tool DAG, answer, and citations
  • [ ] Trace recorder (JSONL) and a run CLI; tests pass offline with the mock backend

Stretch

  • [ ] Embedding retrieval alongside BM25
  • [ ] Second model run end to end for a first comparison