Homework 4: Agentic RAG Trace Audit

Contents

Homework 4: Agentic RAG Trace Audit#

To get started with this homework, accept the GitHub Classroom assignment and clone the repository found above. All detailed instructions, deliverables, and grading criteria are in the homework repository README.

This assignment uses Matt Stockton’s guest-speaker repository, agentic-rag-edgar-demo, to study agentic retrieval, tool choice, and trace evaluation on SEC 10-K filings. Where HW3 measured whether RAG improved factual extraction against Compustat ground truth, HW4 asks a different question: when an LLM is allowed to choose retrieval tools at runtime, what does the trace show about the strategy it used, the tools it needed, and the places where simpler methods win?

The deliverable is short: one trace inventory, one evidence-based audit of a single phase, one small patch with a before/after trace comparison, and a brief reflection plus a golden-set entry. Grading is manual — there is no autograded check on push, so review the “What To Submit” section in the homework README yourself before the deadline.

Prerequisites#

  • Python 3.11+ (Matt’s repo pins 3.12)

  • An LLM API key (OpenAI by default; OpenRouter or Anthropic acceptable with a documented setup-time provider/model swap applied uniformly across all four phase scripts)

  • An SEC identity string for EDGAR requests

  • ~700 MB of local disk for cached filings, the BAAI/bge-base-en-v1.5 embedding model, chunks, and embeddings

See the homework README for setup, the four phase scripts, the trace-helpers CLI, and the full submission checklist.

Contents