02. RAG Benchmark: Can Retrieved Context Fix a Weak Model?#
What is RAG?#
Retrieval-augmented generation (RAG) is a two-stage pattern for getting accurate, grounded answers from a large language model:
Retrieve relevant documents from a corpus (the previous notebook showed how to do this with FAISS and semantic embeddings).
Generate an answer by injecting the retrieved text into the LLM prompt, so the model reads the source material instead of relying on whatever it memorized during training.
The key insight is that even a small, inexpensive model can produce accurate answers when you put the right evidence in front of it. Without that evidence, the model must guess from parametric memory – which is often stale, imprecise, or simply wrong for specific financial figures.
The experiment#
We test this claim with a controlled benchmark:
Ground truth comes from Compustat – the standard source for audited financial data (revenue, net income, total assets).
Baseline: ask
gpt-4o-minieach question with no context. The model must rely on whatever it learned during pre-training.RAG-enhanced: ask the same model the same question, but first retrieve the relevant 10-K filing from our WRDS SEC corpus and inject key sections (financial tables, MD&A) into the prompt.
Evaluate: compare each answer to the Compustat ground truth within 5% and 10% tolerance bands.
The data#
The SEC filings come from the WRDS SEC Analytics Suite, the same corpus indexed in notebook 01. For each question we know the company (CIK) and fiscal year end date, so we can look up the corresponding 10-K filing directly – no vector search needed here. We read the cleaned filing text and extract sections most likely to contain the answer (financial tables, results of operations, MD&A).
Setup#
import os
import re
import time
from pathlib import Path
import pandas as pd
from openai import OpenAI
from settings import config
DATA_DIR = Path(config("DATA_DIR"))
OUTPUT_DIR = Path(config("OUTPUT_DIR"))
client = OpenAI(api_key=config("OPENAI_API_KEY"))
MODEL_ID = "gpt-4o-mini"
MODEL_INPUT_PRICE = 0.15 # per million tokens
MODEL_OUTPUT_PRICE = 0.60
Helper functions#
def run_prompt(api_client, model_id, prompt, system_msg=None, max_tokens=300):
"""Send a prompt and capture the response, timing, and token usage."""
messages = []
if system_msg:
messages.append({"role": "system", "content": system_msg})
messages.append({"role": "user", "content": prompt})
start = time.time()
response = api_client.chat.completions.create(
model=model_id,
messages=messages,
max_tokens=max_tokens,
)
latency = time.time() - start
return {
"output": response.choices[0].message.content,
"latency_s": round(latency, 3),
"input_tokens": response.usage.prompt_tokens,
"output_tokens": response.usage.completion_tokens,
"total_tokens": response.usage.total_tokens,
}
def calculate_cost(input_tokens, output_tokens, input_price, output_price):
"""Cost in USD from token counts and per-million-token prices."""
return round(
(input_tokens * input_price + output_tokens * output_price) / 1_000_000, 6
)
def extract_number(text):
"""Extract a numeric value from LLM response.
Handles dollar signs, commas, and scale words (billion/million).
Returns value in millions to match Compustat units.
"""
if text is None:
return None
cleaned = text.replace(",", "").replace("$", "")
# Try "X billion" first (convert to millions)
m = re.search(r"([\d.]+)\s*billion", cleaned, re.IGNORECASE)
if m:
return float(m.group(1)) * 1000
# Try "X million"
m = re.search(r"([\d.]+)\s*million", cleaned, re.IGNORECASE)
if m:
return float(m.group(1))
# Fall back to last number in the text
numbers = re.findall(r"-?\d+\.?\d*", cleaned)
if numbers:
return float(numbers[-1])
return None
def is_correct(expected, extracted, tolerance=0.05):
"""Check if extracted value is within tolerance of expected."""
if expected is None or extracted is None:
return False
if expected == 0:
return abs(extracted) < 1 # close to zero
return abs(extracted - expected) / abs(expected) <= tolerance
Load Compustat ground truth#
To measure accuracy we need numbers we can trust. Compustat provides
audited annual financial data for public companies. We load revenue
(sale), net income (ni), and total assets (at) – all reported
in millions of USD.
from pull_CRSP_Compustat import load_compustat
comp = load_compustat(DATA_DIR)
# Keep only annual data with non-null key metrics
comp = comp.dropna(subset=["sale"])
print(f"Compustat sample: {len(comp)} company-years, {comp['conm'].nunique()} companies")
print(f"Date range: {comp['datadate'].min().date()} to {comp['datadate'].max().date()}")
comp[["conm", "year", "sale", "ni", "at"]].tail(10)
Compustat sample: 228 company-years, 19 companies
Date range: 2014-01-31 to 2026-01-31
| conm | year | sale | ni | at | |
|---|---|---|---|---|---|
| 218 | AMC ENTERTAINMENT HOLDINGS | 2016 | 3235.846 | 111.667 | 8641.841 |
| 219 | AMC ENTERTAINMENT HOLDINGS | 2017 | 5079.2 | -487.2 | 9805.9 |
| 220 | AMC ENTERTAINMENT HOLDINGS | 2018 | 5460.8 | 110.1 | 9495.8 |
| 221 | AMC ENTERTAINMENT HOLDINGS | 2019 | 5471.0 | -149.1 | 13675.8 |
| 222 | AMC ENTERTAINMENT HOLDINGS | 2020 | 1242.4 | -4589.1 | 10276.4 |
| 223 | AMC ENTERTAINMENT HOLDINGS | 2021 | 2527.9 | -1269.1 | 10821.5 |
| 224 | AMC ENTERTAINMENT HOLDINGS | 2022 | 3911.4 | -973.6 | 9135.6 |
| 225 | AMC ENTERTAINMENT HOLDINGS | 2023 | 4812.6 | -396.6 | 9009.2 |
| 226 | AMC ENTERTAINMENT HOLDINGS | 2024 | 4637.2 | -352.6 | 8247.5 |
| 227 | AMC ENTERTAINMENT HOLDINGS | 2025 | 4848.9 | -632.4 | 8017.8 |
Generate benchmark Q&A pairs#
We create one question per (company, year, metric) combination. Each question asks for a specific financial figure in plain English – the kind of question an analyst might ask. The expected answer comes directly from Compustat.
METRICS = {
"sale": "total revenue (net sales)",
"ni": "net income (loss)",
"at": "total assets",
}
# Use only recent fiscal years where we have 10-K filings on disk.
recent = comp[comp["year"] >= 2024].copy()
test_cases = []
for _, row in recent.iterrows():
for col, label in METRICS.items():
value = row[col]
if pd.isna(value):
continue
test_cases.append(
{
"company_name": row["conm"],
"cik": str(row["cik"]).strip(),
"fiscal_year": int(row["year"]),
"fiscal_year_end": row["datadate"].strftime("%Y-%m-%d"),
"metric": col,
"metric_label": label,
"expected": float(value),
"question": (
f"What was {row['conm']}'s {label} for the fiscal year "
f"ending {row['datadate'].strftime('%Y-%m-%d')}? "
f"Report a single number in millions of US dollars."
),
}
)
print(f"Generated {len(test_cases)} test cases")
pd.DataFrame(test_cases)[["company_name", "fiscal_year", "metric_label", "expected"]].head(10)
Generated 117 test cases
| company_name | fiscal_year | metric_label | expected | |
|---|---|---|---|---|
| 0 | APPLE INC | 2024 | total revenue (net sales) | 391035.0 |
| 1 | APPLE INC | 2024 | net income (loss) | 93736.0 |
| 2 | APPLE INC | 2024 | total assets | 364980.0 |
| 3 | APPLE INC | 2025 | total revenue (net sales) | 416161.0 |
| 4 | APPLE INC | 2025 | net income (loss) | 112010.0 |
| 5 | APPLE INC | 2025 | total assets | 359241.0 |
| 6 | BOEING CO | 2024 | total revenue (net sales) | 66960.0 |
| 7 | BOEING CO | 2024 | net income (loss) | -11817.0 |
| 8 | BOEING CO | 2024 | total assets | 156363.0 |
| 9 | BOEING CO | 2025 | total revenue (net sales) | 89463.0 |
Baseline evaluation (no context)#
First we establish the baseline: ask gpt-4o-mini each question with
no additional context. The model must rely entirely on whatever
financial data it absorbed during pre-training. For recent fiscal years
this is especially challenging – the model’s training data may not
include the latest annual reports.
SYSTEM_MSG = (
"You are a financial analyst. Answer the question with a single numeric "
"value in millions of US dollars. If you are uncertain, give your best "
"estimate. Do not refuse to answer."
)
baseline_results = []
for i, tc in enumerate(test_cases):
result = run_prompt(client, MODEL_ID, tc["question"], system_msg=SYSTEM_MSG)
extracted = extract_number(result["output"])
correct_5 = is_correct(tc["expected"], extracted, tolerance=0.05)
correct_10 = is_correct(tc["expected"], extracted, tolerance=0.10)
baseline_results.append(
{
**tc,
"answer": result["output"],
"extracted": extracted,
"correct_5pct": correct_5,
"correct_10pct": correct_10,
"latency_s": result["latency_s"],
"input_tokens": result["input_tokens"],
"output_tokens": result["output_tokens"],
}
)
if (i + 1) % 10 == 0:
print(f" Baseline: {i + 1}/{len(test_cases)} done")
df_baseline = pd.DataFrame(baseline_results)
n = len(df_baseline)
print(f"\nBaseline accuracy (5%): {df_baseline['correct_5pct'].sum()}/{n} ({df_baseline['correct_5pct'].mean():.1%})")
print(f"Baseline accuracy (10%): {df_baseline['correct_10pct'].sum()}/{n} ({df_baseline['correct_10pct'].mean():.1%})")
Baseline: 10/117 done
Baseline: 20/117 done
Baseline: 30/117 done
Baseline: 40/117 done
Baseline: 50/117 done
Baseline: 60/117 done
Baseline: 70/117 done
Baseline: 80/117 done
Baseline: 90/117 done
Baseline: 100/117 done
Baseline: 110/117 done
Baseline accuracy (5%): 10/117 (8.5%)
Baseline accuracy (10%): 23/117 (19.7%)
Build filing lookup for RAG#
This is the retrieval step. For each question we know the company (CIK) and fiscal year end date. We look up the corresponding 10-K filing from the WRDS metadata index, read the cleaned full text, and extract the sections most likely to contain the answer – financial tables, MD&A, and results of operations.
The retrieval here is structured lookup (match on CIK + date) rather than vector search. In production RAG systems you would typically use the FAISS-based semantic search from notebook 01. We use direct lookup here so the benchmark isolates the effect of having context from the quality of the retrieval itself.
from pull_sec_filings_raw import edgar_to_wrds_path
# Load filing metadata
filing_idx = pd.read_parquet(DATA_DIR / "sec_filings_metadata" / "filing_index.parquet")
filing_idx["fdate"] = pd.to_datetime(filing_idx["fdate"])
tenk_idx = filing_idx[filing_idx["form"] == "10-K"].copy()
print(f"10-K filings in index: {len(tenk_idx)}")
filings_dir = DATA_DIR / "wrds_clean_filings"
def extract_financial_context(text, max_words=3000):
"""Extract financial context from a 10-K filing.
Uses two strategies:
1. Find table rows with key financial figures (revenue, income, assets)
and include surrounding lines for context.
2. Find narrative paragraphs discussing financial results.
"""
keywords = [
"net sales", "total net revenue", "total revenue",
"net income", "net loss", "net earnings",
"total assets",
"results of operations",
"operating income", "gross margin", "gross profit",
]
lines = text.split("\n")
context_parts = []
total_words = 0
seen = set()
# Strategy 1: Find lines with key financial table data and grab context
for i, line in enumerate(lines):
lower = line.lower().strip()
if not lower:
continue
# Look for lines like "Total net sales $ 391,035"
has_keyword = any(kw in lower for kw in keywords)
has_number = bool(re.search(r"[\d,]{3,}", line))
if has_keyword and has_number:
# Grab a window of lines around this match
start = max(0, i - 5)
end = min(len(lines), i + 6)
chunk = "\n".join(lines[start:end]).strip()
chunk_key = chunk[:100]
if chunk_key in seen:
continue
seen.add(chunk_key)
words = chunk.split()
if total_words + len(words) > max_words:
continue
context_parts.append(chunk)
total_words += len(words)
# Strategy 2: Narrative paragraphs with financial discussion
paras = [p.strip() for p in text.split("\n\n") if p.strip()]
for para in paras:
words = para.split()
if len(words) < 20 or len(words) > 500:
continue
lower = para.lower()
hits = sum(1 for kw in keywords if kw in lower)
has_dollar = bool(re.search(r"\$\s*[\d,]+", para))
if hits >= 2 and has_dollar:
para_key = para[:100]
if para_key in seen:
continue
seen.add(para_key)
if total_words + len(words) > max_words:
continue
context_parts.append(para)
total_words += len(words)
return "\n\n---\n\n".join(context_parts)
def find_10k_filing(cik, fiscal_year_end):
"""Find the 10-K filing for a given company and fiscal year end date.
Returns the filing context or None if not found.
"""
fy_end = pd.Timestamp(fiscal_year_end)
# Find 10-Ks for this CIK filed within 6 months after fiscal year end
cik_filings = tenk_idx[tenk_idx["cik"] == cik].copy()
cik_filings = cik_filings[
(cik_filings["fdate"] >= fy_end)
& (cik_filings["fdate"] <= fy_end + pd.Timedelta(days=180))
]
if cik_filings.empty:
return None
# Take the earliest filing after FY end (the actual 10-K for that year)
filing = cik_filings.sort_values("fdate").iloc[0]
wrds_relpath = edgar_to_wrds_path(filing["fname"])
filepath = filings_dir / wrds_relpath
if not filepath.exists():
return None
text = filepath.read_text(encoding="utf-8", errors="ignore")
context = extract_financial_context(text)
if not context:
return None
return {
"context": context,
"company_name": filing["coname"],
"filing_date": str(filing["fdate"].date()),
}
# Quick test
test_result = find_10k_filing("0000320193", "2024-09-30")
if test_result:
print(f"Apple 10-K found: filed {test_result['filing_date']}, "
f"context length: {len(test_result['context'])} chars")
10-K filings in index: 239
Apple 10-K found: filed 2024-11-01, context length: 19465 chars
RAG prompt construction#
The prompt template is straightforward: we prepend the retrieved filing excerpts to the question and instruct the model to extract numbers from the provided text. This is the core of RAG – the model no longer needs to know the answer; it just needs to read it.
def build_rag_prompt(question, filing_context):
"""Build a prompt with retrieved SEC filing context."""
if filing_context is None:
return question
source = f"{filing_context['company_name']}, 10-K, filed {filing_context['filing_date']}"
return (
"Use the following SEC filing excerpts to answer the question. "
"Extract the specific numbers from the excerpts.\n\n"
f"SEC FILING EXCERPTS (Source: {source}):\n"
f"{filing_context['context']}\n\n"
f"QUESTION: {question}\n\n"
"Answer with a single number in millions of US dollars."
)
RAG-enhanced evaluation#
Now we repeat the same questions, but this time each prompt includes the retrieved SEC filing excerpts. The model, the system message, and the questions are identical – the only difference is the added context. Any improvement in accuracy is attributable to RAG.
rag_results = []
filings_found = 0
for i, tc in enumerate(test_cases):
filing_context = find_10k_filing(tc["cik"], tc["fiscal_year_end"])
if filing_context:
filings_found += 1
rag_prompt = build_rag_prompt(tc["question"], filing_context)
result = run_prompt(client, MODEL_ID, rag_prompt, system_msg=SYSTEM_MSG)
extracted = extract_number(result["output"])
correct_5 = is_correct(tc["expected"], extracted, tolerance=0.05)
correct_10 = is_correct(tc["expected"], extracted, tolerance=0.10)
rag_results.append(
{
**tc,
"answer": result["output"],
"extracted": extracted,
"correct_5pct": correct_5,
"correct_10pct": correct_10,
"latency_s": result["latency_s"],
"input_tokens": result["input_tokens"],
"output_tokens": result["output_tokens"],
"filing_found": filing_context is not None,
}
)
if (i + 1) % 10 == 0:
print(f" RAG: {i + 1}/{len(test_cases)} done")
df_rag = pd.DataFrame(rag_results)
n = len(df_rag)
print(f"\nFilings found: {filings_found}/{n}")
print(f"RAG accuracy (5%): {df_rag['correct_5pct'].sum()}/{n} ({df_rag['correct_5pct'].mean():.1%})")
print(f"RAG accuracy (10%): {df_rag['correct_10pct'].sum()}/{n} ({df_rag['correct_10pct'].mean():.1%})")
RAG: 10/117 done
RAG: 20/117 done
RAG: 30/117 done
RAG: 40/117 done
RAG: 50/117 done
RAG: 60/117 done
RAG: 70/117 done
RAG: 80/117 done
RAG: 90/117 done
RAG: 100/117 done
RAG: 110/117 done
Filings found: 90/117
RAG accuracy (5%): 71/117 (60.7%)
RAG accuracy (10%): 76/117 (65.0%)
Results comparison#
print("=" * 60)
print("RAG BENCHMARK RESULTS")
print("=" * 60)
baseline_cost = sum(
calculate_cost(r["input_tokens"], r["output_tokens"], MODEL_INPUT_PRICE, MODEL_OUTPUT_PRICE)
for r in baseline_results
)
rag_cost = sum(
calculate_cost(r["input_tokens"], r["output_tokens"], MODEL_INPUT_PRICE, MODEL_OUTPUT_PRICE)
for r in rag_results
)
summary = pd.DataFrame(
{
"Metric": [
"Accuracy (5% tolerance)",
"Accuracy (10% tolerance)",
"Avg latency (s)",
"Total cost (USD)",
],
"Baseline": [
f"{df_baseline['correct_5pct'].mean():.1%}",
f"{df_baseline['correct_10pct'].mean():.1%}",
f"{df_baseline['latency_s'].mean():.2f}",
f"${baseline_cost:.4f}",
],
"RAG": [
f"{df_rag['correct_5pct'].mean():.1%}",
f"{df_rag['correct_10pct'].mean():.1%}",
f"{df_rag['latency_s'].mean():.2f}",
f"${rag_cost:.4f}",
],
}
)
print(summary.to_string(index=False))
============================================================
RAG BENCHMARK RESULTS
============================================================
Metric Baseline RAG
Accuracy (5% tolerance) 8.5% 60.7%
Accuracy (10% tolerance) 19.7% 65.0%
Avg latency (s) 1.35 1.08
Total cost (USD) $0.0044 $0.0604
# Per-question breakdown
comparison = pd.DataFrame(
{
"Company": [tc["company_name"] for tc in test_cases],
"Year": [tc["fiscal_year"] for tc in test_cases],
"Metric": [tc["metric_label"] for tc in test_cases],
"Expected": [tc["expected"] for tc in test_cases],
"Baseline": df_baseline["extracted"].values,
"Base_OK": df_baseline["correct_10pct"].values,
"RAG": df_rag["extracted"].values,
"RAG_OK": df_rag["correct_10pct"].values,
}
)
comparison.to_csv(OUTPUT_DIR / "rag_benchmark_results.csv", index=False)
print(f"\nSaved detailed results to {OUTPUT_DIR / 'rag_benchmark_results.csv'}")
comparison
Saved detailed results to /Users/jbejarano/GitRepositories/finm-32900-ALL/case_study_sec_filings/_output/rag_benchmark_results.csv
| Company | Year | Metric | Expected | Baseline | Base_OK | RAG | RAG_OK | |
|---|---|---|---|---|---|---|---|---|
| 0 | APPLE INC | 2024 | total revenue (net sales) | 391035.0 | 400000.0 | True | 391035.0 | True |
| 1 | APPLE INC | 2024 | net income (loss) | 93736.0 | 2024.0 | False | 93736.0 | True |
| 2 | APPLE INC | 2024 | total assets | 364980.0 | 2023.0 | False | 364980.0 | True |
| 3 | APPLE INC | 2025 | total revenue (net sales) | 416161.0 | 400000.0 | True | 416161.0 | True |
| 4 | APPLE INC | 2025 | net income (loss) | 112010.0 | 2023.0 | False | 94695.0 | False |
| ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 112 | AMC ENTERTAINMENT HOLDINGS | 2024 | net income (loss) | -352.6 | 100.0 | False | -352.6 | True |
| 113 | AMC ENTERTAINMENT HOLDINGS | 2024 | total assets | 8247.5 | 10000.0 | False | 8247.5 | True |
| 114 | AMC ENTERTAINMENT HOLDINGS | 2025 | total revenue (net sales) | 4848.9 | 2023.0 | False | 5000.0 | True |
| 115 | AMC ENTERTAINMENT HOLDINGS | 2025 | net income (loss) | -632.4 | 200.0 | False | 40.0 | False |
| 116 | AMC ENTERTAINMENT HOLDINGS | 2025 | total assets | 8017.8 | 5000.0 | False | 6000.0 | False |
117 rows × 8 columns
Takeaways#
Without context, the model must rely on memorized (and often stale or imprecise) financial data – resulting in frequent hallucinations. With RAG, the same model receives the actual SEC filing text containing the reported figures and can extract them accurately.
This is the core value proposition of retrieval-augmented generation: ground the model in source documents so it doesn’t have to guess. A cheap, fast model with good retrieval beats an expensive model working from memory alone.
Connecting the two notebooks#
Together, these notebooks illustrate the full RAG pipeline:
Notebook 01 built a semantic search index over SEC filings using vector embeddings and FAISS, and showed how to query it with natural-language questions.
Notebook 02 (this one) demonstrated why retrieval matters: the same LLM that hallucinates without context becomes accurate when we inject the right source documents.
In a production system, the retrieval step from notebook 01 would feed directly into the generation step from notebook 02 – the query identifies the most relevant filing passages, and those passages become the context window for the LLM.