Benchmarking RAG with SEC Filings

Benchmarking RAG with SEC Filings#

This is the third notebook in the series. Notebook 01 inventoried the data; notebook 02 walked through the retrieval and grading code in src/rag_benchmark.py and src/embeddings_index.py with small runnable examples. This notebook runs the full grid of questions and reports the aggregate numbers.

What is RAG?#

Retrieval-Augmented Generation (RAG) is a two-stage pattern for getting accurate, grounded answers from a large language model:

  1. Retrieve relevant documents from a corpus. We benchmark two flavors of retrieval here:

    • Structured lookup – the question carries a CIK and fiscal year end, so we look up the matching 10-K directly and run a keyword chunker over it.

    • Dense semantic search – chunk the corpus once, embed every chunk with text-embedding-3-small, and query a global FAISS index at runtime. This is what most people mean by “classical RAG.”

  2. Generate an answer by injecting the retrieved text into the LLM prompt, so the model reads the source material instead of relying on whatever it memorized during training.

The key insight is that even a small, inexpensive model can produce accurate answers when you put the right evidence in front of it. Without that evidence, the model must guess from parametric memory – which is often stale, imprecise, or simply wrong for specific financial figures.

The experiment#

The WRDS SEC Analytics Suite publishes every filing in two forms:

  • /wrds/sec/warchives/... – the raw submission as filed to EDGAR, with SGML/HTML/XBRL markup intact.

  • /wrds/sec/wrds_clean_filings/... – the same filing after WRDS’s cleaning pipeline strips markup and normalizes whitespace.

We compare four conditions against the Compustat ground truth:

  1. Ground truth comes from Compustat (revenue, net income, total assets), as introduced in notebook 01.

  2. Baseline: ask gpt-4o-mini each question with no context. The model must rely on whatever it learned during pre-training.

  3. RAG (cleaned): structured filing lookup against WRDS’s cleaned corpus, then a keyword-based chunk extractor.

  4. RAG (raw): same structured lookup, same keyword extractor, but pointed at the raw archive. The only difference is whether WRDS’s cleaning pipeline was applied to the input text.

  5. RAG (semantic, cleaned): classical dense retrieval. The 48 cleaned filings are chunked and embedded into one global FAISS index. At query time we embed the question, take the top-k nearest chunks across the entire corpus, and inject them. There is no CIK/year prefilter – the retriever has to surface the right company on its own.

  6. Evaluate: compare each answer to the Compustat ground truth within 5% and 10% tolerance bands.

Two questions fall out of this design:

  • Cleaned vs raw (conditions 3 vs 4): does WRDS’s cleaning pipeline actually earn its keep?

  • Structured vs semantic (conditions 3 vs 5): when the join key is exact, what does swapping it for a vector search cost or buy?

Where the numbers come from#

The API calls happen in the doit task run_rag_benchmark (src/rag_benchmark.py), not in this notebook. The cleaned-corpus embeddings index is built once by the upstream task build_embeddings (src/embeddings_index.py) and cached on disk. This notebook just loads the four per-question CSVs and summarizes them.

from pathlib import Path

import pandas as pd

from settings import config
from rag_benchmark import (
    MODEL_ID,
    MODEL_INPUT_PRICE,
    MODEL_OUTPUT_PRICE,
    calculate_cost,
)

DATA_DIR = Path(config("DATA_DIR"))
OUTPUT_DIR = Path(config("OUTPUT_DIR"))

Load benchmark results#

Each CSV has one row per (company, fiscal year, metric) question.

df_baseline = pd.read_csv(OUTPUT_DIR / "baseline_results.csv")
df_clean = pd.read_csv(OUTPUT_DIR / "rag_cleaned_results.csv")
df_raw = pd.read_csv(OUTPUT_DIR / "rag_raw_results.csv")
df_semantic = pd.read_csv(OUTPUT_DIR / "rag_semantic_cleaned_results.csv")

print(f"Questions per condition: {len(df_baseline)}")
print(f"Model: {MODEL_ID}")
df_baseline[["company_name", "fiscal_year", "metric_label", "expected"]].head()

Questions per condition: 96
Model: gpt-4o-mini
company_name fiscal_year metric_label expected
0 APPLE INC 2023 total revenue (net sales) 383285.0
1 APPLE INC 2023 net income (loss) 96995.0
2 APPLE INC 2023 total assets 352583.0
3 APPLE INC 2024 total revenue (net sales) 391035.0
4 APPLE INC 2024 net income (loss) 93736.0

Headline accuracy comparison#

This is the main scoreboard. Each row is one of the three conditions (baseline, RAG with cleaned filings, RAG with raw filings) and each column summarizes one of the per-question fields the benchmark script wrote to the CSVs.

Accuracy. For each question, run_baseline / run_rag in src/rag_benchmark.py does three things: it sends the prompt to gpt-4o-mini, runs the reply through extract_number (which pulls a single value in millions out of free-text answers like “12.3 billion” or “12,345 million”), and then calls is_correct to compare that number against the Compustat ground truth. is_correct is just relative error: |extracted - expected| / |expected| <= tolerance. We log two tolerances per question:

  • correct_5pct – the strict band. Treats the model as right only if its number is within 5 percent of Compustat.

  • correct_10pct – the lenient band, within 10 percent. 10-Ks and Compustat can disagree at the low-single-digit level for harmless reasons (rounding, slightly different line-item definitions), so the 10 percent band is the more forgiving read of “did the model find the right figure.”

The accuracy_5pct and accuracy_10pct columns in the table are just the means of those boolean columns – the share of questions the condition got right at each tolerance.

Latency and tokens. avg_latency_s is the average wall-clock round trip to the OpenAI API per question, as recorded inside run_prompt. avg_input_tokens and avg_output_tokens are the averages of response.usage.prompt_tokens and completion_tokens – what OpenAI actually billed us for. Input tokens scale with the size of the prompt, so the RAG rows are much larger than baseline (the prompt now carries 10-K excerpts), and the raw row is typically larger than the cleaned row because the raw chunks still contain SGML/HTML/XBRL markup.

Cost. total_cost_usd runs every question’s token counts through calculate_cost at the published gpt-4o-mini rates (0.15 USD per million input tokens, 0.60 USD per million output) and sums them. It is the dollar amount this condition cost to run end-to-end.

Read the table top-to-bottom: baseline shows what the model knows from pre-training alone, the cleaned-RAG row shows what cleaning plus retrieval buys you, and the raw-RAG row shows what happens when you skip the cleaning step.

def summarize(df, name):
    return {
        "condition": name,
        "n": len(df),
        "accuracy_5pct": df["correct_5pct"].mean(),
        "accuracy_10pct": df["correct_10pct"].mean(),
        "avg_latency_s": df["latency_s"].mean(),
        "avg_input_tokens": df["input_tokens"].mean(),
        "avg_output_tokens": df["output_tokens"].mean(),
        "total_cost_usd": sum(
            calculate_cost(r["input_tokens"], r["output_tokens"])
            for _, r in df.iterrows()
        ),
    }


summary = pd.DataFrame([
    summarize(df_baseline, "baseline (no context)"),
    summarize(df_clean,    "RAG (cleaned, structured)"),
    summarize(df_raw,      "RAG (raw, structured)"),
    summarize(df_semantic, "RAG (cleaned, semantic)"),
])
summary_display = summary.copy()
summary_display["accuracy_5pct"] = (summary_display["accuracy_5pct"] * 100).round(1).astype(str) + "%"
summary_display["accuracy_10pct"] = (summary_display["accuracy_10pct"] * 100).round(1).astype(str) + "%"
summary_display["avg_latency_s"] = summary_display["avg_latency_s"].round(2)
summary_display["avg_input_tokens"] = summary_display["avg_input_tokens"].round(0).astype(int)
summary_display["avg_output_tokens"] = summary_display["avg_output_tokens"].round(0).astype(int)
summary_display["total_cost_usd"] = summary_display["total_cost_usd"].round(4)
summary_display
condition n accuracy_5pct accuracy_10pct avg_latency_s avg_input_tokens avg_output_tokens total_cost_usd
0 baseline (no context) 96 22.9% 32.3% 1.56 81 35 0.0032
1 RAG (cleaned, structured) 96 81.2% 82.3% 1.61 4460 4 0.0644
2 RAG (raw, structured) 96 13.5% 21.9% 5.68 20051 4 0.2890
3 RAG (cleaned, semantic) 96 80.2% 82.3% 2.59 5797 3 0.0836

Filing-retrieval coverage#

RAG only helps when the retrieval step lands on the right 10-K. “Coverage” means slightly different things depending on the retrieval flavor:

  • Structured conditions (RAG cleaned, RAG raw): we either find the matching filing on disk or we don’t. If we don’t, we fall back to the bare question and the row degrades to the baseline.

  • Semantic condition (RAG semantic, cleaned): every query returns ten chunks, but those chunks may come from the wrong company. We report the share of questions where the top-1 chunk came from the right CIK – a weaker, soft notion of “found the filing.” Even when this is below 100%, the prompt may still contain useful chunks lower in the top-k.

coverage = pd.DataFrame([
    {
        "condition": "RAG (cleaned, structured)",
        "metric": "filing_found",
        "hits": int(df_clean["filing_found"].sum()),
        "total": len(df_clean),
        "rate": f"{df_clean['filing_found'].mean():.1%}",
    },
    {
        "condition": "RAG (raw, structured)",
        "metric": "filing_found",
        "hits": int(df_raw["filing_found"].sum()),
        "total": len(df_raw),
        "rate": f"{df_raw['filing_found'].mean():.1%}",
    },
    {
        "condition": "RAG (cleaned, semantic)",
        "metric": "top1_chunk_cik_match",
        "hits": int(df_semantic["top_chunk_cik_match"].sum()),
        "total": len(df_semantic),
        "rate": f"{df_semantic['top_chunk_cik_match'].mean():.1%}",
    },
])
coverage
condition metric hits total rate
0 RAG (cleaned, structured) filing_found 96 96 100.0%
1 RAG (raw, structured) filing_found 96 96 100.0%
2 RAG (cleaned, semantic) top1_chunk_cik_match 96 96 100.0%

Accuracy by metric#

Do some financial figures survive RAG better than others? Total assets and revenue tend to appear prominently in the income statement / balance sheet, whereas net income can come with loss adjustments or one-off items that complicate extraction.

def accuracy_by(df, by, tol_col="correct_10pct"):
    return df.groupby(by)[tol_col].mean()


by_metric = pd.DataFrame({
    "baseline":      accuracy_by(df_baseline, "metric_label"),
    "rag_cleaned":   accuracy_by(df_clean,    "metric_label"),
    "rag_raw":       accuracy_by(df_raw,      "metric_label"),
    "rag_semantic":  accuracy_by(df_semantic, "metric_label"),
}).round(3)
by_metric
baseline rag_cleaned rag_raw rag_semantic
metric_label
net income (loss) 0.219 0.844 0.094 0.781
total assets 0.312 0.781 0.219 0.812
total revenue (net sales) 0.438 0.844 0.344 0.875

Per-question detail#

One row per question, showing the ground-truth value, the extracted number from each condition, and whether each was within the 10% band. Useful for spot-checking failures.

detail = pd.DataFrame({
    "company": df_baseline["company_name"],
    "year": df_baseline["fiscal_year"],
    "metric": df_baseline["metric_label"],
    "expected": df_baseline["expected"],
    "baseline_extracted": df_baseline["extracted"],
    "baseline_ok": df_baseline["correct_10pct"],
    "rag_clean_extracted": df_clean["extracted"],
    "rag_clean_ok": df_clean["correct_10pct"],
    "rag_raw_extracted": df_raw["extracted"],
    "rag_raw_ok": df_raw["correct_10pct"],
    "rag_semantic_extracted": df_semantic["extracted"],
    "rag_semantic_ok": df_semantic["correct_10pct"],
    "semantic_top1_match": df_semantic["top_chunk_cik_match"],
})
detail
company year metric expected baseline_extracted baseline_ok rag_clean_extracted rag_clean_ok rag_raw_extracted rag_raw_ok rag_semantic_extracted rag_semantic_ok semantic_top1_match
0 APPLE INC 2023 total revenue (net sales) 383285.0 394000.0 True 383285.0 True 394328.0 True 383285.0 True True
1 APPLE INC 2023 net income (loss) 96995.0 90000.0 True 97000.0 True 94680.0 True 96995.0 True True
2 APPLE INC 2023 total assets 352583.0 350000.0 True 351002.0 True 345767.0 True 352755.0 True True
3 APPLE INC 2024 total revenue (net sales) 391035.0 400000.0 True 391035.0 True 394328.0 True 391035.0 True True
4 APPLE INC 2024 net income (loss) 93736.0 2023.0 False 93736.0 True 100000.0 True 93736.0 True True
... ... ... ... ... ... ... ... ... ... ... ... ... ...
91 WALMART INC 2025 net income (loss) 19436.0 14000.0 False NaN False 11981.0 False 20200.0 True True
92 WALMART INC 2025 total assets 260823.0 252000.0 True 244164.0 True 620000.0 False 260823.0 True True
93 WALMART INC 2026 total revenue (net sales) 710813.0 620000.0 False 713200.0 True 611289.0 False 713163.0 True True
94 WALMART INC 2026 net income (loss) 21893.0 2023.0 False 22270.0 True 9952.0 False 0.0 False True
95 WALMART INC 2026 total assets 284668.0 250000.0 False 272746.0 True 240000.0 False 284668.0 True True

96 rows Ă— 13 columns

Context size: raw vs cleaned#

Even though both conditions use the same chunk extractor and the same word-count budget, the raw chunks still contain SGML/HTML/XBRL tags that WRDS’s cleaning pipeline would have removed. That shows up as more characters per chunk, and usually as more input tokens in the prompt.

ctx_stats = pd.DataFrame({
    "rag_cleaned":  df_clean["context_chars"].describe(),
    "rag_raw":      df_raw["context_chars"].describe(),
    "rag_semantic": df_semantic["context_chars"].describe(),
}).round(0)
ctx_stats
rag_cleaned rag_raw rag_semantic
count 96.0 96.0 96.0
mean 20021.0 67173.0 20025.0
std 491.0 8196.0 972.0
min 18868.0 42712.0 18596.0
25% 19762.0 61681.0 18632.0
50% 20127.0 68741.0 20673.0
75% 20326.0 72967.0 20703.0
max 21094.0 77676.0 20753.0

Cost model#

Cheap models + RAG dominate expensive models + no context on a cost/accuracy basis for this kind of task. gpt-4o-mini is priced at roughly \(0.15 per million input tokens and \)0.60 per million output tokens, which makes each RAG query cost a fraction of a cent.

cost_breakdown = summary[["condition", "avg_input_tokens", "avg_output_tokens", "total_cost_usd"]].copy()
cost_breakdown["avg_input_tokens"] = cost_breakdown["avg_input_tokens"].round(0).astype(int)
cost_breakdown["avg_output_tokens"] = cost_breakdown["avg_output_tokens"].round(0).astype(int)
cost_breakdown["total_cost_usd"] = cost_breakdown["total_cost_usd"].round(4)
cost_breakdown
condition avg_input_tokens avg_output_tokens total_cost_usd
0 baseline (no context) 81 35 0.0032
1 RAG (cleaned, structured) 4460 4 0.0644
2 RAG (raw, structured) 20051 4 0.2890
3 RAG (cleaned, semantic) 5797 3 0.0836

Takeaways#

  • Without context, a small model hallucinates specific financial figures with only occasional luck – the “knowledge” is stale or imprecise, and accuracy is low.

  • With WRDS cleaned filings (structured retrieval), the same model becomes a reliable extractor. It no longer needs to know the answer; it just needs to read it off the 10-K.

  • WRDS raw vs. cleaned (structured): the raw archive still helps relative to the baseline, but chunks carry SGML/HTML/XBRL markup that costs extra tokens and occasionally trips up the keyword extractor. WRDS’s cleaning pipeline pays for itself: same retrieval, same model, better answers at lower token cost.

  • Structured vs semantic (cleaned): when the question carries a clean join key (CIK + fiscal year), the structured lookup is hard to beat – it always lands on the right filing. Semantic retrieval has to find the right filing among ~55k chunks, and any time the top-k bleeds in chunks from other companies, the model can be misled. The top1_chunk_cik_match rate is the key diagnostic. The payoff for semantic retrieval shows up in settings where the question is fuzzy or there is no clean join key – not in this benchmark.

Homework exercises#

  1. Rerun the benchmark with a different model (e.g. gpt-4o, gpt-4.1-mini). Where does the accuracy gap close?

  2. Extend extract_financial_context_* in src/rag_benchmark.py to pull more (or less) context. Does more context help, or does noise drown out the signal?

  3. Tune the semantic retrieval. Vary CHUNK_SIZE / CHUNK_OVERLAP in src/embeddings_index.py and top_k / max_words in run_rag_semantic. Rebuild the embeddings index and rerun – does a larger top-k help, or does it just dilute the prompt with chunks from the wrong companies?

  4. Extend semantic retrieval to the raw corpus. Build a second FAISS index over wrds_raw_filings, add an rag_semantic_raw condition, and see whether dense retrieval is more or less robust to markup noise than the keyword extractor.

  5. Add a fourth metric (e.g. long-term debt) and see whether the accuracy ranking across conditions is stable.

Contents