RAG Case Study: Data Overview#
This is the first of three notebooks for HW3: Benchmarking RAG with SEC Filings. Its job is to orient you to the data we will use in the experiment. No analysis yet – just “what did we pull, and what does it look like?”
The homework uses three datasets:
SEC 10-K filings from the WRDS SEC Analytics Suite. We pull both the raw EDGAR text and the cleaned plaintext for a small sample of 10 well-known companies. Notebooks 02 and 03 will use these as the retrieval corpus for RAG, and compare how cleaned vs. raw context affects accuracy.
Compustat fundamentals for the same companies. These audited revenue, net income, and total asset figures are the ground-truth answers for our benchmark questions.
A filing index tying the two together (CIK, company name, form type, filing date, file path).
from pathlib import Path
import pandas as pd
from settings import config
from pull_CRSP_Compustat import load_compustat
from pull_sec_filings_raw import (
SAMPLE_COMPANIES,
edgar_to_wrds_path,
)
DATA_DIR = Path(config("DATA_DIR"))
CLEAN_DIR = DATA_DIR / "wrds_clean_filings"
RAW_DIR = DATA_DIR / "wrds_raw_filings"
The sample companies#
We picked 10 well-known public companies spanning tech, consumer, finance, and auto. The sample is deliberately small: the point of this homework is the RAG methodology, not statistical coverage.
sample_df = pd.DataFrame(
[{"cik": cik, "company_name": name} for cik, name in SAMPLE_COMPANIES.items()]
)
sample_df
| cik | company_name | |
|---|---|---|
| 0 | 0000320193 | Apple Inc |
| 1 | 0000789019 | Microsoft Corp |
| 2 | 0001018724 | Amazon.com Inc |
| 3 | 0000019617 | JPMorgan Chase & Co |
| 4 | 0000200406 | Johnson & Johnson |
| 5 | 0000104169 | Walmart Inc |
| 6 | 0001318605 | Tesla Inc |
| 7 | 0001065280 | Netflix Inc |
| 8 | 0001045810 | NVIDIA Corp |
| 9 | 0001652044 | Alphabet Inc |
The filing index#
pull_sec_filings_raw.py queries the WRDS wrdssec_all.dforms table
for 10-K metadata and saves the result to
_data/sec_filings_metadata/filing_index.parquet. Each row represents
one filing.
filing_idx = pd.read_parquet(
DATA_DIR / "sec_filings_metadata" / "filing_index.parquet"
)
filing_idx["fdate"] = pd.to_datetime(filing_idx["fdate"])
filing_idx["fiscal_year"] = filing_idx["fdate"].dt.year
print(f"Total 10-K filings in index: {len(filing_idx)}")
print(
f"Date range: {filing_idx['fdate'].min().date()} -> "
f"{filing_idx['fdate'].max().date()}"
)
filing_idx[["cik", "coname", "form", "fdate"]].sort_values("fdate")
Total 10-K filings in index: 48
Date range: 2022-01-27 -> 2026-03-13
| cik | coname | form | fdate | |
|---|---|---|---|---|
| 47 | 0001065280 | NETFLIX INC | 10-K | 2022-01-27 |
| 46 | 0001652044 | Alphabet Inc. | 10-K | 2022-02-02 |
| 45 | 0001018724 | AMAZON COM INC | 10-K | 2022-02-04 |
| 44 | 0001318605 | Tesla, Inc. | 10-K | 2022-02-07 |
| 43 | 0000200406 | JOHNSON & JOHNSON | 10-K | 2022-02-17 |
| 42 | 0000019617 | JPMORGAN CHASE & CO | 10-K | 2022-02-22 |
| 41 | 0001045810 | NVIDIA CORP | 10-K | 2022-03-18 |
| 40 | 0000104169 | Walmart Inc. | 10-K | 2022-03-18 |
| 39 | 0000789019 | MICROSOFT CORP | 10-K | 2022-07-28 |
| 38 | 0000320193 | Apple Inc. | 10-K | 2022-10-28 |
| 37 | 0001065280 | NETFLIX INC | 10-K | 2023-01-26 |
| 36 | 0001318605 | Tesla, Inc. | 10-K | 2023-01-31 |
| 34 | 0001018724 | AMAZON COM INC | 10-K | 2023-02-03 |
| 35 | 0001652044 | Alphabet Inc. | 10-K | 2023-02-03 |
| 33 | 0000200406 | JOHNSON & JOHNSON | 10-K | 2023-02-16 |
| 32 | 0000019617 | JPMORGAN CHASE & CO | 10-K | 2023-02-21 |
| 31 | 0001045810 | NVIDIA CORP | 10-K | 2023-02-24 |
| 30 | 0000104169 | Walmart Inc. | 10-K | 2023-03-17 |
| 29 | 0000789019 | MICROSOFT CORP | 10-K | 2023-07-27 |
| 28 | 0000320193 | Apple Inc. | 10-K | 2023-11-03 |
| 27 | 0001065280 | NETFLIX INC | 10-K | 2024-01-26 |
| 26 | 0001318605 | Tesla, Inc. | 10-K | 2024-01-29 |
| 25 | 0001652044 | Alphabet Inc. | 10-K | 2024-01-31 |
| 24 | 0001018724 | AMAZON COM INC | 10-K | 2024-02-02 |
| 23 | 0000019617 | JPMORGAN CHASE & CO | 10-K | 2024-02-16 |
| 22 | 0000200406 | JOHNSON & JOHNSON | 10-K | 2024-02-16 |
| 21 | 0001045810 | NVIDIA CORP | 10-K | 2024-02-21 |
| 20 | 0000104169 | Walmart Inc. | 10-K | 2024-03-15 |
| 19 | 0000789019 | MICROSOFT CORP | 10-K | 2024-07-30 |
| 18 | 0000320193 | Apple Inc. | 10-K | 2024-11-01 |
| 17 | 0001065280 | NETFLIX INC | 10-K | 2025-01-27 |
| 16 | 0001318605 | Tesla, Inc. | 10-K | 2025-01-30 |
| 15 | 0001652044 | Alphabet Inc. | 10-K | 2025-02-05 |
| 14 | 0001018724 | AMAZON COM INC | 10-K | 2025-02-07 |
| 13 | 0000200406 | JOHNSON & JOHNSON | 10-K | 2025-02-13 |
| 12 | 0000019617 | JPMORGAN CHASE & CO | 10-K | 2025-02-14 |
| 11 | 0001045810 | NVIDIA CORP | 10-K | 2025-02-26 |
| 10 | 0000104169 | Walmart Inc. | 10-K | 2025-03-14 |
| 9 | 0000789019 | MICROSOFT CORP | 10-K | 2025-07-30 |
| 8 | 0000320193 | Apple Inc. | 10-K | 2025-10-31 |
| 7 | 0001065280 | NETFLIX INC | 10-K | 2026-01-23 |
| 6 | 0001318605 | Tesla, Inc. | 10-K | 2026-01-29 |
| 5 | 0001652044 | Alphabet Inc. | 10-K | 2026-02-05 |
| 4 | 0001018724 | AMAZON COM INC | 10-K | 2026-02-06 |
| 3 | 0000200406 | JOHNSON & JOHNSON | 10-K | 2026-02-11 |
| 2 | 0000019617 | JPMORGAN CHASE & CO | 10-K | 2026-02-13 |
| 1 | 0001045810 | NVIDIA CORP | 10-K | 2026-02-25 |
| 0 | 0000104169 | Walmart Inc. | 10-K | 2026-03-13 |
Filings per company#
per_company = (
filing_idx.groupby(["cik", "coname"])
.size()
.reset_index(name="n_filings")
.sort_values("n_filings", ascending=False)
)
per_company
| cik | coname | n_filings | |
|---|---|---|---|
| 0 | 0000019617 | JPMORGAN CHASE & CO | 5 |
| 1 | 0000104169 | Walmart Inc. | 5 |
| 2 | 0000200406 | JOHNSON & JOHNSON | 5 |
| 5 | 0001018724 | AMAZON COM INC | 5 |
| 6 | 0001045810 | NVIDIA CORP | 5 |
| 7 | 0001065280 | NETFLIX INC | 5 |
| 8 | 0001318605 | Tesla, Inc. | 5 |
| 9 | 0001652044 | Alphabet Inc. | 5 |
| 3 | 0000320193 | Apple Inc. | 4 |
| 4 | 0000789019 | MICROSOFT CORP | 4 |
Raw vs. cleaned filings#
The WRDS SEC Analytics Suite publishes two versions of every filing:
Raw (
/wrds/sec/warchives/...) – the filing as submitted to EDGAR, including SGML/HTML markup, XBRL tagging, embedded images encoded as base64, and boilerplate headers.Cleaned (
/wrds/sec/wrds_clean_filings/...) – the same filing with markup stripped and whitespace normalized, so it reads like plain text.
Both downloads are mirrored locally into a directory structure that
matches WRDS’s layout: {cik_prefix}/{cik}/{accession}.txt.
Let’s confirm both versions landed on disk and compare their sizes.
def size_mb(path: Path) -> float:
if not path.exists():
return 0.0
return path.stat().st_size / 1_000_000
rows = []
for _, row in filing_idx.iterrows():
relpath = edgar_to_wrds_path(row["fname"])
clean_path = CLEAN_DIR / relpath
raw_path = RAW_DIR / relpath
rows.append({
"company": row["coname"],
"filing_date": row["fdate"].date(),
"cleaned_mb": round(size_mb(clean_path), 3),
"raw_mb": round(size_mb(raw_path), 3),
"cleaned_path": relpath if clean_path.exists() else None,
"raw_path": relpath if raw_path.exists() else None,
})
size_df = pd.DataFrame(rows)
size_df
| company | filing_date | cleaned_mb | raw_mb | cleaned_path | raw_path | |
|---|---|---|---|---|---|---|
| 0 | Walmart Inc. | 2026-03-13 | 2.112 | 13.080 | 000010/104169/0000104169-26-000055.txt | 000010/104169/0000104169-26-000055.txt |
| 1 | NVIDIA CORP | 2026-02-25 | 1.721 | 11.462 | 000104/1045810/0001045810-26-000021.txt | 000104/1045810/0001045810-26-000021.txt |
| 2 | JPMORGAN CHASE & CO | 2026-02-13 | 6.831 | 64.896 | 000001/19617/0001628280-26-008131.txt | 000001/19617/0001628280-26-008131.txt |
| 3 | JOHNSON & JOHNSON | 2026-02-11 | 2.886 | 24.877 | 000020/200406/0000200406-26-000016.txt | 000020/200406/0000200406-26-000016.txt |
| 4 | AMAZON COM INC | 2026-02-06 | 1.901 | 12.701 | 000101/1018724/0001018724-26-000004.txt | 000101/1018724/0001018724-26-000004.txt |
| 5 | Alphabet Inc. | 2026-02-05 | 2.034 | 15.653 | 000165/1652044/0001652044-26-000018.txt | 000165/1652044/0001652044-26-000018.txt |
| 6 | Tesla, Inc. | 2026-01-29 | 2.346 | 15.755 | 000131/1318605/0001628280-26-003952.txt | 000131/1318605/0001628280-26-003952.txt |
| 7 | NETFLIX INC | 2026-01-23 | 1.953 | 13.533 | 000106/1065280/0001065280-26-000034.txt | 000106/1065280/0001065280-26-000034.txt |
| 8 | Apple Inc. | 2025-10-31 | 1.277 | 9.392 | 000032/320193/0000320193-25-000079.txt | 000032/320193/0000320193-25-000079.txt |
| 9 | MICROSOFT CORP | 2025-07-30 | 1.941 | 34.890 | 000078/789019/0000950170-25-100235.txt | 000078/789019/0000950170-25-100235.txt |
| 10 | Walmart Inc. | 2025-03-14 | 1.760 | 12.095 | 000010/104169/0000104169-25-000021.txt | 000010/104169/0000104169-25-000021.txt |
| 11 | NVIDIA CORP | 2025-02-26 | 1.794 | 12.326 | 000104/1045810/0001045810-25-000023.txt | 000104/1045810/0001045810-25-000023.txt |
| 12 | JPMORGAN CHASE & CO | 2025-02-14 | 6.597 | 74.059 | 000001/19617/0000019617-25-000270.txt | 000001/19617/0000019617-25-000270.txt |
| 13 | JOHNSON & JOHNSON | 2025-02-13 | 2.648 | 24.342 | 000020/200406/0000200406-25-000038.txt | 000020/200406/0000200406-25-000038.txt |
| 14 | AMAZON COM INC | 2025-02-07 | 1.726 | 12.030 | 000101/1018724/0001018724-25-000004.txt | 000101/1018724/0001018724-25-000004.txt |
| 15 | Alphabet Inc. | 2025-02-05 | 1.822 | 14.774 | 000165/1652044/0001652044-25-000014.txt | 000165/1652044/0001652044-25-000014.txt |
| 16 | Tesla, Inc. | 2025-01-30 | 2.134 | 15.789 | 000131/1318605/0001628280-25-003063.txt | 000131/1318605/0001628280-25-003063.txt |
| 17 | NETFLIX INC | 2025-01-27 | 1.522 | 12.059 | 000106/1065280/0001065280-25-000044.txt | 000106/1065280/0001065280-25-000044.txt |
| 18 | Apple Inc. | 2024-11-01 | 1.345 | 9.759 | 000032/320193/0000320193-24-000123.txt | 000032/320193/0000320193-24-000123.txt |
| 19 | MICROSOFT CORP | 2024-07-30 | 1.943 | 30.376 | 000078/789019/0000950170-24-087843.txt | 000078/789019/0000950170-24-087843.txt |
| 20 | Walmart Inc. | 2024-03-15 | 1.943 | 13.086 | 000010/104169/0000104169-24-000056.txt | 000010/104169/0000104169-24-000056.txt |
| 21 | NVIDIA CORP | 2024-02-21 | 1.570 | 11.814 | 000104/1045810/0001045810-24-000029.txt | 000104/1045810/0001045810-24-000029.txt |
| 22 | JOHNSON & JOHNSON | 2024-02-16 | 2.666 | 22.223 | 000020/200406/0000200406-24-000013.txt | 000020/200406/0000200406-24-000013.txt |
| 23 | JPMORGAN CHASE & CO | 2024-02-16 | 6.539 | 69.751 | 000001/19617/0000019617-24-000225.txt | 000001/19617/0000019617-24-000225.txt |
| 24 | AMAZON COM INC | 2024-02-02 | 1.711 | 12.111 | 000101/1018724/0001018724-24-000008.txt | 000101/1018724/0001018724-24-000008.txt |
| 25 | Alphabet Inc. | 2024-01-31 | 1.747 | 13.928 | 000165/1652044/0001652044-24-000022.txt | 000165/1652044/0001652044-24-000022.txt |
| 26 | Tesla, Inc. | 2024-01-29 | 2.102 | 15.528 | 000131/1318605/0001628280-24-002390.txt | 000131/1318605/0001628280-24-002390.txt |
| 27 | NETFLIX INC | 2024-01-26 | 1.245 | 10.066 | 000106/1065280/0001065280-24-000030.txt | 000106/1065280/0001065280-24-000030.txt |
| 28 | Apple Inc. | 2023-11-03 | 1.190 | 9.570 | 000032/320193/0000320193-23-000106.txt | 000032/320193/0000320193-23-000106.txt |
| 29 | MICROSOFT CORP | 2023-07-27 | 1.755 | 40.966 | 000078/789019/0000950170-23-035122.txt | 000078/789019/0000950170-23-035122.txt |
| 30 | Walmart Inc. | 2023-03-17 | 1.855 | 13.431 | 000010/104169/0000104169-23-000020.txt | 000010/104169/0000104169-23-000020.txt |
| 31 | NVIDIA CORP | 2023-02-24 | 1.696 | 13.525 | 000104/1045810/0001045810-23-000017.txt | 000104/1045810/0001045810-23-000017.txt |
| 32 | JPMORGAN CHASE & CO | 2023-02-21 | 5.801 | 72.224 | 000001/19617/0000019617-23-000231.txt | 000001/19617/0000019617-23-000231.txt |
| 33 | JOHNSON & JOHNSON | 2023-02-16 | 2.434 | 24.820 | 000020/200406/0000200406-23-000016.txt | 000020/200406/0000200406-23-000016.txt |
| 34 | AMAZON COM INC | 2023-02-03 | 1.564 | 12.809 | 000101/1018724/0001018724-23-000004.txt | 000101/1018724/0001018724-23-000004.txt |
| 35 | Alphabet Inc. | 2023-02-03 | 1.679 | 15.264 | 000165/1652044/0001652044-23-000016.txt | 000165/1652044/0001652044-23-000016.txt |
| 36 | Tesla, Inc. | 2023-01-31 | 2.544 | 31.445 | 000131/1318605/0000950170-23-001409.txt | 000131/1318605/0000950170-23-001409.txt |
| 37 | NETFLIX INC | 2023-01-26 | 1.027 | 8.639 | 000106/1065280/0001065280-23-000035.txt | 000106/1065280/0001065280-23-000035.txt |
| 38 | Apple Inc. | 2022-10-28 | 1.161 | 10.332 | 000032/320193/0000320193-22-000108.txt | 000032/320193/0000320193-22-000108.txt |
| 39 | MICROSOFT CORP | 2022-07-28 | 1.709 | 26.423 | 000078/789019/0001564590-22-026876.txt | 000078/789019/0001564590-22-026876.txt |
| 40 | Walmart Inc. | 2022-03-18 | 2.019 | 14.199 | 000010/104169/0000104169-22-000012.txt | 000010/104169/0000104169-22-000012.txt |
| 41 | NVIDIA CORP | 2022-03-18 | 1.814 | 12.667 | 000104/1045810/0001045810-22-000036.txt | 000104/1045810/0001045810-22-000036.txt |
| 42 | JPMORGAN CHASE & CO | 2022-02-22 | 6.935 | 69.494 | 000001/19617/0000019617-22-000272.txt | 000001/19617/0000019617-22-000272.txt |
| 43 | JOHNSON & JOHNSON | 2022-02-17 | 2.802 | 25.022 | 000020/200406/0000200406-22-000022.txt | 000020/200406/0000200406-22-000022.txt |
| 44 | Tesla, Inc. | 2022-02-07 | 2.677 | 29.316 | 000131/1318605/0000950170-22-000796.txt | 000131/1318605/0000950170-22-000796.txt |
| 45 | AMAZON COM INC | 2022-02-04 | 2.109 | 13.442 | 000101/1018724/0001018724-22-000005.txt | 000101/1018724/0001018724-22-000005.txt |
| 46 | Alphabet Inc. | 2022-02-02 | 2.021 | 15.045 | 000165/1652044/0001652044-22-000019.txt | 000165/1652044/0001652044-22-000019.txt |
| 47 | NETFLIX INC | 2022-01-27 | 1.221 | 9.175 | 000106/1065280/0001065280-22-000036.txt | 000106/1065280/0001065280-22-000036.txt |
Aggregate size comparison#
Raw filings are dramatically larger than cleaned ones – most of the bulk is HTML/XBRL markup and inline images. This matters for RAG: if you feed raw text into the model’s prompt you burn context window (and tokens) on markup that contributes nothing to the answer.
summary = pd.DataFrame({
"version": ["cleaned", "raw"],
"total_mb": [size_df["cleaned_mb"].sum(), size_df["raw_mb"].sum()],
"mean_mb_per_filing": [
size_df["cleaned_mb"].mean(),
size_df["raw_mb"].mean(),
],
"n_files_present": [
size_df["cleaned_path"].notna().sum(),
size_df["raw_path"].notna().sum(),
],
}).round(3)
summary
| version | total_mb | mean_mb_per_filing | n_files_present | |
|---|---|---|---|---|
| 0 | cleaned | 113.829 | 2.371 | 48 |
| 1 | raw | 1070.163 | 22.295 | 48 |
Side-by-side preview#
Both files start with an identical SGML header (filer info, accession
number, etc.), so a naive text[:800] would look the same in both.
To make the difference concrete, we jump past the header and find the
table of contents – specifically, the “Item 1. Business” entry. The
cleaned file shows it as readable prose; the raw file shows the same
content buried in HTML attributes.
present = size_df[(size_df["cleaned_mb"] > 0) & (size_df["raw_mb"] > 0)]
sample_row = present.iloc[0]
relpath = sample_row["cleaned_path"]
cleaned_text = (CLEAN_DIR / relpath).read_text(encoding="utf-8", errors="ignore")
raw_text = (RAW_DIR / relpath).read_text(encoding="utf-8", errors="ignore")
print(f"File: {relpath}")
print(f"Company: {sample_row['company']} Filed: {sample_row['filing_date']}")
print(f"Cleaned size: {sample_row['cleaned_mb']:.2f} MB "
f"Raw size: {sample_row['raw_mb']:.2f} MB")
# Locate the table of contents in each version. The landmark "Item 1."
# appears at very different byte offsets because the raw file has tens
# of thousands of bytes of XBRL/HTML ahead of the readable text.
LANDMARK = "Item 1."
clean_idx = cleaned_text.find(LANDMARK)
raw_idx = raw_text.find(LANDMARK)
print(f"\n'{LANDMARK}' offset in cleaned: {clean_idx:,}")
print(f"'{LANDMARK}' offset in raw: {raw_idx:,}")
File: 000010/104169/0000104169-26-000055.txt
Company: Walmart Inc. Filed: 2026-03-13
Cleaned size: 2.11 MB Raw size: 13.08 MB
'Item 1.' offset in cleaned: 87,663
'Item 1.' offset in raw: 343,452
Cleaned version (800 chars around “Item 1.”):
print(cleaned_text[clean_idx:clean_idx + 800])
Item 1. Business" above for additional discussion of the competitive landscape of our business.
Further, the protection of our proprietary rights, including our trademarks, copyrights, domain names, patents and trade secrets, is important to our business. Effective protection of our proprietary rights may not be available in every jurisdiction in which we offer our products and services, and we may not be able to prevent or deter third parties from infringing or misappropriating our intellectual property, or ensure that third parties will not independently develop equivalent or superior intellectual property rights, which could affect our ability to maintain a competitive advantage and adversely impact our business.
Certain segments of the retail industry are undergoing consolidation or
Raw version (800 chars around “Item 1.”):
Same words, same place in the document – but wrapped in inline styles, anchor tags, and table layout markup. An LLM would have to parse all of this to extract “Item 1. Business”.
print(raw_text[raw_idx:raw_idx + 800])
Item 1. Business</a></span><span style="color:#000000;font-family:'Times New Roman',serif;font-size:10pt;font-weight:400;line-height:120%">" above for additional discussion of the competitive landscape of our business.</span></div><div style="margin-top:5pt"><span style="color:#000000;font-family:'Times New Roman',serif;font-size:10pt;font-weight:400;line-height:120%">Further, the protection of our proprietary rights, including our trademarks, copyrights, domain names, patents and trade secrets, is important to our business. Effective protection of our proprietary rights may not be available in every jurisdiction in which we offer our products and services, and we may not be able to prevent or deter third parties from infringing or misappropriating our intellectual property, or ensure that
Compustat ground truth#
For each (company, fiscal year) we pull three audited figures from
Compustat: revenue (sale), net income (ni), and total assets
(at). These are reported in millions of USD, and they serve as the
ground truth that the RAG benchmark compares LLM answers against.
comp = load_compustat(DATA_DIR)
comp = comp.dropna(subset=["sale"]).sort_values(["conm", "datadate"])
print(
f"Compustat rows: {len(comp)} "
f"Companies: {comp['conm'].nunique()} "
f"Date range: {comp['datadate'].min().date()} -> "
f"{comp['datadate'].max().date()}"
)
comp[["conm", "datadate", "year", "sale", "ni", "at"]].tail(15)
Compustat rows: 122 Companies: 10 Date range: 2014-01-31 -> 2026-01-31
| conm | datadate | year | sale | ni | at | |
|---|---|---|---|---|---|---|
| 82 | TESLA INC | 2024-12-31 | 2024 | 97690.0 | 7091.0 | 122070.0 |
| 83 | TESLA INC | 2025-12-31 | 2025 | 94827.0 | 3794.0 | 137806.0 |
| 109 | WALMART INC | 2014-01-31 | 2014 | 474259.0 | 16022.0 | 204751.0 |
| 110 | WALMART INC | 2015-01-31 | 2015 | 483521.0 | 16363.0 | 203706.0 |
| 111 | WALMART INC | 2016-01-31 | 2016 | 479962.0 | 14694.0 | 199581.0 |
| 112 | WALMART INC | 2017-01-31 | 2017 | 482154.0 | 13643.0 | 198825.0 |
| 113 | WALMART INC | 2018-01-31 | 2018 | 496785.0 | 9862.0 | 204522.0 |
| 114 | WALMART INC | 2019-01-31 | 2019 | 511729.0 | 6670.0 | 219295.0 |
| 115 | WALMART INC | 2020-01-31 | 2020 | 521426.0 | 14881.0 | 236495.0 |
| 116 | WALMART INC | 2021-01-31 | 2021 | 556933.0 | 13510.0 | 252496.0 |
| 117 | WALMART INC | 2022-01-31 | 2022 | 569962.0 | 13673.0 | 244860.0 |
| 118 | WALMART INC | 2023-01-31 | 2023 | 608481.0 | 11680.0 | 243197.0 |
| 119 | WALMART INC | 2024-01-31 | 2024 | 645737.0 | 15511.0 | 252399.0 |
| 120 | WALMART INC | 2025-01-31 | 2025 | 678338.0 | 19436.0 | 260823.0 |
| 121 | WALMART INC | 2026-01-31 | 2026 | 710813.0 | 21893.0 | 284668.0 |
Questions we will ask the model#
In notebook 02 we walk through how one benchmark question flows through the pipeline, and notebook 03 runs the full grid. Questions are keyed on (company, fiscal year, metric) – which is also the join key between Compustat and the 10-K corpus. Here is what those questions will look like for the most recent fiscal years.
recent = comp[comp["year"] >= 2023].copy()
preview = recent[["conm", "year", "datadate", "sale", "ni", "at"]].copy()
preview["datadate"] = preview["datadate"].dt.date
preview
| conm | year | datadate | sale | ni | at | |
|---|---|---|---|---|---|---|
| 69 | ALPHABET INC | 2023 | 2023-12-31 | 307394.0 | 73795.0 | 402392.0 |
| 70 | ALPHABET INC | 2024 | 2024-12-31 | 350018.0 | 100118.0 | 450256.0 |
| 71 | ALPHABET INC | 2025 | 2025-12-31 | 402836.0 | 132170.0 | 595281.0 |
| 45 | AMAZON.COM INC | 2023 | 2023-12-31 | 574785.0 | 30425.0 | 527854.0 |
| 46 | AMAZON.COM INC | 2024 | 2024-12-31 | 637959.0 | 59248.0 | 624894.0 |
| 47 | AMAZON.COM INC | 2025 | 2025-12-31 | 716924.0 | 77670.0 | 818042.0 |
| 9 | APPLE INC | 2023 | 2023-09-30 | 383285.0 | 96995.0 | 352583.0 |
| 10 | APPLE INC | 2024 | 2024-09-30 | 391035.0 | 93736.0 | 364980.0 |
| 11 | APPLE INC | 2025 | 2025-09-30 | 416161.0 | 112010.0 | 359241.0 |
| 33 | JOHNSON & JOHNSON | 2023 | 2023-12-31 | 85159.0 | 35153.0 | 167558.0 |
| 34 | JOHNSON & JOHNSON | 2024 | 2024-12-31 | 88821.0 | 14066.0 | 180104.0 |
| 35 | JOHNSON & JOHNSON | 2025 | 2025-12-31 | 94193.0 | 26804.0 | 199210.0 |
| 21 | JPMORGAN CHASE & CO | 2023 | 2023-12-31 | 236311.0 | 49552.0 | 3875393.0 |
| 22 | JPMORGAN CHASE & CO | 2024 | 2024-12-31 | 278803.0 | 58471.0 | 4002814.0 |
| 23 | JPMORGAN CHASE & CO | 2025 | 2025-12-31 | 279717.0 | 57048.0 | 4424900.0 |
| 93 | MICROSOFT CORP | 2023 | 2023-06-30 | 211915.0 | 72361.0 | 411976.0 |
| 94 | MICROSOFT CORP | 2024 | 2024-06-30 | 245122.0 | 88136.0 | 512163.0 |
| 95 | MICROSOFT CORP | 2025 | 2025-06-30 | 281724.0 | 101832.0 | 619003.0 |
| 57 | NETFLIX INC | 2023 | 2023-12-31 | 33723.297 | 5407.99 | 48731.992 |
| 58 | NETFLIX INC | 2024 | 2024-12-31 | 39000.966 | 8711.631 | 53630.374 |
| 59 | NETFLIX INC | 2025 | 2025-12-31 | 45183.036 | 10981.201 | 55596.993 |
| 105 | NVIDIA CORP | 2023 | 2023-01-31 | 26974.0 | 4368.0 | 41182.0 |
| 106 | NVIDIA CORP | 2024 | 2024-01-31 | 60922.0 | 29760.0 | 65728.0 |
| 107 | NVIDIA CORP | 2025 | 2025-01-31 | 130497.0 | 72880.0 | 111601.0 |
| 108 | NVIDIA CORP | 2026 | 2026-01-31 | 215938.0 | 120067.0 | 206803.0 |
| 81 | TESLA INC | 2023 | 2023-12-31 | 96773.0 | 14997.0 | 106618.0 |
| 82 | TESLA INC | 2024 | 2024-12-31 | 97690.0 | 7091.0 | 122070.0 |
| 83 | TESLA INC | 2025 | 2025-12-31 | 94827.0 | 3794.0 | 137806.0 |
| 118 | WALMART INC | 2023 | 2023-01-31 | 608481.0 | 11680.0 | 243197.0 |
| 119 | WALMART INC | 2024 | 2024-01-31 | 645737.0 | 15511.0 | 252399.0 |
| 120 | WALMART INC | 2025 | 2025-01-31 | 678338.0 | 19436.0 | 260823.0 |
| 121 | WALMART INC | 2026 | 2026-01-31 | 710813.0 | 21893.0 | 284668.0 |
Recap#
10 companies, one form type (10-K), fiscal years 2022+.
Both raw and cleaned filing text downloaded from WRDS.
Compustat fundamentals for the same companies, providing ground-truth answers.
Notebook 02 walks through how the RAG pipeline is wired together function by function, with small concrete examples. Notebook 03 then scales that up and reports aggregate accuracy, latency, and cost for a weak LLM under three conditions: no context, retrieved cleaned filing text, and retrieved raw filing text.