# Discussion 2: Text Representation and Tokenization

## Session Plan

1. **Housekeeping** — How is HW0 going? Questions or issues?
2. **Exam Prep Material** — New [Exam Preparation](exam_prep.md) page added to the textbook with study guidance and topic outlines
3. **HW1 Released** — [Homework 1: Lopez-Lira, Tang (2023)](HW1.md) is now available
4. **Structure of News** — Exploring how financial news is structured and what makes it useful for NLP
5. **Words to Tokens** — From raw text to model input: tokenization in practice
6. **Embedding Analogies** — Hands-on exploration of word embeddings and analogies
7. **HW1 Review: Headline Coverage and Model Performance** — When you don't have enough headlines, strategy performance suffers. We'll look at how to diagnose this by running a local model on both a smaller sample and the fuller sample and comparing results. See the [Replication Results notebook](notebooks/03_replication_results_ipynb.ipynb) for the full comparison.

## Topics

- **Structure of Financial News** — What makes news text useful for quantitative analysis and how it is typically structured
- **AI Copilots Intro** — Getting started with Claude Code and Cursor for AI-assisted development
- **Words to Tokens** — How tokenizers convert text into the numerical representations that models consume
- **Embedding Analogies** — Exploring semantic relationships captured by word embeddings

```{toctree}
:maxdepth: 1
discussions/structure_of_news.md
discussions/ai_copilots_intro.md
discussions/words_to_tokens.md
discussions/03_embedding_analogies_ipynb.ipynb
discussions/review_w02.md
```