Skip to main content
In this lesson we convert written language into numerical signals that computers can analyze. Rather than interpreting sentences the way humans do, we apply statistical and graph-based techniques to surface patterns in words and sentences. These foundational methods—useful for search, classification, and extractive summarization—include:
  • Term frequency (TF)
  • TF‑IDF (term frequency–inverse document frequency)
  • Bag of words (BOW)
  • TextRank
A slide titled "Statistical Analysis" showing four boxed items numbered 01–04: Term frequency, TF‑IDF, Bag of words, and TextRank. Each method is displayed with a colored circular number above its label.
Each technique inspects text from a different angle: some measure raw frequency, others identify terms that distinguish one document from the corpus, and graph-based methods surface the most central sentences. Below we explain each method, show concise formulas and examples, and note common use cases and limitations.

​
Term frequency (TF)

Term frequency quantifies how often a token appears inside a single document. It’s a simple and effective signal for identifying prominent words in a document. Example review: “excellent keyboard, excellent quality, excellent value”
  • Token counts: excellent = 3, keyboard = 1, quality = 1, value = 1
  • If the document has 6 tokens total, the term frequency for excellent is 3 / 6 = 0.5.
A standard formula:
where count(t, d) is the number of occurrences of term t in document d, and |d| is the total number of terms in d. Why use TF?
  • Quick to compute and interpretable.
  • Useful for surfacing frequent words in a document (e.g., common opinions in reviews).
Limitations:
  • TF does not account for how common a term is across other documents, so very common words can appear important even if they are not discriminative.

​
TF‑IDF

TF‑IDF combines term frequency with inverse document frequency to highlight words that are frequent in a document but rare across the corpus. This helps surface terms that discriminate one document from many. Intuition:
  • TF rewards high frequency in a document.
  • IDF downweights terms that appear in many documents.
A common IDF formula:
where N is the total number of documents and DF(t) is the number of documents that contain term t. TF‑IDF is typically computed as:
An infographic explaining TF‑IDF that shows three sample product reviews and a table of terms (e.g., "keyboard", "excellent", "quality") with dots indicating which reviews they appear in and horizontal bars showing their TF‑IDF scores. The header explains TF‑IDF assigns higher importance to terms frequent in a document but uncommon across a collection.
Example: if keyboard appears across many reviews, its IDF is low and the TF‑IDF score will be reduced; if excellent appears often in one review but not others, it receives a higher TF‑IDF score for that document.
TF‑IDF is a statistical importance score: it surfaces discriminative words but does not capture sentiment or semantic meaning. Use it for keyword extraction and as features for classical ML models.

​
Bag of words (BOW)

The bag-of-words model represents a document as a multiset of tokens and their counts. BOW discards word order and syntax—only token presence and frequency matter. Example:
  • Original: “excellent keyboard, excellent quality, excellent value”
  • Bag-of-words: excellent: 3, keyboard: 1, quality: 1, value: 1
If you shuffle the tokens, the BOW representation remains the same.
A slide titled "Bag of Words" explaining that text is represented as word frequencies while ignoring grammar and word order. It shows an example sentence and its shuffled version mapped to counts: Excellent 3, Keyboard 1, Quality 1, Value 1.
Advantages:
  • Simple and computationally efficient.
  • Works well for many classic tasks such as document classification and spam detection.
Limitations:
  • Loses word order and syntactic relations; cannot distinguish sentences that share the same words but have different meanings.
Bag-of-words discards order and grammatical relationships. Use it only when sequence and syntax are not important for the task.

​
TextRank

TextRank is a graph-based ranking algorithm that identifies the most important sentences for extractive summarization. It models sentences as nodes and connects them with edges weighted by similarity. Typical TextRank pipeline:
  1. Tokenize the document into sentences (nodes).
  2. Compute pairwise similarity between sentences (edges), e.g., based on shared keywords, TF‑IDF vectors, or semantic embeddings.
  3. Run a PageRank-style algorithm to score sentences.
  4. Select the top-ranked sentences to build an extractive summary.
A slide titled "TextRank" describing a graph-based algorithm that identifies the most important sentences, with three example sentence boxes. The middle box—about a package arriving late but good product quality and resolved delivery— is highlighted as ranked most important.
Example: a sentence that touches on several core document points (delivery delay, product quality, resolution) will likely be strongly connected to other sentences and receive a high TextRank score, making it a good candidate for an extractive summary. For background on the ranking algorithm, see PageRank: https://en.wikipedia.org/wiki/PageRank

​
Summary comparison

These techniques provide complementary, measurable ways to represent text. They remain useful today as features for machine learning and as lightweight baselines before moving to semantic language models. We will next explore semantic language models and how embeddings and neural techniques represent meaning beyond these statistical and graph-based approaches.

Watch Video