Skip to main content
This guide explains how generative AI models—particularly large language models (LLMs)—process language and generate text. It preserves the original diagram order and clarifies each step in the pipeline: tokenization, embeddings, transformer processing (attention), and iterative next-token generation.
A language model does not “understand” text like a human. It converts text into numerical forms that a neural network processes. Think of this pipeline as: text → tokens → embeddings → transformer layers with attention → next-token prediction.

​
1. Tokenization — splitting text into model-ready pieces

Start with tokenization. Tokenizers break a sentence into smaller units (tokens) that the model can process. For example, the sentence: “I heard a dog bark at a cat.” can be split into tokens like: I, heard, a, dog, bark, at, a, cat. Tokens may be whole words, parts of words, punctuation, or other fragments depending on the tokenizer (BPE, WordPiece, SentencePiece, etc.). After tokenization, tokens are converted into numerical representations so a neural network can operate on them.
A slide titled "Generative AI Models" explaining that large language models use the Transformer architecture, with the example sentence "I heard a dog bark at a cat." The sentence is shown split into tokens: I, heard, a, dog, bark, at, a, cat, labeled as the basic units the model processes.

​
2. Embeddings — mapping tokens to high-dimensional vectors

Embeddings convert tokens into numerical vectors. Instead of feeding the raw token “dog” to the network, the model maps it to a vector such as [10, 3, 2] (toy example). Real embeddings have many more dimensions—commonly hundreds or thousands (1,536 dims is a common size in some models). Each dimension is a learned feature; semantic information is distributed across the vector.
A slide titled "Generative AI Models" showing the token "dog" being converted via an embedding into a vector [10, 3, 2]. The vector is labeled as encoding semantic meaning and noted to have 1,536 dimensions.
Think of embeddings as points in a very high-dimensional space: similar-context words tend to be near each other. For intuition, here is a tiny illustrative vector table in a 3-D toy space:
This simplistic example shows how closeness in vector space reflects related usage. In production models, embeddings are the starting point; transformer layers create context-aware vectors from those embeddings.

​
3. Contextualization with Transformers and Attention

Embeddings are static for a token in isolation, but meaning depends on context. Transformers use self-attention to update token vectors based on the entire sequence. Attention computes how much each token should influence another token’s representation—allowing the model to capture long-range dependencies. For example, in the sentence “I heard a dog bark at her,” when processing “bark,” the model should weight “dog” more heavily than unrelated words. The attention mechanism learns these weight patterns from data and produces context-sensitive representations.
A slide about the attention mechanism in generative AI showing the token sequence "I heard a dog bark at a" with "bark" highlighted. A bar chart below illustrates how strongly "bark" relates to other tokens (dog strongest, then heard, I, at).
Key properties of attention:
  • It computes pairwise influence scores between tokens (scaled dot-product attention is common).
  • It enables the model to combine information from relevant tokens, even if they are far apart in the sequence.
  • Multi-head attention lets the model attend to different aspects of the context simultaneously.

​
4. Generation — next-token prediction and sampling

Generation is performed by repeatedly predicting the next token conditional on the current context. Given a prefix like: “I heard a dog bark at a” the model computes a probability distribution over possible next tokens (e.g., “cat”, “puppy”, “car”, “skateboard”). It can either:
  • Choose the highest-probability token (greedy decoding), or
  • Sample from the distribution (stochastic decoding) with parameters that control diversity:
  • temperature — lower values make the distribution sharper (more deterministic); higher values increase randomness.
  • top-k — restricts sampling to the k most probable tokens.
  • top-p (nucleus sampling) — restricts sampling to the smallest set of tokens whose cumulative probability ≥ p.
A slide titled "Generative AI Models" showing next-token prediction for the phrase "I heard a dog bark at a ?" with candidate next tokens displayed as probability bars—"cat" marked most likely, followed by "puppy," "car," and "skateboard."
Because generation is iterative, a paragraph is not produced in one calculation; the model updates context after each predicted token and uses that updated context to predict the next token. Sampling parameters and decoding strategy determine whether outputs are repetitive, diverse, or coherent.
Be careful when interpreting embedding distances or attention weights as direct explanations. They provide useful intuition, but internal representations are distributed and learned jointly across many parameters.

​
Quick reference: core pipeline

​
Key takeaways

  • Tokenization converts text into units the model processes.
  • Embeddings provide numerical vector representations of tokens.
  • Transformers (with attention) create context-aware token representations.
  • Generation proceeds via iterative next-token prediction; sampling parameters control output diversity.
  • The whole pipeline is a sequence of learned transformations from text to numbers back to text.
Further reading and references:
  • ChatGPT — OpenAI — example of iterative next-token generation in deployed LLMs.
  • Transformer and attention mechanism papers and tutorials (search for “Attention is All You Need” and “transformer architecture” for in-depth explanations).

Watch Video