> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Semantic Language Models > Explains how semantic language models map words to numerical embeddings, capture relationships in vector space, and integrate with transformer LLM pipelines. Welcome to the topic of semantic language models. Generative AI models transform input data into new content (text, images, structured data). Under the hood, these models work with numerical representations so computers can perform the required calculations. This article explains how meaning in language is mapped into numbers, how similar concepts cluster in a learned space, and how embeddings connect to transformer-based large language models (LLMs). ## Why move beyond frequency counts? Statistical analysis (counts and co-occurrences) is useful, but limited. Frequency alone cannot explain meaning or relationships between words. Example: the words dog, puppy, cat, and kitten might have different frequency counts — e.g., dog 100, cat 80, puppy 20 — but frequency does not reveal that a puppy is a young dog or that a kitten is a young cat. Semantic models aim to capture those relationships so generative AI can produce meaningful responses. If you ask “What is a young dog called?” a model that understands semantics can answer “puppy” by recognizing the relationship between “young” and “dog.” ## From tokens to numbers: vectorization Neural networks operate on numbers, so tokens like "dog" must be converted into numeric form. Vectorization maps each token to a numerical vector (an embedding) in a high-dimensional space. During training, models learn embeddings that reflect usage patterns and semantic relationships. A slide titled "Semantic Language Models" showing example tokens (dog, puppy, cat, kitten, young, ball, tree) mapped to numerical vectors (e.g., dog → [0.8, 0.6, 0.1]). These illustrative vectors help build intuition; real embeddings typically have hundreds or thousands of dimensions. ```python theme={null} # Example token → vector mappings (illustrative only) token_vectors = { "dog": [0.8, 0.6, 0.1], "puppy": [0.9, 0.7, 0.4], "cat": [0.7, 0.5, 0.2], "kitten":[0.8, 0.6, 0.5], "young": [0.1, 0.1, 0.3], "ball": [0.3, 0.9, 0.1], "tree": [0.2, 0.1, 0.9], } ``` ## Why vectors capture meaning Vectors place tokens in a metric space where distance or angle indicates relatedness. Common similarity metrics include cosine similarity and Euclidean distance. In simple visualizations, related words cluster together (e.g., cat, kitten, puppy, dog), while unrelated words (e.g., tree) lie farther away. ## Common relationship types Semantic models capture more than synonyms. They learn hierarchical, categorical, and associative relationships. A slide titled "Semantic Language Models" that shows word-to-related-concept pairs: Dog → Puppy, Cat → Kitten, Car → Vehicle, and Doctor → Hospital. | Relationship type | Description | Example | | - | - | - | | Hierarchical / derivational | One concept is a subtype or derived form of another | `dog → puppy` | | Categorical | Membership in a broader category | `car → vehicle` | | Associative / contextual | Frequent co-occurrence or strong association | `doctor ↔ hospital` | | Morphological / inflectional | Related surface forms or derivations | `run → running` | ## A simple mathematical intuition Some semantic relationships are approximately reflected by arithmetic on vectors. The following simplified example illustrates this intuition: ```python theme={null} # Simplified illustration of additive semantics (for intuition only) dog = [0.8, 0.6, 0.1] cat = [0.7, 0.5, 0.2] young = [0.1, 0.1, 0.3] puppy = [0.9, 0.7, 0.4] # approx. dog + young kitten = [0.8, 0.6, 0.5] # approx. cat + young # dog + young ≈ puppy # cat + young ≈ kitten ``` These arithmetic examples are illustrative and intended to build intuition. Modern transformer-based models learn highly non-linear relationships; they do not rely on simple vector addition to derive meaning. ## How embeddings fit into the transformer LLM pipeline Embeddings are the first numerical representation used by transformers. The end-to-end pipeline in most transformer-based LLMs looks like this: 1. Tokenization: Split input text into tokens (words or subwords, e.g., byte-pair encoding). 2. Embedding: Map tokens to learned vectors; often add positional encodings. 3. Transformer processing: Multiple layers use attention to model context and relationships among tokens. 4. Decoding / prediction: The model outputs probabilities for the next token(s); repeated decoding generates text. The pipeline can be summarized as: text → tokens → vectors → transformer (LLM) → generated text. A slide titled "Semantic Language Models" showing a pipeline: text → tokens → vectors → LLM. It uses the example "Excellent keyboard" → "Excellent | keyboard" → [0.9, 0.2, 0.4]. Table: pipeline steps and purpose | Step | Purpose | Notes | | - | - | - | | Tokenization | Convert raw text to token sequence | Subword tokenizers reduce vocabulary size and handle rare words | | Embedding | Map tokens to numeric vectors | Learned during model training; may include positional encodings | | Transformer layers | Model context via attention and feed-forward networks | Multi-head attention captures different kinds of relationships | | Decoding / generation | Produce next token probabilities and output sequence | Causal/decoder-only models generate left-to-right; encoder-decoder models are used for seq2seq tasks | Inside transformers, attention weights determine how strongly tokens influence each other. For instance, attention helps resolve pronoun references: in “The dog chased the ball because it was excited,” attention helps identify what “it” refers to by weighing token relationships. ## Summary * Frequency counts and co-occurrence statistics are useful baselines but do not capture meaning. * Semantic language models represent tokens as embeddings (vectors) that encode relationships and context. * Related concepts cluster in high-dimensional embedding space; relationships include hierarchical, categorical, associative, and contextual types. * Embeddings feed into transformer-based LLMs, which use attention mechanisms and many layers to generate text token-by-token. ## Links and references * [AI-901: Microsoft Azure AI Fundamentals](https://learn.kodekloud.com/user/courses/ai-901-microsoft-azure-ai-fundamentals) * Transformer overview: [https://arxiv.org/abs/1706.03762](https://arxiv.org/abs/1706.03762) * Introduction to word embeddings and cosine similarity: [https://machinelearningmastery.com/what-are-word-embeddings/](https://machinelearningmastery.com/what-are-word-embeddings/) If you want to experiment with embeddings, try producing small vector visualizations (e.g., via PCA or t-SNE) from pre-trained models to see how similar words cluster in practice. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.