Skip to main content
Picture a situation you’ve almost certainly experienced: you’re trying to return something you bought online and you end up talking to a tiny chatbot assistant at the bottom of the page. No matter how you phrase it — “I want to return my order,” “How do I get a refund?” — the bot keeps replying the same way: “Sorry, I didn’t quite get that.” That frustrating loop is not magic; it’s a system built on hand‑coded rules. For decades many AI systems were rule‑based. In the return-bot example, engineers enumerated the many ways customers might ask about returns and wrote conditional rules to match those phrasings. If an input matched a rule, the bot returned a scripted reply. Slightly different wording, and the system failed. The fundamental problem: natural language has far too many variations to cover exhaustively by hand. Rule-based systems are rigid and brittle; they break when faced with new or unexpected inputs. You can’t reasonably handwrite behavior needed for robust language understanding. Shift to learning from examples People flipped the approach: instead of encoding rules, train machines to learn patterns from labeled examples. For instance, instead of writing rules to detect spam phrases like “free money” or “extended warranty,” you provide many emails labeled “spam” or “not spam.” The model discovers statistical patterns in the data and internalizes them. No single human rule states “this phrase means spam” — the system infers that from examples. Stack layers of learned processing and you get a neural network: early layers detect simple patterns, later layers combine those into higher-level features. This worked well across domains (face recognition, speech recognition), but long-range language understanding remained difficult. Traditional sequence models such as RNNs and LSTMs processed words one at a time and tended to lose early context in long sequences because of limited state capacity and vanishing gradients. The transformer breakthrough In 2017, researchers at Google introduced the transformer architecture, which became the foundation for modern large language models. The key innovation: stop processing one token at a time and instead process the whole sequence in parallel. For every token, the transformer computes how much to attend to every other token in the sequence. This mechanism — self‑attention — creates direct connections between tokens that matter to each other, even when they are far apart. And that change unlocked two major capabilities because the transformer processes tokens in parallel and establishes direct token-to-token connections.
A presenter stands beside a glowing glass whiteboard with handwritten boxes and arrows labeled "Rules," "Learning," and "Transformer." The board contains notes about examples, neural networks, and brief rule examples.
First, you can train transformers on massive text corpora — the public web, books, and curated datasets — so they capture broad statistical patterns of language. Second, because attention links distant tokens directly, the model can preserve and use long-range context across conversations or documents (subject to the model’s context window size). Scale those models (more parameters, more data, more compute) and you get large language models (LLMs). The “T” in GPT stands for Transformer: the transformer is the architectural core enabling coherent, context‑aware text generation. A single arc: rules → learned patterns Viewed broadly, the shift is simple but profound: we moved from hand‑written rules to systems that learn behavior from data. Rule-based systems depend on a developer’s enumerations and remain brittle. Learned systems infer generalizable behavior from examples, which is why modern assistants can understand “I want to return my order” phrased dozens of different ways. Practical question to ask about any AI system: is it rule‑based or learned from data? They behave very differently — learned, transformer‑based models generalize far better across varied language than brittle, rule‑driven systems.
Ask whether a system is rule-based or learned from data. If it’s learned, it likely uses a model (often a transformer) that infers patterns from examples rather than following brittle, hand-coded rules.
Quick comparison Further reading and references These resources provide deeper technical context on self‑attention, transformer internals, and how scale influences emergent language capabilities.

Watch Video