Skip to main content
Imagine this: you ask an AI assistant, “Why did my application deployment fail?” The assistant responds with a vague answer. You rewrite the prompt with more detail and try again—still vague. The problem isn’t only the prompt wording. The model never received the logs, error traces, or configuration files you meant to include. What you tweaked by rewriting the prompt is prompt engineering. But in real-world applications there are two prompt layers to consider:
  • System prompt — instructions the application supplies to the model (for example, “Answer only from the supplied incident data”).
  • User prompt — the question the user types (for example, “Why did our deployment fail?”).
Both matter, but they’re only part of what the model actually sees. Everything that the model receives at call time sits inside the context window — the model’s working memory. The context window is a fixed-size buffer measured in tokens. Every piece of text you place into that buffer consumes tokens and reduces remaining space. Context engineering is the discipline of deciding what fills that space so the model has the right information to answer. Prompt engineering optimizes the system and user prompts; context engineering controls the rest: conversation history, tool outputs, retrieved documents, summaries, and any other data you include before the prompt.

​
Why context engineering matters

  • A polished prompt cannot compensate for missing context — if logs or private documents never reach the model, the answer will be limited.
  • Token limits force trade-offs: include more relevant data or longer instructions, but not both.
  • Retrieval-augmented generation (RAG) is how you bring external knowledge into the model’s working memory without exhausting tokens.

​
Typical sources that occupy the context window

​
Retrieval-augmented generation (RAG) in brief

Because you can’t forward thousands of documents directly, RAG pipelines:
  1. Index a large document collection using embeddings.
  2. Use vector search to find the most relevant passages (top-k).
  3. Insert only those passages into the context window alongside the system and user prompts.
This “R” — retrieval — is the practical way to bring external knowledge into the model’s context without exhausting tokens.

​
Key distinctions

  • Prompt engineering: improves instructions (system prompt) and the user’s explicit question.
  • Context engineering: decides what the model actually sees before it generates a response (everything placed in the context window).
  • Practical impact: better context (right logs, relevant document passages, concise history) yields better answers.

​
Practical techniques for context engineering

  • Prioritize content
    • Include the most relevant logs, error messages, or concise summaries first.
    • Place the highest-value items near the beginning of the context block so they’re less likely to be trimmed.
  • Chunk and summarize
    • Break long documents into smaller passages and summarize each chunk.
    • Use automated summarizers to compress repeated context.
  • Use retrieval (RAG)
    • Index content with embeddings and fetch the top-k relevant passages rather than sending entire documents.
    • Tune k to balance breadth vs token cost.
  • Control token usage
    • Trim boilerplate, shorten system prompts, and remove unnecessary conversation history.
    • Prefer structured outputs from tools (which can be summarized) over verbatim dumps.
  • Use tools where appropriate
    • Invoke log parsers, diagnostics, or structured APIs and insert only their essential outputs into the context.
  • Cache useful summaries
    • Store compressed context (e.g., summarized incident state or metadata) for quick reuse in follow-up queries to avoid re-retrieving everything.

​
Example RAG / API request pattern

A typical flow to assemble context before calling a model:
  • Retrieve top-k passages from the vector store.
  • Summarize or filter those passages.
  • Construct the system prompt, concatenated with prioritized passages and the user question.
  • Send the assembled request to the model.
Example request body (illustrative; backend-specific fields will vary):

​
Practical tuning checklist

  • Limit conversation history to recent, relevant turns.
  • Place critical logs and error summaries near the top of the context.
  • Compress repeated diagnostic content into short summaries.
  • Use vector DBs and embeddings for fast, relevant retrieval.
  • Monitor token usage and adjust top-k or passage lengths to stay within limits.

​
Tools & references

  • Embeddings + vector stores: FAISS, Milvus, Chroma, Pinecone
  • RAG patterns and engineering: see articles on retrieval-augmented generation and production pipelines
  • Token considerations: use model documentation for exact context window sizes (e.g., ~8k, 32k tokens)
Links and further reading:
Context engineering = deciding what the model actually sees. You can tune prompts, but the answer is limited by the context window. Retrieval (RAG), summaries, and selective inclusion of logs/documents are the practical levers to improve answers.
Remember: garbage in, garbage out — and the inverse is true as well. The better the context you provide (the right logs, the right document passages, concise history), the better the model’s answer will be. Context engineering is the practical, application-level work that makes prompt engineering effective.

Watch Video