> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# LCEL Demo 3
> Explains batching inputs for LangChain chains to run multiple prompt-variable dictionaries concurrently, boosting throughput while preserving input order and addressing concurrency limits and error handling.
This lesson explains batching: sending multiple inputs to a chain at once so the runtime can execute them concurrently. It builds on synchronous chain invocation and streaming output, and demonstrates how to provide a list of input dictionaries where each dictionary supplies the prompt variables for one invocation (for example, the `question` key used by the prompt).
Batching is useful for throughput optimization: the runtime unpacks your list and invokes the chain for each item (typically in parallel), returning a list of outputs that maintain the input order.
## How batching works (high level)
* Provide a list of dictionaries. Each dictionary maps prompt variable names to values (e.g., `{"question": "..."}`).
* Call `chain.batch(...)`. The runtime executes the chain once per dictionary in the list.
* The return value is a list of outputs in the same order as the input list.
* The runtime usually parallelizes these invocations, so large batches can complete in roughly the same time as a single invocation (subject to concurrency limits, API rate limits, and client/runtime settings).
## Example: prompt template + LLM + output parser + batch
The example below demonstrates:
1. Creating a chat prompt template that expects a `question` variable.
2. Composing the prompt with an LLM and an output parser.
3. Building a list of input dictionaries.
4. Executing the chain via `chain.batch(...)` to process inputs concurrently.
```python theme={null}
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain.chat_models import ChatOpenAI
# Prompt template expects a variable named `question`
prompt = ChatPromptTemplate.from_template(
"""
You are a helpful assistant.
Answer the following question: {question}
"""
)
llm = ChatOpenAI()
output_parser = StrOutputParser()
# Compose the chain
chain = prompt | llm | output_parser
# A list of input dictionaries; each dict maps the prompt variable `question`
questions = [
{"question": "Tell me about The Godfather Movie"},
{"question": "Tell me about Avatar Movie"}
]
# Batch execution: the runtime will unpack the list and execute the chain for each item,
# typically in parallel (concurrency settings may depend on the runtime and client).
response = chain.batch(questions)
# `response` is a list where each element corresponds to one input in `questions`
print(response[0]) # Answer for the first question
print(response[1]) # Answer for the second question
```
Ensure each element in the batch is a dictionary whose keys exactly match the prompt template variable names (for example, use `{"question": "..."}` if the template refers to `{question}`).
## Key points (quick reference)
| Concept | Details |
| ----------------- | ----------------------------------------------------------------------------------------------------------------- |
| Input format | A list of dictionaries, e.g. `[{ "question": "..." }, { "question": "..." }]` |
| Invocation method | `chain.batch(inputs_list)` |
| Return type | A list of outputs aligned with input order |
| Behavior | Runtime typically parallelizes invocations; actual concurrency depends on runtime/client settings and rate limits |
## Measuring performance: single vs batched runs
To compare single synchronous execution time versus batched execution time, measure wall-clock time for both approaches. The snippet below runs each input individually (synchronously) and then runs the same inputs via `chain.batch(...)`.
```python theme={null}
import time
# Prepare inputs
questions = [
{"question": "Tell me about The Godfather Movie"},
{"question": "Tell me about Avatar Movie"}
]
# Single synchronous runs (run each input individually)
start = time.perf_counter()
single_responses = [chain.run(q) for q in [d["question"] for d in questions]]
single_elapsed = time.perf_counter() - start
# Batched run
start = time.perf_counter()
batch_response = chain.batch(questions)
batch_elapsed = time.perf_counter() - start
print(f"Single-run total time: {single_elapsed:.2f}s")
print(f"Batch-run total time: {batch_elapsed:.2f}s")
```
Note: Depending on the chain API in your environment, you may pass single-run inputs as dictionaries (e.g., `chain.run({"question": "..."})`) or directly as a string when the chain accepts a single positional argument. Use the form that matches your LangChain runtime version.
Batching and concurrency increase throughput but also increase parallel API usage. Be mindful of rate limits, concurrency caps, costs, and downstream system limits when issuing large batches.
## Practical considerations and tips
* Validate input keys: confirm that each dictionary includes exactly the keys referenced by the prompt template.
* Batch size: tune batch sizes to balance latency, throughput, and rate limits.
* Error handling: decide how to handle per-item failures (e.g., retries, partial failures, logging). The runtime may surface errors per-item or for the whole batch depending on implementation.
* Idempotency & costs: repeated or retried batches can increase cost—track requests and consider idempotency keys if supported.
## Summary
Batching with `chain.batch(...)` lets you send multiple prompt-variable dictionaries at once so the runtime can execute them in parallel (subject to concurrency and rate limits). It returns a list of outputs in the same order as the inputs. Use batching to dramatically improve throughput when running many similar invocations.
Further topics you may explore: runnable pass-through, more advanced LCEL concepts, and runtime-specific concurrency configuration.
## Links and references
* [LangChain Documentation](https://langchain.com/docs/)
* [OpenAI API](https://platform.openai.com/docs)