Skip to main content
Welcome. This lesson explains the main types of inferencing used to generate predictions from trained AI models, how they differ, typical use cases, and when to choose each approach. What is inferencing?
  • Inferencing (or inference) is the process of using a trained model to produce predictions, classifications, or probabilities for new input data.
  • After training and validation, a model is deployed and receives fresh inputs; the outputs it returns are the inferences.
  • Example: A spam filter trained on millions of labeled emails inspects a new message and infers whether it is spam. A regression model trained on historical sales data forecasts future sales for new input.
An infographic titled "Inferencing in AI — Introduction" showing two panels: "Training" (brain/gear icon) labeled "Model learns from historical data" and "Inferencing" (lightbulb icon) labeled "Model applies this learning to make real-time decisions." The boxes are connected by a dotted line to show the flow from training to inference.
When to infer in production
  • Once a model reaches acceptable performance on validation data, it’s deployed to serve inferences in production.
  • Production inferencing powers many tasks: fraud detection, product recommendations, customer-service automation, telemetry scoring, and more.
  • The main decision is not model accuracy (same model) but how and when data is delivered to the model and how results are consumed.
Two primary inferencing modes
  • Real-time inferencing: predictions are made as individual inputs arrive, with minimal latency.
  • Batch inferencing: predictions are computed over a large set of inputs processed together on a schedule or in bulk.
An infographic titled "Types of Inferencing" comparing Real-Time and Batch inferencing. Real-Time processes data instantly for use cases like chatbots and fraud detection, while Batch processes data in bulk at scheduled intervals for tasks like social media sentiment analysis.
Real-time inferencing
  • Description: Processes each input as it arrives and returns a prediction immediately. Designed for low-latency, synchronous interactions.
  • Typical use cases: conversational agents and chatbots, voice assistants, fraud detection, autonomous vehicles, and recommendation engines that personalize interactions on-the-fly.
  • Trade-offs: Higher cost per request for always-on, low-latency endpoints; requires careful scaling and latency tuning.
  • Cloud example: On AWS, Amazon SageMaker provides managed real-time endpoints that host models and respond to prediction requests with low latency. Endpoints scale to handle concurrent requests and can be tuned for latency and throughput.
Amazon SageMaker supports both real-time endpoints for low-latency inference and Batch Transform jobs for offline/batch inference. Use real-time endpoints for synchronous, time-sensitive requests and Batch Transform for large, asynchronous workloads. See the SageMaker docs for deployment and scaling guidance.
A presentation slide titled "Real-Time Inferencing" that explains making instant predictions and notes uses like voice assistants and self-driving cars. It includes a speech bubble saying "Alexa, what's the weather today?" between a user icon and a smart speaker icon.
Batch inferencing
  • Description: Processes many inputs together in bulk (as a job). Suitable when latency is not critical and throughput/cost-efficiency matter more.
  • Typical use cases: nightly scoring of large datasets, processing accumulated telemetry or logs, large-scale sentiment analysis collected over hours/days, offline customer segmentation, and risk scoring.
  • Trade-offs: Lower cost per prediction for large volumes, but results are not immediate and are consumed asynchronously.
  • Cloud example: On AWS, SageMaker Batch Transform runs inference on collections of input data without a real-time endpoint, often saving cost for large offline jobs.
A dark-themed presentation slide titled "Batch Inferencing" with two brief bullet points describing batch processing and periodic analysis, accompanied by grouped document icons and a large faded play symbol.
Quick comparison Decision guide — when to choose which
  • Choose real-time if:
    • The business requires immediate decisions (e.g., fraud blocking, chat replies).
    • The application is interactive or user-facing with strict SLAs.
  • Choose batch if:
    • Latency can be relaxed and you need to process large volumes cost-effectively.
    • Processing can be scheduled (nightly, hourly) or run as asynchronous jobs.
  • Hybrid approaches:
    • Use both: real-time endpoints for immediate scoring and periodic batch jobs for re-scoring or bulk analytics.
    • Precompute features or use streaming systems for near-real-time needs that don’t require strict single-request latency.
Summary
  • Real-time inferencing: immediate, low-latency predictions for interactive and time-sensitive scenarios.
  • Batch inferencing: bulk processing for cost-efficient, large-scale scoring where latency is not critical.
  • They use the same trained models but differ in how data is delivered and how results are consumed. Select the mode that aligns with your latency requirements, volume, and cost constraints.
Links and references

Watch Video