Skip to main content
Welcome to this section of the NVIDIA Generative AI LLMs Associate Certification — Software Development.

​
Question 1

When deploying an LLM for a production application with high-throughput requirements, which of the following is the most important performance metric to monitor?
  • GPU memory utilization
  • Tokens per second
  • Model parameter count
  • Cache hit ratio
Answer: Tokens per second Tokens per second (TPS) is the primary metric for high-throughput LLM applications because it measures the actual serving throughput — how many tokens your system generates or processes per unit time. TPS captures end-to-end behavior (including batching, parallelism, and I/O), and directly maps to capacity planning (concurrent users, requests per second, SLA fulfillment). Why TPS is the most important metric
  • TPS reflects real user-facing throughput and is actionable for scaling and optimization decisions.
  • Optimizations that increase TPS (better batching, improved GPU utilization, model parallelism, lower latency per token) directly increase the number of users you can serve.
  • TPS captures effects of system-level factors (serializer/deserializer overhead, network latency, and model compute) that individual resource metrics alone do not show.
How to think about the other metrics
  • GPU memory utilization: Important for determining fit of model and batch sizes on hardware, preventing OOMs, and guiding quantization or model sharding choices. Indirectly impacts TPS by limiting batch sizes or causing memory thrashing.
  • Model parameter count: Describes model capacity and quality, not runtime throughput. Larger models often reduce TPS but may be required for accuracy trade-offs.
  • Cache hit ratio: Improves effective throughput for repeated or similar requests by avoiding computation, but it’s workload-dependent. Cache effectiveness should be interpreted together with TPS.
Practical tips to measure and improve TPS
  • Measure TPS directly: TPS = tokens_processed / elapsed_seconds. Instrument request handlers to emit tokens produced and time taken.
  • Use batching: Group multiple requests into a single batch to amortize compute cost per token.
  • Optimize model execution: Mixed precision (AMP), model quantization, weight sharding, and inference engines (e.g., NVIDIA Triton) often yield higher TPS.
  • Profile end-to-end: Track CPU, GPU, memory, I/O, and network to identify bottlenecks that limit TPS (e.g., small batch sizes, data preprocessing overhead).
  • Autoscale based on TPS and latency targets rather than only CPU/GPU utilization.
Comparison table — what to monitor and why
Monitor tokens per second as your primary throughput metric, and use GPU memory utilization, cache hit ratio, and latency metrics to diagnose and optimize bottlenecks. Instrument your inference pipeline to emit TPS so autoscaling and optimization are driven by the user-facing capacity metric.
The image contains a question about the most important performance metric to monitor when deploying a large language model (LLM) for high throughput applications, with the answer being "Tokens per second" and an explanation provided.
References and further reading

Watch Video