Skip to main content
Welcome back. This lesson explains practical, responsible approaches to selecting models for production AI systems. Model selection is one of the earliest and most consequential decisions in an ML program: it influences system performance, user experience, operational cost, regulatory exposure, and overall business outcomes. A well-chosen, well-tuned model improves accuracy, responsiveness, and customer trust; a misaligned model increases latency, cost, and risk.
A dark-themed slide titled "Model Selection AI Systems – A Critical Step" with a central "Model Selection" box. It shows four connected considerations—Performance, User Experience, Profitability, and Market Strategy—each illustrated with a small icon.
Key trade-offs to evaluate up front:
  • Larger models frequently offer higher capability but also increase latency, energy use, and cost.
  • Commercial third‑party models may have licensing or usage restrictions that conflict with open-source or export strategies.
  • Models not trained or tuned for your target domain can degrade user experience.
A good model choice balances technical capability, user needs, cost constraints, and governance obligations.
A split slide comparing a "Well-Tuned Model" on the left — which "Boosts customer satisfaction" and "Increases sales with accuracy" — with a "Poor Model" on the right that "Reduces performance quality." The left side is light-themed and the right side is dark-themed.

​
1. Start by narrowing the application use case

Define the real application first — driven by business objectives and domain constraints — not by available technology. Distinguish capability (what the model can do) from use case (how you will use it). For example, face recognition is a capability; the use case might be:
  • gallery retrieval to find missing persons, or
  • virtual proctoring to detect exam integrity violations.
The intended application determines success metrics, privacy requirements, and the best model configuration.
An infographic titled "Narrowing the Application Use Case" showing "Face Recognition Technology" in the center with two example applications: "Gallery Retrieval" (assists in finding missing persons) and "Virtual Proctoring" (monitors an exam room).

​
2. Choose metrics that reflect user and business goals

Selecting appropriate evaluation metrics is critical. Two common patterns:
  • Gallery retrieval (find as many possible candidates): prioritize recall — you accept more false positives to avoid missing true matches.
  • Celebrity recognition (high-confidence identification): prioritize precision — minimize incorrect matches to preserve trust.
Match metrics to the user impact you care about.
A slide comparing two systems: the left box describes a "Gallery Retrieval System" that focuses on recall (retrieving many matches, accepting some false positives), and the right box describes "Celebrity Recognition" that focuses on precision (minimizing incorrect matches to maintain user confidence).
Example table: mapping use cases to metric priorities

​
3. Use-case-driven tuning: neutrality vs personalization

Different applications require different model behaviors. For retail, for example:
  • Cataloging products: choose neutral, broad-reaching models that prioritize clarity and coverage.
  • Persuasion or personalized recommendations: choose models tuned for engagement and relevance to a target audience.
Your use case determines whether neutrality or personalization should guide model selection and tuning.
A presentation slide titled "Example: Narrow Use Case for Generative AI in Retail" comparing two use cases: "Cataloging Products" (requires a neutral model, clarity and detail, prioritizes neutrality) and "Persuading a Specific Demographic" (tailors recommendations, targets a narrow audience, focuses on engagement and relevance).

​
4. Validate performance on representative data and realistic environments

Model performance is dataset- and environment-specific. Never assume a model that performs well on benchmark data will generalize to your production data. Always:
  • Test candidate models on representative, realistic datasets (including edge cases).
  • Measure latency, throughput, and accuracy under realistic load and hardware.
  • Validate performance across demographic, geographic, and domain slices to find disparate behavior.
A presentation slide titled "Choosing a Model Based on Performance" with a 3D bar chart comparing Dataset A, B, and C (Dataset B is the highest). A caption below advises testing the model on specific datasets.

​
5. Continuous evaluation and periodic retuning

Model quality results from both architecture and the datasets used to evaluate it. Because models, data, and user behavior evolve, implement continuous evaluation and periodic retuning. A model that was optimal months ago may become suboptimal:
  • Instrument production to capture drift signals.
  • Run offline evaluation pipelines against fresh data slices.
  • Set retraining or redeployment triggers based on performance thresholds.
A slide with a chart titled "Model Performance – A Function of Dataset and Model" showing two upward-sloping lines across model versions: Dataset A (steeper improvement) and Dataset B (more gradual). A note at the bottom stresses continuous evaluation and tuning to adapt to dataset and model changes.
Continuously monitor model drift and dataset changes. Implement automated evaluation pipelines and retraining triggers so model performance remains aligned with evolving data and business goals.

​
6. Environmental, energy, and resource impacts

Large models consume more compute and energy. When selecting models, weigh accuracy gains against environmental and operational costs. Strategies to reduce energy footprint include:
  1. Prefer energy-efficient architectures when feasible.
  2. Use algorithms that require fewer computations.
  3. Optimize hardware utilization and placement (right-sizing, batching, quantization).
  4. Select models that hit accuracy targets with less compute.
Balancing accuracy with energy and cost is often a business decision—sometimes a smaller, efficient model is preferable.
An infographic titled "Reducing AI’s Energy Footprint" showing four numbered strategies. They list energy-efficient model architectures, smarter algorithms with fewer computations, optimized hardware utilization, and energy-efficient AI models with reduced computational power.
Consider hardware lifecycle and sustainability as part of total cost. Manufacturing and disposing of hardware create environmental impacts. Mitigations include sharing/reusing hardware, sustainable lifecycle management, and designing for longevity to reduce e-waste.
A presentation slide titled "Reducing Resource Utilization" showing four numbered cards with icons. The cards advise: encourage reuse of hardware; implement sustainable lifecycle management; plan for the entire lifecycle; and design with sustainability in mind.

​
7. Assess direct and indirect impacts — economic and societal

Responsible model selection requires assessment of both direct benefits and indirect societal impacts. Conduct impact assessments and mitigation planning that consider:
  • Performance benefits and business efficiencies.
  • Privacy, security, and explainability trade-offs.
  • Potential worker displacement, inequality, and concentration of data power.
Use multi-stakeholder reviews to weigh economic gains against societal risks.
A presentation slide titled "Responsible AI – Economic Considerations" showing a balance scale comparing Economic Benefits and Economic Risks. The benefits side lists streamlined processes and automation/efficiency, while the risks side lists societal implications, worker/job displacement and inequality.

​
8. Moral agency, transparency, and governance

No current AI system possesses human-level moral reasoning. Implement guardrails that support transparency, traceability, and accountability:
  • Transparency: Document what the model does, its intended use cases, and its limitations.
  • Traceability: Log inputs, outputs, and key decisions to enable audits and root-cause analysis.
  • Accountability: Assign clear ownership for model behavior, remediation, and change control.
Choose models and processes that enable these properties to satisfy ethical and regulatory expectations.
A presentation slide titled "Responsible AI – Moral Agency and Value Alignment" showing three colored icons and labels: Transparency, Traceability, and Accountability. Each icon (an eye, footprints, and a document with a check) is paired with brief captions about aligning models with ethical standards.
Be careful with third-party models: confirm licensing, data usage restrictions, and compliance obligations before integrating them into products. Failure to do so can create legal and operational risk.

​
Quick checklist for responsible model selection

  • Define the concrete use case and metrics.
  • Test candidate models on representative production-like data.
  • Evaluate latency, throughput, and cost on target hardware.
  • Validate fairness and slice-level performance across demographics.
  • Measure energy and resource costs; include hardware lifecycle in TCO.
  • Ensure transparency, logging, and clear ownership for model outputs.
  • Schedule continuous evaluation and automated retraining mechanisms.
Thanks for reading.

Watch Video