Skip to main content
Welcome back. This lesson focuses on security and privacy considerations specific to generative AI (GenAI) and machine learning systems. These systems introduce unique attack surfaces and failure modes that require targeted mitigation, layered defenses, and continuous operational monitoring. AI systems have specific vulnerabilities that you should understand to secure deployments effectively.
A presentation slide titled "AI Systems – Security and Privacy" showing a shield with a brain icon at the center connected to surrounding security-related icons (fingerprint, lock, alert, eye, etc.). Two short points state that AI systems have specific vulnerabilities and understanding them is key to secure deployments.

​
Common threats to AI systems

  • Training-data attacks (data poisoning and backdoors)
  • Adversarial inputs at inference time
  • Model extraction, inversion, and membership inference
  • Prompt injection and jailbreaks for large language models (LLMs)
  • Data breaches and leakage of sensitive training data
The sections below describe each threat, typical impacts, and recommended countermeasures.

​
Training-data integrity: poisoning and backdoors

Manipulation of training data can change model behavior in subtle or dramatic ways. Common techniques include label flipping, inserting maliciously crafted samples, or embedding backdoors—inputs that trigger attacker-chosen behaviors. Consequences range from misclassification (false negatives/false positives) to targeted failures in critical domains such as healthcare diagnostics, fraud detection, or autonomous systems.
An infographic titled "Threats to Training Data Integrity" illustrating data poisoning, with corrupted data entries (document icons) feeding into a model (gear and computer icons). Text notes that training data manipulation can alter model behavior and gives an example of a fraud detector misclassifying fraudulent transactions.
Mitigations
  • Provenance, cataloging, and versioning of datasets
  • Strict access controls and audit trails for data repositories
  • Automated validation checks for outliers and anomalous labels
  • Red-team testing and targeted adversarial/poisoning tests before deployment

​
Adversarial inputs and misclassification

Attackers can craft small perturbations to inputs (images, audio, or text) that cause models to misclassify. These adversarial examples exploit model sensitivity near decision boundaries—for example, altered face images that bypass facial recognition or slightly perturbed signals that degrade classifier performance.
A presentation slide titled "Adversarial Inputs as a Security Threat" showing side-by-side unaltered and digitally altered photos of a child's face. Text explains attackers can manipulate input images to cause misclassifications and bypass facial recognition models.
Mitigations
  • Adversarial training and augmentation of datasets with adversarial examples
  • Defensive techniques such as randomized smoothing, input preprocessing, and ensemble methods
  • Runtime input filters and anomaly detection to flag suspicious inputs

​
Model inversion, membership inference, and extraction

Repeated or carefully crafted queries can allow attackers to infer training-data properties (model inversion), determine whether a specific record was in a training set (membership inference), or reconstruct a model’s decision surface (model extraction). These attacks can reveal sensitive training records or proprietary model behavior.
A slide titled "Model Inversion and Reverse Engineering Threats" showing input queries and model outputs with an attacker icon recreating training data. Text explains attackers can infer training data by repeated queries and can reverse-engineer replica models with similar behavior.
Mitigations
  • Apply differential privacy or noise injection during training/fine-tuning
  • Enforce rate limiting and query budgets to reduce extraction feasibility
  • Monitor and detect anomalous query patterns
  • Provide coarse-grained APIs rather than full model access where feasible

​
Prompt injection and jailbreaks for LLMs

Prompt injection is a class of attacks where crafted inputs manipulate an LLM to ignore system instructions or reveal sensitive information (internal prompts, data sources, credentials, etc.). These attacks can lead to disclosure of confidential material or the model producing harmful outputs.
A presentation slide titled "Prompt Injection Attacks on Large Language Models" showing a UI mockup with an "Add object" prompt and a highlighted "Attacker prompt" image. Text boxes explain that prompt injection manipulates model responses with malicious inputs and can make models reveal sensitive information.
Prompt injection attacks can bypass intended system prompts and policies. Treat user-supplied content as untrusted, apply layered defenses (policy enforcement, response filtering, and detection models), and avoid exposing sensitive system prompts to user-controllable contexts.
Mitigations
  • Sanitize and normalize all user-provided text before including in prompts
  • Use separate, immutable system prompts that are never concatenated directly with user inputs
  • Apply output filters and classifiers trained to detect malicious prompt patterns
  • Leverage managed-service guardrails when available (for example, Amazon Bedrock guardrails)

​
Threats and mitigations at a glance

​
Mitigations and best practices (detailed)

  • Data hygiene and provenance: validate, catalog, and version training datasets; enforce access controls; maintain audit logs.
  • Input validation and sanitization: use strict schema checks, token and size limits, and content filters for user-provided inputs.
  • Adversarial training and robustification: incorporate adversarial examples and use defensive techniques appropriate for the modality.
  • Differential privacy and noise addition: apply differentially private training or fine-tuning to reduce membership inference risk.
  • Rate limiting and query monitoring: throttle requests and detect anomalous access patterns to limit model extraction.
  • Guardrails and content filtering: implement policy-based filters, engineered system prompts, and response validators for LLMs.
  • Access control and network isolation: enforce least privilege with IAM, use VPC endpoints and private subnets to restrict endpoint access.
  • Encryption and key management: encrypt data at rest and in transit and manage keys centrally using a KMS.
To mitigate prompt injection and unsafe outputs, combine multiple layers: input sanitization, strict system prompt handling, policy-based output filtering, and detection models that identify suspicious prompt patterns. For managed services, enable and configure available guardrail features to reduce risk.

​
Operational monitoring and detection

Continuous monitoring, drift detection, and alerting are essential. Early detection of data-distribution changes or model-performance degradation allows investigation of potential attacks (data poisoning, adversarial inputs) or natural drift and enables timely remediation (retraining, rollback, or throttling). Tools and patterns
  • Baseline profiling and continuous monitoring of incoming data and inference outputs
  • Alerts and automated runbooks for detected anomalies
  • Centralized logging and long-term retention for forensic analysis
Amazon SageMaker Model Monitor can inspect incoming data and model inferences, compare them to baselines, and detect data-quality issues, distribution drift, and performance degradation. Use Model Monitor to generate alerts and metrics when deviations occur so you can trigger retraining, rollback, or investigation.
Amazon SageMaker Model Monitor can automatically profile baseline data, monitor live production traffic, and alert on data drift or quality anomalies so you can act before issues become outages or security incidents.
A presentation slide titled "Amazon SageMaker Model Monitor for Real-Time Threat Detection" with a central Amazon SageMaker icon. It highlights monitoring of data and model quality and detecting data drift, anomalies, and deviations from baselines.
Use monitoring services like Amazon CloudWatch to collect logs, metrics, and alerts. CloudWatch can trigger alarms, runbooks, or automated responses when metrics indicate integrity or quality deviations. Store logs centrally (for example, in Amazon S3 or a log analytics platform) to support root-cause and forensic investigations.
A diagram titled "Amazon CloudWatch for Monitoring and Alerting" showing CloudWatch detecting model quality deviations, sending alerts, and writing logs. It also shows logs stored in an Amazon S3 bucket and notes CloudWatch can log model monitoring data and alert on quality or data-integrity deviations.

​
Encryption, keys, and certificate management

Encrypt data at rest and in transit. Use a centralized key-management service (for example, AWS KMS) to manage encryption keys, and consider envelope encryption for large artifacts. Use AWS Certificate Manager (ACM) to provision and manage TLS certificates for supported endpoints. Combine encryption with strict IAM, network controls, and logging to reduce exposure.
A flowchart titled "Data Security — Encryption at Rest and In Transit" showing steps from user data input through processing, storage, retrieval/decryption, application processing, encryption for transmission, transfer, and client-side decryption. A highlighted note on the right emphasizes the importance of data encryption for AI security and using AWS KMS for encryption management.
Key practices
  • Encrypt training, validation, and production data at rest and in transit
  • Use dedicated KMS keys per environment and rotate keys regularly
  • Audit and log key usage and certificate issuance
  • Limit who can decrypt model artifacts and training data via IAM policies

​
Summary

Securing AI systems requires layered, defense-in-depth approaches across the model lifecycle:
  • Preventive controls: data provenance, input sanitization, robust training practices
  • Detective controls: model and data monitoring, logging, anomaly detection
  • Mitigations and recovery: rate limiting, retraining, rollbacks, and incident response playbooks
  • Foundational controls: encryption, IAM, network isolation, and key/certificate management
Combine these practices with continuous testing (including adversarial and red-team exercises), and leverage managed services and built-in guardrails where available to reduce risk and accelerate secure, compliant AI operations.

Watch Video