AI Security Published: 2026-07-22 · 16-min read

AI Container Security — A Playbook & Open-Source Landscape

Almost 100% of AI workloads run in containers — but their attack surface dwarfs ordinary microservices: exposed GPU devices amplify escape risk, model weights are million-dollar assets, pickle loading equals code execution, and Agent code sandboxes are a brand-new high-risk scenario. This article moves from threat model to a defense-in-depth architecture, and hands you a ready-to-copy open-source tool map plus a 90-day rollout plan.

1. Why AI container security is its own topic

AI training / inference / Agent / RAG almost always run in Docker / K8s / Ray, yet they differ fundamentally from ordinary microservices: multi-GB images (CUDA/PyTorch/weights), very deep dependency graphs, frequent GPU-device and shared-memory needs, model weights as core assets, highly non-deterministic LLM output and Agent behaviour, and heavy egress (HuggingFace, OpenAI, vector DBs).

Five core risks:

  1. Exposed GPU = amplified container escape (NVIDIA Container Toolkit CVE history);
  2. Model = asset: stolen weights equal core-code leakage;
  3. AI-framework RCE: PyTorch, Transformers, vLLM, Triton, Ray all have history;
  4. Pickle deserialization: torch.load / pickle.load load = code execution;
  5. Agent sandbox escape: containers that let Agents run code must assume the code is malicious.

2. Threat model (three layers)

1) Image layer

  • Base-image supply chain: nvidia/cuda:*-devel (10GB+) and pytorch/pytorch often ship critical CVEs; missing signatures allow MITM replacement;
  • Python dependency poisoning: Typosquatting / Dependency Confusion / Slopsquatting (LLM-hallucinated package names squatted);
  • Model files as code: torch.load() defaults to weights_only=False; HuggingFace trust_remote_code=True can load malicious .py.

2) Runtime

  • GPU escape: CVE-2024-0132 (TOCTOU), CVE-2025-23266 ("NVIDIAScape", 3-line Dockerfile escape);
  • --ipc=host and large --shm-size break isolation; GPU-memory side channels;
  • Inference-framework RCE: vLLM, Triton, TorchServe (ShellTorch), Ray Dashboard (ShadowRay, unauth RCE), Gradio, Jupyter;
  • Agent code execution: single-layer containers are not enough — use gVisor / Kata / Firecracker.

3) K8s layer

  • RBAC over-privilege: AI Operators often need cluster-admin; a compromised Pod hands over the cluster;
  • Missing network policy: unauthenticated inference services exposed (Ollama 11434, Ray 8265 on the public internet);
  • GPUs lack cgroup-grade isolation; misconfigured MIG allows cross-instance access; unencrypted PVCs leak models/datasets.

3. Defense-in-depth (CI/CD → Admission → Runtime → AI app)

  • CI/CD: Trivy/Grype image scan + modelscan/picklescan model scan + Syft SBOM + pip-audit/osv-scanner dependency audit + Cosign signing;
  • Admission: Kyverno/OPA verify image signatures, enforce PSS (no privileged/hostPID/hostNetwork), model-source allow-list, GPU/memory quotas;
  • Runtime: gVisor/Kata/Firecracker isolation + Falco/Tetragon eBPF behaviour detection + Cilium L7 network policy + Istio mTLS + MIG/HAMi GPU isolation;
  • AI app layer: AI gateway (LiteLLM + LLM Guard), Tool/MCP sandbox, input/output DLP, OpenTelemetry/Langfuse auditing.

Key Pod security config (essentials)

  • ❌ No privileged, hostNetwork/hostPID/hostIPC, mounting docker.sock or /;
  • ✅ Model dir readOnly, automountServiceAccountToken:false, image by digest not tag;
  • ✅ runAsNonRoot, readOnlyRootFilesystem, drop ALL capabilities, seccomp RuntimeDefault;
  • ✅ For strong isolation use runtimeClassName: gvisor; default-deny egress NetworkPolicy and block the metadata service (incl. Alibaba Cloud 100.100.100.200).

4. Open-source tool map (combine to cover 95% of scenarios)

  • Image/dependency scan: Trivy, Grype, Syft, Dockle, Hadolint;
  • AI-model scan: modelscan, picklescan, fickling, NB Defense, garak (LLM vuln scanner);
  • Admission: Kyverno (K8s-native), OPA Gatekeeper, Kubescape (NSA/CIS);
  • Runtime detection: Falco, Tetragon (eBPF real-time block), Tracee, KubeArmor;
  • Isolation runtimes: gVisor, Kata Containers, Firecracker;
  • Network: Cilium (L7), Calico, Istio Ambient;
  • Supply chain: Cosign/Sigstore, in-toto, SLSA;
  • Secrets: External Secrets, Vault, Infisical, SOPS;
  • AI guardrails: LLM Guard, NeMo Guardrails, Llama Guard, Rebuff, PyRIT (AI red-team);
  • Agent sandbox: E2B, Daytona, Open Interpreter;
  • GPU virtualization: HAMi, NVIDIA GPU Operator, Volcano.

5. Recommended rollout by scenario

  • Small teams (lightweight): Trivy + modelscan + Kyverno (5 core policies) + Falco + Calico default-deny + Vault + LLM Guard;
  • Mid-to-large / multi-tenant: add Tetragon real-time blocking, Cilium L7, MIG/HAMi, gVisor+Firecracker, Kubescape compliance, Langfuse observability;
  • Finance / healthcare strong isolation: Chainguard signed base images, Kata by default, code execution via Firecracker with full network isolation, fully encrypted PVCs, zero-trust mTLS + egress allow-list, aligned to ISO 42001 / EU AI Act.

Kyverno five must-have policies: no privileged, no hostPath, enforce runAsNonRoot, enforce Cosign signature verification, enforce resources.limits.

6. Key CVE quick reference

  • CVE-2025-23266 NVIDIA Container Toolkit "NVIDIAScape" → upgrade to 1.17.4+;
  • CVE-2024-0132 same component TOCTOU escape → 1.16.2+;
  • CVE-2023-48022 Ray Dashboard "ShadowRay" unauth RCE → 2.8.1+ and enable auth;
  • CVE-2023-43654 TorchServe "ShellTorch" → 0.8.2+;
  • CVE-2024-37032 Ollama "Probllama" path traversal → 0.1.34+; runc "Leaky Vessels" CVE-2024-21626 → 1.1.12+.
Four basics block 80% of real attacks: upgrade NVIDIA Toolkit, stop public exposure of Ray/inference services, disable pickle (use safetensors), enable a default-deny NetworkPolicy.

7. 90-day rollout

  1. Days 1-30 (baseline): inventory all AI container assets; scan every image with Trivy and every model with modelscan; find and fix privileged/hostPath; upgrade NVIDIA Container Toolkit.
  2. Days 31-60 (policy & runtime): deploy Kyverno's 5 core policies, Falco behaviour detection, default-deny egress NetworkPolicy; adopt Vault + External Secrets to remove hard-coded keys; enable Cosign signing + verification.
  3. Days 61-90 (AI-specific): put LLM Guard / NeMo Guardrails in front of LLM apps; run garak / PyRIT AI red-team scans; move Code-Interpreter Agents onto gVisor / E2B sandboxes; establish an AI-app pre-launch security review.

8. Maturity self-assessment (L1→L5)

L1 relies only on cloud-provider basics; L2 image scan + PSS + Vault; L3 admission control + runtime detection + network policy + image signing; L4 sandbox isolation + AI guardrails + Cilium L7 + AI red-team; L5 zero-trust + real-time blocking + AI SOC + ISO 42001. Most enterprises' AI workloads lag their traditional business by 1-2 maturity levels.

Bottom line: AI container security = cloud-native security ∪ AI security ∪ GPU specifics. Don't reinvent the wheel — the open-source combination above covers 95% of scenarios; runtime detection beats one-off scans; and when an LLM writes code, a sandbox is the default.

Halocent offers an "AI Container Security Baseline Assessment" for your AI platform — covering images, models, K8s configuration and runtime, with an actionable remediation checklist. Contact us.

← Back to News & Insights

Want an AI container-security baseline assessment for your platform?

We respond within one business day, covering images/models/K8s/runtime with an actionable remediation checklist.