Using eBPF for Real-Time AI Workload Threat Detection in Cloud-Native Environments
Share
Share
eBPF (Extended Berkeley Packet Filter) gives security engineers kernel-level observability into AI workloads without modifying application code. This post explains how to use eBPF to detect the attack classes that matter most for ML systems: data poisoning, model exfiltration, inference manipulation, and supply chain compromise — at up to ~600k events per second on the measured rule path.
Why Traditional Security Fails AI Workloads
AI and ML systems running on Kubernetes expose a different attack surface than conventional web services. The threats that matter most — training data poisoning, model weight exfiltration, inference manipulation, and MLOps pipeline compromise — are largely invisible to network-perimeter tools, container image scanners, and SIEM log analysis.
Consider what a compromised ML training job looks like:
No malicious network signature: The attacker exfiltrates model weights by encoding them in DNS queries — traffic that looks like normal DNS
No process anomaly at image level: The container image is clean; the poisoning happens via a mounted volume with compromised training data
Feng Ning (风宁)
12 min read
...
No log entry: The attack modifies training data at the filesystem level before any ML framework log is written
Traditional detection misses all three. eBPF sees all three — at the kernel system call layer, before any application abstraction can hide the activity.
eBPF Architecture for AI Security Monitoring
eBPF programs attach to kernel hooks and execute in a verified, sandboxed environment. For AI workload monitoring, the relevant attachment points are:
System Output
Kernel Attachment Points for AI Security:
│
├── kprobes/kretprobes (kernel function entry/exit)
│ ├── sys_read / sys_write — I/O to model files, training data
│ ├── sys_connect / sys_sendto — Outbound network from inference pods
│ └── sys_execve — New process spawning inside ML containers
│
├── tracepoints (stable kernel trace events)
│ ├── syscalls:sys_enter_openat — File access patterns (model weight reads)
│ ├── net:netif_receive_skb — Incoming network to training pods
│ └── sched:sched_process_exec — Process execution lineage
│
├── XDP (eXpress Data Path)
│ ├── Pre-stack packet inspection at NIC driver level
│ └── Detect covert channels (DNS tunneling, ICMP payload exfiltration)
│
└── LSM hooks (Linux Security Module via eBPF)
├── file_open / file_permission — Fine-grained model file access control
├── socket_connect — Allowlist outbound connections from ML pods
└── bpf_map_update_elem — Detect eBPF map manipulation by attacker programs
CO-RE (Compile Once — Run Everywhere): Modern eBPF programs using BTF (BPF Type Format) and CO-RE can be compiled once and deployed across kernel versions 5.8+, covering all major Kubernetes distributions (EKS, GKE, AKS, self-managed). This eliminates the kernel-version fragmentation problem that plagued earlier eBPF security tools.
The 7 AI-Specific Attack Classes eBPF Detects
Class 1: Training Data Poisoning via Filesystem
Attack pattern: Attacker compromises a mounted volume or data pipeline and modifies training data files before the ML framework reads them. The modification could inject backdoor triggers, degrade model accuracy, or bias outputs toward attacker-controlled predictions.
eBPF detection approach:
System Output
// Detect unexpected writes to training data directories
SEC("tracepoint/syscalls/sys_exit_write")
int detect_training_data_write(struct trace_event_raw_sys_exit *ctx) {
// Get current process namespace and cgroup (container identity)
struct task_struct *task = (struct task_struct *)bpf_get_current_task();
// Check if write target is in training data path
// Flag writes from unexpected process identities (not the expected data loader PID)
if (is_training_data_path(file_path) && !is_authorized_writer(task)) {
emit_alert(ALERT_TRAINING_DATA_WRITE, task, file_path);
}
return 0;
}
What this catches: A supply chain attack where a compromised data preprocessing library writes modified samples to the training directory. The write syscall is intercepted regardless of what the library claims to be doing at the application layer.
Class 2: Model Weight Exfiltration
Attack pattern: A compromised ML container reads model weight files and transmits them out of the cluster via DNS queries, HTTP to a lookalike domain, or ICMP payload encoding.
eBPF detection approach: Correlate sys_read on model files (.pt, .bin, .safetensors, .onnx) with subsequent sys_sendto/sys_connect calls from the same process within a 500ms window. Flag connections to external IPs not in the approved egress list.
Nora Vision implements this as a weight-read-to-egress correlation rule that fires within one ring buffer flush cycle (~50ms), well before the exfiltration completes over a TCP connection.
Detection signature:
System Output
ALERT: Model Exfiltration Attempt
Process: python3 (PID 4821, container: ml-inference-7d9f)
Sequence: open(/models/gpt-finetuned.bin) → read(2.1GB) → connect(185.220.101.x:443)
Time delta: 340ms
Model file hash: sha256:e3b0c44298fc...
Destination: not in egress allowlist
MITRE: T1041 (Exfiltration Over C2 Channel)
Class 3: Inference Manipulation (Adversarial Input at Runtime)
Attack pattern: Attacker submits crafted adversarial inputs to a live inference API — not to extract training data but to force specific model outputs. In a fraud detection context, this means inputs designed to make the model classify fraudulent transactions as legitimate.
eBPF detection: Monitor the inference request handler at the socket layer. Build a statistical baseline of input vector distributions per model version. Use eBPF ring buffers to stream input feature summaries to a userspace anomaly detector running as a sidecar.
Key insight: Standard application-layer logging only captures what the application chooses to log. eBPF captures the raw bytes received by the inference service socket — the adversarial input is detectable before the inference function executes, giving an opportunity to reject it before it reaches the model.
Class 4: MLOps Pipeline Compromise
Attack pattern: An attacker with access to the CI/CD pipeline (via a compromised GitHub token, Argo Workflows misconfiguration, or Kubeflow RBAC escalation) modifies the training job definition to poison the model during a scheduled re-training run.
eBPF detection:
Attach kprobes to sys_execve inside the training namespace
Baseline the expected process tree for a normal training job
Alert on any sys_execve spawning a shell (/bin/sh, /bin/bash) from inside the training container during a scheduled run — legitimate ML frameworks don't spawn shells
Detect unexpected sys_clone calls indicating fork-exec patterns inconsistent with the ML framework's process model
ML training jobs often run with elevated privileges (GPU device access, shared memory for distributed training). This makes them high-value targets for container escape.
eBPF detects escape attempts via:
sys_unshare with CLONE_NEWUSER flag — namespace manipulation for privilege escalation
Write to /proc/sysrq-trigger, /sys/kernel/debug/ — kernel interface abuse
sys_open on /host/etc/shadow, /host/proc/1/maps — host filesystem access via volume mounts
sys_ptrace targeting PID 1 — breakout via host process attachment
eBPF LSM hooks (bpf_lsm_file_open) can block these operations in real time rather than merely alerting after the fact — converting detection into prevention.
Class 6: Covert Channel Exfiltration (DNS Tunneling)
Attack pattern: The attacker encodes model weights or training data in DNS query payloads, bypassing outbound firewall rules that permit DNS traffic.
eBPF XDP detection:
System Output
SEC("xdp")
int detect_dns_tunnel(struct xdp_md *ctx) {
// Parse DNS query at XDP layer (pre-stack, before any socket buffer)
struct dns_hdr *dns = parse_dns(ctx);
// Flag: query name > 63 chars, or QNAME contains base64-looking subdomains
if (dns->qdcount > 1 || get_qname_len(dns) > 63) {
// Compute entropy of QNAME — high entropy = likely encoded data
u32 entropy = compute_qname_entropy(dns);
if (entropy > DNS_TUNNEL_ENTROPY_THRESHOLD) {
emit_xdp_alert(ALERT_DNS_TUNNEL, ctx);
return XDP_DROP; // Block before kernel networking stack
}
}
return XDP_PASS;
}
XDP operates before the kernel networking stack, providing sub-100µs detection and blocking at line rate — unlike userspace packet capture which has kernel-to-userspace copy overhead.
Class 7: Supply Chain Attack on ML Libraries
Attack pattern: A compromised version of a popular ML library (torch, transformers, safetensors) contains a backdoor that activates when certain model weight patterns are loaded.
eBPF detection: Monitor dlopen and library loading syscalls. Maintain a hash allowlist of approved library versions. Alert immediately when a container loads a library version not on the allowlist — before any backdoor code can execute.
Alert output goes to webhook, syslog, or direct SIEM integration (Splunk, Elastic, Chronicle). Detection rules are hot-reloadable without DaemonSet restart.
Practical: Protecting a Production LLM Inference Fleet
Concrete example: an LLM inference fleet running 40 replicas of a fine-tuned language model on Kubernetes.
Nora Vision policy configuration for LLM inference:
System Output
policies:
- name: model-weight-protection
description: Prevent model weight exfiltration from inference pods
selector:
labels:
workload: llm-inference
rules:
- type: file_open
paths: ["/models/**/*.bin", "/models/**/*.safetensors"]
allowed_processes: ["python3", "uvicorn"] # Only inference server can open weights
- type: egress_correlation
trigger: file_read
path_pattern: "/models/**"
window_ms: 1000
alert_on: external_connection # Flag if model read followed by external connect
- name: no-shell-in-inference
description: Inference containers should never spawn a shell
selector:
labels:
workload: llm-inference
rules:
- type: exec_blacklist
processes: ["/bin/sh", "/bin/bash", "/usr/bin/python3 -c"]
action: block # Prevent execution, not just alert
- name: dns-tunnel-prevention
description: Detect DNS-based exfiltration
rules:
- type: xdp_dns
max_qname_length: 63
entropy_threshold: 4.2
action: drop_and_alert
With this policy active, any attempt to exfiltrate model weights — whether via direct TCP connection, DNS tunneling, or spawning a shell to use curl — is blocked within one kernel round trip, before any data leaves the pod.
Getting Started with eBPF-Based AI Security
Option 1: Deploy Nora Vision — Our managed eBPF security solution for Kubernetes. Pre-built detection rules for AI workloads, MITRE ATT&CK mapping, and alert integrations. Request access.
Option 2: Open-source foundation — Start with Tetragon (Cilium project) for basic eBPF observability. Nora Vision adds the AI-workload-specific detection logic on top of a production-hardened eBPF core.
Option 3: Security audit — If you're unsure about your AI workload's current security posture, our AI security audit service includes an eBPF-based runtime observation phase that maps your actual syscall behaviour against expected baselines.
Summary
eBPF provides the only practical approach to securing AI workloads at kernel level with production-acceptable overhead. The seven attack classes that matter most for ML systems — training data poisoning, model exfiltration, inference manipulation, MLOps pipeline compromise, container escape, DNS tunneling, and supply chain attacks — are all detectable via eBPF syscall monitoring, network-layer inspection, and LSM enforcement. At 250,000+ events/second with sub-1% CPU overhead, eBPF security monitoring can be deployed on GPU training nodes and inference fleets without impacting model performance.
Founder of Innora.ai. Sovereign Architect bridging offensive security and autonomous coding. Building the Autonomous Architect reality with zero external dependencies.
