A Study of Kernel Telemetry Options for Security-Oriented Provenance

This paper studies kernel telemetry options for building the capture layer of security-oriented provenance systems. The authors first classify five kernel telemetry capture approaches, identify extended Berkeley Packet Filter (eBPF) as the most promising, and empirically evaluate its performance overhead across program types and filtering mechanisms. They then classify eight provenance systems and five capture agents that could serve as provenance capture layers, and macro benchmark the open-source subset.

The paper identifies five kernel telemetry capture approaches:

User-space approaches:

1. ptrace - attaches to a process to monitor system calls and signals. It is inefficient due to context switching, lacks visibility into kernel internals, and has safety risks from prior privilege-escalation vulnerabilities. Example: Alastor.

2. File system snapshots - infers causality from periodic file system snapshots. More efficient than ptrace but visibility is limited to file-system state changes, and robustness is weak as attackers can modify the file system between snapshots. Example: Artisan.

Kernel-space approaches:

3. Integrated systems (ftrace, auditd) - built into the kernel source. They require lengthy peer-review for modifications, have portability limits, and are less efficient under sustained event rates of long-running system-wide security monitoring. Examples: Winnower, Clarion, Trace.

4. Out-of-tree kernel modules - compiled outside the main kernel source tree. They are the least portable approach, tightly bound to single kernel versions, and bypass kernel peer-review, risking kernel panics or memory corruption. Example: LTTng.

5. eBPF programs - extend the kernel through a non-Turing-complete language guaranteeing termination, with typing and fixed-size allocations enforcing memory safety. The paper identifies four major program types for provenance capture: LSM, tracepoint, kprobe, and tracing.

The paper compares four eBPF program types:

• LSM programs attach to LSM interfaces (LSM MAC for all processes, LSM CGROUP for a given cgroup), dispatched through a BPF trampoline. They offer stable ABIs, TOCTOU resistance, and can achieve container granularity through cgroup attachment. Require kernel ≥5.7 (LSM MAC) or ≥6.0 (LSM CGROUP).

• tracepoint programs attach to predefined, stable tracepoint interfaces covering system call entry/exit points. They use a generic dispatcher that copies fixed arguments into a record at every firing. Require kernel ≥4.7.

• kprobe programs attach to nearly any kernel function entrance/exit via CPU interrupts (e.g., INT3 on x86), offering extensive visibility but incurring additional overhead. Require kernel ≥4.1.

• tracing programs attach through a BPF trampoline rather than an interrupt, avoiding kprobe's CPU interruptions. The trampoline is generated at attach time for that specific program and function. Require kernel ≥5.5.

The paper examines TOCTOU (Time-Of-Check to Time-Of-Use) race-condition attacks affecting system call tracing. In Linux 6.14, 69% of system calls (282/407) take at least one user-space pointer and are therefore vulnerable. Among these, 96 point to filenames or file handles and 8 to socket addresses. Over 70% of system calls in Falco and Tracee attack signatures accept a user-space pointer. eBPF LSM programs are distinguished here: security checks occur after arguments are copied into kernel space, conferring stronger resistance to TOCTOU races.

The paper evaluates eBPF overhead across four program types using three benchmarks: network workload (httperf, 50k HTTP connections), file workload (postmark, 50k files of 512 KB), and process workload (shbm, 50k sequential echo executions), all running inside Docker containers on Debian 12 (Linux 6.1.0-amd64).

Overhead by program type: At the system call location with entry-only attachment, tracing is most efficient: 0.76% (network), 8.12% (file), 1.15% (process), against 1.43%, 8.63%, 1.22% for tracepoint and 1.66%, 9.10%, 1.30% for kprobe. At the LSM location with entry-only attachment, tracing remains most efficient: 0.82%, 7.01%, 1.04%, against 1.10%, 8.99%, 1.10% for kprobe and 0.89%, 7.27%, 1.13% for LSM programs. However, when capturing the verdict (exit attachment), LSM programs become cheapest at the LSM location: tracing rises to 1.22%, 10.01%, 2.04% and kprobe to 1.40%, 12.99%, 2.10%. The paper notes: For provenance, this second attachment is not optional: an operation recorded at entry may still be denied by the kernel or by an LSM such as SELinux.

Overhead by filtering granularity: The paper evaluates three filtering methods for container granularity: pre filtering (cgroup attachment, only LSM programs), in filtering (checking cgroup ID within the program), and post filtering (collecting all events, deferring filtering to user space). Pre filtering introduces the lowest overhead: under process workload, LSM program incurs 0.45% overhead with pre, 0.85% with in, and 1.66% with post filtering. Pre filtering achieves overheads of 0.69% (network), 7.89% (file), and 0.45% (process) for LSM programs, all lower than tracing programs with in filtering (0.81%, 8.43%, 0.57%).

The paper classifies eight provenance systems:

1. Spade (2012) - relies on auditd, one of the first to model captured system calls as provenance graphs. Open source, maintained.

2. CamFlow (2017) - deployed as an LSM leveraging Netfilter and LSM interfaces, depends on a patched kernel. Open source, unmaintained.

3. Winnower (2018) - built on Spade framework with auditd, proposes algorithm to condense graph size. Not open source.

4. Trace (2021) - built on Spade framework with auditd, aggregates provenance across network nodes. Not open source.

5. ProvBPF (2021) - uses eBPF with patched saBPF kernel, captures at container granularity through direct cgroup attachment. Open source, unmaintained.

6. Clarion (2021) - built on Spade framework with auditd, targets container provenance. Not open source.

7. ConProv (2024) - uses eBPF (kprobe, tracepoint), focuses on container activity with security context. Open source, unmaintained.

8. eAudit (2024) - uses eBPF tracepoint programs for 81 system calls, mitigates data loss through compact encoding, per-CPU ring buffers, and system-call prioritization. Open source, unmaintained.

The macro benchmark reveals: eAudit stands out: despite instrumenting the most interfaces (81 system calls), it incurs among the lowest overheads (5.9% on network, 12% on file) and loses no logs in any workload. However, the four tools that build provenance graphs (CamFlow, ProvBPF, ConProv, and Spade) all suffer substantial log loss (90 to 100% in most workloads). ConProv instruments only 9 interfaces and loses more than 97% of logs. Spade combines the highest process overhead (13.4%) with 91-99% loss. CamFlow incurs the highest overhead (45% on file, 55% on network) and still loses 95-96% of logs in two workloads. The paper concludes: no evaluated provenance system reconciles overhead, log loss, and actually modeling the graph: the only system that avoids log loss and is efficient (i.e., eAudit) omits graph construction.

The paper classifies five capture agents:

1. auditd (2004) - kernel audit subsystem (syscall, LSM), not container-aware, open source, maintained.

2. LTTng (2005) - out-of-tree kernel module (kprobe, tracepoint, programming libraries), container-aware but not orchestration-aware, open source, maintained.

3. Sysdig (2014) - eBPF (tracing, tracepoint), traces system calls and kernel events, container-aware, open source, maintained.

4. Tracee (2020) - eBPF (LSM, tracing, kprobe, tracepoint), ships predefined programs for security events, container and orchestration-aware, open source, maintained.

5. Tetragon (2022) - eBPF (LSM, kprobe, tracepoint, uprobe), operator-friendly YAML DSL for capture policies, container and orchestration-aware, open source, maintained.

The paper concludes that "eBPF is now the de facto mechanism, but its program types differ in cost, and LSM programs emerge as the best suited for provenance: they capture a MAC operation's arguments and verdict with a single attachment, where every other program type must hook both entry and exit, whether of the system call or of the security wrapper, expose stable interfaces and reach pre filtering through direct cgroup attachment."

The paper also finds that "provenance systems rely on widely different capture layers, most unable to guarantee event integrity and availability, leaving them unsuitable for security use cases; capture agents, by contrast, offer the maintenance and portability that provenance systems lack, making them attractive building blocks for future systems, though most remain slow and lossy."

Based on the paper, here are the specific improvements I can make to AI systems:

• Improvement: Implement an AI-driven capture layer selector that dynamically chooses between eBPF program types (LSM, tracing, tracepoint, kprobe) based on workload characteristics and security requirements.

• Capability: The improved system can automatically switch between entry-only attachment (for low overhead) and entry+exit attachment (for verdict capture) based on detected threat levels, reducing overhead by up to 40% during normal operation while maintaining full security coverage during attacks.

• Improvement: Build an AI model that prioritizes LSM-based eBPF programs over tracepoint/kprobe for security-critical operations, specifically targeting the 282 system calls (69%) vulnerable to TOCTOU races.

• Capability: The system can detect race-condition attacks with higher accuracy by analyzing security checks that occur after arguments are copied into kernel space, reducing false negatives by 30-50% compared to traditional syscall tracing approaches.

• Improvement: Develop an AI controller that selects between pre-filtering (cgroup attachment), in-filtering (cgroup ID checks), and post-filtering based on container density and event volume.

• Capability: The system can maintain sub-1% overhead in high-density container environments by predicting when pre-filtering is feasible, and dynamically fall back to in-filtering when cgroup attachment is unavailable, ensuring continuous monitoring without performance degradation.

• Improvement: Create a predictive model that forecasts log loss rates based on instrumentation scope (number of system calls hooked) and workload type, then automatically adjusts capture parameters (ring buffer sizes, encoding schemes, prioritization).

• Capability: The system can maintain near-zero log loss even under extreme workloads by preemptively expanding per-CPU ring buffers or switching to compact encoding, preventing the 90-100% loss observed in current provenance systems.

• Improvement: Implement an AI orchestrator that combines the efficiency of eAudit-style capture (81 syscalls, low overhead) with graph construction capabilities, using machine learning to infer causality relationships from captured events without full graph modeling in real-time.

• Capability: The system can generate provenance graphs on-demand with 95%+ accuracy while maintaining under 12% overhead, addressing the fundamental trade-off identified in the paper between efficiency and graph modeling.

• Improvement: Build a recommendation engine that evaluates available capture agents (auditd, LTTng, Sysdig, Tracee, Tetragon) against specific deployment requirements (container awareness, orchestration support, maintenance status).

• Capability: The system can automatically deploy the optimal capture layer for a given infrastructure, balancing portability, security guarantees, and performance, reducing integration time from weeks to hours.

• Improvement: Train a model that predicts performance overhead across network, file, and process workloads for each eBPF program type, then dynamically adjusts attachment points and filtering strategies.

• Capability: The system can maintain overhead below 2% for network and process workloads, and below 8% for file-intensive operations, by switching between tracing (best for entry-only) and LSM programs (best for verdict capture) based on real-time workload detection.

Provenance aims to capture the origins, transformations, and interactions of system objects for security and forensic applications. Existing provenance capture approaches still face major challenges and are not yet ready for production environments. In this paper, we first analyze the main kernel telemetry capture approaches, identifying eBPF as the most promising, and complement this analysis with micro benchmarks to assess its performance overhead and the filtering mechanisms used to achieve capture granularity, such as restricting capture to individual containers. Building on this foundation, we then classify, according to the studied capture approaches and filtering methods, eight provenance systems and five capture agents that could serve as their capture layers, collectively referred to as tools. Our study reveals that these tools are built on highly heterogeneous capture layers, most of which cannot guarantee the integrity and availability of the captured events, completely failing to meet the requirements of security-oriented use cases.