NiyamAI - An Intent-Bound AI Agent with Cryptographically Verifiable Guardrails using Zero-Knowledge Proofs

The paper addresses a fundamental security gap in autonomous LLM agents: "Giving an AI agent the ability to send emails, query databases, or execute commands is useful — until the agent gets tricked into doing something it shouldn't. Prompt injection, hallucinated reasoning, and unsafe tool calls are the new attack surface for autonomous LLM agents. The authors argue that most current defenses are just software checks: a system prompt, a filter, a policy file. The problem is that all of these live on the same machine the attacker is trying to compromise, so there's no way for an outside party to confirm the safety check actually ran, let alone ran correctly."

Niyam-AI takes a fundamentally different approach: instead of trusting the enforcement logic, we make it provable. The system works as follows:

• At the start of a session, the agent's permitted tools and constraints are locked into an Intent Contract and committed with SHA-256.

• Every tool call the agent attempts is intercepted and checked by an isolated Judge model.

• If the action passes, a zk-SNARK proof — via EZKL — that the check was actually performed and passed is generated.

• The tool only executes once that proof verifies. Nobody has to take our word for it.

The key design insight is that the proof only certifies that a lightweight Judge model made the right call, not the entire multi-billion-parameter LLM, which is what makes this approach computationally realistic instead of a theoretical exercise. This is critical because generating a cryptographic proof for a modern, multi-billion parameter Large Language Model (LLM) remains computationally prohibitive for real-time inference.

The paper lists three main contributions:

1. Intent Binding Protocol: A SHA-256–based mechanism that binds an agent's permitted tools and operational constraints at session initialization, preventing mid-session capability drift.

2. Cryptographic Interception Layer: Tool requests generated by the LLM are intercepted before execution and checked against the session intent using a lightweight Judge model.

3. Verifiable ZK-ML Execution: Safety decisions made by the Judge model are verified through zk-SNARK proofs which are generated using EZKL before a tool-action is allowed to proceed.

The architecture consists of three main layers: Agent Layer, Guardrail Layer, and Verification Layer.

The Intent Contract is defined in a structured JSON format, listing the permissible tools, permissible data scope, and operational limitations. It is hashed using SHA-256 to produce an IntentHash that will be constant throughout the session and will be used to verify all actions by the tools. Any alteration to the agent capabilities results in a hash mismatch, which in turn, blocks the action.

Core intelligence is powered by an open-source Large Language Model (e.g., Llama-3 or Mistral). The system implements a custom Reasoning-Interception Loop — when the LLM determines an external tool is required, the system's execution engine intercepts the output string before it can be parsed or executed by the host operating system.

This acts as security middleware using a lightweight neural-network-based classification engine. When a tool call is intercepted, the system extracts the tool's name and arguments to compute an ActionHash, then concatenates the session's IntentHash with the ActionHash to form an evaluation tuple. This tuple is fed into the Judge Neural Network that classifies the action as Safe (1) or Unsafe (0). If Unsafe, execution is immediately blocked and logged; if Safe, the system proceeds to the proving phase.

The system is formalized as S = (SHA-256, J, Prove, Verify). The public input to the zk-SNARK verifier is the tuple x = (HI, Ha) — the 256-bit IntentHash and 256-bit ActionHash. The Judge function is modeled as a deterministic classifier J: 0,1 512 × W → 0,1, where W denotes the space of private model weights. When J outputs 1, the system generates proof π ← Prove(pk, w, x), and Execution is enforced strictly: Execute(a) ⟺ Verify(vk, π, x) = 1.

The adversary is assumed to operate in an Untrusted Zone with capabilities including: injecting arbitrary malicious prompts into the LLM's context window, attempting to manipulate tool arguments generated by the agent, and attempting software-level execution bypasses against the host environment. Security relies on two cryptographic assumptions: Collision Resistance of SHA-256 and Knowledge Soundness of zk-SNARKs — i.e., the adversary cannot forge a valid proof without actually computing J(x) = 1.

The Judge model was evaluated using 5-fold stratified cross-validation with out-of-fold predictions — every scenario is scored exclusively by a fold-model that never saw that scenario during its own training. On 2,000 Agent-SafetyBench scenarios, Niyam-AI achieved: Accuracy 97.9%, Precision 89.2%, Recall 87.8%, F1 88.5%, FPR 1.1%. The bootstrap 95% CI for F1 was [85.19%, 91.88%] with N=1000 resamples.

Compared against three approaches:

• NeMo Guardrails (Llama-3.1-8B): F1 40.4%, FPR 19.9% — its self-check-input prompt reacts to surface keywords ('email,' 'execute,' 'database') rather than actual intent with precision of just 27.9%, meaning three in four of its blocks are false alarms.

• Llama Prompt Guard 2 (86M): the strongest baseline at F1 66.7%, but with five times Niyam-AI's false-positive rate (5.4%).

• GPT-OSS-Safeguard-20B: F1 46.2%, FPR 21.5%.

The paper includes an important methodological caveat: Niyam-AI's classifier is fitted to Agent-SafetyBench's specific scenario distribution... The three baselines are evaluated fully zero-shot. The authors state readers "should interpret the reported margin as 'a domain-adapted classifier evaluated against general-purpose zero-shot safety models,' not as evidence that Niyam-AI's architecture is inherently more accurate independent of this adaptation."

McNemar's exact paired test confirmed the margin is not due to chance: "Niyam-AI wins 387 discordant scenarios against NeMo versus 20 losses, 112 against Prompt Guard 2 versus 13, and 381 against GPT-OSS-Safeguard versus 19 (p < 0.0001 in all three cases)" — win ratios of 19.4:1, 8.6:1, and 20.1:1 respectively.

• Proof generation time: 2,260.6 ± 218.4 ms per approved action (Table III reports 2160.4 ± 510.6 ms)

• Proof verification time: 53.1 ± 11.8 ms (Table III reports 65.3 ± 18.8 ms), can be performed by any third party without access to the Judge model's weights

• Proof size: 18.71 ± 0.03 KB

• Circuit: 15 logrows, 431 constraints

• One-time setup: gen settings 16.5 ms, compile circuit 2.3 ms, get srs 2353.3 ms, key setup 1973.1 ms

• Gate-only (scope + static rules): Accuracy 96.1%, but F1 Score 0.0% — scope-matching cannot detect intent violations that occur within an already-permitted tool — for example, a legitimate tool called with a malicious payload.

• Gate + Intent Judge: F1 rises to 88.5%, confirming that semantic classification, not scope enforcement, is what actually catches unsafe actions.

• Full Niyam-AI (Gate + Judge + ZK): "Adding the ZK proof layer changes neither accuracy nor F1 — 91.6% in both configurations — which is the expected and correct result: the zk-SNARK proves that the Judge's decision was computed correctly, it does not and should not alter that decision." The cost is 2,260.6 ms additional latency per approved action.

Performance was consistent across seven of eight Agent-SafetyBench categories (97.6%–99.6% accuracy), with one exception: Produce unsafe information/misinformation scored 92.0%, which the authors explain consists of "scenarios where harm arises from the content an LLM generates directly, rather than from a tool call the gate can intercept — precisely the tool-free content-harm class... identified as outside Niyam-AI's intended scope (action integrity, not content moderation)."

The authors constructed 12 adversarial attack vectors across five classes: hash canonicalization, Judge model evasion, control-flow replay, schema/payload boundary conditions, and confused-deputy scope injection. Two implementation-level vulnerabilities were found and fixed:

1. Schema validation gap: the validator accepted IEEE-754 numbers NaN and Infinity and had no upper limit on string field lengths (allowing 1 MB oversized fields and null byte payloads). Fixed with non-finite-value rejections and length and control-character checks.

2. Control-flow session binding deficiency: "there was no cryptographic binding between the sequence guard and the sealed IntentHash of the session, allowing the flow object to be instantiated anew and to reset the sequence completion status independently of the sealed session. Fixed by binding each flow instance to its own IntentHash of the session via a session registry."

After patching, all 12/12 attack vectors (100%) were protected, compared to 9/12 (75%) before patching. Notably, the Judge did not succumb to replacement attacks, character-level obfuscation (using Unicode homoglyphs and zero-width spacing), and dilution attacks that involved hiding the harmful code amidst lots of innocuous text.

Key advantages over traditional approaches include: tamper-resistant execution control, transparent and auditable decision processes, modular security architecture, and deployment flexibility (verification can be local or deployed as a Solidity smart contract on Ethereum Sepolia Testnet).

The paper honestly acknowledges limitations: "The current version of the Judge produces only a binary outcome – safe or unsafe – where a more detailed policy may be required in practice. The proof generation takes about 2.26 seconds which is good enough for high-stakes decisions like money transactions or irreversible system manipulations but would require either hardware acceleration or batching in order to serve high throughput and latency sensitive agents. Multi-agent setups present future challenges around synchronization issues and specifically dealing with the shared/delegated IntentHash when handing off from one agent to another."

The paper also explicitly distinguishes Niyam-AI from constrained decoding frameworks like Outlines and Guidance, noting these operate INSIDE the language model's generation loop, at the token level to enforce syntactic structure, whereas Niyam-AI addresses an architecturally distinct problem: verifying the SEMANTIC INTENT of a fully-formed tool call AFTER the LLM has decided what action to take. The authors note the two approaches are complementary rather than competing.

The paper concludes: "Across 2,000 real-world agent scenarios, it outperformed three established safety systems — NeMo Guardrails, Llama Prompt Guard 2, and GPT-OSS-Safeguard — while adding something none of them provide: a mathematical proof, verifiable in milliseconds, that each safety decision actually happened as claimed. The authors frame this as a viable way of implementing zero knowledge guarantees in the realm of agent security, while emphasizing this is only a first iteration, not the final solution."

1. Commit the Judge model to the proof statement.

Publish a SHA-256 commitment of the Judge model weights, circuit, and version at session setup and include it in the zk-SNARK public inputs. This prevents an attacker from swapping in a different or weaker Judge model while still passing verification.

2. Add fine-grained, multi-label safety outputs instead of a binary Safe/Unsafe decision.

Replace the 0/1 Judge output with a multi-label risk vector: email send ok, data exfil risk, destructive command, misinformation risk, requires human approval. The proof then certifies not just safe but which policy category was applied.

3. Introduce a content-integrity Judge for tool-free LLM output.

The current system only intercepts tool calls; it misses harmful content generated directly as text. Add a lightweight content judge that provably inspects the final output text and proves it does not violate content policy — covering both action integrity and content moderation.

4. Use recursive SNARK aggregation / proof batching.

Generate one aggregate proof for a sequence of N approved tool calls using recursive zk-SNARKs, or batch proofs for repeated/similar approved actions. This amortizes the 2.26-second proving cost and substantially increases throughput for autonomous agents.

5. Add replay and freshness protection inside the ActionHash.

Include a monotonic session nonce, timestamp, conversation digest, and user/session ID in the ActionHash. This prevents replaying an old valid proof for a different session or reusing a proof out of order.

6. Compile richer policy languages — not just static JSON — into the verifiable circuit.

Express constraints as Rego/OPA-style policies or context-dependent rules, compile them into arithmetic constraints, and prove policy evaluation just like the Judge inference. This supports dynamic, conditional, per-user permissions while keeping cryptographic verifiability.

7. Prove human-in-the-loop consent for high-risk actions.

For irreversible operations — money transfer, data deletion, privileged execution — the proof statement can additionally require a signed user-consent message over the exact ActionHash. The verifier then knows not only that the Judge approved, but that a human explicitly authorized that specific action.

8. Add verifiable explanation output.

Instead of only proving Safe=1, the system can reveal selected policy-rule IDs, feature attribution bits, or a compact reason code as public inputs, and prove they are consistent with the Judge decision. This gives auditors an auditable rationale, not just a yes/no answer.

9. Merge structured-output enforcement with intent verification.

Use constrained decoding (Outlines/Guidance) as a front-end to guarantee the LLM produces well-formed, schema-valid JSON tool calls; then feed that JSON into NiyamAI’s semantic Judge. This combines syntactic enforcement with semantic, cryptographically verified safety.

10. Support delegated IntentHashes for multi-agent workflows.

Extend the Intent Contract to include a delegation chain: agent B can act on behalf of agent A only if A’s session proof authorizes B’s sub-intent hash. This enables verifiable multi-agent handoffs without losing the binding between actions and the original session intent.

11. Use hardware-accelerated/GPU proving to reduce latency.

Optimize the EZKL circuit for GPU/FPGA/ASIC backends and use folding/IVC to reduce proof-generation time from 2.26 seconds to tens of milliseconds, making verifiable safety practical for high-frequency, latency-sensitive agents.

12. Add continuous adversarial retraining with versioned model registry.

After each red-team campaign, retrain the Judge on newly discovered attacks, register the new model version with a public commitment, and keep the universal setup. The system can then prove the current audited model version made this decision, adapting to new threats without breaking verifiability.

• Prove, in under 100 ms per action, that a tool call was checked by the exact, audited Judge model committed for that session, and that the session’s IntentHash was not altered.

• Verify the full policy chain — intent binding, Judge classification, human consent, and output-content safety — as a single zk-SNARK, with proof size still under 20 KB.

• Prevent proof replay, Judge substitution, cross-session action reuse, and mid-session capability drift by including session nonces, freshness tokens, and model commitments in the public input.

• Handle high-throughput and latency-sensitive agents via recursive proof aggregation and hardware-accelerated proving, while retaining third-party verifiability.

• Block harmful LLM-generated content even when no tool is called, using a verifiable content Judge.

• Support safe delegation between multiple AI agents without losing the cryptographic link between each sub-action and the original user-approved Intent Contract.

• Provide auditable, human-readable reasons for every blocked or approved action, with the reason itself proven consistent with the Judge’s decision.

• Deploy as on-chain verifier for smart contracts, where a Solidity verifier checks every agent action before releasing funds, executing state changes, or triggering external integrations — all with mathematical certainty rather than trust in a software guardrail.

Giving an AI agent the ability to send emails, query databases, or execute commands is useful--until the agent is tricked into doing something it shouldn't. Prompt injection, hallucinated reasoning, and unsafe tool calls form the primary attack surface for autonomous LLM agents. Existing defenses rely on software checks like system prompts or policy filters running on the same machine the attacker targets, offering no verifiable proof of execution. We introduce Niyam-AI, a framework that makes safety enforcement provable. At session start, permitted tools and constraints are locked into an Intent Contract committed via SHA-256. Every tool call is intercepted and validated by an isolated Judge model; upon passing, a zk-SNARK proof is generated via EZKL. The tool executes only after proof verification, allowing third parties to confirm enforcement without accessing Judge model weights. Evaluating Niyam-AI on 2,000 real-world scenarios from Agent-SafetyBench against NeMo Guardrails, Meta's Llama Prompt Guard 2, and OpenAI's GPT-OSS-Safeguard using 5-fold stratified cross-validation yields an F1 score of 88.5% with a 1.1% false-positive rate (bootstrap 95% CI: [85.19%, 91.88%], N=1000). McNemar's exact paired test confirms significant improvement: Niyam-AI wins 390 discordant scenarios against NeMo (vs 20 losses), 115 against Prompt Guard 2 (vs 13), and 384 against GPT-OSS-Safeguard (vs 19) with p < 0.0001 in all cases. Proof generation adds 2260.6 +/- 218.4 ms per approved action, while verification takes 53.1 +/- 11.8 ms. Niyam-AI provides a guardrail that is both highly accurate and mathematically verifiable--though this reflects a classifier adapted to Agent-SafetyBench evaluated against zero-shot baselines, a distinction discussed in Section IV.C.