Working Around the Compute Ceiling: Byte-Exact Memory in Galahad Makes LLM Reading a One-Time Cost LLM Reading a One-Time Cost

A transformer language model performs a bounded amount of computation per token, and recent work by Vishal Sikka, former CEO of Infosys, argues that this bound limits which tasks a model can carry out or verify. This paper addresses how much of that computational budget is spent on useful work versus repeating work the model has already done.

The gist

Galahad is a memory layer for vLLM, SGLang and llama.cpp that makes reading text a one-time cost by storing the model’s key–value (KV) state and document sections, allowing subsequent requests containing the same data to load this state instead of recomputing it.

System Architecture

Galahad is implemented as a shared library that plugs into three production runtimes: vLLM as a KV connector, SGLang as a HiCache storage backend, and llama.cpp through slot save and restore. The system utilizes two primary memories: Taliesin stores the model’s own KV state for a block of text, while Blaise keeps the documents themselves organized into sections on the CPU.

(Figure 1 illustrates this flow: Blaise selects the section of text a question needs; Taliesin supplies saved KV state for any block the model has already processed, and saves new blocks after their first prefill.)

Memory Mechanisms

The system achieves reuse through two distinct methods:

1. Taliesin removes repeated computation by storing the model’s own KV state for a block of text and loading it when the same bytes appear again. The design rule is serve without memory, never with wrong memory: if a record is missing, damaged, or fails a check, the runtime recomputes the block and the request completes normally. Reuse happens at block level, where token positions match exactly.

2. Blaise removes unnecessary reading by storing each document as byte-exact text divided into sections. For a question, Blaise selects one section on the CPU and passes that section’s text to the model, which then reads it and answers. This allows for reading less, with Blaise passing about 668 tokens per question instead of the full corpus in a recall test.

Correctness and Verification

Exactness is paramount because approximate reuse fails silently. Galahad treats any mismatch as a failed load and recomputes, ensuring that memory never changes an answer. The system incorporates several safety mechanisms to ensure correctness:

(Table 1 summarizes the correctness checks, including saving and restoring 262,144 of 262,144 logits bit-identical.)

The paper details sabotage testing where each safety mechanism (confirmation hash, licence signature, tenant separation, byte comparison on load) was tested three times: with the defense on, with it switched off, and with it restored. A defence is effective only if the attack succeeds when it is off and fails when it is on. Furthermore, analysis tools found one out-of-bounds read which was subsequently fixed before release.

Evaluation Results

The evaluation across seven real-world datasets showed significant gains:

(Table 2 compares accuracy against median time per question for the recall test.)

Taliesin alone (the model reads the whole corpus, loaded instead of recomputed) answered 98 to 100 questions on the three runtimes. On llama.cpp, this resulted in a 3.1× faster response time than the baseline without Galahad and 79% less GPU energy. With Blaise added, the model achieved 100 of 100 answers on every runtime with a median time of 0.64 seconds and 213 Joules per question on vLLM.

Implications for Inference Cost

The results support that reuse turns most prompt computation into loading. For workloads returning to the same documents, the cost grows with the text the model has not read before. This shift moves inference from a stateless computation to one with persistent memory, where a model should pay to read a document once, and every later question about it should cost only the question and the answer. Galahad works around the compute ceiling by keeping repeated work, search, and verification away from the model. The main implication is that this stateful inference can enable smaller models with exact memory of a large corpus to answer questions that would otherwise require a model with a much longer context window.

Availability

Galahad is available as a free, non-commercial beta for one GPU at https://github.com/corbenicai/galahad, and commercial pilots are available on request. The system ships as one shared library (libgalahad.so) for Linux x86-64 with two runtime dependencies (OpenSSL’s libcrypto and zstd).

Here are the specific improvements that can be made to existing AI systems based on this research, along with a description of what those improved systems can achieve:

The core improvement is shifting LLM inference from a stateless, recomputational process to a stateful, memory-persistent process by introducing the Galahad framework. This fundamentally changes the unit of inference cost from total text processed to new text read.

Here are specific improvements categorized by their technical mechanism:

2. Implement a shared memory layer that plugs into existing serving runtimes (vLLM, SGLang, llama.cpp) as a single library (libgalahad.so). This allows for stateful persistence across requests without modifying model weights or runtime source code directly.

4. Incorporate the Taliesin memory component to store the model’s Key-Value (KV) state for specific blocks of prompt text on disk, indexed by byte-exact input fingerprints (model ID + tenant ID + input bytes). This allows subsequent requests containing identical byte sequences to load the KV state instead of recomputing it.

6. Implement the Blaise memory component to store the corpus as byte-exact text, pre-divided into sections. When a question is posed, Blaise selects only the necessary section (e.g., 668 tokens) from the corpus and passes only that exact text to the model, rather than feeding it a massive context window or recomputing the entire prompt.

8. Enforce strict correctness checks on all loaded state using cryptographic hashes and byte comparisons before loading, ensuring that restored state is bit-identical to the freshly computed state (e.g., 262,144 logits match). Any load failure triggers a recomputation, guaranteeing correctness over approximation.

The improved AI systems can achieve the following specific capabilities:

2. Support "Persistent Q&A Agents" for high-frequency tasks (e.g., customer support assistants or coding agents) where the same documents, code snippets, or agreements are queried repeatedly. These agents will drastically reduce latency and energy consumption because repeated queries about known information cost only the question/answer time rather than the full prompt prefill cost.

4. Enable Knowledge-Intensive RAG Systems that operate on large corpora (millions of tokens) without incurring massive per-question costs or requiring excessively long context windows. The system can selectively retrieve and pass only the 668 most relevant tokens, leading to significant reductions in time (up to 14.5x faster for Blaise mode) and GPU energy compared to feeding the entire corpus or a truncated window.

6. Achieve Model Agnostic State Reuse across different hardware and model architectures (e.g., loading KV state saved on an A6000 onto an RTX 4090) due to the strict byte-identity checks, allowing for cross-platform state portability.

8. Create Cost-Aware Inference Pipelines that dynamically determine whether to recompute or load memory based on input similarity, moving inference from a purely stateless computation to one where the cost scales with the novelty of the information requested, not the total text processed in the prompt.

A transformer language model performs a bounded amount of computation per token, and recent work by Vishal Sikka, former CEO of Infosys, argues that this bound limits which tasks a model can carry out or verify (arXiv:2507.07505). We ask how much of the budget beneath that ceiling is spent on work the model has already done. Serving is stateless across requests: a model that answers a second question about a document recomputes the document's attention state from the first token. On seven real-world datasets, 98.7% of prompt tokens were text the model had already read. We present Galahad, a memory layer for vLLM, SGLang and llama.cpp that makes this reading a one-time cost. Taliesin saves the model's key-value (KV) state for a block of text and loads it on the next request that contains the same bytes, instead of recomputing it. Blaise keeps the documents themselves and passes the model only the section a question needs. On a recall test with 100 facts hidden in a 97,000-token corpus (Gemma 4 31B), Taliesin alone let the model attend to the whole corpus and answered 98 of 100 on llama.cpp at 3.0 s and 572 J per question, against 10 of 100, 9.3 s and 2,754 J for the same model without Galahad, which could hold only the last 12,000 tokens. With Blaise added, the model read about 668 tokens per question and answered 100 of 100 on all three runtimes at 0.59-0.64 s and 200-213 J; a tuned RAGFlow pipeline answered 77. Storing the corpus is a one-time cost of about 100 s and 28 kJ, whose energy is recovered after 13 questions. Restored state is bit-identical: all 262,144 output logits matched after restart, rehydration and hot-load. Galahad worked with all 30 models we tested under vLLM, and it fails closed: any load that does not pass its checks is recomputed. Together these results move LLM serving from stateless to stateful inference.