Efficient On-Device Agents via Adaptive Context Management

arXiv:2511.03728 · cs.AI, cs.LG · Submitted 2025-09-24 · Read on arXiv

cs.AI, cs.LG

Submitted: 2025-09-24

Updated: 2026-09-13

Comments: 26 pages, 5 figures

Code: https://github.com/ggerganov/llama.cpp

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Fine-Tuning — 10× in this paper · explained in 5 episodes
• KV cache — 10× in this paper · explained in 3 episodes
• Context State Object — 8× in this paper
• Core — 7× in this paper · explained in 2 episodes
• Benchmark — 5× in this paper · explained in 2 episodes
• AI Agents — 4× in this paper · explained in 2 episodes
• Distillation — 4× in this paper · explained in 2 episodes
• Adaptation — 3× in this paper · explained in 1 episode
• F1 Score — 3× in this paper · explained in 3 episodes
• Failure Modes — 3× in this paper · explained in 3 episodes
• Frustration — 3× in this paper · explained in 2 episodes
• Generalizability — 3× in this paper · explained in 6 episodes
• Accuracy — 2× in this paper · explained in 1 episode
• Fidelity — 2× in this paper · explained in 2 episodes
• Generalization — 2× in this paper · explained in 13 episodes
• LoRA Adapters — 2× in this paper · explained in 3 episodes
• Personalization — 2× in this paper · explained in 2 episodes
• Supervised Fine-Tuning — 2× in this paper
• alpha — 2× in this paper · explained in 1 episode
• AdamW Optimizer — 1× in this paper · explained in 1 episode
• Agentic Evaluation — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• B-parameter Small Language Model — 1× in this paper
• Edge Devices — 1× in this paper · explained in 2 episodes
• Grounding — 1× in this paper · explained in 3 episodes
• Hallucination — 1× in this paper · explained in 4 episodes
• LLM-as-a-Judge — 1× in this paper · explained in 3 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Low-Rank Adaptation (LoRA) — 1× in this paper · explained in 9 episodes
• Multi-Agent System — 1× in this paper · explained in 6 episodes
• Parameter-Efficient Fine-Tuning — 1× in this paper · explained in 2 episodes
• RAG — 1× in this paper · explained in 3 episodes
• Reconstruction Error — 1× in this paper · explained in 2 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Reinforcement Learning (RL) — 1× in this paper · explained in 25 episodes
• Sparse Attention — 1× in this paper · explained in 5 episodes
• State Management — 1× in this paper · explained in 3 episodes
• Supervised Fine-Tuning (SFT) — 1× in this paper · explained in 8 episodes
• Transformer — 1× in this paper · explained in 2 episodes
• Verification — 1× in this paper · explained in 2 episodes

Sources

• Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• Octopus: On-device language model for function calling of software APIs
• TinyAgent: Function Calling at the Edge
• LoRA: Low-Rank Adaptation of Large Language Models
• Collaborating with AI Agents: Field Experiments on Teamwork, Productivity, and Performance
• KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction
• On-Device LLM for Context-Aware Wi-Fi Roaming
• Lost in the Middle: How Language Models Use Long Contexts
• Less is More: Optimizing Function Calling for LLM Execution on Edge Devices
• KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments
• Efficiently Scaling Transformer Inference
• APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Gemma 3 Technical Report
• Arch-Router: Aligning LLM Routing with Human Preferences
• Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models
• OpenHands: An Open Platform for AI Software Developers as Generalist Agents
• Efficient Streaming Language Models with Attention Sinks

Related papers

• MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
• Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
• The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
• MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
• Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
• VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection