Role-aware Heuristic Episodic Attention for Conversational LLMs

arXiv:2610.00958 · cs.CL · Submitted 2026-10-01 · Read on arXiv

cs.CL

Submitted: 2026-10-01

Updated: 2026-10-01

Code: https://github.com/huha12138/rhea

Terminology

• Benchmark — 17× in this paper · explained in 2 episodes
• Transformer — 9× in this paper · explained in 2 episodes
• Episodic Memory — 8× in this paper
• Ablation — 7× in this paper · explained in 1 episode
• Instruction Recognizer — 6× in this paper
• Instructional Memory — 5× in this paper
• Accuracy — 4× in this paper · explained in 1 episode
• Embeddings — 4× in this paper · explained in 2 episodes
• Heuristic Context Retrieval — 4× in this paper
• Fine-Tuning — 3× in this paper · explained in 5 episodes
• Instruction Tuning — 3× in this paper · explained in 2 episodes
• LLM-as-a-Judge — 3× in this paper · explained in 3 episodes
• RAG — 3× in this paper · explained in 3 episodes
• Robustness — 3× in this paper · explained in 5 episodes
• Consistency — 2× in this paper · explained in 3 episodes
• Core — 2× in this paper · explained in 2 episodes
• Decoupling — 2× in this paper · explained in 2 episodes
• Joint Goal Accuracy — 2× in this paper
• AdamW Optimizer — 1× in this paper · explained in 1 episode
• Adaptation — 1× in this paper · explained in 1 episode
• Alignment — 1× in this paper · explained in 1 episode
• Benchmarking — 1× in this paper · explained in 3 episodes
• Cosine Similarity — 1× in this paper · explained in 3 episodes
• Failure Modes — 1× in this paper · explained in 3 episodes
• Gradient Flow — 1× in this paper · explained in 2 episodes
• Hallucination — 1× in this paper · explained in 3 episodes
• Information Gain — 1× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Latent Representation — 1× in this paper · explained in 2 episodes
• Latent Representations — 1× in this paper · explained in 3 episodes
• Reinforcement Learning — 1× in this paper · explained in 4 episodes
• Rulebased Instruction Adherence Rate — 1× in this paper
• Speculative Decoding — 1× in this paper · explained in 2 episodes

Sources

• GPT-4 Technical Report
• Longformer: The Long-Document Transformer
• LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models
• Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
• LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens
• LoRA: Low-Rank Adaptation of Large Language Models
• LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models
• Memory OS of AI Agent
• Compressed Context Memory For Online Language Model Interaction
• MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models
• LLMs Get Lost In Multi-Turn Conversation
• Lost in the Middle: How Language Models Use Long Contexts
• PISCO: Pretty Simple Compression for Retrieval-Augmented Generation
• MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversation
• Long Dialog Summarization: An Analysis
• Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
• On Memory Construction and Retrieval for Personalized Conversational Agents
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMs
• BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage

Related papers

• Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
• Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
• Subliminal Steering: Stronger Encoding of Hidden Signals
• MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
• The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
• Untangling the Mechanisms of Misleading Context in Medical Question Answering