Hydra: Phase-Aware Workload Characterization of LLM Inference across Edge SoC Generations, Backends, and Quantization Levels

arXiv:2608.25053 · cs.AR, cs.AI, cs.DC, cs.PF · Submitted 2026-08-25 · Read on arXiv

cs.AR, cs.AI, cs.DC, cs.PF

Submitted: 2026-08-25

Updated: 2026-08-25

Code: https://github.com/amirtaherin/hydra

Terminology

• Decode — 60× in this paper · explained in 2 episodes
• Quantization — 39× in this paper · explained in 12 episodes
• Transformer — 22× in this paper · explained in 2 episodes
• Tokenization — 18× in this paper · explained in 2 episodes
• Core — 13× in this paper · explained in 2 episodes
• Benchmarking — 10× in this paper · explained in 3 episodes
• Benchmark — 9× in this paper · explained in 2 episodes
• Trace — 7× in this paper · explained in 2 episodes
• Accuracy — 4× in this paper · explained in 1 episode
• Energy Efficiency — 4× in this paper · explained in 3 episodes
• Edge Devices — 3× in this paper · explained in 2 episodes
• Speculative Decoding — 3× in this paper · explained in 2 episodes
• Generative AI — 2× in this paper · explained in 2 episodes
• KV cache — 2× in this paper · explained in 3 episodes
• Meeting of the Association for Computational Linguistics — 2× in this paper
• Time to First Token — 2× in this paper
• Adaptation — 1× in this paper · explained in 1 episode
• Allocator — 1× in this paper · explained in 1 episode
• Annual International Symposium on Computer Architecture — 1× in this paper
• Consistency — 1× in this paper · explained in 3 episodes
• Edge AI — 1× in this paper · explained in 2 episodes
• Edge Deployment — 1× in this paper · explained in 2 episodes
• High-Performance Computer Architecture — 1× in this paper
• IEEE International Symposium on Workload Characterization — 1× in this paper
• International Conference on Learning Representations — 1× in this paper
• Large Language Models (LLMs) — 1× in this paper · explained in 6 episodes
• Quantization-Aware Training — 1× in this paper · explained in 3 episodes
• Verification — 1× in this paper · explained in 2 episodes
• World Model — 1× in this paper · explained in 7 episodes

Sources

• Evaluating Large Language Models Trained on Code
• No Language Left Behind: Scaling Human-Centered Machine Translation
• DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
• Human Cognition in Machines: A Unified Perspective of World Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Splitwise: Efficient generative LLM inference using phase splitting
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving
• ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production
• Large Language Models for Robotics: A Survey
• VOTE: Vision-Language-Action Optimization with Trajectory Ensemble Voting
• A Survey on Efficient Inference for Large Language Models
• DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency
• Squat: Quant Small Language Models on the Edge
• Generative AI on the Edge: Architecture and Performance Evaluation
• PalmBench: A Comprehensive Benchmark of Compressed Large Language Models on Mobile Platforms
• MELTing point: Mobile Evaluation of Language Transformers
• LLM in a flash: Efficient Large Language Model Inference with Limited Memory
• MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
• Mobile Foundation Model as Firmware
• MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI

Related papers

• WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization
• Golden Ruler: A Numeric Format Catalog with Bit-Exact Conformance Vectors for FP8, BF16, MXFP4, and Microscaling Formats
• PoisonCap: Efficient Hierarchical Temporal Safety for CHERI
• Provisioning to Runtime Optimization of a 100 MW-Scale AI Cluster
• Bit-Accurate Modeling of GPU Matrix Multiply-Accumulate Units: Demystifying Numerical Discrepancy and Accuracy
• Optimizing Polynomial Multiplication and Fixed-Weight Sampling for HQC on ARM Cortex-M4