On the Lexical Superstition of Large Language Models for Code Comprehension: Re-evaluation on Code of Low Lexical Quality

arXiv:2609.26388 · cs.SE, cs.CL, cs.LG · Submitted 2026-09-22 · Read on arXiv

cs.SE, cs.CL, cs.LG

Submitted: 2026-09-22

Updated: 2026-09-22

Comments: 27 pages, 9 figures, 12 tables. Submitted to an ACM journal in September 2025. Preprint; manuscript under review. Corresponding author: Ming Li

License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/

Terminology

• Accuracy — 14× in this paper · explained in 1 episode
• Fine-Tuning — 13× in this paper · explained in 5 episodes
• Benchmark — 9× in this paper · explained in 2 episodes
• Robustness — 9× in this paper · explained in 5 episodes
• In-Context Learning — 8× in this paper · explained in 4 episodes
• alpha — 6× in this paper · explained in 1 episode
• Transformer — 4× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 3× in this paper · explained in 6 episodes
• Adversarial Robustness — 2× in this paper · explained in 3 episodes
• Chain-of-Thought Prompting — 2× in this paper · explained in 2 episodes
• Generalization — 2× in this paper · explained in 13 episodes
• Mean Reciprocal Rank — 2× in this paper
• Adaptation — 1× in this paper · explained in 1 episode
• Adversarial Context — 1× in this paper · explained in 1 episode
• Adversarial Examples — 1× in this paper · explained in 2 episodes
• Contrastive Learning — 1× in this paper · explained in 5 episodes
• Embeddings — 1× in this paper · explained in 2 episodes
• F1 Score — 1× in this paper · explained in 3 episodes
• Few-Shot Learning — 1× in this paper · explained in 4 episodes
• Foundation Model — 1× in this paper · explained in 3 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Guided attention — 1× in this paper · explained in 2 episodes
• Interpretability — 1× in this paper · explained in 15 episodes
• Structural Reasoning — 1× in this paper · explained in 2 episodes

Sources

• GPT-4 Technical Report
• Evaluating Large Language Models Trained on Code
• Large Language Models of Code Fail at Completing Code with Potential Bugs
• An Extensive Study on Adversarial Attack against Pre-trained Models of Code
• The Llama 3 Herd of Models
• Two Sides of the Same Coin: Exploiting the Impact of Identifiers in Neural Code Comprehension
• DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
• Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicates
• CodeSearchNet Challenge: Evaluating the State of Semantic Code Search
• A Survey on Large Language Models for Code Generation
• Code Llama: Open Foundation Models for Code
• LLaMA: Open and Efficient Foundation Language Models
• Qwen2 Technical Report
• Qwen2.5 Technical Report
• Transfer Attacks and Defenses for Large Language Models on Coding Tasks
• DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence

Related papers

• Falsification-Based Verification of LLM-Generated Optimization Models: Sound Test Batteries and Their Detection Limits
• GitSkills: A Dataset of Agent Skills on GitHub
• SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
• PackMonitor: Enabling Zero Package Hallucinations Through Decoding-Time Monitoring
• IntentCoding: Amplifying User Intent in Code Generation
• Incentives and Outcomes in Bug Bounties