QC-Stark: A Multi-Task Benchmark Revealing Capability Dissociations in LLMs Evaluated on Quantum Computing Tasks

arXiv:2609.35581 · quant-ph, cs.AI, cs.LG · Submitted 2026-09-28 · Read on arXiv

quant-ph, cs.AI, cs.LG

Submitted: 2026-09-28

Updated: 2026-09-28

Terminology

• Benchmark — 24× in this paper · explained in 2 episodes
• Fidelity — 11× in this paper · explained in 2 episodes
• Accuracy — 9× in this paper · explained in 1 episode
• Item Response Theory — 4× in this paper
• Verification — 3× in this paper · explained in 2 episodes
• Large Language Models (LLMs) — 2× in this paper · explained in 6 episodes
• RAG — 2× in this paper · explained in 3 episodes
• Robustness — 2× in this paper · explained in 5 episodes
• Variational Quantum Eigensolver — 2× in this paper
• Benchmarking — 1× in this paper · explained in 3 episodes
• Decode — 1× in this paper · explained in 2 episodes
• Error Taxonomy — 1× in this paper · explained in 2 episodes
• Fine-Tuning — 1× in this paper · explained in 5 episodes
• Generative Model — 1× in this paper · explained in 2 episodes
• Generative Models — 1× in this paper · explained in 2 episodes
• Retrieval Augmented Generation (RAG) — 1× in this paper · explained in 4 episodes
• Variational Quantum Eigensolver (VQE) — 1× in this paper · explained in 5 episodes
• Verifier — 1× in this paper · explained in 2 episodes

Sources

• Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation
• Quantum-Audit: Evaluating the Reasoning Limits of LLMs on Quantum Computing
• QCircuitBench: A Large-Scale Dataset for Benchmarking Quantum Algorithm Design
• SciCode: A Research Coding Benchmark Curated by Scientists
• CMT-Benchmark: A Benchmark for Condensed Matter Theory Built by Expert Researchers
• Holistic Evaluation of Language Models
• Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models
• Are Emergent Abilities of Large Language Models a Mirage?
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training

Related papers

• Reconquering Bell sampling on qudits: stabilizer learning and testing, quantum pseudorandomness bounds, and more
• Encrypted clones can leak: Classification of informative subsets in Quantum Encrypted Cloning
• Polynomial-time classical and quantum simulation of quantum impurity models
• Theory of quantum-enhanced interferometry with general Markovian light sources
• A convergent hierarchy of spectral gap certificates for qubit Hamiltonians
• Universal Bound and Phase Transition in Many-Body Fermionic Non-Gaussianity