ORCA: Evaluating LLMs on Data Science Code Translation
cs.AI
Submitted: 2026-09-25
Updated: 2026-09-25
Code: https://github.com/bird-bench/livesqlbench
Project page: https://bird-bench.github.io
Terminology
Sources
- On Randomness in Agentic Evals
- Training and Evaluating a Jupyter Notebook Data Science Assistant
- On the evolution of technical lag in the npm package dependency network
- TimeMachine-bench: A Benchmark for Evaluating Model Capabilities in Repository-Level Migration Tasks
- RepoTransAgent: Multi-Agent LLM Framework for Repository-Aware Code Translation
- InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks
- CoTran: An LLM-based Code Translator using Reinforcement Learning with Feedback from Compiler and Symbolic Execution
- DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?
- Capture the Flag: Uncovering Data Insights with Large Language Models
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
- Leveraging Automated Unit Tests for Unsupervised Code Translation
- ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code
- CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference
- Skeleton-Guided-Translation: A Benchmarking Framework for Code Repository Translation with Fine-Grained Quality Evaluation
Related papers
- MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Prediction
- Model Discovery Agent: LLM-assisted Bayesian experiment design for data-efficient discovery of mechanistic world models
- The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
- MindHelper: Closed-Loop Embodied Mental-State Reasoning for Precision Intervention
- Incumbent Advantage: Brand Bias and Cognitive Manipulation Dynamics in LLM Recommendation Systems
- VSAL: A Vision Solver with Adaptive Layouts for Graph Property Detection