Evaluation of OpenAI o1: Opportunities and Challenges of AGI
cs.CL
Submitted: 2024-09-27
Updated: 2026-09-23
Code: https://github.com/UGA-CAID/AGI-Bench
Project page: https://louisapenfold.com/cardboard-art
License: http://creativecommons.org/licenses/by/4.0/
The gist: This comprehensive study evaluates the performance of OpenAI's o1-preview large language model across a diverse array of complex reasoning tasks, spanning multiple domains, including computer
Terminology
Abstract
This comprehensive study evaluates the performance of OpenAI's o1-preview large language model across a diverse array of complex reasoning tasks, spanning multiple domains, including computer science, mathematics, natural sciences, medicine, linguistics, and social sciences. Through rigorous testing, o1-preview demonstrated remarkable capabilities, often achieving human-level or superior performance in areas ranging from coding challenges to scientific reasoning and from language processing to creative problem-solving. Key findings include: -83.3% success rate in solving complex competitive programming problems, surpassing many human experts. -Superior ability in generating coherent and accurate radiology reports, outperforming other evaluated models. -100% accuracy in high school-level mathematical reasoning tasks, providing detailed step-by-step solutions. -Advanced natural language inference capabilities across general and specialized domains like medicine. -Impressive performance in chip design tasks, outperforming specialized models in areas such as EDA script generation and bug analysis. -Remarkable proficiency in anthropology and geology, demonstrating deep understanding and reasoning in these specialized fields. -Strong capabilities in quantitative investing. O1 has comprehensive financial knowledge and statistical modeling skills. -Effective performance in social media analysis, including sentiment analysis and emotion recognition. The model excelled particularly in tasks requiring intricate reasoning and knowledge integration across various fields. While some limitations were observed, including occasional errors on simpler problems and challenges with certain highly specialized concepts, the overall results indicate significant progress towards artificial general intelligence.
Sources
- GPT-4 Technical Report
- Self-Driving Cars: A Survey
- On the Measure of Intelligence
- Training Diffusion Models with Reinforcement Learning
- Exploring Multimodal Approaches for Alzheimer's Disease Detection Using Patient Speech Transcript and Audio Data
- MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
- AD-AutoGPT: An Autonomous GPT for Alzheimer's Disease Infodemiology
- AugGPT: Leveraging ChatGPT for Text Data Augmentation
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challenges
- A Survey of Data Augmentation Approaches for NLP
- CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Study
- REALM: Retrieval-Augmented Language Model Pre-Training
- Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey
- Measuring Massive Multitask Language Understanding
- Benchmarking a foundation LLM on its ability to re-label structure names in accordance with the AAPM TG-263 report
- Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics
- LoRA: Low-Rank Adaptation of Large Language Models
- In-Context Analogical Reasoning with Pre-Trained Language Models
- MathPrompter: Mathematical Reasoning using Large Language Models
Related papers
- Exploring Solution Divergence and Its Effect on Large Language Model Problem Solving
- Ishigaki-IDS-Bench: A Benchmark for Generating Information Delivery Specification from BIM Information Requirements
- Subliminal Steering: Stronger Encoding of Hidden Signals
- MedStruct-S: A Benchmark for Key Discovery, Key-Conditioned QA and Semi-Structured Extraction from OCR Clinical Reports
- The End of Transformers? On Challenging Attention and the Rise of Sub-Quadratic Architectures
- Untangling the Mechanisms of Misleading Context in Medical Question Answering