Olapa-MCoT: Enhancing the Chinese Mathematical Reasoning Capability of LLMs

arXiv:2312.17535 · cs.AI, cs.CL, cs.HC · Submitted 2023-12-29 · Read on arXiv

Listen

Radio episode about this paper

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Olapa-MCoT: Enhancing the Chinese Mathematical Reasoning Capability of LLMs".

Jane: The paper was written by Shaojie Zhu, Zhaobin Wang, Chengxiang Zhuo, Hui Lu, Bo Hu et al. from Tencent and Shanghai Jiao Tong University.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Jane: We also have Lu with us today — senior AI researcher at Tsinghua.

Tom: We also have Meng with us today — lead engineer at a mysterious AI startup.

Jane: We also have Lalam with us today — the in-house Large Language Model.

Tom: Alright, let's get started.

Summary: Tom: Okay, so we’ve talked about the title and the general idea of improving math reasoning specifically for Chinese contexts. Now that we're looking at the summary section of "Olapa-MCoT: Enhancing the Chinese Mathematical Reasoning Capability of LLMs," what are we learning about their core methodology?

Jane: The summary seems to point toward a specific mechanism they are using, moving beyond just feeding it more data. It sounds like they've engineered a way for the model to structure its thinking process more deliberately when solving these problems.

Lu: Right, it’s not just about *what* information is available; it’s about optimizing the *path* of thought. The summary suggests they are integrating advanced Chain-of-Thought techniques in a way that is highly tailored to mathematical problem decomposition.

Meng: When they talk about structuring the thinking process, are we talking about adding explicit steps, like "Step one: Identify variables," and "Step two: Apply formula X"? Or is it more subtle scaffolding? I need to know if this is trainable or if it requires hand-coded logic.

Lalam: What strikes me from reading the summary is how they are trying to make the reasoning process *visible*. For AI to truly assist humans, we need to see the 'why' behind every conclusion, not just the final answer.

Tom: So, it seems like their main contribution in this section is detailing this improved structural guidance—it’s an enhancement of how the model generates its internal monologue while solving problems. Jane, can you simplify what that structural guidance means for a listener who doesn't read these papers?

Jane: Imagine trying to assemble IKEA furniture using only vague instructions; it’s hard. This paper is giving the LLM like perfectly illustrated, step-by-step diagrams showing exactly how the pieces fit together logically, piece by piece.

Lu: And what makes it advanced is that this scaffolding isn't just a template; it adapts based on the mathematical genre presented in the Chinese context, making it flexible rather than rigid. That adaptive nature is where the real breakthrough lies.

Meng: If this scaffolding is adaptive, does that mean the system needs to dynamically identify which type of thinking sequence—algebraic, geometric, combinatorial—is required for any given problem instance? That requires some kind of meta-reasoning layer.

Lalam: From a cultural standpoint, making the reasoning visible honors the human process of learning. It turns a black box into a teachable model, which is vital if we want AI to become true collaborators rather than just answer engines.

Improvements: Tom: We’ve looked at the title and the summary, and now we’re heading into "Olapa-MCoT: Enhancing the Chinese Mathematical Reasoning Capability of LLMs" section discussing specific improvements. Jane, what is the key improvement they are proposing here that builds on their earlier findings?

Jane: It seems like they are refining *how* the model learns to use those thinking steps. They aren't just showing it correct examples; they might be teaching it how to self-correct or how to manage ambiguity within the problem statement itself.

Lu: Precisely! The improvements go beyond just solving the test set problems. They are improving the *robustness* of the reasoning path itself, making it less susceptible to minor variations in phrasing or data input that often trip up standard models.

Meng: When they talk about robustness, are they suggesting a form of adversarial training? Like intentionally feeding it slightly corrupted versions of known math problems to see where the chain breaks down and then patching those specific failure points?

Lalam: I’m really excited by the idea of improved robustness because it speaks to reliability in high-stakes situations. If this AI is used in, say, medical diagnostics that involve complex formulas, we cannot afford for it to fail gracefully; we need it to fail *predictably*.

Tom: So, these improvements are about hardening the system against failure points rather than just maximizing correct answers on a clean test set. Lu mentioned robustness—can you elaborate on what kind of 'weakness' in

Paper discussion segment 3: Tom: So, we’ve seen how Olapa-MCoT uses a structured approach to tackle Chinese math, and now we want to talk about the specific improvements that make this model work so much better than previous attempts.

Jane: These improvements really boil down to making the learning process smarter and more reliable for both human interaction and internal logic.

Meng: From an engineering standpoint, I’m especially interested in SimRRHF; how much computational power did they save by replacing those complex RLHF setups?

Lu: It's a huge reduction in complexity because instead of needing four separate models running simultaneously, SimRRHF uses a single model guided by similarity loss to ensure the performance remains stable and focused.

Tom: That sounds like they found a way to achieve high fidelity without the massive infrastructure cost, which is a major win for scalability.

Jane: Exactly, so it’s not just about speed; it making sure the quality of the guidance is consistent across every step.

Lalam: And when we talk about stability, that translates into reliability in culture—we aren't building an AI that occasionally hallucinates or drifts away from a core set a reliable truth.

Meng: But what’s truly interesting to me is IDRL, the idea of actively learning from its own mistakes. How does the system actually decide which errors are worth re-learning?

Lu: The model identifies instances where it made incorrect inferences during training, and that data gets put back into the training set for a second pass, forcing a deeper understanding.

Jane: It’s like practicing difficult concepts in math; you don't just read the solution, you drill the exact problem areas where you previously failed until your intuition corrects itself.

Tom: So, by actively reintroducing those errors, they are improving its grasp of complex logic that standard models usually skip over.

Lalam: This shift is profound because it implies that AI can evolve past mere pattern recognition and start developing a genuine capacity for self-correction.

Meng: It’s definitely a powerful way to build robustness into an LLM, but we need to think about the implications of moving forward with this level of complex reasoning.

Lu: The potential for this model to solve highly specialized, multi-step problems is immense, opening doors in fields that require rigorous logical consistency.

Jane: It suggests that the future isn't just about bigger models, but smarter training loops.

Conclusion: Tom: So what we're left with after talking through "Olapa-MCoT: Enhancing the Chinese Mathematical Reasoning Capability of LLMs" is this massive leap in how specialized AI models can become.

Jane: Exactly, Tom. It really shows that giving these large language models specific, high-quality structure—like advanced prompting or targeted data—can dramatically boost their ability to handle complex reasoning tasks in a very specific cultural and academic context.

Meng: The biggest takeaway for me is the engineering implication; it confirms that generalized models aren't enough if you need peak performance in a niche, yet critical, domain like Chinese mathematics.

Lu: But I think the real potential goes way beyond just math problems, don't you see? If we can fine-tune this sophisticated reasoning for one area, we can apply that architecture to literally any complex human knowledge system.

Jane: Lu’s right; it suggests a blueprint for domain adaptation that is incredibly powerful. It’s not just about better math; it's about building better cognitive scaffolding for the AI itself.

Tom: And from a practical standpoint, Meng brought up the niche aspect, which is key because specialized tools are what really move adoption forward in industry.

Meng: Right. Because if we want an AI to assist students or even professionals using localized curriculum, we can't just throw a massive general model at it; you need this level of focused enhancement.

Lalam: Thinking about the impact on society, I see this capability enhancing educational equity across China and beyond. Making advanced reasoning accessible fundamentally improves cultural understanding and opportunity for millions of people who might otherwise struggle with resource limitations.

Lu: And imagine applying that same logic to historical linguistics or regional law codes—it's a gateway to unlocking deeply complex human knowledge that was previously siloed.

Jane: It makes you feel really optimistic about the future of AI education. We’re moving toward tools that genuinely help people learn, rather than just giving them answers.

Tom: It’s been an absolute blast talking through this deep dive into "Olapa-MCoT," guys, and I think we all agree that these advancements are going to change how we approach AI training.

Meng: Yeah, it’s clear the next generation of models will be highly modular and specialization will be king.

Lalam: For me, the most profound shift will be in how culture values knowledge acquisition itself, making learning a more structured and technologically supported process.

Lu: We're talking about a paradigm shift in what 'intelligence' means when we consider human expertise.

Jane: Well, folks, that wraps up our look at this paper for today, but stay tuned because next time we're diving into something totally different...

Shaojie Zhu, Zhaobin Wang, Chengxiang Zhuo, Hui Lu, Bo Hu, Zang Li

Tencent · Shanghai Jiao Tong University

cs.AI, cs.CL, cs.HC

Submitted: 2023-12-29

Updated: 2026-08-25

Code: https://github.com/baichuan-inc/Baichuan-13B

Importance score: 81/100

The gist: This paper introduces Olapa-MCoT, a method designed to enhance the "Chinese mathematical reasoning ability of LLMs" using a llama2-13B base model.

Key concepts

structural guidance
the method's core mechanism for shaping the model's reasoning
Chain-of-Thought techniques
the scaffolding the paper integrates for mathematical problem decomposition
SimRRHF
the training simplification replacing four-model RLHF setups
robustness
the stability goal of the training improvements

Terminology

Summary

This paper introduces Olapa-MCoT, a method designed to enhance the Chinese mathematical reasoning ability of LLMs using a llama2-13B base model. It addresses the limitations of existing models, such as Llama2's very unsatisfactory Chinese math performance and the high training costs associated with large-scale Chinese corpora, by providing an efficient fine-tuning and alignment framework.

The Olapa-SFT Stage

To address the lack of Chinese reasoning ability in the pretrained llama2 model, the researchers first implement a supervised fine-tuning stage. This stage focuses on constructing high-quality Chinese mathematical reasoning samples to build a foundation for subsequent learning. The construction process involves:

  • Designing an inference prompt using the ape210K QA dataset as a construction seed set.

  • Obtaining reasoning steps and answers from models like Baichuan1 and ChatGLM2 as candidate results.

  • Utilizing a high-quality discriminator to filter out data with correct answers and better reasoning steps.

The model is fine-tuned using the QLoRA method to reduce the GPU consumption during this high-demand process.

SimRRHF Alignment

The alignment learning phase introduces SimRRHF, an optimization of the Rank Responses to align Human Feedback (RRHF) paradigm. Unlike RLHF, which requires loading four models and is sensitive to hyperparameters, SimRRHF maintains only one model while achieving similar functions through a weighted sum of three specific losses:

  1. Ranking loss (L rank) to optimize the model's ability to rank responses.

  2. SFT loss (L sft) using the length-normalized value of the top-rated response.

  3. Similarity loss (L similarity) which calculates the semantic distance between the response generated by model pi and the top rated response.

This approach ensures that the performance of current finetuning model [does] not deviate from the best object, leading to more stable and faster convergence while avoiding uncontrollable model performance caused by excessive liberalization of the learning process.

Incorrect Data Relearning

To improve the model's ability to handle difficult reasoning knowledge, the authors propose Incorrect Data Relearning (IDRL). This method is inspired by how humans sort out common mistakes then practice repeatedly. During alignment learning, the researchers collect data where the model made incorrect inferences in the training dataset and incorporate these samples into subsequent rounds of training.

The process involves:

  • Collecting incorrect inferences from previous training rounds to form an important part of new samples.

  • Supplementing this with new samples to create a complete training dataset for the next round.

This iterative approach allows Olapa-MCoT to gain a more accurate understanding and learning of difficult knowledge, specifically helping the model overcome complex reasoning logic.

Experimental Results

The effectiveness of the Olapa-MCoT method is demonstrated through significant improvements in both Chinese and English reasoning tasks. On the Chinese evaluation dataset, accuracy reached 50%, representing a 36% rise compared to llama2-13B. Notably, the model's performance surpassed several prominent models:

  • It achieved 1% more accuracy than gpt-3.5-turbo in Chinese math reasoning.

  • It outperformed Baichuan and Chatglm LLMs by 3% to 12%.

Furthermore, the English mathematical reasoning accuracy also increased by nearly 4% compared to the llama2-13B-chat baseline. Ablation studies confirm that SimRRHF improves stability and that IDRL provides a 5% higher accuracy boost over models trained without it.

Improvements for AI systems

Improvement 1: Automated Reasoning Synthesis Pipeline

Implement a data construction architecture that utilizes a seed dataset to trigger multi-model reasoning generation (using diverse LLM architectures), followed by a high-quality discriminator-based filtering stage to select only trajectories with correct answers and logically consistent intermediate steps.

  • Improved AI Capability: The system can rapidly and autonomously scale high-quality, multi-step Chain-of-Thought (CoT) training datasets for specialized domains or low-resource languages without the prohibitive cost of human annotation.

Improvement 2: SimRRHF (Semantic Similarity Augmented Alignment)

Integrate a similarity loss (L similarity) into the preference alignment process, specifically calculating the cosine distance between the mean embeddings of the model’s generated response and the top-rated reference response, combined with length-normalized ranking loss.

  • Improved AI Capability: The system can achieve stable, high-precision alignment with human preferences while maintaining low memory overhead; it prevents model drift (uncontrollable performance degradation) during reasoning optimization without the computational complexity of maintaining four separate models (Policy, Value, Reward, Reference) as required by PPO.

Improvement 3: Iterative Incorrect Data Relearning (IDRL) Loop

Incorporate a closed-loop training mechanism where the model’s failed inference attempts and incorrect reasoning paths are automatically harvested from the training set to form a hard sample dataset, which is then supplemented with new data and used for subsequent fine-tuning rounds.

  • Improved AI Capability: The system can autonomously identify its own logical weak points and self-correct through targeted practice, leading to significantly higher accuracy in complex, multi-step mathematical and symbolic reasoning tasks.

Sources

Related papers