MUSE: A Full-Text Cross-Domain Knowledge Base of Scientific Problems, Solutions, and Rationales

MUSE: A Full-Text Cross-Domain Knowledge Base of Scientific Problems, Solutions, and Rationales

Abstract

Scientific papers contain fine-grained records of problem solving: authors mention technical obstacles and methods that were used to address them, often along with reasoning on why those methods were chosen. We introduce MUSE (Mining Underlying Scientific Explanations), a full-text, multi-domain resource of scientific Problem–Solution–Rationale (P–S–R) triplets. We curate 579 expert-annotated full-text paragraphs, with a rich annotation schema covering salient problem, solution, and rationale spans, solves and rationale of links, and conceptual coreference. A modular extraction pipeline scales this annotation to build a high-quality knowledge base of 37K source-grounded P–S–R triplets. We evaluate the extraction components and include a preliminary experiment training a rationale-supervised LLM for scientific problem solving. Interestingly, we find that rationale supervision improves performance on complex, multiconstraint problems but can harm performance on simpler ones.

Introduction

A fundamental goal in Scientific NLP is to convert scholarly text into structured representations that support retrieval, synthesis, and computational analysis. This objective has motivated construction of large scholarly corpora, as well as information-extraction benchmarks for scientific keyphrases, typed entities, semantic relations, coreference, and document-level n-ary relations. Full-text resources annotate entities and relations in complete papers, such as dataset, method, task and metric, and discourse-oriented resources annotate rhetorical or communicative roles such as background, motivation, method, result, and conclusion.

Scientific knowledge-bases built on information extraction include problems and solutions, contributions, challenges, and inspirations. These resources support many important tasks, but they do not directly represent a common structure in scientific papers: a fine-grained concrete technical sub-problem (which can appear deep in the full text, not being the main problem addressed by the paper), the solution used for that problem, and the rationale that explains why the solution is appropriate. This is more specific than a paper-level contribution—indeed, the problems and solutions may not even be part of the paper's contribution per se.

We introduce MUSE (Mining Underlying Scientific Explanations), a dataset and knowledge base for full-text scientific Problem–Solution–Rationale (P–S–R) extraction. In MUSE, a problem is a limitation, obstacle, gap, or desideratum; a solution is a method, mechanism, formulation, design choice, or procedure that was used to address it; and a rationale is the author-stated motivation, justification or explanation connecting the solution to the problem. The annotation scheme captures both textual evidence and relational structure: both sentences and salient spans for problems, solutions, and rationales; solves links; rationale of links; and conceptual coreference among fragmented or repeated mentions.

The resource has two intended contributions as an IE artifact. First, it is full-text and broad-domain: the extracted triplets are mined from paragraphs throughout full-text scientific papers from multiple domains, rather than only abstracts; this allows the resource to capture technical justifications that often appear in methods, analysis, appendix, and limitation sections. Second, it is structurally rich. Each example retains source provenance and multiple annotation layers: paragraph relevance, span boundaries, typed relations, concept-level coreference groups, and post-processed standalone problem, solution, and rationale fields. This makes MUSE useful both as a structured knowledge base and as training or evaluation data for IE.

We carefully train a dedicated extraction pipeline on MUSE, and use the pipeline to construct a KB with roughly 37K P–S–R triplets across all domains in arXiv. We envisage that this KB may be used for many future applications, including training and evaluation sets for AI scientific problem solving, cross-domain ideation through analogous rationales, tools for surfacing expert reasoning patterns, and metascientific analyses of how different fields formulate constraints and justify methods. As one demonstration, we study whether extracted rationales can supervise LLMs for scientific problem solving. We find that rationale supervision is useful when a problem contains multiple constraints that must be aligned with the solution design, but can be harmful on simple problems. These experiments are preliminary and are primarily presented to illustrate an application of the resource.

Our contributions are:

1. A full-text scientific IE task and expert-annotated seed dataset for Problem–Solution–Rationale extraction, including salient spans, solves links, rationale of links, and conceptual coreference.

2. A large structured knowledge base of 36,960 source-grounded P–S–R triplets mined from full-text scientific papers across domains and represented as self-contained problem, solution, and rationale fields.

3. A preliminary application study showing how MUSE rationales can be used for rationale-supervised LLM training for scientific problem solving.

Related Work

Scientific information extraction involves abstract-level or paper-level entity and relation extraction tasks. ScienceIE focuses on scientific keyphrases and semantic relations such as synonym and hyponym links. SciERC annotates scientific entities, relations, and coreference clusters in abstracts. SciREX targets document-level IE over scientific articles, including salient entity identification and n-ary relation extraction. Recent full-text resources such as SciER and SciNLP extend this line to complete papers, annotating entities and relations involving datasets, methods, tasks, metrics, and other scientific artifacts.

Discourse-oriented resources represent the rhetorical function of scientific passages. CoreSC-style annotation identifies conceptual zones such as background, motivation, goal, method, result, and conclusion; SciDTB annotates discourse dependencies in scientific abstracts; and CODA-19 scales research-aspect annotation to COVID-19 abstracts.

Heffernan and Teufel create a corpus of positive and negative problem and solution phrases and train classifiers for problemhood and solutionhood. Lahav et al. annotate challenges and (independently) research directions in full-length COVID-19-related papers and build a search engine for scientific discovery.

MUSE differs in several ways. MUSE extracts problem-solving structures and the rationale connecting solutions to the problems they address, in full-text passages, from many domains. It operates at paragraph and span level rather than phrase or sentence level; and it links solutions to the specific problems they address and adds rationales and rationale relations.

Scientific knowledge-base construction has increasingly moved toward richer representations of contributions. CHIMERA mines idea recombinations and inspirations for scientific ideation; the Scientific Contribution Graph models contributions and prerequisite relations for literature-based technological roadmapping. MUSE is complementary to such resources: it extracts local problem–solution–rationale units that can be indexed, compared, and reused.

Natural-language rationales have been studied in interpretability and explanation datasets, including eSNLI for natural language inference explanations and ERASER for evaluating rationalized NLP models across tasks. In contrast, MUSE rationales are not annotations of why a model predicted a label; they are author-written scientific justifications that explain why a technical solution addresses a technical problem. This makes them useful for scientific search, problem-solving evaluation, and analysis of expert justification patterns, and potentially also as additional supervision for training reasoning LLMs.

The P–S–R Task and Dataset

Task Definition

Given a scientific paragraph D, the task is to recover a set of P–S–R structures. Each structure contains:

• Problem: a concrete limitation, obstacle, gap, or desideratum;

• Solution: a method, mechanism, design choice, formulation, or procedure used to address the problem;

• Rationale: the stated motivation, justification or explanation why the solution was chosen to address the problem.

At the annotation level, problems and solutions are marked as salient spans, problem salient and solution salient, and rationales are marked as rationale spans. We additionally annotate three relations: solves links a solution to the problem it addresses; rationale of links a rationale to the solution it explains; and coreference links multiple spans that refer to the same underlying problem or solution concept.

Annotated Seed Dataset

We sample full-text paragraphs from arXiv, using high-recall keyword filters to make annotation more efficient than random sampling of full-text paragraphs. The seed set contains 579 manually annotated paragraphs, a number comparable to or larger than sets used in related scientific IE resources (e.g., CHIMERA and CARE). Unlike these abstract-level resources, we annotate full-text paragraphs. Annotation was conducted by a PhD-level scientific annotator after screening and training and an NLP expert. Another NLP expert then independently annotated a held-out quality-control subset; disagreements were discussed, guidelines were refined, and the annotator revised examples accordingly. Agreement was very high. We use an 80%/10%/10% train/validation/test split at the paper level.

The MUSE Extraction Pipeline

The design differentiates between labels that are local and boundary-based, and outputs that require conceptual grouping or source-grounded rewriting.

Why not end-to-end LLM extraction? Before designing the modular pipeline, we explored prompting a single LLM to extract complete Problem–Solution–Rationale structures from a paragraph in one pass. On simple paragraphs with a single problem and solution the model was often adequate, but it degraded sharply on the structured cases that motivate this work: it frequently conflated solutions with their rationales, dropped the relations between multiple problem–solution pairs, and failed to preserve the correspondence among problems, solutions, and rationales when a paragraph interleaved several concepts. Rationale extraction was particularly brittle — given only the paragraph, the model tended to summarize the paper's overall contribution rather than the local justification for a specific solution. These failures motivate our decomposition into paragraph filtering, span extraction, relation extraction, and post-processing, so that each stage solves a narrower, better-posed problem; in particular, rationale construction is conditioned on already-extracted problem and solution anchors rather than on the raw paragraph alone.

Paragraph filtering. The first stage identifies paragraphs likely to contain a problem, a solution, and a rationale. A single holistic classifier was unreliable on the small seed set, so we use a cascade: first detect Problem–Solution paragraphs, then detect whether a rationale is present among those paragraphs. For large-scale construction, each sample receives a probability score from the cascade. If this probability is higher than 0.5, the sample is classified as relevant; otherwise, it is considered irrelevant. To prioritize precision during KB construction, we retain only the top 10% of paragraphs ranked by the cascaded classifier's confidence score.

Problem and solution extraction. For paragraphs that pass filtering, we extract problem salient and solution salient spans with token-level classifiers. We evaluate multiple encoder architectures and labeling formulations, and select the best-performing multi-class DeBERTa-v3-large model for both problem and solution spans. Rationale spans are substantially less boundary-stable: they are often distributed across clauses or require synthesis. We therefore do not rely on token classification for the final rationale field.

Relation extraction. We fine-tune Mistral-7B-Instruct for two relational steps. First, conceptual coreference groups problem or solution spans that refer to the same underlying concept. Second, a solves classifier links each solution group to the problem group it addresses. This produces structured problem–solution candidates with provenance in the original paragraph.

Post-processing and rationale construction. The final fields in the KB must be readable outside the source paragraph. We prompt an LLM (GPT-4o) with the paragraph and extracted spans to produce concise, self-contained problem and solution descriptions. We then prompt GPT-4o with the paragraph and the post-processed problem–solution pair to extract a faithful, source-grounded rationale. This step rewrites and contextualizes the author-described rationale rather than treating the generated text as an independent model explanation. We validate this LLM step with human annotation over 30 examples, finding high agreement.

The span-level output is then post-processed into a self-contained triplet t = (problem: P, solution: S, rationale: R), where P, S, and R can be read independently of the source paragraph while preserving provenance.

Extraction Results

Paragraph Classification

The holistic P–S–R classifier performs poorly on the seed set, motivating the cascaded design. The cascaded classifier improves the relevant-class F1, and the top-confidence version prioritizes precision for KB construction. Human agreement for relevant class is F1=0.91, while the cascaded classifier achieves F1=0.73, and the cascaded @10 achieves F1=0.73 for relevant and F1=0.82 for irrelevant.

Entity Extraction

Encoder classifiers are evaluated with strict token-level matching. Multi-class DeBERTa-v3-large is strongest for problem and solution spans (F1=0.66 and F1=0.64 respectively), while rationale spans are not reliably captured by token classification (best F1=0.13).

Relation Extraction and Post-Processing

For conceptual coreference, the fine-tuned Mistral-7B model achieves strong link-based performance (F1 = 0.92), indicating that it can reliably group fragmented or repeated mentions of the same problem or solution concept. The solves classifier also performs well on positive links (F1 = 0.87), although performance is lower for the negative class (F1 = 0.69), suggesting that the model is better at identifying plausible problem–solution relations than rejecting difficult non-relations.

The GPT-4o post-processing stage produces highly self-contained and faithful problem and solution descriptions: problems achieve scores of 4.97 for standalone quality and 4.52 for fidelity, while solutions achieve 4.82 and 4.51, respectively. Completeness is lower, particularly for solutions (3.53), reflecting the challenge of preserving all technical details when condensing complex solution spans. Rationale construction is generally grounded and coherent, with high scores for groundedness (4.70) and coherence (4.84), as well as a SciBERT BERTScore F1 of 0.78 against human rationale annotations. However, rationale specificity and completeness remain more moderate, indicating that rationale generation is still the least mature component of the pipeline.

During evaluation, we further observed cases where solution and rationale descriptions were not sufficiently separated, with generated fields containing overlapping content from both components. To address this issue, we applied an additional refinement step using Claude Opus 4.8 over the generated P–S–R records. This step focused on improving the semantic distinction between solution and rationale fields, ensuring that solution descriptions primarily capture the proposed technical approach, while rationale descriptions capture the underlying motivation, justification, or explanation for the approach. Following this refinement, we manually reviewed 50 randomly sampled P–S–R records and found that 43 examples exhibited a clear separation between solution and rationale descriptions, corresponding to an 86% success rate.

KB Construction

For KB construction, we apply the pipeline to arXiv full-text corpus covering papers from across all arXiv categories through 2026. We construct the KB from the top 10% of candidate paragraphs classified as relevant, ranked by the cascaded classifier's confidence score. Applying the pipeline to full-text scientific papers in arXiv yields 36,960 P–S–R triplets. Each triplet contains a self-contained problem, solution, and rationale, while retaining the source paragraph and intermediate extraction structure. The KB is therefore usable both as a text resource and as a structured graph of problem–solution–rationale relations.

Preliminary Application: LLM Rationale Tuning

To illustrate one use of MUSE, we conduct an exploratory study of rationale-supervised fine-tuning. The goal is not to claim a state-of-the-art reasoning model, but to test whether the rationale field provides a useful training signal beyond solution-only supervision.

Setup

For two open-source models—Qwen-3-32B and DeepSeek-R1-Distill-Qwen-32B—we train two variants:

• PS: problem → solution, omitting rationale;

• PRS: problem → rationale → solution, using the full P–S–R triplet.

Both variants use the same examples and decoding settings. Training uses supervised fine-tuning followed by Group Relative Policy Optimization (GRPO), where we optimize for correctness, divergence and format with the following reward: Rtotal (x, y) = 0.7 Rcorr (x, y) + 0.1 Rfmt (x, y) + 0.2 Rdiv (x, y), where Rcorr uses an LLM as judge, Rdiv penalizes near-copying of the prompt using sentence embeddings, and x, y are input and output, respectively. We optimized hyperparameters with grid search. To prevent data contamination in this preliminary study, we construct the test and validation sets from papers published in arXiv after the models' stated knowledge cutoff dates, yielding 298 test and 296 validation examples. Evaluation uses a blinded LLM-as-a-judge protocol. Base, PS, and PRS models are prompted identically to produce a rationale and solution; responses are anonymized and randomized before scoring. The judge scores overall quality, rationale quality, solution quality, technical detail, novelty, feasibility, and relative ranks.

Results and Analysis

We find that rationale supervision helps Qwen-3-32B relative to solution-only training: overall quality rises from 5.12 to 6.41 and solution quality from 5.10 to 6.37. DeepSeek-R1-32B shows a mixed pattern: the PRS variant slightly lowers holistic scores but modestly improves novelty, feasibility, and technical detail scores.

However, for both model families the untuned base models are strongest in absolute overall score. To understand this dynamic, we stratified the evaluation set into the 10% most complex and 10% least complex problems, as rated by Claude Opus 4.8. The rating scores each problem primarily on its constraint load — the number of interacting requirements a correct solution must satisfy. These ratings agreed with human complexity scores 83.3% of the time.

The two strata behave in opposite directions. On the most complex problems, the rationale-supervised model consistently outperformed the base model across key metrics, suggesting that rationale supervision is most useful when a problem contains multiple constraints that must be aligned with the solution design. On the least complex problems the effect reverses: rationale supervision consistently degraded performance relative to the base model, suggesting that enforcing rationale generation on trivial problems may lead the model to over-engineer its responses and underperform the already-competitive baseline.

To understand the mechanism behind this divergence, we inspected model outputs across the two strata. Both effects appear to stem from a single underlying behavior: rationale supervision teaches the model to reason explicitly about problem constraints before committing to a solution. On complex problems the PRS model tends to keep all of them in view and choose a mechanism that satisfies each, whereas the base model often resolves the most salient constraint while dropping or violating another. On trivial problems the model manufactures constraints that are not present and reaches for heavier machinery than the task warrants.

Conclusion

We introduced MUSE, a full-text, cross-domain resource for representing scientific problem solving as structured Problem–Solution–Rationale (P–S–R) triplets. Unlike resources centered on paper-level contributions or broad rhetorical roles, MUSE captures local technical reasoning that may appear throughout a paper: a concrete issue, the method used to address it, and the author-stated rationale for that choice. We curated an expert-annotated seed dataset of full-text paragraphs with a rich schema, achieving high inter-annotator agreement. We used the annotated data to train extraction models: we designed and evaluated a modular extraction pipeline whose decomposition addresses systematic failure modes we identified in end-to-end LLM extraction, such as conflating solutions with rationales and misaligning entities across multiple P–S–R structures. We then used the pipeline to extract a high-precision KB of 36,960 source-grounded and post-processed P-S-R triplets from across all arXiv categories.

MUSE supports scientific IE, as well as potential additional applications, e.g., meta-scientific analysis of how disciplines justify methodological choices, and LLM-based scientific ideation. Our preliminary rationale-supervised fine-tuning study surfaced a potentially interesting finding for future exploration: that rationale supervision is not uniformly beneficial. It improved solution generation on complex, multi-constraint problems but degraded performance on trivial problems. This suggests that the value of explanation-style supervision may be conditional on problem structure. Future work may improve our extraction methods, use our extraction pipeline to collect larger KBs, and integrate the extracted rationales into LLM post-training pipelines for seeding scientist-like reasoning patterns. We release the dataset, models, and KB to support these directions.

Limitations

Scope. Generalization to medicine, social science or humanities literature remains to be tested.

Preliminary LLM-training study. The fine-tuning experiments use constrained compute, LoRA adaptation, small effective batch sizes. They should be read as a demonstration of one possible application of the dataset, not as a conclusive result about reasoning-model training.

Based on the paper, here are specific improvements you can make to AI systems and what the improved systems can do:

1. Constraint-Aware Reasoning Module

• Improvement: Train the AI to explicitly enumerate and track all constraints in a problem before generating a solution, using the MUSE rationale structure as supervision. Add a constraint load detection mechanism that estimates problem complexity (number of interacting requirements) and adjusts reasoning depth accordingly.

• What it can do: On multi-constraint problems (e.g., design a memory-efficient algorithm that also handles streaming data and maintains O(log n) query time), the AI will keep all constraints in view and produce solutions satisfying each, rather than dropping or violating some. On trivial problems, it will avoid over-engineering and produce simpler, more direct answers.

2. Conditional Rationale Generation

• Improvement: Implement a gating mechanism that decides whether to generate a rationale based on problem complexity. For complex problems, force explicit rationale generation; for simple ones, skip or shorten it. Use the paper's finding that rationale supervision helps on complex but harms on simple problems.

• What it can do: The AI will dynamically adjust its explanation depth—providing detailed justifications for difficult, multi-step problems while giving concise, direct answers for easy ones—improving both accuracy and user satisfaction across a range of query difficulties.

3. Solution–Rationale Separation Module

• Improvement: Add a post-processing layer (similar to the Claude Opus refinement step in the paper) that semantically distinguishes the what (solution: the technical approach) from the why (rationale: the motivation/justification). Train a classifier to detect overlap and rewrite fields to ensure separation.

• What it can do: When generating technical responses, the AI will produce clearly delineated sections: one describing the method/mechanism and another explaining why it was chosen. This reduces confusion in outputs like we use X because it's efficient where the rationale is embedded in the solution description, making outputs more modular and easier for downstream systems to parse.

4. Cross-Domain Analogy Engine

• Improvement: Build a retrieval system that indexes the 37K P–S–R triplets and, given a new problem, retrieves analogous rationales from different domains (e.g., a physics problem's rationale about minimizing energy dissipation applied to a computer vision problem about reducing computational cost). Use the rationale of links to match problem structures across fields.

• What it can do: The AI can suggest novel solutions to a problem by borrowing justification patterns from unrelated disciplines—e.g., applying a biological homeostasis rationale to a network load-balancing problem—enabling cross-domain ideation and creative problem-solving that a single-domain model would miss.

5. Rationale-Grounded Evaluation Metric

• Improvement: Create an automated evaluator that scores AI-generated solutions not just on correctness but on whether the rationale aligns with the problem's constraints (using the MUSE schema). Train a judge model on the P–S–R triplets to detect missing justifications, unsupported claims, or rationales that don't connect to the specific problem.

• What it can do: The AI system can self-assess its outputs before delivery, flagging when a solution lacks a proper justification or when the rationale doesn't address the stated constraints. This reduces hallucinated reasoning and improves trustworthiness in high-stakes applications like medical or engineering advice.

6. Complexity-Adaptive Training Curriculum

• Improvement: Use the paper's stratification finding (complex vs. simple problems) to design a training curriculum that exposes the AI to a balanced mix of both, with explicit labels for constraint load. Fine-tune with a loss function that weights rationale supervision more heavily on complex examples and less on simple ones.

• What it can do: The AI will learn to generalize better across problem difficulties—performing strongly on both simple queries (where it stays concise) and complex ones (where it reasons thoroughly)—without the performance degradation seen in the paper's uniformly rationale-supervised models.

Scientific papers contain fine-grained records of problem solving: authors mention technical obstacles and methods that were used to address them, often along with reasoning on why those methods were chosen. We introduce MUSE (Mining Underlying Scientific Explanations), a full-text, multi-domain resource of scientific Problem-Solution-Rationale (P-S-R) triplets. We curate 579 expert-annotated full-text paragraphs, with a rich annotation schema covering salient problem, solution, and rationale spans, solves and rationale of links and conceptual coreference. A modular extraction pipeline scales this annotation to build a high-quality knowledge base of 37K source-grounded P-S-R triplets. We evaluate the extraction components and include a preliminary experiment training a rationale-supervised LLM for scientific problem solving. Interestingly, we find that rationale supervision improves performance on complex, multi-constraint problems but can harm performance on simpler ones.