LLM4Mat-Bench: Benchmarking Large Language Models for Materials Property Prediction

LLM4Mat-Bench is the largest benchmark to date for evaluating the performance of large language models (LLMs) in predicting the properties of crystalline materials. It contains about 1.9M crystal structures in total, collected from 10 publicly available materials data sources, and 45 distinct properties. LLM4Mat-Bench features different input modalities: crystal composition, CIF, and crystal text description, with 4.7M, 615.5M, and 3.1B tokens in total for each modality, respectively. The benchmark is used to fine-tune models with different sizes, including LLM-Prop and MatBERT, and to provide zero-shot and few-shot prompts to evaluate the property prediction capabilities of LLM-chat-like models, including Llama, Gemma, and Mistral. The results highlight the challenges of general-purpose LLMs in materials science and the need for task-specific predictive models and task-specific instruction-tuned LLMs in materials property prediction.

The dataset comprises approximately two million samples, sourced from ten publicly available materials sources, each containing between 10K and 1M structure samples. LLM4Mat-Bench encompasses several tasks, including the prediction of electronic, elastic, and thermodynamic properties based on a material’s composition, crystal information file (CIF), or textual description of its structure. The data sources include hMOF, Materials Project, OQMD, OMDB, JARVIS-DFT, QMOF, JARVIS-QETB, GNoME, Cantor HEA, and SNUMAT. Crystal structure files (CIFs), material compositions, and material properties were collected from these publicly accessible sources, facilitated by APIs and direct download links. For databases such as Materials Project, OMDB, SNUMAT, JARVIS-DFT, and JARVIS-QETB, user registration is required for access, while databases like hMOF, QMOF, OQMD, and GNoME allow direct data access without registration.

Crystal structures are typically described in file formats such as Crystallographic Information File (CIF) which include predominantly numbers describing lattice vectors and atomic coordinates and are less amenable to LLMs. Instead of directly using these as inputs, the authors use Robocrystallographer to deterministically generate texts that are more descriptive of crystal structures from CIF files. Robocrystallographer leverages predefined rules and existing libraries to extract chemical and structural information, including oxidation states, global structural descriptions (symmetry information, prototype matching, structural fingerprint calculations etc.), and local structural descriptions (e.g. bonding and neighbor analysis, connectivity). This method not only generates deterministic and human-readable texts, but also ensures no data contamination in the fine-tuned LLMs, as the data sources mentioned do not include these crystal text descriptions.

The total samples for each dataset in LLM4Mat-Bench are randomly split into 80%, 10%, and 10% for training, validation, and testing, respectively. OQMD has the highest number of samples at 964,403, while QMOF has the fewest with 7,656 samples. On average, each dataset in LLM4Mat-Bench contains approximately 200,000 samples. In LLM4Mat-Bench, when combined, textual descriptions contain 3.1 billion tokens, crystal structures 615 million, and compositions 4.7 million. OQMD leads in composition tokens (964K), while hMOF has the most description tokens (581M). For CIFs, both OQMD and hMOF have around 96M tokens. On average, compositions have 8 subword tokens per sample, CIFs 1600, and descriptions 1700. hMOF averages the longest inputs for compositions (14.9) and descriptions (5629), while QMOF leads in structures (5876.4). JARVIS-DFT has the most tasks with 20 properties, followed by Materials Project with 10, and OMDB with one.

The authors conducted about 1,235 experiments, evaluating the performance of five models and three material representations on each property for each data source. Consistent with standard practices in materials science, they evaluated performance separately for each data source rather than combining samples from different sources for the same property. This approach accounts for variations in techniques and settings used by different data sources, which can result in discrepancies, such as differing band gaps for the same material. The models benchmarked include CGCNN (a GNN baseline), MatBERT (a BERT-base model with 109 million parameters, pretrained on two million materials science articles), LLM-Prop (a model based on the encoder part of T5-small model with 35 million parameters), and Llama, Gemma, and Mistral variants (3B, 7B, 8B, and 9B parameters).

The main observations from the results are as follows: Small, task-specific predictive LLMs exhibit significantly better performance than larger, generative general-purpose LLMs. This performance disparity is evident across both regression and classification tasks on all 10 datasets. Specifically, LLM-Prop and MatBERT outperform conversational LLMs by a substantial margin, despite being approximately 200 and 64 times smaller in size, respectively. In regression tasks, LLM-Prop achieves the highest accuracy on 8 out of 10 datasets, with MatBERT leading on the remaining 2 datasets. For classification tasks, both LLM-Prop and MatBERT deliver the best performance on 1 out of 2 datasets. LLM-Prop surpasses MatBERT by 1.8% on the SNUMAT dataset, whereas MatBERT outperforms LLM-Prop by 0.8% on the other dataset.

General-purpose generative LLMs hallucinate and often fail to generate valid property values. As shown in the results, Llama, Gemma, and Mistral models produce invalid outputs on multiple tasks, where the expected property value is missing. This issue occurs less frequently when the input is a description or chemical formula, but more commonly when the input is a CIF file. One reason may be that descriptions and chemical formulas resemble natural language, which LLMs can more easily interpret compared to CIF files. This may also indicate that when the input modality during inference differs significantly from the modalities encountered during pretraining, fine-tuning is necessary to achieve reasonable performance. Another key observation is that these models often generate the same property value for different inputs (i.e. hallucinate), contributing to their poor performance across multiple tasks.

Representing materials with their textual descriptions improves the performance of LLM-based property predictors compared to other representations. A significant performance improvement is observed when the input is a description compared to when it is a CIF file or a chemical formula. One of the possible reasons for this might be that LLMs are more adept at learning from natural language data. On the other hand, although material compositions appear more natural to LLMs compared to CIF files, they lack sufficient structural information. This is likely why LLMs with CIF files as input significantly outperform those using chemical formulas.

More advanced, general-purpose generative LLMs do not necessarily yield better results in predicting material properties. The results indicate that, despite being trained on substantially larger and higher-quality datasets, more advanced versions of generative LLMs show limited improvements in performance and validity of predictions for material properties. For instance, Llama 3 and 3.1 8b models were trained on over 15 trillion tokens—around eight times more data than the 2 trillion tokens used for the Llama 2 7b models. This finding highlights the ongoing challenges of leveraging LLMs in material property prediction and underscores the need for further research to harness the potential of these robust models in this domain.

The performance on energetic properties is consistently better across all datasets compared to other properties. This is consistent with the trend observed in the community benchmarks such as MatBench and JARVIS-Leaderboard, where energetic properties are among those that can be most accurately predicted. This is not surprising because energy is known to be relatively well predicted from e.g., compositions and atom coordination (bonding), which is inherently represented in GNNs and also presented in text descriptions.

Task-specific predictive LLM-based models excel with shorter textual descriptions, while CGCNN performs better on datasets with longer descriptions. For regression tasks, LLM-Prop outperforms CGCNN on only 4 out of 10 datasets (MP, JARVIS-DFT, JARVIS-QETB, and SNUMAT), and MatBERT outperforms CGCNN on just 2 out of 10 datasets (MP and JARVIS-QETB). In contrast, CGCNN achieves the best performance on 5 out of 10 datasets (GNoME, hMOF, Cantor HEA, OQMD, and OMDB). Further analysis reveals that CGCNN tends to perform better than LLM-based models on datasets with relatively longer textual descriptions, while LLM-based models excel on datasets with shorter descriptions. The performance gain on shorter descriptions may stem from LLM-based models’ ability to leverage more context from compact text, while CGCNN consistently benefits from training on the entire crystal structure.

The authors conclude that LLMs are increasingly being utilized in materials science, particularly for materials property prediction and discovery. However, the absence of standardized evaluation benchmarks has impeded progress in this field. They introduced LLM4Mat-Bench, a comprehensive benchmark dataset designed to evaluate LLMs for predicting properties of atomic and molecular crystals and MOFs. Their results demonstrate the limitations of general-purpose LLMs in this domain and underscore the necessity for task-specific predictive models and instruction-tuned LLMs tailored for materials property prediction. These findings emphasize the importance of using LLM4Mat-Bench to advance the development of more effective LLMs in materials science.

Due to computational constraints and the number of experiments, the authors were unable to conduct thorough hyperparameter searches for each property and dataset. The reported settings were optimized on the MP dataset and then fixed for other datasets. Additionally, they could not include results from SOTA commercial LLMs such as GPT-4o or Claude 3.5 Sonnet due to budget constraints. They also encountered issues with chat-based models, which sometimes failed to follow the output format, producing invalid or incomplete outputs. Extracting property values was therefore challenging. Furthermore, they did not include comparisons with dataset-specific retrieval-augmented generation (RAG) models, such as the recently developed LLaMP, a RAG-based model tailored for interaction with the MP dataset.

Based on the paper, here are the specific improvements I can implement in AI systems, along with the resulting capabilities:

1. Improved AI System: Task-Specific Predictive LLM for Materials Property Prediction

• Improvements:

• Architecture: Use a small, encoder-only transformer (e.g., T5-small, 35M parameters) instead of large generative models. This is based on the finding that LLM-Prop and MatBERT significantly outperform larger chat models.

• Input Representation: Accept and process three distinct input modalities: chemical composition, CIF structure, and text description generated by Robocrystallographer. The system should prioritize text descriptions as they yield the best performance.

• Training: Fine-tune the model on the LLM4Mat-Bench dataset. Use the provided train/validation/test splits to ensure reproducibility. For CIF inputs, implement xVal encoding to handle numerical values efficiently.

• Evaluation: Use the MAD:MAE ratio for regression tasks and AUC for classification tasks, as defined in the paper.

• Capabilities:

• Accurate Property Prediction: Predict 45 distinct properties (e.g., band gap, formation energy, elastic moduli) with high accuracy, outperforming general-purpose LLMs by a significant margin (e.g., achieving MAD:MAE ratios above 5.0, which is considered good).

• Multi-Modal Input: Can predict properties from a simple chemical formula (e.g., NaCl), a full CIF file, or a natural language description of the crystal structure.

• Efficient and Fast: Due to its small size, it can be trained and deployed much faster than large models (e.g., 2.5 days for 300K data points vs. 0.5 days for inference on 40K samples for a 7B model).

• Reliable Output: It will not hallucinate or produce invalid outputs, a common failure mode of generative models, especially when processing CIF files.

2. Improved AI System: Reliable Property Extraction from Generative LLMs

3. Improved AI System: Benchmarking and Model Selection Tool

Large language models (LLMs) are increasingly being used in materials science. However, little attention has been given to benchmarking and standardized evaluation for LLM-based materials property prediction, which hinders progress. We present LLM4Mat-Bench, the largest benchmark to date for evaluating the performance of LLMs in predicting the properties of crystalline materials. LLM4Mat-Bench contains about 1.9M crystal structures in total, collected from 10 publicly available materials data sources, and 45 distinct properties. LLM4Mat-Bench features different input modalities: crystal composition, CIF, and crystal text description, with 4.7M, 615.5M, and 3.1B tokens in total for each modality, respectively. We use LLM4Mat-Bench to fine-tune models with different sizes, including LLM-Prop and MatBERT, and provide zero-shot and few-shot prompts to evaluate the property prediction capabilities of LLM-chat-like models, including Llama, Gemma, and Mistral. The results highlight the challenges of general-purpose LLMs in materials science and the need for task-specific predictive models and task-specific instruction-tuned LLMs in materials property prediction.