A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model

In an era where large language models are becoming ubiquitous, it is crucial to consider their environmental impact due to their extreme size and resource use. This work proposes a holistic assessment of the total carbon footprint of Noor, an extremescale Arabic language model project, by evaluating costs from data collection through future inference estimates.

The gist

We evaluate the total carbon bill of the entire project: starting with data collection and storage costs, including research and development budgets, pretraining costs, future serving estimates, and other exogenous impacts sparked by this international cooperation.

Holistic Assessment Scope

The assessment goes beyond traditional focus on pretraining costs to capture the entire project lifecycle. The evaluation includes:

1. Data collection, curation, and storage costs.

2. Research and development budgets and hyper-parameters tuning budgets.

3. Pretraining costs for the four models (1.5B, 2.7B, 6.7B, and 13B parameters).

4. Future serving estimates for inference costs as the models are put into use in the wild.

5. Other exogenous impacts necessary for this international cooperation (e.g., flights, personnel).

Key Findings on Emissions Drivers

The study identifies several key factors influencing the carbon footprint:

We identify pretraining compute as driving more than half of the emissions of the project.

However, other components still contribute significantly:

"all combined, other R&D, storage, and personnel counts still amount for 35% of the carbon footprint."

The paper notes that in scenarios with a low-impact training electric mix, costs beyond pretraining may become the main sources of emissions.

Factors Influencing Carbon Footprint

Several factors directly related to the models and their deployment are analyzed:

1. Model size: The compute budget is approximated by the formula, C = 6ND (Kaplan et al., 2020), and compute budget will scale more or less linearly with model size.

2. Hardware characteristics: Throughput per Watt is a critical metric, noting that More efficient hardware will have more throughput per Watt.

3. Modelling decisions: This includes the number of tokens processed (for training or inference) and hardware throughput, which are impacted by decisions like choosing a more fertile tokenizer or implementing specific parallelism techniques.

4. Data center efficiency: The Power Usage Effectiveness (PUE) is used to assess overall efficiency, with a worldwide average around 1.8, though some providers report lower figures like Google's 1.11.

5. Electricity mix: This is a crucial factor as it determines the carbon emissions per kWh of electricity.

Carbon Footprint Breakdown

The total estimated electricity consumption for the Noor project is approximately 59.14 MWh, with data preprocessing accounting for 20% of it. The breakdown shows:

Nearly a third of the energy consumed (30%) went to tasks outside of main models pretraining.

The carbon footprint distribution is highly dependent on location and electricity mix, as shown in Table 2. For instance, the training on the Noor-HPC in the UAE resulted in a footprint of 23.7 tCO2e. The study also accounts for exogenous costs like international flights, which account for 18% of the total carbon emission of the whole project.

Pathways to Reduction

The paper suggests several recommendations to lower future footprints:

Efficient architectures,

such as Mixture-of-experts (MoE) models, which can bring significant energy savings during training and inference.

Quantization (Yang et al., 2019)

to reduce numerical precision at inference time and accelerate processing.

Distillation (i.e., training a smaller model from the outputs of a larger one)

as a promising direction for making models leaner for inference.

Additionally, focusing on hardware choices is advised: selecting data centers with a PUE of 1.1 and choosing accelerators like T4s over A100s for smaller models can reduce energy consumption. Finally, the paper recommends that the AI community start reporting the full energy consumption and the CO2e of their projects.

Conclusion

The development of the suite of four Noor models is estimated to have emitted 36.5 tons of CO2, with 65% attributed to training. The main driver remains the carbon intensity of the mix used for model training, emphasizing that Appropriately selecting the location of calculations can significantly reduce the environmental impact. Furthermore, it is stressed that large-scale inference could also rapidly outtake pretraining costs in terms of carbon impact.

Based on the provided scientific paper, here are specific improvements for AI systems, categorized by modeling, infrastructure, and operational practices:

)Efficient Architectures (Modelling & Engineering):

1. Implement Mixture-of-Experts (MoE) models for future large language models (LLMs). This will allow for significant energy savings during both training and inference by sparsely activating only the relevant experts.

2. Focus on developing more lean architectures specifically optimized for inference, potentially through techniques like model distillation (training a smaller, efficient model from the outputs of a larger one) to reduce token generation requirements and computational load during deployment.

3. Invest in research into better embeddings and activation functions that have a non-negligible impact on the overall carbon footprint of models.

)Efficient Implementations (Modelling & Engineering):

1. Develop highly optimized distributed training implementations that minimize idle hardware consumption, potentially by incorporating fine-grained effects such as wave and tile quantization to maximize throughput on existing hardware.

2. Prioritize efficient scaling techniques, such as switch transformers using simple and efficient sparsity, for scaling models to trillion-parameter sizes while maintaining efficiency gains over dense models.

)Hardware Selection (Infrastructure):

1. When selecting data centers for training, mandate the choice of platforms with a Power Usage Effectiveness (PUE) significantly lower than the world average (e.g., PUE of 1.1 or better).

2. For inference workloads involving smaller models (<3B parameters), strategically utilize more energy-efficient accelerators, such as T4s instead of A100s, leveraging their superior energy efficiency per FLOP when appropriate for the task.

3. Prioritize locating major training operations in regions with a low-carbon electricity mix (e.g., France's mix reported in the study) to drastically reduce the carbon intensity factor of the total footprint.

)Operational Practices (Exogenous Costs & Lifecycle):

1. Mandate comprehensive, end-to-end carbon cost reporting for all large model development projects, including data storage, R&D budgets, pretraining costs, and future serving/inference estimates. This allows for accurate carbon accounting and potential emissions offsetting.

2. Establish rigorous protocols to minimize the environmental impact of international personnel travel (e.g., limiting flights to essential workshops) as these exogenous costs can represent a significant percentage of the total footprint (up to 18% in the Noor project example).

3. Implement best practices for inference deployment by advising end-users and platforms on efficient inference strategies, such as selecting low-impact cloud regions for serving and choosing hardware optimized for the model size.

This paper suggests that future AI systems should be designed with a holistic lifecycle perspective, moving beyond just pretraining compute to include data sourcing, infrastructure choices (PUE), regional electricity mix, and operational efficiency (quantization/MoE) to achieve a significantly reduced total carbon footprint.

As ever larger language models grow more ubiquitous, it is crucial to consider their environmental impact. Characterised by extreme size and resource use, recent generations of models have been criticised for their voracious appetite for compute, and thus significant carbon footprint. Although reporting of carbon impact has grown more common in machine learning papers, this reporting is usually limited to compute resources used strictly for training. In this work, we propose a holistic assessment of the footprint of an extreme-scale language model, Noor. Noor is an ongoing project aiming to develop the largest multi-task Arabic language models -- with up to 13B parameters -- leveraging zero-shot generalisation to enable a wide range of downstream tasks via natural language instructions. We assess the total carbon bill of the entire project: starting with data collection and storage costs, including research and development budgets, pretraining costs, future serving estimates, and other exogenous costs necessary for this international cooperation. Notably, we find that inference costs and exogenous factors can have a significant impact on total budget. Finally, we discuss pathways to reduce the carbon footprint of extreme-scale models.