Climate Impacts of Scientific Computing in Particle Physics

Snowmass 2021

From 2020 to 2022, the US particle physics community collectively worked through a community planning process, which is called Snowmass for historical reasons. This consisted of many small groups working together to write white papers on various topics, such as what kind of experiments should be built in the next 10 years, what the technological needs of the field are, and how to improve inclusion, diversity, equity, and accessibility in the field. The process culminated in a 10 day meeting at the University of Washington in Seattle (in years past this was at Snowmass, hence the name), where all of the white papers were distilled into a series of reports, which will themselves be distilled by a committee of community leaders and experts into a single report that will be shared with the US Congress and Department of Energy. This process has its own website, where you can find all the information you could possibly ever want on the state of US particle physics in 2022.

As a part of this process, I worked on a white paper that assessed the contributions of particle physics research to climate change, in particular looking at places where greenhouse gasses are directly or indirectly emitted. I authored a section on the ways in which scientific computing creates GHG emissions, which I’ll now share.

Climate Impacts of Scientific Computing in Particle Physics

High-performance computing (HPC) is an essential part of physics research. It is also a growing source of greenhouse gas emissions, primarily due to the large amount of electricity used by computation itself. Across all sectors, data centers and computing already contribute approximately 2-4% of global GHG emissions, and that fraction is only predicted to grow in the next 10 years. While the environmental costs of HPC are not unknown, they are often not prioritized, or even discussed, when planning and scheduling computational projects. We believe the particle physics community should improve its accounting of this issue. We provide several specific suggestions below, many of which are inspired by Lannelongue et al.

The best way to reduce the carbon emissions of high-performance computing is to ensure that compute facilities are powered by carbon-free energy sources. In the early planning stages of a compute facility, when siting decisions are being made, the carbon intensity of the electricity used to power the facility should be a major factor into the decision making process. In many places, renewable electricity is already cheaper than fossil-fueled electricity, making this a cost-effective decision as well. Local installation of renewable energy generating capacity can also reduce the carbon impact of both future and existing computing facilities, most easily with rooftop solar panels. Other options may also be available depending on the location of the compute center. These strategies have a direct and immediate impact on the carbon emissions associated with a compute center, and do not rely on the cooperation of users as several of our other suggestions do, making them likely more effective.

Reductions in GHG emissions can also be achieved on the demand side with current technology, through a combination of optimization of resource use and careful planning of timing and siting of computation. These suggestions give individual users tools to directly reduce their own carbon emissions impact. However, it is often not obvious how to successfully follow them, since the information required is unavailable or difficult to access. The website green-algorithms.org can provide a useful estimate of the cost of running a particular algorithm, but it requires knowledge of the specific hardware a computation will be using, and also the specific energy mix the compute center uses, to calculate the most accurate emissions estimate. This information could be provided by compute centers, and while the former frequently is, the latter often is not. If they aren’t already doing so, compute centers should provide information on the GHG intensity of their computation, ideally in a standardized format to facilitate easy comparisons. This information could be integrated into the green-algorithms website, or else made available in a standalone website, and statistics from cloud computing services could also be included if that information is available.

Going a step further, compute centers, and the scientific collaborations that make use of them, could even provide a simple tool to estimate the GHG emissions impact of a particular job request, perhaps as a simple command-line utility. A precise estimate would be difficult to achieve, but with knowledge of the hardware, energy mix, and particulars of the job request, a reasonably accurate range of possible GHG emissions should be possible to produce, likely ranging from zero to full resource utilization. Compute centers could then report their CO2e intensity to a common location, ideally a centralized website of some sort. A version of this already exists as the Green500, an offshoot of the better-known TOP500. However, the Green500 compares the power efficiency of various systems, which is related to, but not the same as, the CO2e intensity. For example, while HiPerGator AI at the University of Florida has a slightly better power efficiency rating than Perlmutter at the National Energy Research Supercomputing Center (NERSC, Berkeley, CA), 80.3% of electricity consumption in Florida in 2019 was fossil fueled while only 35.9% California’s electricity consumption over the same period was fossil fueled, meaning that the per-FLOP CO2e emissions impact of Perlmutter is likely less than that of HiPerGator AI.¹

While obtaining better information about compute center GHG emissions can be very useful for long-term planning, such as an experiment deciding where to do the bulk of their computing, it is less useful for individual users who often do not have the ability to choose where to do their large-scale computing. Users do usually have the ability to improve the efficiency of their code, which can lead to substantial improvements in emissions if done properly. However, frequently users are writing code that makes use of large libraries, such as Geant4 or ROOT, that they do not have control over, which can limit potential improvements from optimization. Developers of these libraries should be sure to provide information on how to minimize GHG emissions, such as optimal hardware, and scaling of memory utilization. Tracking the GHG emissions efficiency over software releases with some kind of standard benchmark would provide an incentive to minimize emissions.

Even if one makes optimal choices to minimize the GHG emissions of their computation, some emissions will still inherently result from the electricity used. Even if the electricity is produced from renewable sources, the life cycle emissions of those power sources, and also of the compute center itself, are nonzero. Purchasing carbon offsets, while never a true substitute for emissions reductions, can help to mitigate the remaining unavoidable emissions. Incentivizing the purchasing of carbon offsets for unavoidable emissions is something funding agencies could do. Some Department of Energy compute centers, such as NERSC, use an allocation-and-charging framework for distributing computational resources. Carbon offsets could be folded into the cost of batch jobs by the compute center, drawn directly from the user’s allocation. This would incentivize experiments to choose greener compute centers, and users to ensure that their code runs as efficiently as possible. At minimum, compute centers could provide links to legitimate offset companies, or other agencies that rate and verify them. Finally, compute centers could coordinate their power loads with their electricity suppliers, scheduling more jobs when electricity is cheap and relatively clean (e.g. midday or nighttime), and reducing their loads when electricity is expensive and relatively dirty (e.g. late afternoon through the early evening). For the greatest impact, this would require real-time information from the electricity supplier, but if the compute center is a sufficiently large electricity customer, the supplier would likely be happy to provide this information since it would improve their demand response. Ahmed et al. have completed a study of this possibility.

1. This is only an example to illustrate the concept; obviously the exact energy mix for each compute center is likely not identical to the statewide average. ↩