In order to support the large computation effort to study black hole accretion physics and explain EHT observations, PIRE submitted a Large-scale Community Partnerships (LSCP) proposal to the Texas Advanced Computing Center (TACC) for a computation and storage allocation on the supercomputer, Frontera. The proposal was successful and PIRE has been awarded 168 million core hours of computing time on Frontera and its Longhorn subsystem, and 850 TB of storage for a three-year period.
At a cost of $60 million, Frontera is the 5th most powerful supercomputer in the world and the fastest supercomputer on a university campus. Its primary computing system consists of 8008 compute nodes interconnected with InfiniBand. With 56 core, 192GB of memory, and 480 GB of local SSD per node, it provides half-million CPU cores and 1.5 PB of memory on the main system. Frontera also has additional GPU-accelerated subsystems. Among them, PIRE is given access to the Longhorn subsystem, which is powered by IBM POWER-9 CPUs and 448 NVIDIA V100 GPUs in total and has a peak performance at 3.5 PetaFLOPS. Finally, the recently updated Ranch storage system consists of a 30PB front-end disk system and a back-end tape library.
The awarded allocation would, in principle, allow PIRE to use the full Frontera system for two weeks, a value equivalent to more than $2 million. In practice, we will spread the allocation of this 3-year partnership to support the EHT to achieve its science objectives by carrying out state-of-the-art numerical simulations; building, maintaining, and releasing the most complete black hole accretion simulation libraries; enabling EHT researchers to model, interpret, and understand its current and future observation results; and advancing the forefront of black hole astrophysics research in the U.S. A uniform set of high-resolution GRMHD simulations are currently running on Frontera. Planned to run to 30,000 gravitational time scales by mid-October, these simulations will provide steady-state solutions even for the magnetically arrested disk (MAD) state, and become a key data set for EHT’s Sgr A* theory paper. Altogether, this allocation provides for substantial computational resources that will enable the EHT to place tighter constraints on Einstein’s general theory of relativity.