EXASCALE SKY SIMULATIONS FOR SQUARE KILOMETER ARRAY OBSERVATORY

PIs in SKACH and PASC SPH-EXA: 
Florina Ciorba (University of Basel)
Lucio Mayer (University of Zurich)
Rubén Cabezón (University of Basel) 

Project members:
Osman Seckin Simsek (University of Basel)
Lukas Schmidt (University of Basel)
Yiqing Zhu (University of Basel)

Collaborators:
Noah Kubli (University of Zurich)
Darren Reed (University of Zurich)
Sebastian Keller (ETH Zurich/CSCS)
Jean-Guillaume Piccinali (ETH Zurich/CSCS)
Jean M. Favre (ETH Zurich/CSCS)
Benjamin Cumming (ETH Zurich/CSCS)
John Biddiscombe (ETH Zurich/CSCS)

External collaborators:
José Escartin (European Southern Observatory)
Axel Sanz (Universitat Politècnica de Catalunya, Spain)
Joseph Touzet (Université Paris-Saclay)

Funding agency: The State Secretariat for Education, Research and Innovation (SERI) (https://www.sbfi.admin.ch/sbfi/en/home/seri/seri.html)

Duration: 01.09.2021-31.12.2024

Software: The SPH-EXA simulation framework is publicly available here.

Swiss SKA Consortium (SKACH):  https://skach.org

Project Summary

The Square Kilometre Array Observatory  (SKAO) is a next-generation radio astronomy facility, involving partners around the globe, that will lead to groundbreaking new insights in astrophysics and cosmology. Established on March 12, 2019 the SKAO is the second inter-governmental organisation dedicated to astronomy in the world. It will be operated over three sites: the Global Headquarters in the UK, the mid-frequency array in South Africa (SKA-mid), and the low-frequency array in Australia (SKA-low). 

The two telescopes under construction, SKA-Mid and SKA-Low, will combine the signals received from thousands of small antennae spread over a distance of several thousand kilometres to simulate a single giant radio telescope capable of extremely high sensitivity and angular resolution, using a technique called aperture synthesis. Some of the sub-arrays of the SKA will also have a very large field-of-view (FOV), making it possible to survey very large areas of the sky at once!

Switzerland has become the eighth country to join the intergovernmental nations that will collaborate in building the Square Kilometre Array Observatory (SKAO), to be built in Australia and South Africa. Swiss involvement is organized through a strong consortium of research institutions, called SKACH, including, Fachhochschule Nordwestschweiz (FHNW), Universität Zürich (UZH), Eidgenössische Technische Hochschule Zürich (ETHZ), École Polytechnique Fédérale de Lausanne (EPFL), Zürcher Hochschule für Angewandte Wissenschaften (ZHAW), Universität Basel (UniBas), Université de Genève (UniGE), Haute École spécialisée de Suisse Occidentale (HES-SO), Centro Svizzero di Calcolo Scientifico (CSCS).

The SKA telescopes will look at the history of the Universe as far back as the Cosmic Dawn, when the very first stars and galaxies formed. These key facilities will help Swiss scientists discover answers to the burning questions throughout several key topics in the field of astrophysics, including: dark energy, cosmic reionization, dark matter, galaxy evolution, cosmic magnetic fields, tests of gravity, solar physics, and others.

During its operation, the SKAO will collect unprecedented amounts of data, requiring the world’s fastest supercomputers to process this data in near real time. Swiss data scientists are working on complex Big Data algorithms enhanced by High-Performance Computing and  machine learning techniques to handle these large data streams.

As part of SKACH, the aim of our group is to extend the SPH-EXA simulation framework to include proper cosmological physics to reach trillion particle simulations on hybrid Tier-0 computing architectures. To this end we aim at coupling relevant physics modules with our SPH framework enabling the possibility of addressing both long-standing and cutting-edge problems via beyond state state-of-the-art simulations at extreme scales in the fields of Cosmology and Astrophysics. Such simulations include the formation, growth, and mergers of supermassive black holes in the early universe which would greatly impact the scientific community (for instance, the 2020 Nobel Prize in Physics has been awarded for pioneering research on super-massive black holes). Moreover, the ability to simulate planet formation with high-resolution models will play an important role in consolidating Switzerland’s position as a leader in experimental physics and observational astronomy. Additional targets will be related to explosive scenarios such as core-collapse and Type Ia supernovas, in which Switzerland has also maintained a long record of international renown. These simulations would be possible with a Tier-0-ready SPH code and would have a large impact on projects such as the current NCCR PlanetS funded by the SNF. 

Publications

O.S. Simsek, JG. Piccinali, F.M. Ciorba. “Increasing Energy Efficiency of Astrophysics Simulations Through GPU Frequency Scaling” In Proceedings of The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC24) Workshops Sustainable Supercomputing, November 2024. (to appear)

O.S. Simsek, JG. Piccinali, F.M. Ciorba. “Accurate Measurement of Application-level Energy Consumption for Energy-Aware Large-Scale Simulations.” In Proceedings of The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC23) Workshops Sustainable Supercomputing, November 2023. [C75.bib]

S. Keller, A. Cavelan, R. Cabezon, L. Mayer, F.M. Ciorba. “Cornerstone: Octree Construction Algorithms for Scalable Particle Simulations.” In Proceedings of the Platform for Advanced Scientific Computing Conference, June 2023.

W. Elwasif, W. Godoy, N. Hagerty, J. A. Harris, O. Hernandez, B. Joo, P. Kent, D. Lebrun-Grandie, E. Maccarthy, V. M. Vergara, B. Messer, R. Miller, S. Oral, S. Bastrakov, M. Bussmann, A. Debus, K., J. Stephan, R. Widera, S. Bryngelson, H. Le Berre, A. Radhakrishnan, J. Young, S. Chandrasekaran, F. M. Ciorba, O.S. Simsek, K. Clark, F. Spiga, J. Hammond, J. Stone, D. Hardy, S. Keller, J.G. Piccinali, and C. Trott. “Application Experiences on a GPU-Accelerated Arm-based HPC Testbed”. International Workshop on Arm-based HPC: Practice and Experience (IWAHPCE-2023), February 2023.

H. Brunst, S. Chandrasekaran, F. Ciorba, N. Hagerty, R. Henschel, G. Juckeland, J. Li, V. Vergara, S. Wienke, M. Zavala. “First Experiences in Performance Benchmarking with the New SPEChpc 2021 Suites”, 22nd IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), May 2022.

You may find previous related publications in publications from SPH-EXA (2017-2021).

Talks

Simsek, O. S.; “Sustainability of Extreme-Scale Simulations with SPH-EXA”, Talk in SKA days, 2024 September 2-4, Geneva, Switzerland.

Simsek, O. S.; “Increasing Energy Efficiency of SPH Codes Through Dynamic GPU Frequency Scaling“, Talk at the 18th International SPHERIC Workshop, June 18-20 2024, Berlin.

Ciorba, F. M.; “Sustainable and Scalable Simulations for Computational Sciences”, Invited panel talk at the Sibiu Innovation Days (SID 2023), 2023 October 5-6, Sibiu, Romania. (video)

Simsek, O. S. “Unlocking Data Locality in Imbalanced Supercomputers: Unveiling the Energy Perspective”, Talk in 6th Programming and Abstractions for Data Locality Workshop (PADAL), September 4-6, 2023, Istanbul, Turkey.

Simsek, O. S. “Accurately Measuring Energy Consumption of Large Cosmological Simulations”, Talk in PASC23 Mini-symposium on Green Computing Architectures and Tools for Scientific Computing, June 26-28, 2023, Davos, Switzerland.

Ciorba, F. M.; “SPH-EXA: A Framework for Scalable, Flexible, and Extensible Astrophysical and Cosmological Simulations”, Invited talk at the 35th Workshop on Sustained Simulation Performance Workshop, 2023 April 13-14, Stuttgart, Germany.

Ciorba, F. M.; “SPH-EXA: A Framework for Scalable, Flexible, and Extensible Astrophysical and Cosmological Simulations”, Talk in SIAM Conference on Computational Science and Engineering (CSE23), 2023 February, Amsterdam, Netherlands

Ciorba, F. M.; “Analysis of Load Imbalance in SPH-EXA Simulations”, Talk in SKACH Winter Meeting, 2023 January, Basel, Switzerland

Touzet, J.; “Multiphysics/Hydrodynamical Simulations with Attached and Detached Data: Simulation of Nuclear Networks in SPH-EXA”, Talk in SKACH Winter Meeting, 2023 January, Basel, Switzerland

Kubli, N.; “Fragmenting galactic disk simulations with SPH-EXA”, Talk in SKACH Winter Meeting, 2023 January, Basel, Switzerland

Cabezón, R.; “SPH-EXA: Bringing Computational Astrophysics to Exascale”, Talk in Universitat Politècnica de Catalunya, 2022 October, Barcelona, Spain.

Keller, S.; Simsek, O.; “SPH-EXA: A Framework for Smoothed Particle Hydrodynamics and Gravity at Exascale”, Talk in SKA days, 2022 October, Lugano, Switzerland.

You may also find previous talks from SPH-EXA (2017-2021).