14 institutions from six European countries are developing a new adaptive storage system for high-performance computing as part of the ADMIRE project. Her goal is to significantly improve the runtime of applications in areas such as weather forecasting, molecular dynamics, turbulence modeling, cartography, brain research and software cataloging. The project, coordinated by the Carlos III University of Madrid (UC3M), is funded by the European High-Performance Computing Joint Undertaking (EuroHPC JU) and the participating countries.
The processing of huge amounts of data, as is often required for artificial intelligence, is one of the driving forces behind the concept of high-performance computing – a trend that challenges traditional architecture, which mainly focuses on computationally intensive tasks. Your flat storage hierarchies with a central parallel file system are increasingly proving to be inadequate. New technologies that provide fast, non-volatile and, at the same time, energy-efficient mass storage, open up opportunities to meet the requirements of data-intensive applications. However, there is currently a lack of adequate control mechanisms for the available resources and specially tailored file systems to realize this potential. Solving this challenge is the main objective of the European ADMIRE project.
The ADMIRE software stack and applications
Four German institutions are significantly involved in the project and will be working on various components of the overall ADMIRE system over the next three years.
* Johannes Gutenberg University Mainz (JGU):
Researchers at JGU are working on so-called ad-hoc storage systems, which can be controlled by the overall ADMIRE system if necessary. With ad-hoc storage systems, the load on the central parallel file system can be reduced and, at the same time, significantly higher data and metadata throughputs possible. Therefore, these file systems can help meet the I / O needs of a wide variety of applications. At the center of the JGU contribution is the GekkoFS file system, which was developed in previous projects. In the ADMIRE project, GekkoFS is being expanded to meet the requirements of modern high-performance computing applications, including with regard to their semantic and consistency requirements and their I / O access patterns. This enables the file system to offer the best possible I / O performance and to react dynamically to decisions made by the ADMIRE system.
* Darmstadt University of Technology:
TU Darmstadt develops algorithms and tools to adapt the resource consumption of a program during runtime so that both the runtime of individual programs and the computing power of the overall system are optimized. Extra-P, a performance modeling tool created by the TU Darmstadt in previous projects, is being expanded as part of ADMIRE to include a functionality for modeling I / O.
* Research center Julich:
The Jülich Supercomputing Center (JSC) at Forschungszentrum Jülich generates scalable and efficient processing workflows for Earth Observation (EO) applications based on machine learning and deep learning methods. As its contribution to ADMIRE, the JSC will optimize the I / O performance of entire processing pipelines, which will then be able to automatically update existing land use maps by analyzing a series of multispectral satellite images captured in real time. The new ADMIRE software stack will accelerate the access to and analysis of data in the processing stages from data acquisition to the end product. The aim of the JSC is to provide efficient EO processing workflows that can process huge amounts of remote sensor data from multiple sources in operational scenarios to provide decision makers with clear, up-to-date and useful information.
* The Max Planck Computing and Data Facility (MPCDF):
The MPCDF is working on developing in-situ methods with which data can be processed, e.g. compressed, and analyzed during ongoing simulations. As a result, many tasks that are usually only carried out after a simulation has been completed and the data has been saved in the file system can already be taken over for active data in the main memory, which means that the amount of data that ultimately arises is significantly reduced. In-situ methods were originally developed for visualization applications, but will be adapted for a more general application in the course of the ADMIRE project in order to facilitate the use of artificial intelligence and other data analysis techniques.
ADMIRE is an EU funded project. It was launched on April 1, 2021 and has a budget of 7.9 million euros over three years. Coordinated by the Carlos III University of Madrid (Spain), the multidisciplinary collaborative research team consists of members of the Barcelona Supercomputing Center (Spain), the Johannes Gutenberg University Mainz (Germany), the Technical University of Darmstadt (Germany), the Max Planck Society ( Germany), Forschungszentrum Jülich GmbH (Germany), DataDirect Networks (France), Paratools (France), Institut national de recherche en sciences et technologies du numérique (France), Consorzio Interuniversitario Nazionale per l’Informatica (Italy), CINECA Consorzio Interuniversitario ( Italy), E4 Computer Engineering SpA (Italy), Instytut Chemii Bioorganicznej Polskiej Akademii Nauk (Poland) and Kungliga Tekniska Högskolan (Sweden).
The ADMIRE project was funded by the European High-Performance Computing Joint Undertaking (JU) under grant agreement No. 956748. The JU is supported by the research and innovation program Horizon 2020 of the European Union and the participating countries Spain, Germany, France, Italy, Poland and Sweden.
Related Link: https: /
Disclaimer: AAAS and EurekAlert! are not responsible for the accuracy of press releases sent to EurekAlert! by contributing institutions or for the use of information via the EurekAlert system.