By: Michael Feldman

The European Commission is funding a project to build middleware that optimizes data movement for high performance computing  and high performance data analytics applications.

The thrust of the project, known as Maestro, will be to efficiently orchestrate the movement of data across the various levels of memory and storage found in HPC systems. Thanks to the growing gap between computational performance and memory/storage bandwidth, more and more HPC applications have become data-bound. That’s true even when the data hierarchy is expanded with things like flash-based storage and storage-class memory. The Maestro website encapsulates the problem as follows:

“Though HPC and HPDA applications pose a broad variety of efficiency challenges, it would be fair to say that the performance of both has become dominated by data movement through the memory and storage systems, as opposed to floating point computational capability. Despite this shift, current software technologies remain severely limited in their ability to optimise data movement.”

The problem has its roots from an earlier era when computation, especially numerical computation, was expensive. That resulted in a software stack that was designed to optimize floating-point operations, but largely ignored data management issues. One of the principle challenges in making applications data-aware is the need to know about the hardware attributes of the memory and storage hierarchy. Maestro’s solution is to create a middleware framework that is able to isolate the application software from these underlying details.

The Maestro consortium is made up of the Jülich Supercomputing Centre (JSC), Cray, the European Centre for Medium-Range Weather Forecasts (ECMWF), the French Alternative Energies and Atomic Energy Commission (CEA), the Swiss National Supercomputing Centre (CSCS), and Seagate Technologies, and Appentra Solutions. Each organization has a specific role:

• JSC will manage the project and coordinate the efforts undertaken by the other partners.
• Cray will provide technical coordination and will lead the core middleware design work.
• ECMWF will provide their production and research numerical weather prediction workflows as a use-case for the project and will co-design many aspects of the middleware.
• CEA will develop an in-situ framework based on the Maestro middleware.
• CSCS will develop a dynamic provisioning system targeting data-intensive applications.
• Seagate Technologies will develop an object storage backend to the middleware and interface Maestro to it.
• Appentra will enhance their Parallelware software for data awareness to support the Maestro middleware.

The consortium work is being funded by a three-year grant from the European Commission’s H2020 Future Enabling Technologies for HPC (FETHPC) program.