The following energy industry HPC codes are being used and optimised by ENERXICO:

WRF (Weather Research and Forecasting Model) 

DEVELOPER(S) NCAR (National Center for Atmospheric Research), The National Centers for Environmental Prediction (NCEP), Forecast Systems Laboratory (FSL), Air Force Weather Agency (AFWA), Naval Research Laboratory, Oklahoma University, Federal Aviation Administration (FAA)
LINK https://github.com/wrf-model/WRF/releases
SHORT DESCRIPTION WRF is a mesoscale numerical weather prediction system designed for both atmospheric research and operational forecasting applications. It features two dynamical cores, a data assimilation system, and a software architecture supporting parallel computation and system extensibility. The model serves a wide range of meteorological applications across scales from tens of meters to thousands of kilometres
  • WRF produces simulations based on actual atmospheric conditions (i.e., from observations and analyses) or idealized conditions

  • WRF offers operational forecasting of atmospheric conditions

PERFORMANCE RESULTS Permadi et al. WRF Performance Analysis and Scalability on Multicore High Performance Computing Systems. https://doi.org/10.1002/9781119720492.ch18


BSIT (Barcelona Subsurface Imaging Tools) 


Currently at BSC: Mauricio Hanzich, Josep de la Puente, Juan E. Rodríguez and Albert Farrés

No longer at BSC: Jean Kormann, Natalia Gutierrez, Miguel Ferrer, Samuel Rodríguez, Claudio Márquez, Vladimir Puzyrev, Raúl de la Cruz, Feliz Rubio and Genís Aguilar

LINK https://github.com/albertfc/FWI_enerxico
SHORT DESCRIPTION BSIT is a production geophysical imaging application with the capacity to run on extremely large systems implementing Full Waveform Inversion. FWI is a cutting-edge technique that aims to acquire the physical properties of the subsoil from a set of seismic measurements. Starting from a guess (initial model) of the variables being inverted (e.g., sound transmission velocity), the stimulus introduced and the recorded signals, Full Waveform Inversion performs several phases of iterative computations to reach the real value of the set of variables being inverted with an acceptable error threshold

Main results:


  • Hanzich, M., Rodriguez, J., Gutierrez, N., de la Puente, J., & Cela, J. (2014). Using HPC software frameworks for developing BSIT: a geophysical imaging tool. Proceedings of WCCM XI, ECCM V, ECFD VI, 3, 2019-2030

HPC evaluations and improvements:



DEVELOPER(S) Guillaume Houzeaux, Mariano Vazquez, Jose Maria Cela, Ricard Borrell, Herbert Owen, Oriol Lehmkuhl, Matias Avila
LINK https://gitlab.bsc.es/alya/alya/-/wikis/home
SHORT DESCRIPTION Alya is a code developed by the Barcelona Supercomputing Center (BSC). It is a parallel multi-physics CFD code of the PRACE Benchmark Suite for HPC. The numerical discretization is based on a second-order spatial low-dissipation finite element scheme with an explicit temporal third-order Runge-Kutta method for momentum and scalar transport. Alya has been designed to run on leading HPC systems and includes both the MPI and OpenMP models to take advantage of the distributed and the shared memory paradigms. Accelerators like GPUs are also exploited to further enhance the performance of the code

Co-design activities: 

  • Coupling of microscale simulations using Alya with WRF outputs of atmospheric simulations


Implementation of Mesoscale to Microscale coupling using RANS and LES turbulence models.

Reference: Rodrigo et al. The ALEX17 diurnal cycles in complex terrain benchmark. doi:10.1088/1742-6596/1934/1/012002


Mariano Vázquez et al. Alya: Multiphysics engineering simulation toward exascale. JCS 14, 15-27.2016. https://doi.org/10.1016/j.jocs.2015.12.007.


ExaHyPE (An Exascale Hyperbolic PDE Engine) 

DEVELOPER(S) ExaHyPE consortium, see https://exahype.eu/consortium
Developers in ENERXICO: Jean-Matthieu Gallard (TUM), Anne Reinarz (Durham University, formerly at TUM). Contact person: Michael Bader
LINK http://www.exahype.org/
SHORT DESCRIPTION ExaHyPE is an engine for solving systems of first-order hyperbolic partial differential equations. Due to the robustness and shock capturing abilities of ExaHyPE’s numerical methods, both linear and non-linear hyperbolic PDEs can be simulated with very high accuracy.

Co-design activities: 

Optimisation of backends for small tensor and matrix operations on various CPU architectures (Intel, AMD, ARM)


Implementation of UQ-based scenarios based on the MIT UQ library.


L. Seelinger, A. Reinarz, L. Rannabauer, M. Bader, P. Bastian, R. Scheichl: High Performance Uncertainty Quantification with Parallelized Multilevel Markov Chain Monte Carlo, Accepted for SC21 Supercomputing Conference.


Algorithmic and HPC improvements:

  • Improved vectorization due to algorithmic modification of the ADER-DG scheme and change of memory layout (“Array of Struct of Array”)
  • Analysed energy consumption of optimisations on various CPU architectures
  • Evaluation of mixed precision in ADER-DG’s space-time-predictor kernel


Black Hole code (extension of the DualSPHysics code)

DEVELOPER(S) Jaime Klapp (ININ, México), Leonardo Di G. Sigalotti (UAM-A, México), Moncho Gómez Gesteira and José Manuel Domínguez Alonso (Universidad de Vigo, Spain)
LINK https://github.com/DualSPHysics/DualSPHysics/tree/develop
SHORT DESCRIPTION The Black Hole code is a multiphase extension of the DualSPHysics code, it is the first SPH based code for the numerical simulation of oil reservoirs and which has also important benefits versus commercial codes based on other numerical techniques
MAIN RESULTS AND REFERENCES The code is able to perform efficient numerical simulations of fractured oil reservoirs and includes a fluid characterization package

The esults for 1 GPU and multi-GPU (up to 32 GPUs) shows a high level of parallelization (97%), see:

D1.1 Report on efficiency assessment of HPC codes in  intra and multi-node 

D1.2 Report on intra-node and multi-node optimizations for HPC codes

D1.3 Report on enabling computational and energy efficient codes for the Exascale



DEVELOPER(S) Romain Brossier (Univ. Grenoble Alpes), Jian Cao (Univ. Grenoble Alpes), Phuong Thu Trinh (Univ. Grenoble Alpes)
LINK Contact: romain.brossier@univ-grenoble-alpes.fr
SHORT DESCRIPTION SEM46 is a seismic full waveform modelling and inversion code developed at Universite Grenoble Alpes. It aims to provide high resolution 3D models of the subsurface physical properties, from active seismic recordings. Such models can then be interpreted as geological formations, rock types, reservoirs and/or fluid content, in order to evaluate resource quantities and monitor production
MAIN RESULTS AND REFERENCES The code has been extended to fluid-solid coupled media and various types of anisotropy in the solid part (isotropic, VTI, Orthorhombic, Triclinic), for both the modeling and the inversion modes

Various optimizations (D1.3) leading to 35% (total gain) improvement on a realistic and large test-case:

  • Improvement of vectorization over multiple elements

  • Reduction of memory bandwidth by exploiting the stiffness properties depending on the anisotropy choice

  • Use of fixed-point coding for some elements of the  stiffness  table

  • Recombination of global field update routine




SeisSol Team, see https://seissol.github.io/team/

In ENERXICO: Sebastian Wolf (TUM).

Contact persons: Michael Bader and  Alice-Agnes Gabriel

LINK http://www.seissol.org/

SeisSol is a software package for earthquake simulation of highly complex scenarios. Characteristics of the SeisSol simulation software are:

  • high-order discontinuous Galerkin discretisation with ADER time stepping (ADER-DG)

  • use of tetrahedral meshes to approximate complex 3D model geometries (faults & topography) and rapid model generation supports elastic, viscoelastic and viscoplastic material to approximate realistic geological subsurface properties


Co-design activities: 

Optimisation of backends for small tensor and matrix operations on various CPU architectures (Intel, AMD, ARM)


  • Extension to anisotropic-elastic and poroelastic material properties
  • Novel algorithmic approach for space-time-predictor for poroelastic wave equations (with stiff source term) 

  • NUMA analysis and optimisation, in particular for AMD-based supercomputers

  • Evaluation of energy consumption on various CPU architectures


S. Wolf, A.-A. Gabriel, M. Bader: Optimization and Local Time Stepping of an ADER-DG Scheme for Fully Anisotropic Wave Propagation in Complex Geometries. Computational Science - ICCS 2020 (LNCS 12139), Springer, 2020, p. 32-45.