Towards Heterogeneous Multi-scale Computing on Large Scale Parallel Supercomputers


  • Saad A. Alowayyed University of Amsterdam King Abdulaziz City for Science and Technology
  • Maxime Vassaux
  • Ben Czaja
  • Peter V. Coveney
  • Alfons G. Hoekstra



New applications that can exploit emerging exascale computing resources efficiently, while providing meaningful scientific results, are eagerly anticipated. Multi-scale models, especially multi-scale applications, will assuredly run at the exascale. We have established that a class of multi-scale applications implementing the heterogeneous multi-scale model follows, a heterogeneous multi-scale computing (HMC) pattern, which typically features a macroscopic model synchronising numerous independent microscopic model simulations. Consequently, communication between microscopic simulations is limited. Furthermore, a surrogate model can often be introduced between macro-scale and micro-scale models to interpolate required data from previously computed micro-scale simulations, thereby substantially reducing the number of micro-scale simulations. Nonetheless, HMC applications, though versatile, remain constrained by load balancing issues. We discuss two main issues: the a priori unknown and variable execution time of microscopic simulations, and the dynamic number of micro-scale simulations required. We tackle execution time variability using a pilot job mechanism to handle internal queuing and multiple sub-model execution on large-scale supercomputers, together with a data-informed execution time prediction model. To dynamically select the number of micro-scale simulations, the HMC pattern automatically detects and identifies three surrogate model phases that help control the available and used core amount. After relevant phase detection and micro-scale simulation scheduling, any idle cores can be used for surrogate model update or for processor release back to the system. We demonstrate HMC performance by testing it on two representative multi-scale applications. We conclude that, considering the subtle interplay between the macroscale model, surrogate models and micro-scale simulations, HMC provides a promising path towards exascale for many multiscale applications.


Abdulle, A., Weinan, E., Engquist, B., Vanden-Eijnden, E.: The heterogeneous multiscale method. Acta Numerica 21, 1–87 (2012), DOI: 10.1017/S0962492912000025

Aktulga, H.M., Fogarty, J.C., Pandit, S.A., Grama, A.Y.: Parallel reactive molecular dynamics: Numerical methods and algorithmic techniques. Parallel Computing 38(4), 245–259 (Apr 2012), DOI: 10.1016/j.parco.2011.08.005

Alowayyed, S., Piontek, T., Suter, J.L., Hoenen, O., Groen, D., Luk, O., Bosak, B., Kopta, P., Kurowski, K., Perks, O., Brabazon, K., Jancauskas, V., Coster, D., Coveney, P.V., Hoekstra, A.G.: Patterns for high performance multiscale computing. Future generation computer systems 91, 335–346 (2019), DOI: 10.1016/j.future.2018.08.045

Alowayyed, S.: Patterns for multiscale computing. Ph.D. thesis, University of Amsterdam (2018),

Alowayyed, S., Groen, D., Coveney, P.V., Hoekstra, A.G.: Multiscale computing in the exascale era. Journal of Computational Science 22, 15–25 (2017), DOI: 10.1016/j.jocs.2017.07.004

ASTM: Test method for short-beam strength of polymer matrix composite materials and their laminates, DOI: 10.1520/d2344 d2344m-16, accessed: 2019-11-29

Axner, L., Bernsdorf, J., Zeiser, T., Lammers, P., Linxweiler, J., Hoekstra, A.G.: Performance evaluation of a parallel sparse lattice Boltzmann solver. Journal of Computational Physics 227(10), 4895–4911 (2008), DOI: 10.1016/

Bangerth, W., Hartmann, R., Kanschat, G.: Deal.II–A general-purpose object-oriented finite element library. ACM Trans. Math. Softw. 33(4) (Aug 2007), DOI: 10.1145/1268776.1268779

Borgdorff, J., Ben Belgacem, M., Bona-Casas, C., Fazendeiro, L., Groen, D., Hoenen, O., Mizeranschi, A., Suter, J.L., Coster, D., Coveney, P.V., Dubitzky, W., Hoekstra, A.G., Strand, P., Chopard, B.: Performance of distributed multiscale simulations. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 372 (2014), DOI: 10.1098/rsta.2013.0407

Borgdorff, J., Lorenz, E., Bona-Casas, C., Hoekstra, A.G., Falcone, J.L., Chopard, B.: Foundations of distributed multiscale computing: Formalization, specification, and analysis. J. Parallel Distrib. Comput. Journal of Parallel and Distributed Computing 73(4), 465–483 (2013), DOI: 10.1016/j.jpdc.2012.12.011

Borgdorff, J., Mamonski, M., Bosak, B., Kurowski, K., Belgacem, M.B., Chopard, B., Groen, D., Coveney, P., Hoekstra, A.: Distributed multiscale computing with MUSCLE 2, the Multiscale Coupling Library and Environment. Journal of Computational Science 5(5), 719–731 (2014), DOI: 10.1016/j.jocs.2014.04.004

Bouvard, J.L., Ward, D.K., Hossain, D., Nouranian, S., Marin, E.B., Horstemeyer, M.F.: Review of hierarchical multiscale modeling to describe the mechanical behavior of amorphous polymers. Journal of Engineering Materials and Technology 131(4) (Sep 2009), DOI: 10.1115/1.3183779

Cheng, L.T., Weinan, E.: The heterogeneous multi-scale method for interface dynamics. Contemporary mathematics. 330, 43–54 (2003), DOI: 10.1007/978-94-007-0412-1_18

Chopard, B., Falcone, J.L., Kunzli, P., Veen, L., Hoekstra, A.: Multiscale modeling: recent progress and open questions. Multiscale and Multidiscip. Model. Exp. and Des. Multiscale and Multidisciplinary Modeling, Experiments and Design 1(1), 57–68 (2018), DOI: 10.1007/s41939-017-0006-4

Coveney, P.V., Boon, J.P., Succi, S.: Bridging the gaps at the physics–chemistry–biology interface. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 374(2080), 335–339 (2016), DOI: 10.1098/rsta.2016.0335

Engquist, B., L¨otstedt, P., Runborg, O.: Multiscale modeling and simulation in science. Springer Science & Business Media, Heidelberg (2009), DOI: 10.1007/978-3-540-88857-4

Hoekstra, A.G., Chopard, B., Coster, D., Portegies Zwart, S., Coveney, P.V.: Multiscale computing for science and engineering in the era of exascale performance. Phil. Trans. R. Soc. A Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 377(2142), 20180144 (2019), DOI: 10.1098/rsta.2018.0144

Hoekstra, A.G., Chopard, B., Coveney, P.V.: Multiscale modelling and simulation: a position paper. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 372, 30377–30385 (2014), DOI: 10.1098/rsta.2013.0377

Hoekstra, A.G., Portegies Zwart, S., Coveney, P.V.: Multiscale modelling, simulation and computing: from the desktop to the exascale. Phil. Trans. R. Soc. A Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 377(2142), 20180355 (2019), DOI: 10.1098/rsta.2018.0355

Knap, J., Spear, C.E., Borodin, O., Leiter, K.W.: Advancing a distributed multi-scale computing framework for large-scale high-throughput discovery in materials science. Nanotechnology 26(43), 434004–434016 (2015), DOI: 10.1088/0957-4484/26/43/434004

Kopta, P., Bosak, B.: QCG-PilotJob. (2018)

Leiter, K.W., Barnes, B.C., Becker, R., Knap, J.: Accelerated scale-bridging through adaptive surrogate model evaluation. Journal of Computational Science 27, 91–106 (2018), DOI: 10.1016/j.jocs.2018.04.010

Lorenz, E., Hoekstra, A.G.: Heterogeneous multiscale simulations of suspension flow. Multiscale Modeling and Simulation 9(4), 1301–1326 (2011), DOI: 10.1137/100818522

Lorenz, E., Hoekstra, A.G., Caiazzo, A.: Lees-edwards boundary conditions for lattice Boltzmann suspension simulations. Phys. Rev. E Physical Review E 79(3) (2009), DOI: 10.1103/PhysRevE.79.036706

Luk, O., Hoenen, O., Perks, O., Brabazon, K., Piontek, T., Kopta, P., Bosak, B., Bottino, A., Scott, B.D., Coster, D.P.: Application of the extreme scaling computing pattern on multiscale fusion plasma modelling. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 377(2142), 20180152 (2019), DOI: 10.1098/rsta.2018.0152

Nikishova, A., Veen, L., Zun, P., Hoekstra, A.G.: Uncertainty quantification of a multiscale model for in-stent restenosis. Cardiovascular Engineering and Technology (2018), DOI: 10.1007/s13239-018-00372-4

Plimpton, S.: Fast parallel algorithms for short-range molecular dynamics. Journal of Computational Physics 117(1), 1–19 (Mar 1995), DOI: 10.1006/jcph.1995.1039

Sloot, P.M.A., Hoekstra, A.G.: Multi-scale modelling in computational biomedicine. Briefings in bioinformatics 11(1), 142–152 (2009), DOI: 10.1093/bib/bbp038

Turilli, M., Santcroos, M., Jha, S.: A comprehensive perspective on Pilot-Job systems. ACM Computing Surveys 51(2), 1–32 (2018), DOI: 10.1145/3177851

Vassaux, M., Richardson, R., Coveney, P.V.: The heterogeneous multiscale method applied to inelastic polymer mechanics. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 377(2142) (2019), DOI: 10.1098/rsta.2018.0150

Wang, C., Duan, Q., Gong, W., Ye, A., Di, Z., Miao, C.: An evaluation of adaptive surrogate modeling based optimization with two benchmark problems. Environmental Modelling and Software 60, 167–179 (2014), DOI: 10.1016/j.envsoft.2014.05.026

Weinan, E.: Principles of multiscale modeling. Cambridge University Press, Cambridge, New York (2011), DOI: 10.1063/PT.3.1609

Weinan, E., Engquist, B., Huang, Z.: Heterogeneous multiscale method: a general methodology for multiscale modeling. Physical Review B 67(9), 092101 (2003), DOI: 10.1103/PhysRevB.67.092101

Z´avodszky, G., van Rooij, B., Azizi, V., Hoekstra, A.G.: Cellular level in-silico modeling of blood rheology with an improved material model for red blood cells. Frontiers in physiology 8 (2017), DOI: 10.3389/fphys.2017.00563

Z´avodszky, G., van Rooij, B., Azizi, V., Alowayyed, S., Hoekstra, A.G.: Hemocell: a highperformance microscopic cellular library. Procedia Computer Science 108, 159–165 (2017), DOI: 10.1016/j.procs.2017.05.084




How to Cite

Alowayyed, S. A., Vassaux, M., Czaja, B., Coveney, P. V., & Hoekstra, A. G. (2020). Towards Heterogeneous Multi-scale Computing on Large Scale Parallel Supercomputers. Supercomputing Frontiers and Innovations, 6(4), 20–43.