Towards Heterogeneous Multi-scale Computing on Large Scale Parallel Supercomputers
DOI:
https://doi.org/10.14529/jsfi190402Abstract
New applications that can exploit emerging exascale computing resources efficiently, while providing meaningful scientific results, are eagerly anticipated. Multi-scale models, especially multi-scale applications, will assuredly run at the exascale. We have established that a class of multi-scale applications implementing the heterogeneous multi-scale model follows, a heterogeneous multi-scale computing (HMC) pattern, which typically features a macroscopic model synchronising numerous independent microscopic model simulations. Consequently, communication between microscopic simulations is limited. Furthermore, a surrogate model can often be introduced between macro-scale and micro-scale models to interpolate required data from previously computed micro-scale simulations, thereby substantially reducing the number of micro-scale simulations. Nonetheless, HMC applications, though versatile, remain constrained by load balancing issues. We discuss two main issues: the a priori unknown and variable execution time of microscopic simulations, and the dynamic number of micro-scale simulations required. We tackle execution time variability using a pilot job mechanism to handle internal queuing and multiple sub-model execution on large-scale supercomputers, together with a data-informed execution time prediction model. To dynamically select the number of micro-scale simulations, the HMC pattern automatically detects and identifies three surrogate model phases that help control the available and used core amount. After relevant phase detection and micro-scale simulation scheduling, any idle cores can be used for surrogate model update or for processor release back to the system. We demonstrate HMC performance by testing it on two representative multi-scale applications. We conclude that, considering the subtle interplay between the macroscale model, surrogate models and micro-scale simulations, HMC provides a promising path towards exascale for many multiscale applications.
References
Abdulle, A., Weinan, E., Engquist, B., Vanden-Eijnden, E.: The heterogeneous multiscale method. Acta Numerica 21, 1–87 (2012), DOI: 10.1017/S0962492912000025
Aktulga, H.M., Fogarty, J.C., Pandit, S.A., Grama, A.Y.: Parallel reactive molecular dynamics: Numerical methods and algorithmic techniques. Parallel Computing 38(4), 245–259 (Apr 2012), DOI: 10.1016/j.parco.2011.08.005
Alowayyed, S., Piontek, T., Suter, J.L., Hoenen, O., Groen, D., Luk, O., Bosak, B., Kopta, P., Kurowski, K., Perks, O., Brabazon, K., Jancauskas, V., Coster, D., Coveney, P.V., Hoekstra, A.G.: Patterns for high performance multiscale computing. Future generation computer systems 91, 335–346 (2019), DOI: 10.1016/j.future.2018.08.045
Alowayyed, S.: Patterns for multiscale computing. Ph.D. thesis, University of Amsterdam (2018), http://hdl.handle.net/11245.1/9cf904f8-fcc7-4b8a-a105-622a865359d8
Alowayyed, S., Groen, D., Coveney, P.V., Hoekstra, A.G.: Multiscale computing in the exascale era. Journal of Computational Science 22, 15–25 (2017), DOI: 10.1016/j.jocs.2017.07.004
ASTM: Test method for short-beam strength of polymer matrix composite materials and their laminates, DOI: 10.1520/d2344 d2344m-16, accessed: 2019-11-29
Axner, L., Bernsdorf, J., Zeiser, T., Lammers, P., Linxweiler, J., Hoekstra, A.G.: Performance evaluation of a parallel sparse lattice Boltzmann solver. Journal of Computational Physics 227(10), 4895–4911 (2008), DOI: 10.1016/j.jcp.2008.01.013
Bangerth, W., Hartmann, R., Kanschat, G.: Deal.II–A general-purpose object-oriented finite element library. ACM Trans. Math. Softw. 33(4) (Aug 2007), DOI: 10.1145/1268776.1268779
Borgdorff, J., Ben Belgacem, M., Bona-Casas, C., Fazendeiro, L., Groen, D., Hoenen, O., Mizeranschi, A., Suter, J.L., Coster, D., Coveney, P.V., Dubitzky, W., Hoekstra, A.G., Strand, P., Chopard, B.: Performance of distributed multiscale simulations. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 372 (2014), DOI: 10.1098/rsta.2013.0407
Borgdorff, J., Lorenz, E., Bona-Casas, C., Hoekstra, A.G., Falcone, J.L., Chopard, B.: Foundations of distributed multiscale computing: Formalization, specification, and analysis. J. Parallel Distrib. Comput. Journal of Parallel and Distributed Computing 73(4), 465–483 (2013), DOI: 10.1016/j.jpdc.2012.12.011
Borgdorff, J., Mamonski, M., Bosak, B., Kurowski, K., Belgacem, M.B., Chopard, B., Groen, D., Coveney, P., Hoekstra, A.: Distributed multiscale computing with MUSCLE 2, the Multiscale Coupling Library and Environment. Journal of Computational Science 5(5), 719–731 (2014), DOI: 10.1016/j.jocs.2014.04.004
Bouvard, J.L., Ward, D.K., Hossain, D., Nouranian, S., Marin, E.B., Horstemeyer, M.F.: Review of hierarchical multiscale modeling to describe the mechanical behavior of amorphous polymers. Journal of Engineering Materials and Technology 131(4) (Sep 2009), DOI: 10.1115/1.3183779
Cheng, L.T., Weinan, E.: The heterogeneous multi-scale method for interface dynamics. Contemporary mathematics. 330, 43–54 (2003), DOI: 10.1007/978-94-007-0412-1_18
Chopard, B., Falcone, J.L., Kunzli, P., Veen, L., Hoekstra, A.: Multiscale modeling: recent progress and open questions. Multiscale and Multidiscip. Model. Exp. and Des. Multiscale and Multidisciplinary Modeling, Experiments and Design 1(1), 57–68 (2018), DOI: 10.1007/s41939-017-0006-4
Coveney, P.V., Boon, J.P., Succi, S.: Bridging the gaps at the physics–chemistry–biology interface. Philosophical Transactions of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 374(2080), 335–339 (2016), DOI: 10.1098/rsta.2016.0335
Engquist, B., L¨otstedt, P., Runborg, O.: Multiscale modeling and simulation in science. Springer Science & Business Media, Heidelberg (2009), DOI: 10.1007/978-3-540-88857-4
Hoekstra, A.G., Chopard, B., Coster, D., Portegies Zwart, S., Coveney, P.V.: Multiscale computing for science and engineering in the era of exascale performance. Phil. Trans. R. Soc. A Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 377(2142), 20180144 (2019), DOI: 10.1098/rsta.2018.0144
Hoekstra, A.G., Chopard, B., Coveney, P.V.: Multiscale modelling and simulation: a position paper. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 372, 30377–30385 (2014), DOI: 10.1098/rsta.2013.0377
Hoekstra, A.G., Portegies Zwart, S., Coveney, P.V.: Multiscale modelling, simulation and computing: from the desktop to the exascale. Phil. Trans. R. Soc. A Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 377(2142), 20180355 (2019), DOI: 10.1098/rsta.2018.0355
Knap, J., Spear, C.E., Borodin, O., Leiter, K.W.: Advancing a distributed multi-scale computing framework for large-scale high-throughput discovery in materials science. Nanotechnology 26(43), 434004–434016 (2015), DOI: 10.1088/0957-4484/26/43/434004
Kopta, P., Bosak, B.: QCG-PilotJob. https://github.com/compat-project/QCG-PilotJob (2018)
Leiter, K.W., Barnes, B.C., Becker, R., Knap, J.: Accelerated scale-bridging through adaptive surrogate model evaluation. Journal of Computational Science 27, 91–106 (2018), DOI: 10.1016/j.jocs.2018.04.010
Lorenz, E., Hoekstra, A.G.: Heterogeneous multiscale simulations of suspension flow. Multiscale Modeling and Simulation 9(4), 1301–1326 (2011), DOI: 10.1137/100818522
Lorenz, E., Hoekstra, A.G., Caiazzo, A.: Lees-edwards boundary conditions for lattice Boltzmann suspension simulations. Phys. Rev. E Physical Review E 79(3) (2009), DOI: 10.1103/PhysRevE.79.036706
Luk, O., Hoenen, O., Perks, O., Brabazon, K., Piontek, T., Kopta, P., Bosak, B., Bottino, A., Scott, B.D., Coster, D.P.: Application of the extreme scaling computing pattern on multiscale fusion plasma modelling. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 377(2142), 20180152 (2019), DOI: 10.1098/rsta.2018.0152
Nikishova, A., Veen, L., Zun, P., Hoekstra, A.G.: Uncertainty quantification of a multiscale model for in-stent restenosis. Cardiovascular Engineering and Technology (2018), DOI: 10.1007/s13239-018-00372-4
Plimpton, S.: Fast parallel algorithms for short-range molecular dynamics. Journal of Computational Physics 117(1), 1–19 (Mar 1995), DOI: 10.1006/jcph.1995.1039
Sloot, P.M.A., Hoekstra, A.G.: Multi-scale modelling in computational biomedicine. Briefings in bioinformatics 11(1), 142–152 (2009), DOI: 10.1093/bib/bbp038
Turilli, M., Santcroos, M., Jha, S.: A comprehensive perspective on Pilot-Job systems. ACM Computing Surveys 51(2), 1–32 (2018), DOI: 10.1145/3177851
Vassaux, M., Richardson, R., Coveney, P.V.: The heterogeneous multiscale method applied to inelastic polymer mechanics. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 377(2142) (2019), DOI: 10.1098/rsta.2018.0150
Wang, C., Duan, Q., Gong, W., Ye, A., Di, Z., Miao, C.: An evaluation of adaptive surrogate modeling based optimization with two benchmark problems. Environmental Modelling and Software 60, 167–179 (2014), DOI: 10.1016/j.envsoft.2014.05.026
Weinan, E.: Principles of multiscale modeling. Cambridge University Press, Cambridge, New York (2011), DOI: 10.1063/PT.3.1609
Weinan, E., Engquist, B., Huang, Z.: Heterogeneous multiscale method: a general methodology for multiscale modeling. Physical Review B 67(9), 092101 (2003), DOI: 10.1103/PhysRevB.67.092101
Z´avodszky, G., van Rooij, B., Azizi, V., Hoekstra, A.G.: Cellular level in-silico modeling of blood rheology with an improved material model for red blood cells. Frontiers in physiology 8 (2017), DOI: 10.3389/fphys.2017.00563
Z´avodszky, G., van Rooij, B., Azizi, V., Alowayyed, S., Hoekstra, A.G.: Hemocell: a highperformance microscopic cellular library. Procedia Computer Science 108, 159–165 (2017), DOI: 10.1016/j.procs.2017.05.084
Downloads
Published
How to Cite
Issue
License
Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution-Non Commercial 3.0 License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.