Direct Numerical Simulation of Stratified Turbulent Flows and Passive Tracer Transport on HPC Systems: Comparison of CPU Architectures

Evgeny V. Mortikov; Andrey V. Debolskiy

doi:10.14529/jsfi210405

Authors

Evgeny V. Mortikov Lomonosov Moscow State University
Andrey V. Debolskiy Lomonosov Moscow State University Research Computing Center

DOI:

https://doi.org/10.14529/jsfi210405

Keywords:

turbulence, direct numerical simulation, ARM, supercomputing

Abstract

In this paper we assess the influence of CPU architectures commonly used in HPC systems on the efficiency of the implementation of algorithms used for direct numerical simulation (DNS) of turbulent flows. We consider a stably stratified turbulent plane Couette flow as a benchmark problem supplemented with the additional transport of passive substances. The comparison includes the Intel Xeon, AMD Rome x86 CPU architecture processors and the Huawei Kunpeng ARM CPU processor. We discuss the role of memory-oriented optimizations on the efficiency of tracer transport implementation on each platform.

References

Afanasyev, I., Lichmanov, D.: Evaluating the performance of Kunpeng 920 processors on modern HPC applications. In: Parallel Computing Technologies 2021, Proceedings. pp. 301–321. Springer International Publishing (2021). https://doi.org/10.1007/978-3-030-86359-3_23

Ayala, A., Tomov, S., Haidar, A., Dongarra, J.: heFFTe: Highly efficient FFT for exascale. In: Computational Science – ICCS 2020. ICCS 2020. Lecture Notes in Computer Science. pp. 262–275. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50371-0_19

Besnard, J., Malony, A., Shende, S., et al.: An MPI halo-cell implementation for zero-copy abstraction. In: EuroMPI ’15: Proceedings of the 22nd European MPI Users’ Group Meeting. pp. 1–9. ACM Press (2015). https://doi.org/10.1145/2802658.2802669

Brown, D., Cortez, R., Minion, M.: Accurate projection methods for the incompressible Navier-Stokes equations. J. Comp. Phys. 168, 464–499 (2001). https://doi.org/10.1006/jcph.2001.6715

Dongarra, J., Heroux, M., Luszczek, P.: A new metric for ranking high performance computing systems. Nat. Sci. Rev. 3(1), 30–35 (2016). https://doi.org/10.1093/nsr/nwv084

Gladskikh, D., Stepanenko, V., Mortikov, E.: The effect of the horizontal dimensions of inland water bodies on the thickness of the upper mixed layer. Water Res. 48, 226–234 (2021). https://doi.org/10.1134/S0097807821020068

Glazunov, A., Mortikov, E., Barskov, K., et al.: Layered structure of stably stratified turbulent shear flows. Izv., Atmos. Ocean. Phys. 55(4), 312–323 (2019). https://doi.org/10.1134/S0001433819040042

Ibeid, H., Olson, L., Gropp, W.: FFT, FMM, and multigrid on the road to exascale: Performance challenges and opportunities. J. Parallel Distrib. Comput. 136, 63–74 (2020). https://doi.org/10.1016/j.jpdc.2019.09.014

Kadantsev, E., Mortikov, E., Zilitinkevich, S.: The resistance law for stably stratified atmospheric planetary boundary layers. Q. J. R. Meteorol. Soc. 147(737), 2233–2243 (2021). https://doi.org/10.1002/qj.4019

Larsson, J., Lien, F., Yee, E.: Conditional semicoarsening multigrid algorithm for the Poisson equation on anisotropic grids. J. Comp. Phys. 208, 368–383 (2005). https://doi.org/10.1016/j.jcp.2005.02.020

LeMone, M., Angevine, W., Bretherton, C., et al.: 100 years of progress in boundary layer meteorology. Meteorological Monographs 59, 9.1–9.85 (2019). https://doi.org/10.1175/AMSMONOGRAPHS-D-18-0013.1

Moin, P., Mahesh, K.: Direct numerical simulation: A tool in turbulence research. Annu. Rev. Fluid Mech. 30, 539–578 (1998). https://doi.org/10.1146/annurev.fluid.30.1.539

Monin, A., Yaglom, A.: Statistical fluid mechanics: The mechanics of turbulence. MIT Press, Cambridge (1971)

Morinishi, Y., Lund, T., Vasilyev, O., Moin, P.: Fully conservative higher order finite difference schemes for incompressible flows. J. Comp. Phys. 143, 90–124 (1998). https://doi.org/10.1006/jcph.1998.5962

Mortikov, E.: Numerical simulation of the motion of an ice keel in a stratified flow. Izv., Atmos. Ocean. Phys. 52(1), 108–115 (2016). https://doi.org/10.1134/S0001433816010072

Mortikov, E., Glazunov, A., Lykosov, V.: Numerical study of plane Couette flow: turbulence statistics and the structure of pressure-strain correlations. Russ. J. Numer. Analysis Math. Model. 34(2), 119–132 (2019). https://doi.org/10.1515/rnam-2019-0010

Pirozzoli, S., Bernardini, M., Orlandi, P.: Turbulence statistics in Couette flow at high reynolds number. J. Fluid Mech. 758, 327–343 (2014). https://doi.org/10.1017/jfm.2014.529

Porter, A., Appleyard, J., Ashworth, M., et al.: Portable multi- and many-core performance for finite-difference or finite-element codes – application to the free-surface component of NEMO (NEMOLite2D 1.0). Geosci. Model Dev. 11, 3447–3464 (2018). https://doi.org/10.5194/gmd-2017-150

Rajovic, N., Rico, A., Puzovic, N., Adeniyi-Jones, C., Ramirez, A.: Tibidabo: Making the case for an ARM-based HPC system. Future Generation Computer Systems 36, 322–334 (2014). https://doi.org/10.1016/j.future.2013.07.013

Sofiev, M., Vira, J., Kouznetsov, R., et al.: Construction of the SILAM Eulerian atmospheric dispersion model based on the advection algorithm of Michael Galperin. Geosci. Model Dev. 8, 3497–3522 (2015). https://doi.org/10.5194/gmd-8-3497-2015

Soustova, I., Troitskaya, Y., Gladskikh, D., et al.: A simple description of the turbulent transport in a stratified shear flow as applied to the description of thermohydrodynamics of inland water bodies. Izv., Atmos. Ocean. Phys. 56, 603–612 (2020). https://doi.org/10.1134/S0001433820060109

Thorpe, S.: An introduction to ocean turbulence. Cambridge University Press, Cambridge (2007)

Tkachenko, E., Debolskiy, A., Mortikov, E.: Intercomparison of subgrid scale models in largeeddy simulation of sunset atmospheric boundary layer turbulence: computational aspects. Lobachevskii Journal of Mathematics 42, 1580–1595 (2021). https://doi.org/10.1134/S1995080221070234

Trottenberg, U., Oosterlee, C., Schüller, A.: Multigrid. Academic Press, London (2001)

Vasilyev, O.: High order finite difference schemes on non-uniform meshes with good conservation properties. J. Comp. Phys. 157, 746–761 (2000). https://doi.org/10.1006/jcph.1999.6398

Vichi, M., Pinardi, N., Masina, S.: A generalized model of pelagic biogeochemistry for the global ocean ecosystem. Part I: theory. J. Mar. Sys. 64, 89–109 (2007). https://doi.org/10.1016/j.jmarsys.2006.03.006

Van der Vorst, H.: Bi-CGSTAB: A Fast and Smoothly Converging Variant of Bi-CG for the Solution of Nonsymmetric Linear Systems. SIAM J. Sci. Stat. Comput. 13(2), 631–644 (1992). https://doi.org/10.1137/0913035

Yli-Juuti, T., Barsanti, K., Hildebrandt Ruiz, L., et al.: Model for acid-base chemistry in nanoparticle growth (MABNAG). Atmos. Chem. Phys. 13(24), 12507–12524 (2013). https://doi.org/10.5194/acp-13-12507-2013

Zasko, G., Glazunov, A., Mortikov, E., Nechepurenko, Y.: Large-scale structures in stratified turbulent Couette flow and optimal disturbances. Russ. J. Numer. Analysis Math. Model. 35(1), 37–53 (2020). https://doi.org/10.1515/rnam-2020-0004

Zilitinkevich, S., Druzhinin, O., Glazunov, A., et al.: Dissipation rate of turbulent kinetic energy in stably stratified sheared flows. Atmos. Chem. Phys. 19, 2489–2496 (2019). https://doi.org/10.5194/acp-19-2489-2019

Zoric, D., Sandborn, V.: Similarity of large reynolds number boundary layers. Bound. Layer-Meteorol. 2, 326–333 (1972). https://doi.org/10.1007/BF02184773