Recurrent Monitoring of Supercomputer Noise

Vadim V. Voevodin; Dmitry A. Nikitenko

doi:10.14529/jsfi230304

Authors

Vadim V. Voevodin Lomonosov Moscow State University, Moscow, Russian Federation https://orcid.org/0000-0003-1897-1828
Dmitry A. Nikitenko Lomonosov Moscow State University, Moscow, Russian Federation https://orcid.org/0000-0002-2864-7995

DOI:

https://doi.org/10.14529/jsfi230304

Keywords:

supercomputing, high-performance computing, monitoring, noise, noise measurement, noise level

Abstract

The presence of noise in supercomputers has long been known, but its scale and impact on the behavior of user applications are nevertheless considerably unclear. Therefore, we decided to develop an approach to determining the noise level on an ongoing basis, which makes it possible to assess the global impact of noise over a long time period. This paper describes a method for recurrent monitoring the noise level and analyzing collected statistics on a real modern supercomputer, and also presents the implementation and evaluation of this method on the Lomonosov-2 supercomputer. The usage of the proposed approach in practice made it possible to identify previously unknown issues and peculiarities, like detection of a faulty compute node, presence of nodes that tend to be more susceptible to noise or the global nature of the noise, which leads to the appearance of noise at multiple nodes simultaneously. This method can as well be ported to other similar computing systems without significant changes.

References

Backfill scheduling, https://slurm.schedmd.com/sched_config.html#backfill

Characteristics of Lomonosov-2 supercomputer, https://parallel.ru/cluster/lomonosov2.html

Current rating of the 50 most powerful supercomputers in CIS, http://top50.supercomputers.ru/?page=rating

Frontier supercomputer debuts as world’s fastest, breaking exascale barrier, https://www.ornl.gov/news/frontier-supercomputer-debuts-worlds-fastest-breaking-exascale-barrier

Redash homepage, https://redash.io/

TOP500 list, https://top500.org/lists/top500/

Afzal, A., Hager, G., Wellein, G.: Propagation and decay of injected one-off delays on clusters: a case study. In: 2019 IEEE International Conference on Cluster Computing (CLUSTER). pp. 1–10. IEEE (2019). https://doi.org/10.1109/CLUSTER.2019.8890995

Agarwal, S., Garg, R., Vishnoi, N.K.: The impact of noise on the scaling of collectives: A theoretical approach. In: High Performance Computing – HiPC 2005. HiPC 2005. Lecture Notes in Computer Science, vol. 3769, pp. 280–289. Springer (2005). https://doi.org/10.1007/11602569_31

Akkan, H., Lang, M., Liebrock, L.: Understanding and isolating the noise in the Linux kernel. The International Journal of High Performance Computing Applications 27(2), 136–146 (2013). https://doi.org/10.1177/1094342013477892

De, P., Kothari, R., Mann, V.: Identifying sources of operating system jitter through finegrained kernel instrumentation. In: Proceedings of the 2007 IEEE International Conference on Cluster Computing, ICCC. pp. 331–340. IEEE (2007). https://doi.org/10.1109/CLUSTR.2007.4629247

De, P., Mann, V.: jitSim: A simulator for predicting scalability of parallel applications in presence of OS jitter. In: Euro-Par 2010 - Parallel Processing, 16th International Euro-Par Conference, Proceedings, Part I. Lecture Notes in Computer Science, vol. 6271, pp. 117–130. Springer (2010). https://doi.org/10.1007/978-3-642-15277-1_12

De, P., Mann, V., Mittal, U.: Handling OS jitter on multicore multithreaded systems. In: Proceedings of the 2009 IEEE International Parallel and Distributed Processing Symposium, IPDPS 2009. pp. 1–12. IEEE (2009). https://doi.org/10.1109/IPDPS.2009.5161046

Ferreira, K.B., Bridges, P., Brightwell, R.: Characterizing application sensitivity to OS interference using kernel-level noise injection. In: Proceedings of the ACM/IEEE Conference on High Performance Computing, SC 2008. pp. 1–12. IEEE/ACM (2008). https://doi.org/10.1109/SC.2008.5219920

Garg, R., De, P.: Impact of noise on scaling of collectives: An empirical evaluation. In: High Performance Computing - HiPC 2006, 13th International Conference, Proceedings. Lecture Notes in Computer Science, vol. 4297, pp. 460–471. Springer (2006). https://doi.org/10.1007/11945918_45

Hoefler, T., Mehlan, T., Lumsdaine, A., Rehm, W.: Netgauge: A network performance measurement framework. In: High Performance Computing and Communications, Third International Conference, HPCC 2007, Proceedings. Lecture Notes in Computer Science, vol. 4782, pp. 659–671. Springer (2007). https://doi.org/10.1007/978-3-540-75444-2_62

Hoefler, T., Schneider, T., Lumsdaine, A.: Characterizing the influence of system noise on large-scale applications by simulation. In: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, SC’10. pp. 1–11. IEEE (2010). https://doi.org/10.1109/SC.2010.12

Jones, T.: Linux kernel co-scheduling for bulk synchronous parallel applications. In: Proceedings of the 1st International Workshop on Runtime and Operating Systems for Supercomputers. pp. 57–64. ACM (2011). https://doi.org/10.1145/1988796.1988805

Khudoleeva, A., Stefanov, K., Voevodin, V.: Evaluating the Impact of MPI Network Sharing on HPC Applications. In: Parallel Computational Technologies. PCT 2023. Communications in Computer and Information Science, vol. 1868, pp. 3–18. Springer (2023). https://doi.org/10.1007/978-3-031-38864-4_1

Mondragon, O.H., Bridges, P.G., Levy, S., Ferreira, K.B., Widener, P.: Understanding performance interference in next-generation HPC systems. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2016. pp. 384–395. IEEE (2016). https://doi.org/10.1109/SC.2016.32

Nikitenko, D., Mohr, B., Wolf, F., et al.: Influence of Noisy Environments on Behavior of HPC Applications. Lobachevskii Journal of Mathematics 42(7), 1560–1570 (2021). https://doi.org/10.1134/S1995080221070192

Nikitenko, D., Antonov, A., Shvets, P., et al.: JobDigest – Detailed System Monitoring-Based Supercomputer Application Behavior Analysis. In: Supercomputing. Third Russian Supercomputing Days, RuSCDays 2017, Moscow, Russia, September 25-26, 2017, Revised Selected Papers. Communications in Computer and Information Science, vol. 793, pp. 516–529. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-71255-0_42

Shvets, P., Voevodin, V., Nikitenko, D.: Approach to Workload Analysis of Large HPC Centers. In: Parallel Computational Technologies. PCT 2020. Communications in Computer and Information Science, vol. 1263, pp. 16–30. Springer (2020). https://doi.org/10.1007/978-3-030-55326-5_2

Stefanov, K., Voevodin, V., Zhumatiy, S., Voevodin, V.: Dynamically Reconfigurable Distributed Modular Monitoring System for Supercomputers (DiMMon). Procedia Computer Science 66, 625–634 (2015). https://doi.org/10.1016/j.procs.2015.11.071

Tsafrir, D., Etsion, Y., Feitelson, D.G., Kirkpatrick, S.: System noise, OS clock ticks, and fine-grained parallel applications. In: Proceedings of the 19th Annual International Conference on Supercomputing. pp. 303–312. ACM (2005). https://doi.org/10.1145/1088149.1088190

Voevodin, V., Antonov, A., Nikitenko, D., et al.: Supercomputer Lomonosov-2: Large scale, deep monitoring and fine analytics for the user community. Supercomputing Frontiers and Innovations 6(2), 4–11 (2019). https://doi.org/10.14529/jsfi190201