A General Neural-Networks-Based Method for Identification of Partial Differential Equations, Implemented on a Novel AI Accelerator

Mikhail A. Krinitskiy; Victor M. Stepanenko; Alexey O. Malkhanov; Mikhail E. Smorkalov

doi:10.14529/jsfi220302

Authors

Mikhail A. Krinitskiy Shirshov Institute of Oceanology, Russian Academy of Sciences, Moscow, Russian Federation; Lomonosov Moscow State University, Moscow, Russian Federation https://orcid.org/0000-0001-5943-0695
Victor M. Stepanenko Lomonosov Moscow State University, Moscow, Russian Federation; Moscow Center for Fundamental and Applied Mathematics, Moscow, Russian Federation https://orcid.org/0000-0003-3033-6712
Alexey O. Malkhanov Huawei Nizhny Novgorod Research Center, Nizhny Novgorod, Russian Federation https://orcid.org/0000-0001-5210-8281
Mikhail E. Smorkalov Skolkovo Institute of Science and Technology, Moscow, Russian Federation; Huawei Nizhny Novgorod Research Center, Nizhny Novgorod, Russian Federation https://orcid.org/0000-0002-0454-5249

DOI:

https://doi.org/10.14529/jsfi220302

Keywords:

partial differential equations, artificial neural networks, machine learning, inverse problems, land surface model, Richards equation, Ascend platform

Abstract

Partial differential equations (PDEs) are pervasive in vast domains of science and engineering. Although there is huge legacy of numerical methods for solving direct and inverse PDE problems, these methods are computationally expensive for many fundamental and real-life applications, demanding supercomputer resources. Moreover, existing methods for PDEs identification assume concrete functional forms for the coefficients to be found, significantly limiting the range of possible solutions. The mentioned circumstances lead to increasing interest in AI-based methods for direct solving and identification of PDEs. In this study, we propose a novel method based on artificial neural networks (ANNs) for the identification of partial differential equations. The method does not require any strong a priori assumptions regarding the family of the functions approximating PDE coefficients. It allows one to approximate the coefficients of a PDE based on the observed evolution of PDE direct solution. We demonstrate efficacy and high accuracy of ANN-based method in case of diffusion equation and nonlinear diffusion-advection equation (Richards equation) applied to the simulation of heat and moisture transfer in soil. We demonstrate that the novel method implemented on Ascend platform using the mixed precision floating point operations overperforms the classical gradient descent method in Barzilai–Borwein stabilized modification (BBstab, realized on a conventional central processor), in terms of MAPE (mean absolute percentage error) and RMSE (root mean square error) of approximated coefficients at least an order of magnitude. We also found that ANN-method is much less sensitive to initial guess of parameters compared to BBstab approach. Since the considered equations are of generic form, we anticipate that the proposed ANN-based method can be successfully exploited in other applications. These potential applications include hydrodynamic-type problems, e.g., optimization of turbulence closures, where the assumed reference solutions of PDEs are usually obtained from high-resolution direct Navier-Stokes simulations.

References

Mixed precision training, http://docs.nvidia.com/deeplearning/frameworks/mixed-precision-training/index.html

Abadi, M., Agarwal, A., Barham, P., et al.: TensorFlow: Large-scale machine learning on heterogeneous systems (2015), http://tensorflow.org/

Akiba, T., Sano, S., Yanase, T., et al.: Optuna: A next-generation hyperparameter optimization framework. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, Anchorage, AK, USA, August 4-8, 2019. pp. 2623–2631. ACM (2019). https://doi.org/10.1145/3292500.3330701

Baek, M., DiMaio, F., Anishchenko, I., et al.: Accurate prediction of protein structures and interactions using a three-track neural network. Science 373(6557), 871–876 (2021). https://doi.org/10.1126/science.abj8754

Bandai, T., Ghezzehei, T.A.: Physics-Informed Neural Networks With Monotonicity Constraints for Richardson-Richards Equation: Estimation of Constitutive Relationships and

Soil Water Flux Density From Volumetric Water Content Measurements. Water Resources Research 57(2) (feb 2021). https://doi.org/10.1029/2020WR027642

Barzilai, J., Borwein, J.M.: Two-Point Step Size Gradient Methods. IMA Journal of Numerical Analysis 8(1), 141–148 (1988). https://doi.org/10.1093/imanum/8.1.141

Brooks, R., Corey, A.: Hydraulic Properties of Porous Media. Tech. rep., Colorado State University, Fort Collins (1964)

Burdakov, O., Dai, Y.H., Huang, N.: Stabilized Barzilai-Borwein method. Journal of Computational Mathematics 37(6), 916–936 (2019). https://doi.org/10.4208/jcm.1911-m2019-01

Burdine, N.: Relative Permeability Calculations From Pore Size Distribution Data. Journal of Petroleum Technology 5(03), 71–78 (mar 1953). https://doi.org/10.2118/225-G

Camporeale, E., Wilkie, G.J., Drozdov, A., Bortnik, J.: Machine-learning based discovery of missing physical processes in radiation belt modeling (2021)

Côté, J., Konrad, J.M.: A generalized thermal conductivity model for soils and construction materials. Canadian Geotechnical Journal 42(2), 443–458 (apr 2005). https://doi.org/10.1139/t04-106

Du, C.: Comparison of the performance of 22 models describing soil water retention curves from saturation to oven dryness. Vadose Zone Journal 19(1) (jan 2020). https://doi.org/10.1002/vzj2.20072

Fadeev, R.Y., Ushakov, K.V., Tolstykh, M.A., Ibrayev, R.A.: Design and development of the SLAV-INMIO-CICE coupled model for seasonal prediction and climate research. Russian Journal of Numerical Analysis and Mathematical Modelling 33(6), 333–340 (dec 2018).

https://doi.org/10.1515/rnam-2018-0028

Fuentes, C., Chávez, C., Brambila, F.: Relating Hydraulic Conductivity Curve to SoilWater Retention Curve Using a Fractal Model. Mathematics 8(12), 2201 (dec 2020). https://doi.org/10.3390/math8122201

Gardner, W.R.: Field Measurement of Soil Water Diffusivity. Soil Science Society of America Journal 34(5), 832 (1970). https://doi.org/10.2136/sssaj1970.03615995003400050045x

Gasmi, C.F., Tchelepi, H.: Physics informed deep learning for flow and transport in porous media (2021)

van Genuchten, M.T.: A Closed-form Equation for Predicting the Hydraulic Conductivity of Unsaturated Soils. Soil Science Society of America Journal 44(5), 892–898 (sep 1980). https://doi.org/10.2136/sssaj1980.03615995004400050002x

Ghorbani, A., Sadeghi, M., Jones, S.B.: Towards new soil water flow equations using physicsconstrained machine learning. Vadose Zone Journal 20(4) (jul 2021). https://doi.org/10.1002/vzj2.20136

Haghighat, E., Raissi, M., Moure, A., et al.: A deep learning framework for solution and discovery in solid mechanics: linear elasticity. CoRR abs/2003.02751 (2020), https://arxiv.org/abs/2003.02751

Harris, C.R., Millman, K.J., van der Walt, S.J., et al.: Array programming with NumPy. Nature 585(7825), 357–362 (Sep 2020). https://doi.org/10.1038/s41586-020-2649-2

He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV) (December 2015), https://www.cv-foundation.org/openaccess/content_iccv_2015/html/He_Delving_Deep_into_ICCV_2015_paper.html

Jagtap, A., Karniadakis, G.: Extended Physics-Informed Neural Networks (XPINNs): A Generalized Space-Time Domain Decomposition Based Deep Learning Framework for Nonlinear Partial Differential Equations. Communications in Computational Physics 28, 2002–2041 (11 2020). https://doi.org/10.4208/cicp.OA-2020-0164

Jia, W., Wang, H., Chen, M., et al.: Pushing the limit of molecular dynamics with ab initio accuracy to 100 million atoms with machine learning. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. SC ’20, IEEE Press (2020). https://doi.org/10.5555/3433701.3433707

Johansen, O.: Thermal conductivity of soils. Ph.D. thesis, University of Trondheim (1975)

Jumper, J., Evans, R., Pritzel, A., et al.: Highly accurate protein structure prediction with alphafold. Nature (2021), https://doi.org/10.1038/s41586-021-03819-2

Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd Int. Conf. on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015), http://arxiv.org/abs/1412.6980

Kokoreva, A.A., Dembovetskiy, A.V., Ezhelev, Z.S., et al.: Simulating water transport in porous media of urban soil. IOP Conference Series: Earth and Environmental Science 862(1), 012042 (oct 2021). https://doi.org/10.1088/1755-1315/862/1/012042

Kurth, T., Treichler, S., Romero, J., et al.: Exascale deep learning for climate analytics. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018, Dallas, TX, USA, November 11-16, 2018. pp. 51:1–51:12. IEEE / ACM (2018). https://doi.org/10.5555/3291656.3291724

Li, Z., Kovachki, N.B., Azizzadenesheli, K., et al.: Fourier neural operator for parametric partial differential equations. CoRR abs/2010.08895 (2020), https://arxiv.org/abs/2010.08895

Li, Z., Kovachki, N.B., Azizzadenesheli, K., et al.: Fourier neural operator for parametric partial differential equations. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net (2021), https://openreview.net/forum?id=c8P9NQVtmnO

Loshchilov, I., Hutter, F.: SGDR: stochastic gradient descent with warm restarts. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net (2017), https://openreview.net/forum?id=Skq89Scxx

Lu, L., Jin, P., Karniadakis, G.E.: DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. CoRR abs/1910.03193 (2019), http://arxiv.org/abs/1910.03193

Mishkin, D., Matas, J.: All you need is a good init. In: 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings (2016), http://arxiv.org/abs/1511.06422

Misra, D.: Mish: A self regularized non-monotonic activation function (2020), http://arxiv.org/abs/1908.08681

Mortikov, E.V., Glazunov, A.V., Lykosov, V.N.: Numerical study of plane Couette flow: turbulence statistics and the structure of pressure–strain correlations. Russian Journal of Numerical Analysis and Mathematical Modelling 34(2), 119–132 (apr 2019). https://doi.org/10.1515/rnam-2019-0010

Mualem, Y.: A new model for predicting the hydraulic conductivity of unsaturated porous media. Water Resources Research 12(3), 513–522 (jun 1976). https://doi.org/10.1029/WR012i003p00513

Pfau, D., Spencer, J.S., Matthews, A.G.D.G., Foulkes, W.M.C.: Ab initio solution of the many-electron Schrödinger equation with deep neural networks. Physical Review Research 2(3) (Sep 2020). https://doi.org/10.1103/physrevresearch.2.033429

Raissi, M., Perdikaris, P., Karniadakis, G.E.: Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics 378, 686–707 (2019). https://doi.org/https://doi.org/10.1016/j.jcp.2018.10.045

Sitzmann, V., Martel, J.N.P., Bergman, A.W., et al.: Implicit neural representations with periodic activation functions. CoRR abs/2006.09661 (2020), https://arxiv.org/abs/2006.09661

Tancik, M., Srinivasan, P.P., Mildenhall, B., et al.: Fourier features let networks learn high frequency functions in low dimensional domains. CoRR abs/2006.10739 (2020), https://arxiv.org/abs/2006.10739

Van Rossum, G., Drake, F.L.: Python 3 Reference Manual. CreateSpace, Scotts Valley, CA (2009)

Volodin, E.M., Gritsun, A.S.: Simulation of Possible Future Climate Changes in the 21st Century in the INM-CM5 Climate Model. Izvestiya, Atmospheric and Oceanic Physics 56(3), 218–228 (may 2020). https://doi.org/10.1134/S0001433820030123

Weyn, J.A., Durran, D.R., Caruana, R.: Improving data-driven global weather prediction using deep convolutional neural networks on a cubed sphere. Journal of Advances in Modeling Earth Systems 12(9) (Sep 2020). https://doi.org/10.1029/2020ms002109

Wiecha, P.R., Arbouet, A., Girard, C., Muskens, O.L.: Deep learning in nano-photonics: inverse design and beyond. Photonics Research 9(5), B182 (Apr 2021). https://doi.org/10.1364/prj.415960

Ziogas, A.N., Ben-Nun, T., Fernández, G.I., et al.: A data-centric approach to extremescale ab initio dissipative quantum transport simulations. Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (Nov 2019). https://doi.org/10.1145/3295500.3357156