Cloud Service for Solution of Promising Problems of Nanotechnology

The paper presents the problem of creating a cloud service designed to solve promising nanotechnology problems on supercomputer systems. The motivation for creating such a service was the need to integrate ideas, knowledge and computing technologies related to this applied problem, as well as the need to involve specialists in solving problems of this type. The preliminary result of the work is a prototype of the cloud environment, implemented as a KIAM Multilogin service and an application software accessible from users virtual machines. The first applications of the service were the software packages GIMM NANO and Flow and Particles, designed to solve the actual problems of nanoelectronics, laser nanotechnology, multiscale problems of applied gas dynamics. The implementation of the service took into account such aspects as support for parallel computations on the park of remote supercomputers, improving the efficiency of parallelization, very large data sets processing, visualization of supercomputer modeling results. With the help of the implemented service, it was possible to optimize the process of solving the applied problems associated with calculating the parameters of gas-dynamic flows in the microchannels of industrial spraying systems. In particular, it was possible to carry out a series of studies devoted to the analysis of gas-dynamic processes at gas-metal boundary. In these studies it was shown that in the presence of microcapillaries in a technical system, it is necessary to use direct modeling of gas dynamic processes on the basis of the first principles in the Knudsen layers, for example, using molecular dynamics methods.


Introduction
The present work is devoted to the development of applied cloud services intended for solving complex nanotechnology problems by computer methods.Fundamental nature and relevance of the common problem comprise the fact that at present in context of the introduction of nanotechnology in many industries there is an urgent need to combine various mathematical approaches, information and computing resources into a unified tool of computer and supercomputer modeling.The most successful way of such an association is to create relevant cloud environments and services in which each user can have access to all possible information materials, modeling programs, computing resources and industrial CAD.
The specific task is to create a cloud service using heterogeneous clusters and supercomputers for multiscale modeling of nonlinear processes in polydisperse multicomponent media used in the implementation of industrial nanotechnologies.To solve the problem, the KIAM Multilogin cloud platform has been developed and implemented in the previous two years, it allows accessing the created modeling environment.The architecture of the platform and the individual components were described in sufficient detail in [1][2][3].The application part of the service was represented by the parallel program Flow and Particles [4][5][6], intended for multiscale molecular modeling of processes of interacting gas particles and the walls of microchannels of technical systems.However, in connection with the evolution of this application to the scale of the software package and the availability of other software tools which the developers had, including the GIMM NANO [7] software, the problem of integrating such programs into large systems and/or environments appeared.
The article is organized as follows.Section 1 is devoted to describing the problems and formulations of specific tasks that need to be solved.Section 2 describes the architecture of the cloud service being developed and user scenarios of work.In Section 3, the prototype of the cloud service and its individual components are described.Section 4 illustrates the application of the developed service to the example of solving the problem of modeling gas flows in the microchannels of technical systems.Conclusion summarizes the results and determines the further development of the service.

Problems and Tasks
In this paper, the task was to create a cloud-based application service.The motivation for this task was the general trends in the development of software for mass use.Within the framework of cloud technologies, many problems are solved: hierarchical user authentication, the creation of a common protected information space, the work of users with personal desktops of virtual machines, migration of virtual machines in the constantly expanding and modifying field of calculators, optimal management of computing resources, convenience of developing specific applications, guaranteed secure storage of critical information, joint software development, training in work with services, and much more.
The specific task was, first of all, the development of a general concept of the applied part of the medium associated with supercomputer modeling of multiscale nonlinear processes in real technical microsystems.The complexity of this kind of problem is due to the fact that the mathematical models used contain heterogeneous descriptions of the physical processes under consideration that pertain to different scale levels.These heterogeneity and different scales give rise to a number of problems that have to be solved simultaneously.
Let us explain what has been said on a certain example.For this, let us consider the problem of calculating gas flow through a metallic channels system of a real three-dimensional geometry.In the case of a macroscale problem (when the sizes of the modeled system are much larger than the mean free path of molecules, that is, the Knudsen number is much less than 1) and simple Cartesian geometry, the description of the flow will require the use of one of the models of continuum mechanics (for example, the systems of Navier-Stokes equations for describing the gas flow, as well as the heat conduction equation for describing the heat exchange of a gas with the walls of the channel).The final initial-boundary task can be solved either with the help of ready-made software (for example, with the help of packages ANSYS CFX, StarCD, OpenFOAM, FlowVision, etc.), or by developing own solution.
In essence, in the case of simple geometry, it is necessary to use the grid finite difference/volume method (FDM/FVM) or the finite element method (FEM) in combination with the method of markers and cells on a suitable structured grid.Parallel implementation of the resulting numerical algorithm is usually not particularly complicated.However, this requires a good library of parallel algorithms for solving systems of linear and/or nonlinear algebraic equations (for example, Aztec, PETSc, etc.).The use of hybrid computing, for example, systems with central and graphics processors, may present some difficulties.However, there are already a lot of tools from NVidia, Intel and other developers.
If you need to take into account the curvilinear geometry of the system, it is necessary to use unstructured grids and specialized versions of FVM or FEM.If the geometry of the problem is rather complicated, and the degree of anisotropy of the computational domain and grid is high, then for high-precision parallel calculation a parallel grid generation, its optimal partitioning according to calculators, a special scheme for storing calculation control points and calculation results, an efficient parallel algorithms for solving algebraic problems with singular matrices will be required.In this situation, widespread software may not be suitable either because of its high cost, or because there are no reliable algorithms for solving the problem at all stages (assigning and/or importing and analyzing the geometry of the computational domain, constructing and partitioning the grid or multiple grids, forming a set of algebraic tasks, solving the algebraic problems, preservation, processing and visualization of results).
If it is necessary to take into account the more complex physics of the process, then the problem of the formation of a realistic mathematical model is added to the problems of processing geometric data.In the case under consideration, it is necessary to take into account the nonideality of the gas and the walls of the channel, the mutual influence of the gas and solid media at the boundaries.As a result, it is necessary to modify the model and methods of its numerical implementation, introducing the material coefficients of the medium, including the real equations of state for the gas, the dependences of the viscosity (shear and volume), thermal conductivity and diffusion coefficients from temperature and pressure (for both gas and material of the channel walls), special algebraic models of the boundary layer.
Specified complication of the mathematical model is impossible without knowledge of the properties of substances in a wide range of parameters.This information is not always available in full, even for well-researched materials.For new composite materials, it is practically nonexistent.Therefore, in order to carry out the basic study, additional studies are needed on the properties of the gas and solid media used in the model.At present, in addition to natural experiments, there are three main approaches for obtaining information about the properties of matter: quantum-mechanical, statistical and hybrid.
The first approach is related to ab initio methods (from the first principles), which allow obtaining detailed information about substances at the molecular and submolecular levels by solving the corresponding quantum-mechanical problems.However, this approach is the most computationally capacious and does not allow obtaining in real time an information about large particle systems (in the present situation, the size of the quantum-mechanical systems being studied amounts to several thousand structural units -molecules, atoms, electrons, protons, neutrons, etc.).Therefore, it is necessary to simplify the corresponding models, moving to hybrid schemes.
Within the framework of the statistical approach, it is possible to consider very large systems (currently the number of random variables can reach 15-20 decimal orders), but the accuracy of the data obtained in large systems first increases with the number of trials, but then decreases sharply due to increasing parasitic noise.As a result, and in the framework of this approach, we have to look for hybrid solutions.
Among the most effective hybrid approaches, we select methods for solving problems based on the Boltzmann kinetic equation or Fokker-Planck equation in conjunction with statistical Monte Carlo methods, as well as methods of classical and quantum molecular dynamics.These approaches successfully combine models from the first principles with statistical and variational calculations of moderate volume and allow with good accuracy to determine the properties of different media and materials.Based on these approaches, databases on the properties of matter are usually formed.In particular, the NIST database (USA), as well as the Russian EPIDIF database, etc. are widely known.However, the use of such databases is possible only in a limited range of parameters and in off-line mode, when the user is forced to manually collect the necessary data.Therefore, researchers have to independently create similar databases using different packages for quantum-mechanical, statistical and molecular-dynamics calculations.Among them we note AB INITIO, GAMESS, GAUSSIAN, VASP, HyperChem, LAMMPS, GROMACS and many others.
In case of a meso-or microscale problem (when in some zones of the flow the Knudsen number approaches or even exceeds 1), an additional problem is the violation of the hypothesis of continuity of the medium.In this situation it is necessary to use mixed models that combine descriptions of continuum mechanics and alternative descriptions (meso-and microscopic).At the program level, this means, in the general calculation, using the algorithms of various kinds implemented within the framework of different libraries and/or packages.The only correct possibility of such computations is the method of splitting by physical processes.Within the framework of this method, it is possible to interface almost any computational schemes (it is only necessary to satisfy the stability conditions of the general algorithm).In the presented paper, exactly this concept is laid in the basis of cloud service that is being developed.

The Architecture of the Cloud Service and the Scenarios of Users Work
Let us briefly consider the architecture of the application cloud service that is being developed.It assumes the existence of a distributed computing platform hidden from users and a single scheme for accessing it via the Internet/Intranet using a specialized web portal.Standard user authentication mechanisms (login-password, login-public key, etc.) and their role differentiation (novice user, user-researcher, developer, administrator, etc.) are assumed here.Inside this cloud architecture, in addition to access servers, there are an information portal, a training portal, a research portal, a developer portal, and computing resources in the form of database servers and working computers (clusters and supercomputers).
Access servers form a decentralized system in which individual elements can be spaced territorially, but synchronized according to a certain principle.Access servers contain, in addition to the authentication service, a platform for launching and operating virtual machines (VM) of users.User virtual machines are stored in one or more databases (DBVM).They can be started by the service at any time and on any access server, but only in a single copy.Simultaneous launch of a user's virtual machine on multiple access servers is excluded.In this case, it is possible to migrate the user's virtual machine to a less-loaded access server, performed by storing all VM data and cold restart.Also, each user can have several variants of VM, distinguished by the type of OS and/or intended purpose, for example, by the role function (user, developer, administrator), as well as several saved states of a particular VM (snaphots).
The user's scenario of work starts with the procedure for registering it on the access servers as a novice user, preparing one or more virtual machines (usually selecting the VM from the repository according to the user's specific preferences), exploring the capabilities of a cloud environment on the information portal, learning to work within the framework of specific projects.After passing the initial training course, the user gets the status of a user-researcher and/or a developer user and can start working within one or several projects.At the same time, he or she has the opportunity to initiate a new project.
The scenario of the user-researcher's work is connected with preparation and carrying out of long cycles of calculations within the framework of one or several projects.This involves all the main application services of the cloud environment.Relating to the problems of computational fluid dynamics, the following commercial or open software is necessary to use: CAD systems for specifying geometry (for example, SolidWorks, ParaSolid, etc.), mesh generators (for example, GAMBIT, TetGen, Ani3D, etc.), programs of grid partitioning by calculators (for example, METIS, ParMETIS, Jostle, etc.), specific solvers from the set of available packages (for example, ANSYS CFX, StarCD, Comsol, OpenFOAM, FemLab, etc.), sequential and parallel visualizers of the received intermediate and resulting data (for example, TecPlot , ParaView, MolDraw, ChemDraw, etc.), as well as programs for pre-and post-processing of data at all stages of calculations.Programs for pre-and post-processing are necessary for combining separate stages of computation by data.The work scenario of the user-developer of the software is associated with the use of specialized virtual machines (equipped with the necessary licensed and/or freely propagated programming tools), installing ready-made software packages on computers, as well as developing and testing new programs and services in various OS and environments.

Cloud Service as a Way of Integrating Technology
The above-described service architecture and users work scenarios are inherent in many cloud systems.However, in the created service variant such a standard solution is only a part of the overall implementation.More complex and relevant parts in this case are the systems of managing informational, educational and computing resources, as well as a project management system.To date, in the evolving service, in a full-scale version, there is only a system for managing computing resources and computing tasks for users.It is called KIAM Job Control and was partially described in [2].
At this stage of the cloud project development, it allows users to perform large-scale calculations on the basis of separate parallel programs from the GIMM NANO and Flow and Particles packages, and also store the results in the corresponding databases, post-process and visualize them [8].
Informational and educational resources management systems are under development and will eventually represent a distributed knowledge base on all aspects of the application service: from technologies of its realization to modeling technologies implemented within user-accessible application program packages and separate applications.
The project management system has not yet been implemented, although such systems are widely used in business and in foreign government institutions.Apparently, there is a strong non-determinism of the processes of scientific research.Therefore, within the framework of this direction it is supposed to start with the automation of application development processes.
To implement this task, there is a certain reserve obtained when creating the GIMM NANO package.Within the framework of this project, the architecture and individual elements of the designer of parallel hybrid applications in the field of solving nanotechnology problems on the basis of ready library functions were determined.In particular, the program interface for C++, C and Fortran languages was developed, which allows assembling a hybrid MPI application from some ready-made components and a newly developed calculation part that determines the novelty and target function of the code.
The ready-made components of the application include: -input/output functions (both sequential and distributed) of text and binary data related to the description of the geometry of the calculation area, grid parameters, physical parameters of the problem, parameters of the numerical approach, parameters of parallel computations, fields of calculated data, etc.; -parallel generators of grids of a certain type (Cartesian, triangular, tetrahedral, hybrid, block, etc.); and -parallel solvers of systems of linear and nonlinear equations supporting calculations on central processing units (CPU), vector processing units (VPU) and graphics processing units (GPU).
The development of the application as a whole is carried out in accordance with the concept of a hybrid parallel platform [9], the distinguishing feature of which is the desire to maximize accounting the features of the problem being solved and the architecture of a calculator with minimal differences in the code from the implementation on the CPU.Achievement of such parameters is realized due to writing of special functions and macroses, taking into account specific parameters of algorithms and capabilities of calculators.Basically, the success or failure of such an implementation depends on the possibilities of vectorization of calculations and the use of block processing of poorly structured data.
When integrating the solutions of the GIMM NANO package based on the models of continuum mechanics with molecular dynamics applications from the Flow and Particles package, it was possible to implement multiscale two-level parallel algorithms and corresponding programs that allow solving new types of problems of technical gas dynamics characteristic of modern nanotechnology.An example of a solution to one such problem is presented in [10].It is important to emphasize that rapid development of this application was accomplished through the use of technologies implemented in the GIMM NANO package.

Modeling of Gas Flows in Microchannels of Technical Systems
Let us briefly consider the application of the developed service to the solution of the problem of modeling gas flows in microchannels of technical systems.The basis of parallel application developed for these purposes is a combination of a gas dynamic code and a program of direct molecular modeling [6,11,12].The calculated geometry is shown in Fig. 1.The essence of the problem is to calculate the gas flow from the high-pressure chamber (located on the left) through the micronozzle (located in the center) into the free space (located on the right).The complexity of the calculation is due to the fact that when the size of the nozzle decreases, the real properties of the gas and the walls of the chamber and the micronozzle turn out to be important, and not only the similarity theory is violated, but also the criterion for the continuity of the medium.
In the calculations below, we used a cylindrical micronozzle with diameter D 0 ≈ 310 µm, and length L 0 = 6D 0 ≈ 1860 µm.It connects the chamber of nitrogen and the open space of the vacuum chamber (zone of free space), which was initially filled with the same highly diluted gas.The dimensions of the computational domain were chosen as follows.The diameters of the computational parts in the chamber and in the vacuum chamber were equal to D 1 = D 2 = 6D 0 .The length of the calculated part in the chamber was L 1 = 10L 0 ; the length of the calculated part in the vacuum chamber was equal to L 2 = 50L 0 .Along with the true size of the investigated system, much smaller ones were considered: D 0 ∼ 1 µm.
At the initial moment, the gas is at rest: u 1 = u 2 = 0.In this case, it is in the chamber under standard normal conditions: T 1 = 295.15K, p 1 = 101325 P a; in the nozzle and the vacuum chamber it is at the same temperature, but at lower pressures: T 2 = 295.15K, p 2 = δ 0 p 1 , δ 0 ∼ 10 −3 ÷ 10 −5 -pumping parameter.The nozzle on the left is blocked by a partition, which at the beginning of the calculation opens instantly.
To carry out realistic calculations, it was necessary to find out the real characteristics of the gas: the parameters of the equation of state (compressibility factor, heat capacity) and its kinetic coefficients (viscosity, thermal conductivity, diffusion, etc.).For this purpose for a sufficiently long time we were accumulating a database on the properties of nitrogen (in the temperature range from 80 to 400 K and the pressure range from 0.00001 to 1 atm), which was used as a gaseous medium (see [10,13,14]).
Next, a series of two-dimensional and three-dimensional calculations based on the quasigasdynamic (QGD) [15] model were performed [10], taking into account the real properties of nitrogen.One of the results is shown in Fig. 2 and Fig. 3.It illustrates the passage of a shock wave through a micronozzle.The figures show the results of two-dimensional modeling based on the QGD model, taking into account the real equation of state and the kinetic properties of the gas environment.
The concentration distributions for two characteristic times are represented in Fig. 2. The longitudinal velocity distributions for the same moments of time are represented in Fig. 3.The analysis of the obtained data showed that the numerical approach used reflects many real characteristics of the flow.However, some clarification is required.In particular, a more detailed study of the passage of gas through the micronozzle in case of its very small dimensions is necessary.
In connection with the foregoing, a series of numerical experiments was carried out, connected with direct molecular modeling of the processes of interacting of the gas and the micronozle walls.For this purpose, we have chosen nickel as the nozzle material, which is often used in vacuum micro-and nanotechnology.As a gas, nitrogen was still used.The conducted molecular dynamics (MD) [16] experiments comprised 3 stages.
At the first stage, the equilibrium parameters of bulk nickel were clarified, including the lattice step at 273.15 K, which amounted to a N i ≈ 0.35314 nm [17,18].Also, the equilibrium state of nitrogen at a given temperature was calculated, including the mean free path of nitrogen molecules λ N2 ≈ 75 nm [15].
At the second stage, the interaction of resting nitrogen with a nickel plate with dimensions 288x288x24 in units of a N i or 101.7x101.7x8.48 in nanometers [19][20][21].The dimensions of the calculated region in the gas phase were 1.36x1.3640.68 in units of λ N2 or 101.7x101.7x1525.6 in nanometers.The number of molecules in the considered gas-metal system was about 8.55 million particles.These calculations showed that, under normal conditions, a layer of high density nitrogen molecules is adsorbed on the nickel surface.The thickness of the layer is comparable to the triple height of the nickel crystal, that is 3a N i .The analysis of the obtained data was carried out using the visualizer Flow and Particles View [8,22,23] embedded in the cloud environment (Fig. 4).Note that this required processing of five thousand checkpoints of the Flow and Particles application with a total volume of 3.3 TB.At the third stage, a small middle part of the micronozle was reconstructed based on the equilibrium state of the nickel plate and the nitrogen layer (Fig. 5).For reconstruction the microstructure editor KIAM MicroStructure Editor was used.The total geometry actually contained two plates of nickel (top and bottom) with layers of adsorbed nitrogen adhering to them and a free layer of nitrogen between them.The parameters of the new microsystem were: 1) the dimensions of the top and bottom plates of nickel were 288028824 in units of a N i or 1017.0x101.7x8.5 in nanometers; 2) the dimensions of the gaseous medium between the plates were 13.6x1.368.2 in units of λ N2 or 1017.0x101.7x614.5 in nanometers.The number of nickel atoms in both plates is 162.57million, the number of nitrogen molecules on the plates and between them is 42.382 million.
The new microsystem was again brought to a state of equilibrium.Then, with the help of the Langevin thermostat, the middle part (a layer of the width of w = 1440 • a N i ≈ 508.5 nm) of the gas in it was accelerated to the velocity of 0.4 nm/ps, which is slightly higher than the speed The simulation results are shown in Fig. 6, where the profiles of the longitudinal velocity component V x (z) averaged over the coordinate x are shown at different moments of time.They show first the acceleration of the gas to supersonic speed in the middle zone of the microchannel, and then the formation of a profile type of the Poiseuille profile.Dynamics with time also shows the beating of the velocity profile near the microchannel wall (Fig. 7).It is important to emphasize that these calculations would not be possible both without the use of supercomputers, and the created cloud service.In the above calculations the supercomputer K60 of Keldysh Institute of Applied Mathematics of the Russian Academy of Sciences with a performance of 60 TFlops and the supercomputer MVS10P-PII of Joint Supercomputer Center of the Russian Academy of Sciences with a performance of 167 TFlops were used.The

Conclusion
The paper presents a prototype of cloud service, intended for solving promising nanotechnology problems on supercomputer systems.The prototype is implemented as an access service KIAM Multilogin and application software accessible from users virtual machines.The first experience of parallel applications integration within the framework of the created service was using separate programs from the GIMM NANO and Flow and Particles packages intended for solving complex problems of nanoelectronics, laser nanotechnology, and multiscale problems of technical gas dynamics.The service capabilities are demonstrated using the example of the problem of calculating the gas flow from a high-pressure chamber through a micronozzle into a free space.Specificity of the problem consists in its multiple scales and nonlinearity.To solve the problem, a two-level multiscale approach is used, realized by grid methods at the macrolevel and molecular dynamics methods at the microlevel.To achieve the necessary accuracy, the high-grid resolution and a large number of particles at the microlevel had to be used within the framework of the proposed approach.Simultaneous realization of these requirements became possible due to the use of supercomputers and the developed cloud service.
Further development of the service consists in improving the management systems of information, educational and computing resources, adding new packages and applications, creating a flexible project management system, increasing the capabilities of the visualization system.

Figure 2 .Figure 3 .
Figure 2. Distributions of the concentration of gas molecules at time points t = 0.0553, 0.553 µs (respectively, a and b)

Figure 4 .
Figure 4. Visualization of effect of nitrogen adsorption on nickel surface

Figure 6 .
Figure 6.Evolution of the distribution of the average longitudinal velocity as a function of the coordinate z during the acceleration of the flow (on the left) and the establishment of the regular regime (right)

Figure 7 .
Figure 7. Distribution of the modulus of gas flow velocity in the middle part of the micronozle