Representation of Spatial Data Processing Pipelines Using Relational Database

Igor G. Okladnikov

doi:10.14529/jsfi210404

Authors

Igor G. Okladnikov Institute of Monitoring of Climatic and Ecological Systems of the Siberian Branch of the Russian Academy of Sciences (IMCES SB RAS)

DOI:

https://doi.org/10.14529/jsfi210404

Keywords:

spatial data, information systems, databases, workflow, directed multigraph, processing pipeline, climate research

Abstract

A methodology for representation of spatial data processing pipelines using relational database within the framework of the computing backend of the online information-analytical system “Climate” (http://climate.scert.ru) is proposed. Each pipeline is represented by a sequence of instructions for the computing backend describing how to run data processing modules and pass datasets between them (from the output of one module to the input of another one), including raw data and final computational results obtained in graphical or binary formats. Using relational database for storing descriptions of processing pipelines used in the “Climate” system provides flexibility and efficiency while adding and developing spatial data processing modules. It also provides computing pipelines scaling for further implementation for multiprocessor systems.

References

Gordov, E., Shiklomanov, A., Okladnikov, I., et al.: Development of Distributed Research Center for analysis of regional climatic and environmental changes. IOP Conference Series: Earth and Environmental Science 48, 012033 (2016). https://doi.org/10.1088/1755-1315/48/1/012033

Okladnikov, I.G., Gordov, E.P., Titov, A.G.: Development of climate data storage and processing model. IOP Conference Series: Earth and Environmental Science 48, 012030 (2016). https://doi.org/10.1088/1755-1315/48/1/012030

Okladnikov, I.G.: Computing core of a software package for “cloud” analysis of climate change and the environment. IOP Conference Series: Earth and Environmental Science 611, 012058 (2020). https://doi.org/10.1088/1755-1315/611/1/012058

Swamy, M.N.S., Thulasiraman, K.: Graphs, Networks, and Algorithms. Wiley-Blackwell, New York (1980)

Allamanis, M., Brockschmidt, M., Khademi, M.: Learning to represent programs with graphs. International Conference on Learning Representations (ICLR) (2018). https://openreview.net/pdf?id=BJOFETxR-

Chang, C.L.: Interpretation and execution of fuzzy programs. In: Zadeh, L.A., et al. (eds.) Fuzzy Sets and Their Applications to Cognitive and Decision Process, pp. 191–218. Academic Press, New York (1975)

Averkin, A.N., Batyrshin, I.Z., Blishun, A.F., et al.: Fuzzy sets in models of control and artificial intelligence. Nauka, Moscow (1986)

Li, Yu., Tarlow, D., Brockschmidt, M., Zemel, R.: Gated graph sequence neural networks. International Conference on Learning Representations (ICLR) (2015). https://arxiv.org/pdf/1511.05493.pdf

Balakrishnan, V.K.: Graph Theory. McGraw-Hill (1997)

Halpin, T.: Entity Relationship modeling from an ORM perspective: Part 1. Object Role Modeling. http://www.orm.net/pdf/JCM11.pdf, accessed: 2020-02-25

Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms (3rd ed.). MIT Press and McGraw-Hill, Cambridge (2009)

Dee, D.P., Uppala, S.M., Simmons, A.J., et al.: The ERAInterim reanalysis: Configuration and performance of the data assimilation system. Quarterly Journal of the royal meteorological society 137(656), 553–597 (2011). https://doi.org/10.1002/qj.828