Scientific Big Data Visualization: a coupled tools approach

Antoni Artigues, Fernando Martin Cucchietti, Carlos Tripiana Montes, David Vicente, Hadrien Calmet, Guillermo Marin, Guillaume Houzeaux, Mariano Vazquez

Abstract


We designed and implemented a parallel visualisation system for the analysis of large scale time-dependent particle type data. The particular challenge we address is how to analyse a high perfor- mance computation style dataset when a visual representation of the full set is not possible or useful, and one is only interested in finding and inspecting smaller subsets that fulfil certain complex criteria. We used Paraview as the user interface, which is a familiar tool for many HPC users, runs in parallel, and can be conveniently extended. We distributed the data in a supercomputing environment using the Hadoop file system. On top of it, we run Hive or Impala, and implemented a connection between Paraview and them that al- lows us to launch programmable SQL queries in the database di- rectly from within Paraview. The queries return a Paraview-native VTK object that fits directly into the Paraview pipeline. We find good scalability and response times. In the typical supercomputer environment (like the one we used for implementation) the queue and management system make it difficult to keep local data in be- tween sessions, which imposes a bottleneck in the data loading stage. This makes our system most useful when permanently in- stalled on a dedicated cluster.

Full Text:

PDF

References


H. Childs, E. Brugger, K. Bonnell, J. Meredith, M. Miller, B. Whitlock, and N. Maxi. A contract based system for large data visualization. In Visualiza- tion, 2005. VIS 05. IEEE, 2005.

Christopher Mitchell, James Ahrens, and Jun Wang. Visio: Enabling in- teractive visualization of ultra-scale, time series data via high-bandwidth dis- tributed i/o systems. In Parallel Distributed Processing Symposium (IPDPS), 2011 IEEE International, 2011.

Tableau software. http://www.tableausoftware.com/. Tableau is developed by Tableau Software. http://www.tableausoftware. com.

Alya system - large scale computational mechanics. http://www.bsc. es/computer-applications/alya-system.

Paraview software. http://www.paraview.org. Paraview is devel- oped by Kitware company. http://www.kitware.com.

VTK api. http://www.vtk.org. VTK is developed by Kitware com- pany. http://www.kitware.com.

Hadoop. http://hadoop.apache.org/.

Yang Hung-Chih, Ali Dasdan, Hsiao Ruey-Lung, and D. Stott Parker. Map- reduce-merge simplified relational data processing on large clusters. In Proc. of ACM SIGMOD, 2007.

Hive database engine. http://hive.apache.org/.

Information technology database languages sql. http://www. iso.org/iso/iso_catalogue/catalogue_tc/catalogue_ detail.htm?csnumber=53681. ISO/IEC 9075-1:2011 standard.

Impala database engine. http://impala.io. Impala is developed by Cloudera company. http://www.cloudera.com.

G. Houzeaux, R. Aubry, and M. Va ́zquez. Extension of fractional step tech- niques for incompressible flows: The preconditioned orthomin(1) for the pressure schur complement. Computers Fluids, 44:297–313, 2011.

G. Houzeaux, R. de la Cruz, H. Owen, and M. Va ́zquez. Parallel uniform mesh multiplication applied to a navier-stokes solver. Computers & Fluids, 80:142–151, 7 2013.

G. Houzeaux, H. Owen, B. Eguzkitza, C. Samaniego, R. de la Cruz, H. Cal- met,M.Va ́zquez,andM.A ́vila.DevelopmentsinParallel,Distributed,Grid and Cloud Computing for Engineering, volume 31 of Computational Sci- ence, Engineering and Technology Series, chapter Chapter 8: A Parallel In- compressible Navier-Stokes Solver: Implementation Issues, pages 171–201. Saxe-Coburg Publications, b.h.v. topping and p. iva ́nyi (Editor) edition,

M.Va ́zquez,G.Houzeaux,S.Koric,A.Artigues,J.Aguado-Sierra,R.Ar ́ıs, D. Mira, H. Calmet, F. Cucchietti, H. Owen, A. Taha, and J.M. Cela. Alya: Towards exascale for engineering simulation codes. arXiv.org, 2014.

G. Houzeaux. Numerical simulation of air flow in the human large airways. PRACE 4th regular call. 20m core hours on tier-0, Fermi (I)., 2012.

Supercomputer description. http://en.wikipedia.org/wiki/ Supercomputer.

Minotauro - nvidia gpu cluster. http://www.bsc.es/ marenostrum-support-services/other-hpc-facilities/ nvidia-gpu-cluster.




Publishing Center of South Ural State University (454080, Lenin prospekt, 76, Chelyabinsk, Russia)