EdgeCFD: a parallel residual-based variational multiscale code for multiphysics. (English) Zbl 07474517

Summary: High fidelity multiphysics simulations are ubiquitous in science and engineering but still face many challenges to run efficiently in today’s supercomputers. This work reports advanced technologies present in EdgeCFD, a hybrid parallel variational multiscale multiphysics finite element software capable of running industrial problems involving turbulent incompressible, compressible, free-surface flows, advection-diffusion of multiple scalar fields, fluid-object interaction using an Arbitrary Lagrangian-Eulerian formulation, and, more recently, used to supporting uncertainty quantification. We also show the strategies to support the solution of large scale problems, including in-situ visualisation and how EdgeCFD handles computational steering. Industrial applications in the oil and gas industry are shown to demonstrate EdgeCFD capabilities. We present the analysis of vortex-induced vibration in cylindrical risers with appendages to reduce drag and lift, and the simulation of successive discharges of turbidity currents, a critical process for understanding the formation of oil reservoirs.


76-XX Fluid mechanics
74-XX Mechanics of deformable solids
Full Text: DOI


[1] Arndt, D.; Bangerth, W.; Clevenger, T. C.; Davydov, D.; Fehling, M.; Garcia-Sanchez, D.; Harper, G., The deal.II Library, Version 9.1, Journal of Numerical Mathematics, 27, 4, 203-213 (2019)
[2] Ayachit, U., Bauer, A., Geveci, B., O’Leary, P., Moreland, K., Fabian, N., and Mauldin, J.. 2015. “Paraview Catalyst: Enabling In Situ Data Analysis and Visualization.” In Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization, ISAV2015, 25-29. New York, NY: ACM.
[3] Bazilevs, Y.; Calo, V. M.; Cottrell, J. A.; Hughes, T. J.; Reali, A.; Scovazzi, G., Variational Multiscale Residual-based Turbulence Modeling for Large Eddy Simulation of Incompressible Flows, Computer Methods in Applied Mechanics and Engineering, 197, 1, 173-201 (2007)
[4] Bey, J., Simplicial Grid Refinement: On Freudenthal’s Algorithm and the Optimal Number of Congruence Classes, Numerische Mathematik, 85, 1-29 (2000)
[5] Camata, J. J.; Silva, V.; Valduriez, P.; Mattoso, M.; Coutinho, A. L., In Situ Visualization and Data Analysis for Turbidity Currents Simulation, Computers & Geosciences, 110, 23-31 (2018)
[6] Casagrande, M. V.; Alves, J. L.; Silva, C. E.; Alves, F. T.; Elias, R. N.; Coutinho, A. L., A Hybrid Fem-dem Approach to the Simulation of Fluid Flow Laden with Many Particles, Computational Particle Mechanics, 4, 2, 213-227 (2017)
[7] Choi, J. Y., Chang, C.-S., Dominski, J., Klasky, S., Merlo, G., Suchyta, E., Ainsworth, M., et al. 2018. “Coupling Exascale Multiphysics Applications: Methods and Lessons Learned.” In 2018 IEEE 14th International Conference on e-Science (e-Science), 442-452. IEEE.
[8] Christon, M. A., A Domain-Decomposition Message-Passing Approach to Transient Viscous Incompressible Flow Using Explicit Time Integration, Computer Methods in Applied Mechanics and Engineering, 148, 3, 329-352 (1997)
[9] Codina, R., Badia, S., Baiges, J., and Principe, J.. 2017. “Variational Multiscale Methods in Computational Fluid Dynamics.” In Encyclopedia of Computational Mechanics, 2nd ed. John Wiley & Sons.
[10] Coutinho, A. L.; Alves, J. L.; Ebecken, N. F., A Study of Implementation Schemes for Vectorized Sparse EBE Matrix-Vector Multiplication, Advances in Engineering Software and Workstations, 13, 3, 130-134 (1991)
[11] Dalcin, L.; Collier, N.; Vignal, P.; Côrtes, A.; Calo, V., PetIGA: A Framework for High-Performance Isogeometric Analysis, Computer Methods in Applied Mechanics and Engineering, 308, 151-181 (2016)
[12] D’Amico, M., Garcia-Gasulla, M., López, V., Jokanovic, A., Sirvent, R., and Corbalan, J.. 2018. “Drom: Enabling Efficient and Effortless Malleability for Resource Managers.” In Proceedings of the 47th International Conference on Parallel Processing Companion, 1-10.
[13] Elias, R. N.; Coutinho, A. L. G. A., Stabilized Edge-Based Finite Element Simulation of Free-Surface Flows, International Journal for Numerical Methods in Fluids, 54, 6-8, 965-993 (2007)
[14] Elias, R. N., Martins, M. A. D., and Coutinho, A. L. G. A.. 2005. In Parallel Edge-Based Inexact Newton Solution of Steady Incompressible 3D Navier-Stokes Equations, Chapter 1, 1237-1245. Berlin: Springer.
[15] Elias, R. N.; Martins, M. A.; Coutinho, A. L., Simple Finite Element-Based Computation of Distance Functions in Unstructured Grids, International Journal for Numerical Methods in Engineering, 72, 9, 1095-1110 (2007)
[16] Elias, R. N.; Paraizo, P. L.; Coutinho, A. L., Stabilized Edge-Based Finite Element Computation of Gravity Currents in Lock-Exchange Configurations, International Journal for Numerical Methods in Fluids, 57, 9, 1137-1152 (2008)
[17] Fluid & Dynamics Research Group. 2017. Experimental Investigation of VIV on Risers Fitted with Helicoidal Segmented Strakes. Technical Report. Universidade de São Paulo (Private Communication).
[18] Francisco, E.; Espath, L.; Laizet, S.; Silvestrini, J., Reynolds Number and Settling Velocity Influence for Finite-Release Particle-Laden Gravity Currents in a Basin, Computers & Geosciences, 110 (2017)
[19] Garcia-Gasulla, M.; Houzeaux, G.; Ferrer, R.; Artigues, A.; López, V.; Labarta, J.; Vázquez, M., MPI+X: Task-Based Parallelisation and Dynamic Load Balance of Finite Element Assembly, International Journal of Computational Fluid Dynamics, 33, 3, 115-136 (2019)
[20] Gaston, D.; Newman, C.; Hansen, G.; Lebrun-Grandié, D., MOOSE: A Parallel Computational Framework for Coupled Systems of Nonlinear Equations, Nuclear Engineering and Design, 239, 10, 1768-1778 (2009)
[21] Geuzaine, C.; Remacle, J.-F., Gmsh: A 3-D Finite Element Mesh Generator with Built-in Pre- and Post-Processing Facilities, International Journal for Numerical Methods in Engineering, 79, 11, 1309-1331 (2009)
[22] Guerra, G. M.; Zio, S.; Camata, J. J.; Dias, J.; Elias, R. N.; Mattoso, M.; Paraizo, P. L.; Coutinho, A. L.; Rochinha, F. A., Uncertainty Quantification in Numerical Simulation of Particle-Laden Flows, Computational Geosciences, 20, 1, 265-281 (2016)
[23] Guerra, G. M.; Zio, S.; Camata, J. J.; Rochinha, F. A.; Elias, R. N.; Paraizo, P. L.; Coutinho, A. L., Numerical Simulation of Particle-Laden Flows by the Residual-Based Variational Multiscale Method, International Journal for Numerical Methods in Fluids, 73, 8, 729-749 (2013)
[24] Hecht, F., New Development in Freefem++, Journal of Numerical Mathematics, 20, 3-4, 251-265 (2012)
[25] Houzeaux, G., Borrell, R., Fournier, Y., Garcia-Gasulla, M., Göbbert, J. H., Hachem, E., Mehta, V., Mesri, Y., Owen, H., and Vázquez, M.. 2018. “High-Performance Computing: Dos and Don”ts.” In Computational Fluid Dynamics – Basic Instruments and Applications in Science.
[26] Hughes, T. J. R.; Ferencz, R. M.; Hallquist, J. O., Large-scale Vectorized Implicit Calculations in Solid Mechanics on a Cray X-MP/48 Utilizing EBE Preconditioned Conjugate Gradients, Computer Methods in Applied Mechanics and Engineering, 61, 2, 215-248 (1987)
[27] Hughes, T. J. R., Scovazzi, G., and Franca, L. P.. 2017. “Multiscale and Stabilized Methods.” In Encyclopedia of Computational Mechanics, 2nd ed. John Wiley & Sons.
[28] Johan, Z.; Hughes, T. J.; Mathur, K. K.; Johnsson, S., A Data Parallel Finite Element Method for Computational Fluid Dynamics on the Connection Machine System, Computer Methods in Applied Mechanics and Engineering, 99, 1, 113-134 (1992)
[29] Karanam, A. K.; Jansen, K. E.; Whiting, C. H., Geometry Based Pre-Processor for Parallel Fluid Dynamic Simulations Using a Hierarchical Basis, Engineering with Computers, 24, 17-26 (2008)
[30] Karypis, G., and Kumar, V.. 1998. “METIS* A Software Package for Partitioning Unstructured Graphs, Partitioning Meshes, and Computing Fill-Reducing Orderings of Sparse Matrices.” In Manual.
[31] Keyes, D. E.; McInnes, L. C.; Woodward, C.; Gropp, W.; Myra, E.; Pernice, M.; Bell, J., Multiphysics Simulations: Challenges and Opportunities, The International Journal of High Performance Computing Applications, 27, 1, 4-83 (2013)
[32] Kirk, B. S.; Peterson, J. W.; Stogner, R. H.; Carey, G. F., libMesh: A C++ Library for Parallel Adaptive Mesh Refinement/Coarsening Simulations, Engineering with Computers, 22, 3-4, 237-254 (2006)
[33] Klasky, S., Suchyta, E., Ainsworth, M., Liu, Q., Whitney, B., Wolf, M., Choi, J., et al. 2017. “Exacution: Enhancing Scientific Data Management for Exascale.” In 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), 1927-1937. IEEE.
[34] Korkischko, I.; Meneghini, J. R., Experimental Investigation of Flow-Induced Vibration on Isolated and Tandem Circular Cylinders Fitted with Strakes, Journal of Fluids and Structures, 26, 4, 611-625 (2010)
[35] Korkischko, I.; Meneghini, J. R., Volumetric Reconstruction of the Mean Flow Around Circular Cylinders Fitted with Strakes, Experiments in Fluids, 51, 4, 1109-1122 (2011)
[36] Lins, E. F.; Elias, R. N.; Guerra, G. M.; Rochinha, F. A.; Coutinho, A. L. G. A., Edge-Based Finite Element Implementation of the Residual-Based Variational Multiscale Method, International Journal for Numerical Methods in Fluids, 61, 1, 1-22 (2009)
[37] Lins, E. F.; Elias, R. N.; Rochinha, F. A.; Coutinho, A. L. G. A., Residual-Based Variational Multiscale Simulation of Free Surface Flows, Computational Mechanics, 46, 4, 545-557 (2010)
[38] Logg, A.; Mardal, K.-A.; Wells, G. N., Automated Solution of Differential Equations by the Finite Element Method (2012), Springer
[39] Löhner, R., Applied Computational Fluid Dynamics Techniques: An Introduction Based on Finite Element Methods (2008), Chichester: John Wiley & Sons, Chichester
[40] Löhner, R., Yang, C., Cebral, J., Soto, O., Camelli, F., Baum, J. D., Luo, H., et al. 2001. “Advances in FEFLO.” In 39th Aerospace Sciences Meeting and Exhibit.
[41] Martins, M., Elias, R., and Coutinho, A.. 2006. “Edgepack: A Parallel Vertex and Node Reordering Package for Optimizing Edge-Based Computations in Unstructured Grids.” In International Conference on High Performance Computing for Computational Science, 292-304. Springer.
[42] Meiburg, E.; Kneller, B., Turbidity Currents and Their Deposits, Annual Review of Fluid Mechanics, 42, 1, 135-156 (2010)
[43] Miras, T.; Camata, J. J.; Elias, R. N.; Alves, J. L. D.; Rochinha, F. A.; Coutinho, A. L. G. A., A Staggered Procedure for Fluid-Object Interaction with Free Surfaces, Large Rotations and Driven by Adaptive Time Stepping, Journal of the Brazilian Society of Mechanical Sciences and Engineering, 40, 4, 239 (2018)
[44] Moreland, K., Ayachit, U., Geveci, B., Quammen, C., Demarle, D., Moreland, K., Bauer, A., et al. 2018. The ParaView Guide. Sandia National Laboratories.
[45] Ribeiro, F. L.; Coutinho, A. L., Comparison Between Element, Edge and Compressed Storage Schemes for Iterative Solutions in Finite Element Analyses, International Journal for Numerical Methods in Engineering, 63, 4, 569-588 (2005)
[46] Saad, Y.2003. Iterative Methods for Sparse Linear Systems, Vol. 82. SIAM.
[47] Sagaut, P.2006. “Large Eddy Simulation for Incompressible Flows: An Introduction.” In Scientific Computation. Berlin: Springer.
[48] Sahni, O., Zhou, M., Shephard, M. S., and Jansen, K. E.. 2009. “Scalable Implicit Finite Element Solver for Massively Parallel Processing with Demonstration to 160k Cores.” In Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, 1-12. IEEE.
[49] Silva, V.; Leite, J.; Camata, J. J.; de Oliveira, D.; Coutinho, A. L.; Valduriez, P.; Mattoso, M., Raw Data Queries During Data-Intensive Parallel Workflow Execution, Future Generation Computer Systems, 75, 402-422 (2017)
[50] Silva, R. M., Lima, B. S., Camata, J. J., Elias, R. N., and Coutinho, A. L.. 2019. “Communication-Free Parallel Mesh Multiplication for Large Scale Simulations.” In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics).
[51] Tezduyar, T.; Aliabadi, S.; Behr, M.; Johnson, A.; Mittal, S., Parallel Finite-Element Computation of 3d Flows, Computer, 26, 10, 27-36 (1993)
[52] Valli, A. M.; Carey, G. F.; Coutinho, A. L., Control Strategies for Timestep Selection in Finite Element Simulation of Incompressible Flows and Coupled Reaction-Convection-Diffusion Processes, International Journal for Numerical Methods in Fluids, 47, 3, 201-231 (2005)
[53] Vázquez, M.; Houzeaux, G.; Koric, S.; Artigues, A.; Aguado-Sierra, J.; Arís, R.; Mira, D., Alya: Multiphysics Engineering Simulation Toward Exascale, Journal of Computational Science, 14, 15-27 (2016)
[54] Zio, S.; da Costa, H. F.; Guerra, G. M.; Paraizo, P. L.; Camata, J. J.; Elias, R. N.; Coutinho, A. L.; Rochinha, F. A., Bayesian Assessment of Uncertainty in Viscosity Closure Models for Turbidity Currents Computations, Computer Methods in Applied Mechanics and Engineering, 342, 653-673 (2018)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.