FEMPAR: an object-oriented parallel finite element framework. (English) Zbl 1392.65005

Summary: FEMPAR is an open source object oriented Fortran200X scientific software library for the high-performance scalable simulation of complex multiphysics problems governed by partial differential equations at large scales, by exploiting state-of-the-art supercomputing resources. It is a highly modularized, flexible, and extensible library, that provides a set of modules that can be combined to carry out the different steps of the simulation pipeline. FEMPAR includes a rich set of algorithms for the discretization step, namely (arbitrary-order) grad, div, and curl-conforming finite element methods, discontinuous Galerkin methods, B-splines, and unfitted finite element techniques on cut cells, combined with \(h\)-adaptivity. The linear solver module relies on state-of-the-art bulk-asynchronous implementations of multilevel domain decomposition solvers for the different discretization alternatives and block-preconditioning techniques for multiphysics problems. FEMPAR is a framework that provides users with out-of-the-box state-of-the-art discretization techniques and highly scalable solvers for the simulation of complex applications, hiding the dramatic complexity of the underlying algorithms. But it is also a framework for researchers that want to experience with new algorithms and solvers, by providing a highly extensible framework. In this work, the first one in a series of articles about FEMPAR, we provide a detailed introduction to the software abstractions used in the discretization module and the related geometrical module. We also provide some ingredients about the assembly of linear systems arising from finite element discretizations, but the software design of complex scalable multilevel solvers is postponed to a subsequent work.


65-04 Software, source code, etc. for problems pertaining to numerical analysis
65N30 Finite element, Rayleigh-Ritz and Galerkin methods for boundary value problems involving PDEs
65Y05 Parallel numerical computation
65Y15 Packaged methods for numerical algorithms
Full Text: DOI arXiv


[1] Guo, B; Babuska, I, The h-p version of the finite element method, Comput Mech, 1, 21-41, (1986) · Zbl 0634.73058
[2] Ainsworth M, Oden JT (2011) A posteriori error estimation in finite element analysis. Wiley, New York · Zbl 0895.76040
[3] Melenk, JM; Wohlmuth, BI, On residual-based a posteriori error estimation in hp-FEM, Adv Comput Math, 15, 311-331, (2001) · Zbl 0991.65111
[4] Nedelec, JC, Mixed finite elements in \(\cal{R}^3\), Numer Math, 35, 315-341, (1980) · Zbl 0419.65069
[5] Brezzi F, Fortin M (1991) Mixed and hybrid finite element methods. Springer, Berlin · Zbl 0788.73002
[6] Arnold, DN; Falk, RS; Winther, R, Finite element exterior calculus, homological techniques, and applications, Acta Numer, 15, 1-155, (2006) · Zbl 1185.65204
[7] Neilan, M; Sap, D, Stokes elements on cubic meshes yielding divergence-free approximations, Calcolo, 53, 263-283, (2016) · Zbl 1388.76148
[8] Hughes, TJR; Cottrell, JA; Bazilevs, Y, Isogeometric analysis: CAD, finite elements, NURBS, exact geometry and mesh refinement, Comput Methods Appl Mech Eng, 194, 4135-4195, (2005) · Zbl 1151.74419
[9] Cockburn, B; Gopalakrishnan, J; Lazarov, R, Unified hybridization of discontinuous Galerkin, mixed, and continuous Galerkin methods for second order elliptic problems, SIAM J Numer Anal, 47, 1319-1365, (2009) · Zbl 1205.65312
[10] Demkowicz, L; Gopalakrishnan, J, A class of discontinuous Petrov-Galerkin methods. part I: the transport equation, Comput Methods Appl Mech Eng, 199, 1558-1572, (2010) · Zbl 1231.76142
[11] Ang J, Barrett R, Benner R, Burke D, Chan C, Cook J, Donofrio D, Hammond S, Hemmert K, Kelly S, Le H, Leung V, Resnick D, Rodrigues A, Shalf J, Stark D, Unat D, Wright N (2014) Abstract machine models and proxy architectures for exascale computing. In: Co-HPC’14 Proceedings of the 1st international workshop on hardware-software co-design for high performance computing. IEEE, pp 25-32 · Zbl 1364.65237
[12] Gropp W (2015) Is MPI+X enough for exascale? Keynote for international high performance computing forum, Tianjin, China
[13] Kale, LV; Krishnan, S, CHARM++: a portable concurrent object oriented system based on C++, ACM SIGPLAN Not, 28, 91-108, (1993)
[14] Bauer M, Treichler S, Slaughter E, Aiken A (2012) Legion: expressing locality and independence with logical regions. In: Proceedings of the international conference on high performance computing, networking, storage and analysis, SC ’12, Los Alamitos, CA, USA, IEEE Computer Society Press, pp 66:1-66:11 · Zbl 06645426
[15] Kaiser H, Heller T, Adelstein-Lelbach B, Serio A, Fey D (2014) Hpx: A task based programming model in a global address space. In Proceedings of the 8th international conference on partitioned global address space programming models, PGAS ’14, New York, NY, USA, ACM, pp 6:1-6:11 · Zbl 1228.76079
[16] Janine Bennett PI, Robert Clay PM, Baker G, Gamell M, Hollman D, Knight S, Kolla H, Sjaardema G, Slattengren N, Teranishi K et al (2015) ASC ATDM level 2 milestone# 5325: asynchronous many-task runtime system analysis and assessment for next generation platforms. Technical Report SAND2015-8312, Sandia National Laboratories
[17] Bangerth, W; Hartmann, R; Kanschat, G, Deal. II-a general-purpose object-oriented finite element library, ACM Trans Math Softw, 33, 24, (2007) · Zbl 1365.65248
[18] Bangerth, W; Davydov, D; Heister, T; Heltai, L; Kanschat, G; Kronbichler, M; Maier, M; Turcksin, B; Wells, D, The deal. II library, version 8.4, J Numer Math, 24, 135-141, (2016) · Zbl 1348.65187
[19] Alnæs, M; Blechta, J; Hake, J; Johansson, A; Kehlet, B; Logg, A; Richardson, C; Ring, J; Rognes, ME; Wells, GN, The fenics project version 1.5, Arch Numer Softw, 3, 9-23, (2015)
[20] Bauman, P; Stogner, R, GRINS: a multiphysics framework based on the libmesh finite element library, SIAM J Sci Comput, 38, s78-s100, (2016)
[21] Cantwell, CD; Moxey, D; Comerford, A; Bolis, A; Rocco, G; Mengaldo, G; Grazia, D; Yakovlev, S; Lombard, JE; Ekelschot, D; Jordi, B; Xu, H; Mohamied, Y; Eskilsson, C; Nelson, B; Vos, P; Biotto, C; Kirby, RM; Sherwin, SJ, Nektar++: an open-source spectral/element framework, Comput Phys Commun, 192, 205-219, (2015) · Zbl 1380.65465
[22] MOOSE (Multiphysics Object-Oriented Simulation Environment) Framework. http://mooseframework.org/
[23] MFEM—a free, lightweight, scalable C++ library for finite element methods. http://mfem.org/ · Zbl 0634.73058
[24] Hecht, F, New development in freefem++, J Numer Math, 20, 251-265, (2012) · Zbl 1266.68090
[25] Dedner, A; Nolte, M; Dedner, A (ed.); Flemisch, B (ed.); Klöfkorn, R (ed.), Construction of local finite element spaces using the generic reference elements, 3-16, (2012), Berlin
[26] Balay S, Abhyankar S, Adams MF, Brown J, Brune P, Buschelman K, Dalcin L, Eijkhout V, Gropp WD, Kaushik D, Knepley MG, McInnes LC, Rupp K, Smith BF, Zampini S, Zhang H, Zhang H (2016a) PETSc web page
[27] Balay S, Abhyankar S, Adams MF, Brown J, Brune P, Buschelman K, Dalcin L, Eijkhout V, Gropp WD, Kaushik D, Knepley MG, McInnes LC, Rupp K, Smith BF, Zampini S, Zhang H, Zhang H (2016b) PETSc users manual. Technical report ANL-95/11—Revision 3.7, Argonne National Laboratory
[28] Balay, S; Gropp, WD; McInnes, LC; Smith, BF; Arge, E (ed.); Bruaset, AM (ed.); Langtangen, HP (ed.), Efficient management of parallelism in object oriented numerical software libraries, 163-202, (1997), Berlin
[29] Falgout RD, Yang UM (2002) hypre: a library of high performance preconditioners. In: Computational science—ICCS 2002. Springer, Berlin, pp 632-641 · Zbl 1056.65046
[30] Heroux, MA; Bartlett, RA; Howle, VE; Hoekstra, RJ; Hu, JJ; Kolda, TG; Lehoucq, RB; Long, KR; Pawlowski, RP; Phipps, ET; Salinger, AG; Thornquist, HK; Tuminaro, RS; Willenbring, JM; Williams, A; Stanley, KS, An overview of the trilinos project, ACM Trans Math Softw, 31, 397-423, (2005) · Zbl 1136.65354
[31] The Trilinos Project. https://trilinos.org · Zbl 1136.65354
[32] Toselli, A, Dual-primal FETI algorithms for edge finite-element approximations in 3D, IMA J Numer Anal, 26, 96-130, (2006) · Zbl 1088.65102
[33] Dohrmann, CR, An approximate BDDC preconditioner, Numer Linear Algebra Appl, 14, 149-168, (2007) · Zbl 1199.65088
[34] Badia, S; Martín, AF; Principe, J, On the scalability of inexact balancing domain decomposition by constraints with overlapped coarse/fine corrections, Parallel Comput, 50, 1-24, (2015)
[35] Efendiev Y, Hou TY (2009) Multiscale finite element methods: theory and applications. Springer, New York · Zbl 1163.65080
[36] Klawonn, A; Lanser, M; Rheinbach, O, Toward extremely scalable nonlinear domain decomposition methods for elliptic partial differential equations, SIAM J Sci Comput, 37, c667-c696, (2015) · Zbl 1329.65294
[37] Badia S, Martín A, Principe J (2016) Multilevel balancing domain decomposition at extreme scales. SIAM J Sci Comput, pp C22-C52 · Zbl 1334.65217
[38] Zampini, S, PCBDDC: a class of robust dual-primal methods in petsc, SIAM J Sci Comput, 38, s282-s306, (2016) · Zbl 1352.65632
[39] Badia, S; Olm, M, Space-time balancing domain decomposition, SIAM J Sci Comput, 39, c194-c213, (2017) · Zbl 1365.65217
[40] Brune, P; Knepley, M; Smith, B; Tu, X, Composing scalable nonlinear algebraic solvers, SIAM Rev, 57, 535-565, (2015) · Zbl 1336.65030
[41] Falgout, R; Friedhoff, S; Kolev, T; MacLachlan, S; Schroder, J, Parallel time integration with multigrid, SIAM J Sci Comput, 36, c635-c661, (2014) · Zbl 1310.65115
[42] FEMPAR: Finite Element Multiphysics PARallel solvers. https://gitlab.com/fempar/fempar · Zbl 1038.65039
[43] Elman HC, Silvester DJ, Wathen AJ (2005) Finite elements and fast iterative solvers: with applications in incompressible fluid dynamics. Oxford University Press, Oxford · Zbl 1083.76001
[44] Badia, S; Martín, AF; Planas, R, Block recursive LU preconditioners for the thermally coupled incompressible inductionless MHD problem, J Comput Phys, 274, 562-591, (2014) · Zbl 1351.76046
[45] Cyr, E; Shadid, J; Tuminaro, R, Teko: a block preconditioning capability with concrete example applications in Navier-Stokes and MHD, SIAM J Sci Comput, 38, s307-s331, (2016) · Zbl 06645426
[46] Colomés, O; Badia, S; Codina, R; Principe, J, Assessment of variational multiscale models for the large eddy simulation of turbulent incompressible flows, Comput Methods Appl Mech Eng, 285, 32-63, (2015) · Zbl 1423.76152
[47] Colomés, O; Badia, S, Segregated Runge-Kutta methods for the incompressible Navier-Stokes equations, Int J Numer Methods Eng, 105, 372-400, (2016)
[48] Colomés, O; Badia, S; Principe, J, Mixed finite element methods with convection stabilization for the large eddy simulation of incompressible turbulent flows, Comput Methods Appl Mech Eng, 304, 294-318, (2016)
[49] Colomés, O; Badia, S, Segregated Runge-Kutta time integration of convection-stabilized mixed finite element schemes for wall-unresolved LES of incompressible flows, Comput Methods Appl Mech Eng, 313, 189-215, (2017)
[50] Badia, S; Codina, R; Planas, R, On an unconditionally convergent stabilized finite element approximation of resistive magnetohydrodynamics, J Comput Phys, 234, 399-416, (2013) · Zbl 1284.76248
[51] Badia, S; Planas, R; Gutiérrez-Santacreu, JV, Unconditionally stable operator splitting algorithms for the incompressible magnetohydrodynamics system discretized by a stabilized finite element formulation based on projections, Int J Numer Methods Eng, 93, 302-328, (2013) · Zbl 1352.76122
[52] Planas, R; Badia, S; Codina, R, Approximation of the inductionless MHD problem using a stabilized finite element method, J Comput Phys, 230, 2977-2996, (2011) · Zbl 1316.76110
[53] Smolentsev, S; Badia, S; Bhattacharyay, R; Bühler, L; Chen, L; Huang, Q; Jin, HG; Krasnov, D; Lee, DW; Valls, EM; Mistrangelo, C; Munipalli, R; Ni, M-J; Pashkevich, D; Patel, A; Pulugundla, G; Satyamurthy, P; Snegirev, A; Sviridov, V; Swain, P; Zhou, T; Zikanov, O, An approach to verification and validation of MHD codes for fusion applications, Fusion Eng Des, 100, 65-72, (2015)
[54] Badia, S; Codina, R; Planas, R, Analysis of an unconditionally convergent stabilized finite element formulation for incompressible magnetohydrodynamics, Arch Comput Methods Eng, 22, 621-636, (2015) · Zbl 1348.76097
[55] Badia, S; Hierro, A, On discrete maximum principles for discontinuous Galerkin methods, Comput Methods Appl Mech Eng, 286, 107-122, (2015) · Zbl 1423.76205
[56] Badia, S; Hierro, A, On monotonicity-preserving stabilized finite element approximations of transport problems, SIAM J Sci Comput, 36, a2673-a2697, (2014) · Zbl 1312.65143
[57] Hierro, A; Badia, S; Kus, P, Shock capturing techniques for adaptive finite elements, Comput Methods Appl Mech Eng, 309, 532-553, (2016)
[58] Badia, S; Bonilla, J, Monotonicity-preserving finite element schemes based on differentiable nonlinear stabilization, Comput Methods Appl Mech Eng, 313, 133-158, (2017)
[59] Badia, S; Bonilla, J; Hierro, A, Differentiable monotonicity-preserving schemes for discontinuous Galerkin methods on arbitrary meshes, Comput Methods Appl Mech Eng, 320, 582-605, (2017)
[60] Badia S, Verdugo F (2017) Robust and scalable domain decomposition solvers for unfitted finite element methods. arXiv:1703.06323 [math] · Zbl 1462.65216
[61] Chiumenti M, Neiva E, Salsi E, Cervera M, Badia S, Davies C, Chen Z, Lee C (2017) Numerical modelling and experimental validation in selective laser melting (submitted)
[62] Badia, S; Martín, AF; Principe, J, Implementation and scalability analysis of balancing domain decomposition methods, Arch Comput Methods Eng, 20, 239-262, (2013) · Zbl 1354.65261
[63] Badia, S; Martín, AF; Príncipe, J, Enhanced balancing Neumann-Neumann preconditioning in computational fluid and solid mechanics, Int J Numer Meth Eng, 96, 203-230, (2013) · Zbl 1352.74322
[64] Badia, S; Nguyen, H, Balancing domain decomposition by constraints and perturbation, SIAM J Numer Anal, 54, 3436-3464, (2016) · Zbl 1354.65262
[65] Badia S, Martín AF, Nguyen H (2016) Physics-based balancing domain decomposition by constraints for heterogeneous problems. Working paper or preprint
[66] Badia, S; Martín, A; Principe, J, A highly scalable parallel implementation of balancing domain decomposition by constraints, SIAM J Sci Comput, 36, c190-c218, (2014) · Zbl 1296.65177
[67] Adams JC, Brainerd WS, Hendrickson RA, Maine RE, Martin JT, Smith BT (2009) The Fortran 2003 handbook. Springer, London, · Zbl 1176.68042
[68] Rouson D, Xia J, Xu X (2011) Scientific software design: the object-oriented way, 1st edn. Cambridge University Press, New York · Zbl 1228.68005
[69] Ern A, Guermond J-L (2004) Theory and practice of finite elements. Springer, Berlin · Zbl 1059.65103
[70] Brenner SC, Scott R (2010) The mathematical theory of finite element methods. Springer, softcover reprint of hardcover 3rd ed. 2008 edn
[71] Quarteroni A (2014) Numerical models for differential problems. Springer Milan, Milano · Zbl 1285.65054
[72] Monk P (2003) Finite element methods for Maxwell’s equations. Clarendon Press, Oxford · Zbl 1024.78009
[73] Duffy, M, Quadrature over a pyramid or cube of integrands with a singularity at a vertex, SIAM J Numer Anal, 19, 1260-1262, (1982) · Zbl 0493.65011
[74] Dunavant, DA, High degree efficient symmetrical Gaussian quadrature rules for the triangle, Int J Numer Meth Eng, 21, 1129-1148, (1985) · Zbl 0589.65021
[75] Olm M, Badia S, Martín AF (2017) Simulation of high temperature superconductors and experimental validation. arXiv:1707.09783 [physics]
[76] Badia, S; Quaini, A; Quarteroni, A, Modular vs. non-modular preconditioners for fluid-structure systems with large added-mass effect, Comput Methods Appl Mech Eng, 197, 4216-4232, (2008) · Zbl 1194.74058
[77] Badia, S; Nobile, F; Vergara, C, Fluid-structure partitioned procedures based on Robin transmission conditions, J Comput Phys, 227, 7027-7051, (2008) · Zbl 1140.74010
[78] Badia, S; Quaini, A; Quarteroni, A, Splitting methods based on algebraic factorization for fluid-structure interaction, SIAM J Sci Comput, 30, 1778, (2008) · Zbl 1368.74021
[79] Badia, S; Nobile, F; Vergara, C, Robin-Robin preconditioned Krylov methods for fluid-structure interaction problems, Comput Methods Appl Mech Eng, 198, 2768-2784, (2009) · Zbl 1228.76079
[80] Rognes, M; Kirby, R; Logg, A, Efficient assembly of \(H\)(div) and \(H\)(curl) conforming finite elements, SIAM J Sci Comput, 31, 4130-4151, (2009) · Zbl 1206.65248
[81] Agelek R, Anderson M, Bangerth W, Barth W (2017) On orienting edges of unstructured two- and three-dimensional meshes. ACM Trans Math Softw (to appear) · Zbl 1484.65320
[82] Bangerth, W; Kayser-Herold, O, Data structures and requirements for hp finite element software, ACM Trans Math Softw, 36, 1-31, (2009) · Zbl 1364.65237
[83] Badia, S; Baiges, J, Adaptive finite element simulation of incompressible flows by hybrid continuous-discontinuous Galerkin formulations, SIAM J Sci Comput, 35, a491-a516, (2013) · Zbl 1372.76066
[84] Dryja, M; Galvis, J; Sarkis, M, BDDC methods for discontinuous Galerkin discretization of elliptic problems, J Complex, 23, 715-739, (2007) · Zbl 1133.65097
[85] Freeman E, Robson E, Sierra K, Bates B (eds) (2004) Head first design patterns. O’Reilly, Sebastopol
[86] FPL—Fortran parameter list. https://gitlab.com/fempar/FPL
[87] Beall, MW; Shephard, MS, A general topology-based mesh data structure, Int J Numer Meth Eng, 40, 1573-1596, (1997)
[88] Gamma E, Helm R, Johnson R, Vlissides J (1995) Design patterns: elements of reusable object-oriented software. Addison-Wesley, Boston · Zbl 0887.68013
[89] Bangerth, W; Burstedde, C; Heister, T; Kronbichler, M, Algorithms and data structures for massively parallel generic adaptive finite element codes, ACM Trans Math Softw, 38, 14:1-14:28, (2012) · Zbl 1365.65247
[90] Burstedde, C; Wilcox, LC; Ghattas, O, P4est : scalable algorithms for parallel adaptive mesh refinement on forests of octrees, SIAM J Sci Comput, 33, 1103-1133, (2011) · Zbl 1230.65106
[91] GiD—the personal pre and post processor. http://www.gidhome.com
[92] Filippone, S; Buttari, A, Object-oriented techniques for sparse matrix computations in Fortran 2003, ACM Trans Math Softw, 38, 23:1-23:20, (2012) · Zbl 1365.65127
[93] Saad Y (2003) Iterative methods for sparse linear systems, 2nd edn. Society for Industrial and Applied Mathematics · Zbl 1031.65046
[94] FLAP—Fortran command Line Arguments Parser for poor people. https://github.com/szaghi/FLAP
[95] Childs H, Brugger E, Whitlock B, Meredith J, Ahern S, Pugmire D, Biagas K, Miller M, Harrison C, Weber GH, Krishnan H, Fogal T, Sanderson A, Garth C, Bethel EW, Camp D, Rübel O, Durant M, Favre JM, Navrátil P (2012) VisIt: an end-user tool for visualizing and analyzing very large data. In: High performance visualization-enabling extreme-scale scientific insight, pp 357-372 · Zbl 1231.76142
[96] Ayachit U (2015) The paraview guide: a parallel visualization application. Kitware Inc, Clifton Park
[97] Schroeder W, Martin KM, Lorensen WE (1998) The visualization toolkit: an object-oriented approach to 3D graphics, 2nd edn. Prentice-Hall, Inc., Upper Saddle River
[98] Lib_VTK_IO—Pure Fortran (2003+) library to write and read data conforming the VTK standard. https://gitlab.com/fempar/Lib_VTK_IO · Zbl 1185.65204
[99] XDMF—eXtensible data model and format. http://www.xdmf.org/index.php/Main_Page
[100] The HDF Group. Hierarchical data format version 5. http://www.hdfgroup.org/HDF5, 2000-2017
[101] XH5For—XDMF parallel partitioned mesh Input/Output on top of HDF5. https://gitlab.com/fempar/XH5For
[102] Brömmel D, Wylie BJN, Frings W (2015) JUQUEEN extreme scaling workshop 2015. Technical Report FZJ-2015-01645, Jülich Supercomputing Center · Zbl 0419.65069
[103] Dohrmann, CR, A preconditioner for substructuring based on constrained energy minimization, SIAM J Sci Comput, 25, 246-258, (2003) · Zbl 1038.65039
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.