Performance of fully-coupled algebraic multigrid preconditioners for large-scale VMS resistive MHD. (English) Zbl 1443.76152

Summary: This work explores the current performance and scaling of a fully-implicit stabilized unstructured finite element (FE) variational multiscale (VMS) capability for large-scale simulations of 3D incompressible resistive magnetohydrodynamics (MHD). The large-scale linear systems that are generated by a Newton nonlinear solver approach are iteratively solved by preconditioned Krylov subspace methods. The efficiency of this approach is critically dependent on the scalability and performance of the algebraic multigrid preconditioner. This study considers the performance of the numerical methods as recently implemented in the second-generation Trilinos implementation that is 64-bit compliant and is not limited by the 32-bit global identifiers of the original Epetra-based Trilinos. The study presents representative results for a Poisson problem on 1.6 million cores of an IBM Blue Gene/Q platform to demonstrate very large-scale parallel execution. Additionally, results for a more challenging steady-state MHD generator and a transient solution of a benchmark MHD turbulence calculation for the full resistive MHD system are also presented. These results are obtained on up to 131,000 cores of a Cray XC40 and one million cores of a BG/Q system.


76M10 Finite element methods applied to problems in fluid mechanics
65M55 Multigrid methods; domain decomposition for initial value and initial-boundary value problems involving PDEs
76W05 Magnetohydrodynamics and electrohydrodynamics
65F08 Preconditioners for iterative methods
65Y05 Parallel numerical computation
Full Text: DOI Link


[1] Shadid, J. N.; Pawlowski, R. P.; Banks, J. W.; Chacón, L.; Lin, P. T.; Tuminaro, R. S., Towards a scalable fully-implicit fully-coupled resistive MHD formulation with stabilized FE methods, J. Comput. Phys., 229, 20, 7649-7671, (2010) · Zbl 1425.76312
[2] Lin, P. T.; Shadid, J. N.; Tuminaro, R. S.; Sala, M.; Hennigan, G. L.; Pawlowski, R. P., A parallel fully-coupled algebraic multilevel preconditioner applied to multiphysics PDE applications: drift-diffusion, flow/transport/reaction, resistive MHD, Internat. J. Numer. Methods Fluids, 64, 1148-1179, (2010) · Zbl 1427.65036
[3] Shadid, J.; Pawlowski, R.; Cyr, E.; Tuminaro, R.; Chacón, L.; Weber, P., Scalable implicit incompressible resistive MHD with stabilized FE and fully-coupled Newton-Krylov-AMG, Comput. Methods Appl. Mech. Engrg., 304, 1-25, (2016)
[4] M. Heroux, R. Bartlett, V. Howle, R. Hoekstra, J. Hu, T. Kolda, R. Lehoucq, K. Long, R. Pawlowski, E. Phipps, A. Salinger, H. Thornquist, R. Tuminaro, J. Willenbring, A. Williams, An Overview of Trilinos, Tech. Rep. SAND2003-2927, Sandia National Laboratories, 2003.
[5] Edwards, H. C.; Sunderland, D.; Porter, V.; Amsler, C.; Mish, S., Manycore performance portability: kokkos multidimensional array library, Sci. Program., 20, 2, 89-114, (2012)
[6] Balay, S.; Gropp, W. D.; McInnes, L. C.; Smith, B. F., Efficient management of parallelism in object oriented numerical software libraries, (Arge, E.; Bruaset, A. M.; Langtangen, H. P., Modern Software Tools in Scientific Computing, (1997), Birkhäuser Press), 163-202 · Zbl 0882.65154
[7] Baker, A.; Falgout, R.; Gamblin, T.; Kolev, T.; Schulz, M.; Yang, U., Scaling algebraic multigrid solvers: on the road to exascale, (Bischof, C.; Hegering, H.; Nagel, W.; Wittum, G., Competence in High Performance Computing 2010, (2011), Sringer), 215-226
[8] Badia, S.; Martín, A.; Principe, J., Multilevel balancing domain decomposition at extreme scales, SIAM J. Sci. Stat., 38, C22-C52, (2016) · Zbl 1334.65217
[9] Chacón, L., An optimal, parallel, fully implicit Newton-Krylov solver for three-dimensional viscoresistive magnetohydrodynamics, Phys. Plasmas, 15, 5, 056103, (2008)
[10] Goedbloed, H.; Poedts, S., Principles of Magnetohydrodynamics with Applications to Laboratory and Astrophysical Plasmas, (2004), Cambridge Univ. Press
[11] Hughes, T., Multiscale phenomena: green’s functions, the Dirichlet-to-Neumann formulation, subgrid scale models, bubbles and the origins of stabilized methods, Comput. Methods Appl. Mech. Engrg., 127, 387-401, (1995) · Zbl 0866.76044
[12] Hughes, T.; Feijoo, G.; Mazzei, L.; Quincy, J., The variational multiscale method: A paradigm for computational mechanics, Comput. Methods Appl. Mech. Engrg., 166, 3-24, (1998) · Zbl 1017.65525
[13] Shadid, J.; Tuminaro, R.; Devine, K.; Hennigan, G.; Lin, P., Performance of fully-coupled domain decomposition preconditioners for finite element transport/reaction simulations, J. Comput. Phys., 205, 1, 24-47, (2005) · Zbl 1087.76069
[14] Brown, P. N.; Saad, Y., Hybrid Krylov methods for nonlinear systems of equations, SIAM J. Sci. Stat. Comput., 11, 450-481, (1990) · Zbl 0708.65049
[15] Knoll, D. A.; Keyes, D. E., Jacobian-free Newton-Krylov methods: a survey of approaches and applications, J. Comput. Phys., 193, 357-397, (2004) · Zbl 1036.65045
[16] Benzi, M., Preconditioning techniques for large linear systems: a survey, J. Comput. Phys., 182, 418-477, (2002) · Zbl 1015.65018
[17] Quarteroni, A.; Valli, A., Domain Decomposition Methods for Partial Differential Equations, (1999), Oxford University Press · Zbl 0931.65118
[18] Smith, B.; Bjorstad, P.; Gropp, W., Domain Decomposition: Parallel Multilevel Methods for Elliptic Partial Differential Equations, (1996), Cambridge University Press · Zbl 0857.65126
[19] Trottenberg, U.; Oosterlee, C.; Schüller, A., Multigrid, (2001), Academic Press London
[20] Briggs, W.; Henson, V.; McCormick, S., A Multigrid Tutorial, (2000), SIAM Philadelphia
[21] Lin, P. T.; Sala, M.; Shadid, J. N.; Tuminaro, R. S., Performance of fully coupled algebraic multilevel domain decomposition preconditioners for incompressible flow and transport, Internat. J. Numer. Methods Engrg., 67, 2, 208-225, (2006) · Zbl 1110.76315
[22] Lin, P. T.; Shadid, J. N.; Sala, M.; Tuminaro, R. S.; Hennigan, G. L.; Hoekstra, R. J., Performance of a parallel algebraic multilevel preconditioner for stabilized finite element semiconductor device modeling, J. Comput. Phys., 228, 6250-6267, (2009) · Zbl 1175.82077
[23] Lin, P. T., Improving multigrid performance for unstructured mesh drift-diffusion simulations on 147,000 cores, Internat. J. Numer. Methods Engrg., 91, 971-989, (2012)
[24] A. Prokopenko, J. Hu, T. Wiesner, C. Siefert, R. Tuminaro, MueLu User’s Guide 1.0, Tech. Rep. SAND2014-18874, Sandia National Laboratories, 2014.
[25] M. Gee, C. Siefert, J. Hu, R. Tuminaro, M. Sala, ML 5.0 Smoothed Aggregation User’s Guide, Tech. Rep. SAND2006-2649, Sandia National Laboratories, 2006.
[26] Bavier, E.; Hoemmen, M.; Rajamanickam, S.; Thornquist, H., Amesos2 and belos: direct and iterative solvers for large sparse linear systems, Sci. Program., 20, 3, 241-255, (2012)
[27] E. Boman, K. Devine, V. Leung, S. Rajamanickam, L.A. Riesen, M. Deveci, Ü. Çatalyürek, Zoltan2: Next-generation combinatorial toolkit, Tech. rep., Sandia National Laboratories, 2012.
[28] Deveci, M.; Rajamanickam, S.; Devine, K.; Çatalyürek, Ü., Multi-jagged: A scalable parallel spatial partitioning algorithm, IEEE Trans. Parallel Distrib. Syst., 27, 803-817, (2016)
[29] Baker, C.; Heroux, M., Tpetra and the use of generic programming in scientific computing, Sci. Program., 20, 2, 115-128, (2012)
[30] Ballard, G.; Siefert, C.; Hu, J., Reducing communication costs for sparse matrix multiplication within algebraic multigrid, SIAM J. Sci. Comput., 38, 3, C203-C231, (2016) · Zbl 1339.65058
[31] Thakur, R.; Gropp, W., Improving the performance of collective operations in MPICH, (Dongarra, J.; Laforenza, D.; Orlando, S., Recent Advances in Parallel Virtual Machine and Message Passing Interface. EuroPVM/MPI 2003, Lecture Notes in Computer Science, vol. 2840, (2003), Springer), 257-267
[32] Rudi, J.; Malossi, A.; Isaac, T.; Stadler, G.; Gurnis, M.; Staar, P.; Ineichen, Y.; Bekas, C.; Curioni, A.; Ghattas, O., An extreme-scale implicit solver for complex PDEs: highly heterogeneous flow in earth’s mantle, (SC ’15 Proc. of the Inter. Conf. for High Perf. Computing, Networking, Storage and Analysis, (2015), ACM)
[33] Lin, P.; Bettencourt, M.; Domino, S.; Fisher, T.; Hoemmen, M.; Hu, J.; Phipps, E.; Prokopenko, A.; Rajamanickam, S.; Siefert, C.; Kennon, S., Towards extreme-scale simulations for low Mach fluids with second-generation trilinos, Parallel Process. Lett., 24, 4, 1-20, (2014)
[34] Sala, M.; Tuminaro, R. S., A new Petrov-Galerkin smoothed aggregation preconditioner for nonsymmetric linear systems, SIAM J. Sci. Stat., 31, 143-166, (2008) · Zbl 1183.76673
[35] Pouquet, A.; Lee, E.; Brachet, M.; Mininni, P.; Rosenberg, D., The dynamics of unforced turbulence at high Reynolds number for Taylor-Green vortices generalized to MHD, Geophys. Astrophys. Fluid Dyn., 104, 115-134, (2010)
[36] Lee, E.; Brachet, M.; Pouquet, A.; Mininni, P.; Rosenberg, D., Lack of universality in decaying magnetohydrodynamic turbulence, Phys. Rev. E, 81, 016318, (2010)
[37] Sondak, D.; Shadid, J.; Oberai, A.; Pawlowski, R.; Cyr, E.; Smith, T., A new class of finite element variational multiscale turbulence models for incompressible magnetohydrodynamics, J. Comput. Phys., 295, 596-616, (2015) · Zbl 1349.76140
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.