Tusas: a fully implicit parallel approach for coupled phase-field equations. (English) Zbl 07516818

Summary: We develop a fully-coupled, fully-implicit approach for phase-field modeling of solidification in metals and alloys. Predictive simulation of solidification in pure metals and metal alloys remains a significant challenge in the field of materials science, as microstructure formation during the solidification process plays a critical role in the properties and performance of the solid material. Our simulation approach consists of a finite element spatial discretization of the fully-coupled nonlinear system of partial differential equations at the microscale, which is treated implicitly in time with a preconditioned Jacobian-free Newton-Krylov method. The approach is algorithmically scalable as well as efficient due to an effective preconditioning strategy based on algebraic multigrid and block factorization. We implement this approach in the open-source Tusas framework, which is a general, flexible tool developed in C++ for solving coupled systems of nonlinear partial differential equations. The performance of our approach is analyzed in terms of algorithmic scalability and efficiency, while the computational performance of Tusas is presented in terms of parallel scalability and efficiency on emerging heterogeneous architectures. We demonstrate that modern algorithms, discretizations, and computational science, and heterogeneous hardware provide a robust route for predictive phase-field simulation of microstructure evolution during additive manufacturing.


65Fxx Numerical linear algebra
65Hxx Nonlinear algebraic or transcendental equations
76Mxx Basic methods in fluid mechanics
Full Text: DOI


[1] Rappaz, M.; Dantzig, J. A., Solidification, Engineering Sciences (2009), EPFL Press: EPFL Press Lausanne
[2] Kurz, W.; Fisher, D. J., Fundamentals of Solidification (1998), Trans Tech Publications: Trans Tech Publications Zurich
[3] Francois, M. M.; Sun, A.; King, W. E.; Henson, N. J.; Tourret, D.; Bronkhorst, C. A.; Carlson, N. N.; Newman, C. K.; Haut, T. S.; Bakosi, J., Modeling of additive manufacturing processes for metals: challenges and opportunities, Curr. Opin. Solid State Mater. Sci., 21 (2017)
[4] Ghosh, S., Predictive modeling of solidification during laser additive manufacturing of nickel superalloys: recent developments, future directions, Mater. Res. Express, 5, Article 012001 pp. (2018)
[5] Keller, T.; Lindwall, G.; Ghosh, S.; Ma, L.; Lane, B.; Zhang, F.; Kattner, U. R.; Lass, E. A.; Heigel, J. C.; Idell, Y.; Williams, M. E.; Allen, A. J.; Guyer, J. E.; Levine, L. E., Application of finite element, phase-field, and CALPHAD-based methods to additive manufacturing of Ni-based superalloys, Acta Mater., 139, 244-253 (2017)
[6] Ji, Y.; Chen, L.; Chen, L.-Q., Understanding microstructure evolution during additive manufacturing of metallic alloys using phase-field modeling, (Thermo-Mechanical Modeling of Additive Manufacturing (2018), Elsevier: Elsevier NY), 93-116
[7] Acharya, R.; Sharon, J. A.; Staroselsky, A., Prediction of microstructure in laser powder bed fusion process, Acta Mater., 124, 360-371 (2017)
[8] Kundin, J.; Mushongera, L.; Emmerich, H., Phase-field modeling of microstructure formation during rapid solidification in Inconel 718 superalloy, Acta Mater., 95, 343-356 (2015)
[9] Ghosh, S.; Ma, L.; Ofori-Opoku, N.; Guyer, J. E., On the primary spacing and microsegregation of cellular dendrites in laser deposited Ni-Nb alloys, Model. Simul. Mater. Sci. Eng., 25, Article 065002 pp. (2017)
[10] Ghosh, S.; Ma, L.; Levine, L. E.; Ricker, R. E.; Stoudt, M. R.; Heigel, J. C.; Guyer, J. E., Single-track melt-pool measurements and microstructures in Inconel 625, JOM, 1-6 (2018)
[11] Ghosh, S.; McReynolds, K.; Guyer, J. E.; Banerjee, D., Simulation of temperature, stress and microstructure fields during laser deposition of Ti-6Al-4V, Model. Simul. Mater. Sci. Eng., 26, Article 075005 pp. (2018)
[12] Karayagiz, K.; Johnson, L.; Seede, R.; Attari, V.; Zhang, B.; Huang, X.; Ghosh, S.; Duong, T.; Karaman, I.; Elwany, A.; Arroyave, R., Finite interface dissipation phase field modeling of Ni-Nb under additive manufacturing conditions, Acta Mater., 185, 320-339 (2020)
[13] Chen, L. Q., Phase-field models for microstructure evolution, Annu. Rev. Mater. Res., 32, 113-140 (2002)
[14] Boettinger, W. J.; Warren, J. A.; Beckermann, C.; Karma, A., Phase-field simulation of solidification, Annu. Rev. Mater. Res., 32, 163-194 (2002)
[15] Moelans, N.; Blanpain, B.; Wollants, P., An introduction to phase-field modeling of microstructure evolution, Calphad, 32, 268-294 (2008)
[16] Steinbach, I., Phase-field models in materials science, Model. Simul. Mater. Sci. Eng., 17, Article 073001 pp. (2009)
[17] Lambert, J., Numerical Methods for Ordinary Differential Systems: The Initial Value Problem (1991), John Wiley & Sons: John Wiley & Sons Hoboken, NJ · Zbl 0745.65049
[18] Brenner, S.; Scott, R., The Mathematical Theory of Finite Element Methods, Vol. 15 (2007), Springer Science & Business Media: Springer Science & Business Media New York
[19] Braess, D., Finite Elements: Theory, Fast Solvers, and Applications in Solid Mechanics (2001), Cambridge University Press: Cambridge University Press Cambridge · Zbl 0976.65099
[20] Hübner, B.; Walhorn, E.; Dinkler, D., A monolithic approach to fluid-structure interaction using space-time finite elements, Comput. Methods Appl. Mech. Eng., 193, 2087-2104 (2004) · Zbl 1067.74575
[21] Michler, C.; Hulshoff, S.; Van Brummelen, E.; De Borst, R., A monolithic approach to fluid-structure interaction, Comput. Fluids, 33, 839-848 (2004) · Zbl 1053.76042
[22] Matthies, H. G.; Steindorf, J., Partitioned strong coupling algorithms for fluid-structure interaction, Comput. Struct., 81, 805-812 (2003)
[23] Erbts, P.; Hartmann, S.; Düster, A., A partitioned solution approach for electro-thermo-mechanical problems, Arch. Appl. Mech., 85, 1075-1101 (2015) · Zbl 1347.74025
[24] Joshi, V.; Jaiman, R. K., An adaptive variational procedure for the conservative and positivity preserving Allen-Cahn phase-field model, J. Comput. Phys., 366, 478-504 (2018) · Zbl 1406.76049
[25] Joshi, V.; Jaiman, R. K., A positivity preserving and conservative variational scheme for phase-field modeling of two-phase flows, J. Comput. Phys., 360, 137-166 (2018) · Zbl 1391.76651
[26] Knoll, D. A.; Keyes, D. E., Jacobian-free Newton-Krylov methods: a survey of approaches and applications, J. Comput. Phys., 193, 357-397 (2004) · Zbl 1036.65045
[27] Kelley, C. T., Iterative Methods for Linear and Nonlinear Equations (1995), SIAM: SIAM Philadeplhia, PA · Zbl 0832.65046
[28] Knoll, D. A.; Rider, W. J., A multigrid preconditioned Newton-Krylov method, SIAM J. Sci. Comput., 21, 691-710 (2000) · Zbl 0952.65102
[29] Saad, Y., Iterative Methods for Sparse Linear Systems, The PWS Series in Computer Science (1995), PWS Publishing Company: PWS Publishing Company Boston, MA · Zbl 1002.65042
[30] Joshi, V.; Jaiman, R. K., A hybrid variational Allen-Cahn/ALE scheme for the coupled analysis of two-phase fluid-structure interaction, Int. J. Numer. Methods Eng., 117, 405-429 (2019)
[31] Knoll, D. A.; Mousseau, V. A.; Chacón, L.; Reisner, J. M., Jacobian-free Newton-Krylov methods for the accurate time integration of stiff wave systems, J. Sci. Comput., 25, 213-230 (2005) · Zbl 1203.65071
[32] Park, H.; Nourgaliev, R. R.; Martineau, R. C.; Knoll, D. A., On physics-based preconditioning of the Navier-Stokes equations, J. Comput. Phys., 228, 9131-9146 (2010) · Zbl 1395.65028
[33] Mousseau, V. A.; Knoll, D. A.; Reisner, J. M., An implicit nonlinearly consistent method for the two-dimensional shallow-water equations with Coriolis force, Mon. Weather Rev., 130, 2611-2625 (2002)
[34] Knoll, D. A.; Chac’on, L.; Margolin, L. G.; Mousseau, V. A., On balanced approximations for time integration of multiple time scales systems, J. Comput. Phys., 185, 583-611 (2003) · Zbl 1047.76074
[35] Gropp, W.; Lusk, E.; Skjellum, A., Using MPI: Portable Parallel Programming with the Message-Passing Interface, Vol. 1 (1999), MIT Press
[36] Dagum, L.; Menon, R., OpenMP: an industry-standard API for shared-memory programming, IEEE Comput. Sci. Eng., 5, 46-55 (1998)
[37] NVIDIA Corporation, NVIDIA CUDA C programming guide, 2010, Version 3.2.
[38] Jacobsen, D.; Thibault, J.; Senocak, I., An MPI-CUDA implementation for massively parallel incompressible flow computations on multi-GPU clusters, (48th AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition (2010)), 522
[39] Glaser, J.; Nguyen, T. D.; Anderson, J. A.; Lui, P.; Spiga, F.; Millan, J. A.; Morse, D. C.; Glotzer, S. C., Strong scaling of general-purpose molecular dynamics simulations on GPUs, Comput. Phys. Commun., 192, 97-107 (2015)
[40] Komatitsch, D.; Erlebacher, G.; Göddeke, D.; Michéa, D., High-order finite-element seismic wave propagation modeling with MPI on a large GPU cluster, J. Comput. Phys., 229, 7692-7714 (2010) · Zbl 1194.86019
[41] Yang, C.; Xu, Q.; Liu, B., GPU-accelerated three-dimensional phase-field simulation of dendrite growth in a nickel-based superalloy, Comput. Mater. Sci., 136, 133-143 (2017)
[42] Yamanaka, A.; Aoki, T.; Ogawa, S.; Takaki, T., GPU-accelerated phase-field simulation of dendritic solidification in a binary alloy, J. Cryst. Growth, 318, 40-45 (2011)
[43] Sakane, S.; Takaki, T.; Ohno, M.; Shimokawabe, T.; Aoki, T., GPU-Accelerated 3d Phase-Field Simulations of Dendrite Competitive Growth During Directional Solidification of Binary Alloy, IOP Conference Series: Materials Science and Engineering, vol. 84, 012063 (2015), IOP Publishing
[44] Gaston, D.; Newman, C.; Hansen, G.; Lebrun-Grandie, D., MOOSE: a parallel computational framework for coupled systems of nonlinear equations, Nucl. Eng. Des., 239, 1768-1778 (2009)
[45] (2019), PACE3D (Parallel Algorithms for Crystal Evolution in 3D)
[46] MMSP, The mesoscale microstructure simulation project (2014)
[47] MICRESS, The microstructure evolution simulation software (2019)
[48] OpenPhase (2007)
[49] (2019), PhasePot Microstructure Simulation Tool
[50] Guyer, J. E.; Wheeler, D.; Warren, J. A., FiPy: partial differential equations with Python, Comput. Sci. Eng., 11, 6-15 (2009)
[51] DeWitt, S.; Rudraraju, S.; Montiel, D.; Andrews, W. B.; Thornton, K., PRISMS-PF: a general framework for phase-field modeling with a matrix-free finite element method, npj Comput. Mater., 6 (2020)
[52] Dorr, M. R.; Fattebert, J.-L.; Wickett, M.; Belak, J.; Turchi, P., A numerical algorithm for the solution of a phase-field model of polycrystalline materials, J. Comput. Phys., 229, 626-641 (2010) · Zbl 1183.82084
[53] Radhakrishnan, B.; Gorti, S. B.; Turner, J. A.; Acharya, R.; Sharon, J. A.; Staroselsky, A.; El-Wardany, T., Phase field simulations of microstructure evolution in IN718 using a surrogate Ni-Fe-Nb alloy during laser powder bed fusion, Metals, 9, 14 (2019)
[54] Zhu, C.; Liu, J.; Zhu, M.; Feng, L., Multi-GPU hybrid programming accelerated three-dimensional phase-field model in binary alloy, AIP Adv., 8, Article 035312 pp. (2018)
[55] Shimokawabe, T.; Aoki, T.; Takaki, T.; Endo, T.; Yamanaka, A.; Maruyama, N.; Nukada, A.; Matsuoka, S., Peta-scale phase-field simulation for dendritic solidification on the TSUBAME 2.0 supercomputer, (Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis (2011), ACM), 3
[56] Caginalp, G., Stefan and Hele-Shaw type models as asymptotic limits of the phase-field equations, Phys. Rev. A, 39, 5887 (1989) · Zbl 1027.80505
[57] Sekerka, R. F., Morphology: from sharp interface to phase field models, J. Cryst. Growth, 264, 530-540 (2004)
[58] Arfken, G. B.; Weber, H. J.; Harris, F. E., Mathematical Methods for Physicists (2013), Elsevier: Elsevier New York · Zbl 1239.00005
[59] Ewing, G. M., Calculus of Variations with Applications (1969), W. W. Norton and Co. Inc.: W. W. Norton and Co. Inc. New York · Zbl 0198.44501
[60] Allen, S. M.; Cahn, J. W., A microscopic theory for antiphase boundary motion and its application to antiphase domain coarsening, Acta Metall., 27, 1085-1095 (1979)
[61] Cahn, J. W.; Hilliard, J. E., Free energy of a nonuniform system. I. Interfacial free energy, J. Chem. Phys., 28, 258-267 (1958) · Zbl 1431.35066
[62] Dennis, J. E.; Schnabel, R. B., Numerical Methods for Unconstrained Optimization and Nonlinear Equations (1996), SIAM: SIAM Philadelphia, PA · Zbl 0847.65038
[63] Pawlowski, R. P.; Shadid, J. N.; Simonis, J. P.; Walker, H. F., Globalization techniques for Newton-Krylov methods and applications to the fully coupled solution of the Navier-Stokes equations, SIAM Rev., 48, 700-721 (2006) · Zbl 1110.65039
[64] Pernice, M.; Walker, H. F., NITSOL: a Newton iterative solver for nonlinear systems, SIAM J. Sci. Comput., 19, 302-318 (1998) · Zbl 0916.65049
[65] Brown, P. N.; Saad, Y., Hybrid Krylov methods for nonlinear systems of equations, SIAM J. Sci. Stat. Comput., 11, 450-481 (1990) · Zbl 0708.65049
[66] Dembo, R.; Eisenstat, S. C.; Steihaug, T., Inexact Newton methods, SIAM J. Numer. Anal., 19, 400-408 (1982) · Zbl 0478.65030
[67] Eisenstat, S. C.; Walker, H. F., Choosing the forcing terms in a inexact Newton method, SIAM J. Sci. Comput., 17, 16-32 (1996) · Zbl 0845.65021
[68] Chacón, L.; Knoll, D. A.; Finn, J. M., An implicit, nonlinear reduced resistive MHD solver, J. Comput. Phys., 178, 15-36 (2002) · Zbl 1139.76328
[69] Provatas, N.; Elder, K., Phase-Field Methods in Materials Science and Engineering (2011), John Wiley & Sons: John Wiley & Sons Hoboken, NJ
[70] Briggs, W. L.; McCormick, S. F., A Multigrid Tutorial, Vol. 72 (2000), SIAM: SIAM Philadelphia, PA
[71] Hackbusch, W., Multi-Grid Methods and Applications, Vol. 4 (2013), Springer Science & Business Media: Springer Science & Business Media New York
[72] Trottenberg, U.; Oosterlee, C. W.; Schüller Multigrid, A. (2000), Academic Press: Academic Press New York
[73] Gee, M.; Siefert, C.; Hu, J.; Tuminaro, R.; Sala, M., ML 5.0 Smoothed Aggregation User’s Guide (2006), Sandia National Laboratories, Technical Report SAND2006-2649
[74] Berger-Vergiat, L.; Glusa, C. A.; Hu, J. J.; Mayr, M.; Prokopenko, A.; Siefert, C. M.; Tuminaro, R. S.; Wiesner, T. A., MueLu User’s Guide (2019), Sandia National Laboratories, Technical Report SAND2019-0537
[75] Berger-Vergiat, L.; Glusa, C. A.; Hu, J. J.; Mayr, M.; Prokopenko, A.; Siefert, C. M.; Tuminaro, R. S.; Wiesner, T. A., MueLu multigrid framework (2019)
[76] Heroux, M. A., An overview of the Trilinos project, ACM Trans. Math. Softw., 31, 397-423 (2005) · Zbl 1136.65354
[77] Trilinos (2019)
[78] Tusas (2020)
[79] Exodus II (2007)
[80] Schoof, L.; Yarberry, V., EXODUS II: A Finite Element Data Model (1996), Sandia National Laboratories, Technical Report SAND92-2137
[81] NOX & LOCA, Object-oriented nonlinear solver and continuation packages (2010)
[82] Bavier, E.; Hoemmen, M.; Rajamanickam, S.; Thornquist, H., Amesos2 and Belos: direct and iterative solvers for large sparse linear systems, Sci. Program., 20, 241-255 (2012)
[83] M.A. Heroux, AztecOO user guide.
[84] Rythmos (2019)
[85] Gaston, D.; Newman, C.; Hansen, G., MOOSE: a parallel computational framework for coupled systems of nonlinear equations, (American Nuclear Society 2009 International Conference on Advances in Mathematics, Computational Methods, and Reactor Physics. American Nuclear Society 2009 International Conference on Advances in Mathematics, Computational Methods, and Reactor Physics, Saratoga Springs, NY (2009))
[86] CMake (2019)
[87] Boost C++ Libraries (2019)
[88] Hierarchical data format, version 5 (1997)
[89] Rew, R.; Davis, G., Data management: NetCDF: an interface for scientific data access, IEEE Comput. Graph. Appl., 10, 76-82 (1990)
[90] (2020), Kokkos C++ Performance Portability Programming EcoSystem: The Programming Model - Parallel Execution and Memory Abstraction
[91] Edwards, H. C.; Trott, C. R.; Sunderland, D., Kokkos: enabling manycore performance portability through polymorphic memory access patterns, J. Parallel Distrib. Comput., 74, 3202-3216 (2014)
[92] Falgout, R. D.; Yang, U. M., HYPRE: a library of high performance preconditioners, (International Conference on Computational Science (3) (2002)), 632-641 · Zbl 1056.65046
[93] Hindmarsh, A. C.; Brown, P. N.; Grant, K. E.; Lee, S. L.; Serban, R.; Shumaker, D. E.; Woodward, C. S., SUNDIALS: suite of nonlinear and differential/algebraic equation solvers, ACM Trans. Math. Softw., 31, 363-396 (2005) · Zbl 1136.65329
[94] Jokisaari, A. M.; Voorhees, P. W.; Guyer, J. E.; Warren, J. A.; Heinonen, O. G., Phase field benchmark problems for dendritic growth and linear elasticity, Comput. Mater. Sci., 149, 336-347 (2018)
[95] Cummins, S. J.; Quirk, J. J.; Kothe, D. B., An exploration of the phase field technique for microstructure solidification modeling, (Proceedings of IMECE 2002 ASME Microstructural Developments in Solidification (2002), ASME: ASME New Orleans)
[96] Kessler, D. A.; Levine, H., Velocity selection in dendritic growth, Phys. Rev. B, 33, 7867 (1986)
[97] Echebarria, B.; Folch, R.; Karma, A.; Plapp, M., Quantitative phase-field model of alloy solidification, Phys. Rev. E, 70, Article 061604 pp. (2004)
[98] Farzadi, A.; Do-Quang, M.; Serajzadeh, S.; Kokabi, A.; Amberg, G., Phase-field simulation of weld solidification microstructure in an Al-Cu alloy, Model. Simul. Mater. Sci. Eng., 16, 1-18 (2008)
[99] Karma, A., Phase-field formulation for quantitative modeling of alloy solidification, Phys. Rev. Lett., 87, Article 115701 pp. (2001)
[100] Ghosh, S.; Ofori-Opoku, N.; Guyer, J. E., Simulation and analysis of γ-Ni cellular growth during laser powder deposition of Ni-based superalloys, Comput. Mater. Sci., 144, 256-264 (2018)
[101] Ghosh, S.; Stoudt, M. R.; Levine, L. E.; Guyer, J. E., Formation of Nb-rich droplets in laser deposited Ni-matrix microstructures, Scr. Mater., 146, 36-40 (2018)
[102] Summit (2018)
[103] Sierra (2018)
[104] Liu, H.; Yang, B.; Chen, Z., Accelerating algebraic multigrid solvers on NVIDIA GPUs, Comput. Math. Appl., 70, 1162-1181 (2015) · Zbl 1443.65046
[105] Haase, G.; Liebmann, M.; Douglas, C. C.; Plank, G., A parallel algebraic multigrid solver on graphics processing units, (High Performance Computing and Applications (2010), Springer: Springer NY), 38-47
[106] Richter, C.; Schöps, S.; Clemens, M., GPU acceleration of algebraic multigrid preconditioners for discrete elliptic field problems, IEEE Trans. Magn., 50, 461-464 (2014)
[107] Gandham, R.; Esler, K.; Zhang, Y., A GPU accelerated aggregation algebraic multigrid method, Comput. Math. Appl., 68, 1151-1160 (2014) · Zbl 1367.65049
[108] Tuminaro, R. S.; Shadid, J. N.; Cyr, E. C.; Lin, P.; Pawlowski, R. P.; Phillips, E. G.; Wiesner, T.; Adler, J.; Benson, T.; Chacon, L.; Farrell, P.; Rappaport, A.; Maclachlan, S.; Hu, J. J.; Berger-Vergiat, L.; Glusa, C. A.; Siefert, C., Scalable preconditioners to improve time to solution for magnetohydrodynamics applications (2019)
[109] Kothe, D.; Lee, S.; Qualters, I., Exascale computing in the United States, Comput. Sci. Eng., 21, 17-29 (2018)
[110] Alexander, F., Exascale applications: skin in the game, Philos. Trans. R. Soc. A, 378 (2020) · Zbl 1462.65230
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.