SIMPLE adjoint message passing. (English) Zbl 1453.65397

Summary: In the context of numerical simulation a system of partial differential equations is typically solved by discretizing it into a set of linear equations. The non-linear effects in the solution are then approximated by repeatedly discretizing, solving and correcting the equations using some outer iteration scheme. One example for such a scheme is the SIMPLE algorithm. In this paper, we show how we obtained an algorithmic adjoint version of the SIMPLE algorithm in OpenFOAM with particular focus on the adjoint message passing interface communication inside the linear solvers. Due to the sparse matrix storage and specifics of the implementation of the linear solvers, this method requires special attention to the inner workings of said structures and algorithms. We show the feasibility of the chosen method by presenting two case studies. In addition to the SIMPLE algorithm, the presented ideas are applicable to a wide range of algorithms whenever linear systems are embedded into an overlying non-linear context.


65N22 Numerical solution of discretized equations for boundary value problems involving PDEs
76-04 Software, source code, etc. for problems pertaining to fluid mechanics
Full Text: DOI


[1] Consistent treatment of incompletely converged iterative linear solvers in reverse-mode AD, To appear in this special issue, 2017
[2] Axtmann, G.; Ulrich, R., Scalability of OpenFOAM with large Eddy simulations and DNS on high-performance systems, 413-424, (2016), Springer International Publishing, Cham
[3] Parallelism in the reverse mode, in Computational Differentiation: Techniques, Applications, and Tools, SIAM, Philadelphia, PA, 1996, pp. 37–147
[4] Automatically differentiating MPI-1 datatypes: The complete story, in Automatic Differentiation of Algorithms: From Simulation to Optimization, George Corliss, Christele Faure, Andreas Griewank, Laurent Hascoet, and Uwe Naumann, eds., Computer and Information Science, chapter 25, Springer, New York, NY, 2002, pp. 215–222
[5] A duality between forward and adjoint MPI communication routines, in Computational Methods in Science and Technology, Polish Academy of Sciences, Poznan, 2006, pp. 23–24
[6] Christianson, B., reverse accumulation and attractive fixed points, Optim. Methods Softw., 3, 4, 311-326, (1994)
[7] Current bottlenecks in the scalability of OpenFOAM on massively parallel clusters, 2011. PRACE white paper to appear on
[8] Duran, A.; Celebi, M. S.; Piskin, S.; Tuncel, M., scalability of openfoam for bio-medical flow simulations, J. Supercomput., 71, 3, 938-951, (2015)
[9] Odysée and Parallelism: Extension and Validation, Proceedings of the 3rd European Conference on Numerical Mathematics and Advanced Applications, Jyväskylä, Finland, July 26–30, 1999, World Scientific, 2000, pp. 478–485
[10] Ferziger, J. H.; Peric, M., Computational Methods for Fluid Dynamics, (1999), Springer, Heidelberg · Zbl 0943.76001
[11] Collected matrix derivative results for forward and reverse mode algorithmic differentiation, in Advances in Automatic Differentiation, Springer, Heidelberg, 2008, pp. 35–44 · Zbl 1154.65308
[12] Griewank, A.; Walther, A., algorithm 799: revolve: an implementation of checkpointing for the reverse or adjoint mode of computational differentiation, ACM Trans. Math. Softw., 26, 1, 19-45, (2000) · Zbl 1137.65330
[13] Griewank, A.; Walther, A., Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, (2008), SIAM, Philadelphia, PA · Zbl 1159.65026
[14] Griewank, A.; Juedes, D.; Utke, J., algorithm 755: ADOL-C: A package for the automatic differentiation of algorithms written in C/C++, ACM Trans. Math. Softw., 22, 2, 131-167, (1996) · Zbl 0884.65015
[15] Hascoët, L.; Pascual, V., the tapenade automatic differentiation tool: principles, model, and specification, ACM Trans. Math. Softw., 39, 3, 20:1-20:43, (2013) · Zbl 1295.65026
[16] OpenFOAM: A C++ library for complex physics simulations, International Workshop on Coupled Methods in Numerical Dynamics, Vol. 1000, 2007, pp. 1–20
[17] Derivative code by overloading in C++ (dco/c++): Introduction and summary of features, Technical Report AIB-2016-08, RWTH Aachen University, September 2016
[18] Naumann, U., The Art of Differentiating Computer Programs. An Introduction to Algorithmic Differentiation, (2012), SIAM, Philadelphia, PA · Zbl 1275.65015
[19] A framework for proving correctness of adjoint message passing programs, Proceedings of EUROPVM/MPI 2008, Heidelberg, 2008, pp. 316–321
[20] Othmer, C., A continuous adjoint formulation for the computation of topological and surface sensitivities of ducted flows, Int. J. Numer. Methods Fluids, 58, 8, 861-877, (2008) · Zbl 1152.76025
[21] Patankar, S., Numerical Heat Transfer and Fluid Flow, (1980), CRC Press, Boca Raton, FL · Zbl 0521.76003
[22] Pitz, R. W.; Daily, J. W., combustion in a turbulent mixing layer formed at a rearward-facing step, AIAA J., 21, 11, 1565-1570, (1983)
[23] Semantics driven adjoints of the message passing interface, PhD thesis, RWTH Aachen University, 2014
[24] A wish list for efficient adjoints of one-sided MPI communication, in Recent Advances in the Message Passing Interface, Springer, Heidelberg, 2012, pp. 248–257
[25] Second-order algorithmic differentiation by source transformation of MPI code, in Recent Advances in the Message Passing Interface, Lecture Notes in Computer Science, Vol. 6305, Springer, Heidelberg, 2010, pp. 257–264
[26] Schanen, M.; Naumann, U.; Hascoët, L.; Utke, J., interpretative adjoints for numerical simulation codes using MPI, Procedia Comput. Sci., 1, 1, 1825-1833, (2010)
[27] A discrete adjoint version of an unsteady incompressible solver for OpenFOAM using algorithmic differentiation, Proceedings of the Jointly Organised 11th World Congress on Computational Mechanics (WCCM XI): 5th European Conference on Computational Mechanics (ECCM V), 6th European Conference on Computational Fluid Dynamics (ECFD VI), Barcelona, Spain, Barcelona, 2014, International Center for Numerical Methods in Engineering (CIMNE)
[28] Stokes, G. G., On the Effect of the Internal Friction of Fluids on the Motion of Pendulums, 9, (1851), Pitt Press Cambridge, Cambridge
[29] Towara, M.; Naumann, U., A discrete adjoint model for openfoam, Procedia Comput. Sci., 18, 429-438, (2013)
[30] Towara, M.; Schanen, M.; Naumann, U., MPI-parallel discrete adjoint openfoam, Procedia Comput. Sci., 51, 19-28, (2015)
[31] Likwid: A lightweight performance-oriented tool suite for x86 multicore environments, Proceedings of PSTI2010, the First International Workshop on Parallel Software Tools and Tool Infrastructures, San Diego, CA, 2010
[32] Toward Adjoinable MPI, Proceedings of IPDPS 2009, Rome, 2009
[33] Versteeg, H. K.; Malalasekera, W., An Introduction to Computational Fluid Dynamics: The Finite Volume Method, (2007), Pearson Education, Harlow
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.