A communication-avoiding implicit-explicit method for a free-surface ocean model. (English) Zbl 1349.86010

Summary: We examine a nonlinear elimination method for the free-surface ocean equations based on barotropic-baroclinic decomposition. The two dimensional scalar continuity equation is treated implicitly with a preconditioned Jacobian-free Newton-Krylov method (JFNK). The remaining three dimensional equations are subcycled explicitly within the JFNK residual evaluation with a method known as nonlinear elimination. In this approach, the memory footprint of the underlying Krylov vector is greatly reduced over that required by fully coupled implicit methods. The method is second-order accurate and scales algorithmically, with allowed timesteps much larger than fully explicit methods. Moreover, the hierarchical nature of the algorithm lends itself readily to emerging architectures. In particular, we introduce a communication staging strategy for the three dimensional explicit system that greatly reduces the communication costs of the algorithm and provides a key advantage as communication costs continue to dominate relative to floating point costs in emerging architectures.


86-08 Computational methods for problems pertaining to geophysics
65M06 Finite difference methods for initial value and initial-boundary value problems involving PDEs
65M22 Numerical solution of discretized equations for initial value and initial-boundary value problems involving PDEs
86A05 Hydrology, hydrography, oceanography
Full Text: DOI


[1] Maltrud, M.; McClean, J., An eddy resolving global 1/10 Ocean simulation, Ocean Model., 8, 31-54, (2005)
[2] McClean, J.; Bader, D.; Bryan, F.; Maltrud, M.; Dennis, J.; Mirin, A.; Jones, P.; Kim, Y.; Ivanova, D.; Vertenstein, M., A prototype two-decade fully-coupled fine-resolution CCSM simulation, Ocean Model., (2011)
[3] Dukowicz, J.; Smith, R., Implicit free-surface method for the Bryan-Cox-semtner Ocean model, J. Geophys. Res., 99, 7991-8014, (1994)
[4] Hallberg, R., Stable split time stepping schemes for large-scale Ocean modeling, J. Comput. Phys., 135, 54-65, (1997) · Zbl 0889.76044
[5] Higdon, R.; de Szoeke, R., Barotropic-baroclinic time splitting for Ocean circulation modeling, J. Comput. Phys., 135, 30-53, (1997) · Zbl 0888.76055
[6] Bernsen, E.; Dijkstra, H.; Thies, J.; Wubs, F., The application of jacobian-free Newton-Krylov methods to reduce the spin-up time of Ocean general circulation models, J. Comput. Phys., 229, 8167-8179, (2010) · Zbl 1196.86004
[7] Dijkstra, H.; Oksuzoglu, H.; Wubs, F.; Botta, E., A fully implicit model of the three-dimensional thermohaline Ocean circulation, J. Comput. Phys., 173, 685-715, (2001) · Zbl 1051.86004
[8] Newman, C.; Knoll, D. A., Physics-based preconditioners for Ocean simulation, SIAM J. Sci. Comput., 35, S445-S464, (2013) · Zbl 1406.86005
[9] Thies, J.; Wubs, F.; Dijkstra, H., Bifurcation analysis of 3D ocean flows using a parallel fully-implicit Ocean model, Ocean Model., 30, 287-297, (2009)
[10] Weijer, W.; Dijkstra, H.; Öksüzoğlu, H.; Wubs, F.; de Niet, A., A fully-implicit model of the global Ocean circulation, J. Comput. Phys., 192, 452-470, (2003) · Zbl 1032.76557
[11] Wubs, F.; de Niet, A.; Dijkstra, H., The performance of implicit Ocean models on B- and C-grids, J. Comput. Phys., 211, 210-228, (2006) · Zbl 1083.86002
[12] Evans, K.; Rouson, D.; Salinger, A.; Taylor, M.; Weijer, W.; White, J., A scalable and adaptable solution framework within components of the community climate system model, (Computational Science - ICCS 2009, (2009)), 332-341
[13] Knoll, D. A.; Keyes, D. E., Jacobian-free Newton-Krylov methods: a survey of approaches and applications, J. Comput. Phys., 193, 357-397, (2004) · Zbl 1036.65045
[14] Nadiga, B.; Taylor, M.; Lorenz, J., Ocean modelling for climate studies: eliminating short time scales in long-term, high-resolution studies of Ocean circulation, Math. Comput. Model., 44, 870-886, (2006) · Zbl 1132.86310
[15] Smith, R.; Gent, P., Reference manual for the parallel Ocean program (POP), (2002), Los Alamos National Laboratory, Technical Report Los Alamos Technical Report LA-UR-02-2484
[16] Weijer, W., Reference manual for the implicit version of the parallel Ocean program (impop), (2009), Los Alamos National Laboratory, Technical Report Los Alamos Technical Report LA-UR-09-7040
[17] Knoll, D. A.; Mousseau, V. A.; Chacón, L.; Reisner, J. M., Jacobian-free Newton-Krylov methods for the accurate time integration of stiff wave systems, J. Sci. Comput., 25, 213-230, (2005) · Zbl 1203.65071
[18] Park, H.; Nourgaliev, R. R.; Martineau, R. C.; Knoll, D. A., On physics-based preconditioning of the Navier-Stokes equations, J. Comput. Phys., 228, 9131-9146, (2010) · Zbl 1395.65028
[19] Chen, G.; Chacón, L.; Barnes, D. C., An energy- and charge-conserving, implicit, electrostatic particle-in-cell algorithm, J. Comput. Phys., 230, 7018-7036, (2011) · Zbl 1237.78006
[20] Ying, W.; Rose, D. J.; Henriquez, C. S., Efficient fully implicit time integration methods for modeling cardiac dynamics, IEEE Trans. Biomed. Eng., 55, 2701-2711, (2008)
[21] Kadioglu, S. Y.; Knoll, D. A., A fully second order implicit/explicit time integration technique for hydrodynamics plus nonlinear heat conduction problems, J. Comput. Phys., 229, 3237-3249, (2010) · Zbl 1307.76056
[22] Kadioglu, S. Y.; Knoll, D. A.; Lowrie, R. B.; Rauenzahn, R. M., A second order self-consistent IMEX method for radiation hydrodynamics, J. Comput. Phys., 229, 8313-8332, (2010) · Zbl 1381.76262
[23] Lemieux, J.-F.; Knoll, D. A.; Losch, M.; Girard, C., A second-order accurate in time implicit-explicit (IMEX) integration scheme for sea ice dynamics, J. Comput. Phys., 263, 375-392, (2014) · Zbl 1349.86006
[24] Chen, G.; Chacón, L.; Barnes, D. C., An efficient mixed-precision, hybrid CPU-GPU implementation of a nonlinearly implicit one-dimensional particle-in-cell algorithm, J. Comput. Phys., 231, 5374-5388, (2012)
[25] Taitano, W. T.; Knoll, D. A.; Chacón, L.; Chen, G., Development of a consistent and stable fully implicit moment method for Vlasov-Ampère particle in cell (PIC) system, SIAM J. Sci. Comput., 35, S126-S149, (2013) · Zbl 1282.82038
[26] Knoll, D. A.; Park, H.; Smith, K., Application of the Jacobian-free Newton-Krylov method to nonlinear acceleration of transport source iteration in slab geometry, Nucl. Sci. Eng., 167, 122, (2011)
[27] Park, H.; Knoll, D.; Newman, C., Nonlinear acceleration of transport criticality problems, Nucl. Sci. Eng., 172, 52, (2012)
[28] Park, H.; Knoll, D.; Rauenzahn, R.; Newman, C.; Densmore, J.; Wollaber, A., An efficient and time accurate, moment-based scale-bridging algorithm for thermal radiative transfer problems, SIAM J. Sci. Comput., 35, S18-S41, (2013) · Zbl 1285.65092
[29] Park, H.; Knoll, D.; Rauenzahn, R.; Wollaber, A.; Densmore, J., A consistent, moment-based, multiscale solution approach for thermal radiative transfer problems, Transp. Theory Stat. Phys., 41, 284-303, (2012) · Zbl 1278.82051
[30] Wallcraft, A. J.; Chassignet, E. P.; Hurlburt, H. E.; Townsend, T. L., \(1 / 25 \operatorname{°}\) atlantic Ocean simulation using HYCOM, (2005), Naval Research Laboratory, Stennis Space Center, MS, Oceanography Div., Technical Report No. NRL/PP/7320-05-5263
[31] Datta, K.; Kamil, S.; Williams, S.; Oliker, L.; Shalf, J.; Yelick, K., Optimization and performance modeling of stencil computations on modern microprocessors, SIAM Rev., 51, 129-159, (2009) · Zbl 1160.65359
[32] Wittmann, M.; Hager, G.; Treibig, J.; Wellein, G., Leveraging shared caches for parallel temporal blocking of stencil codes on multicore processors and clusters, Parallel Process. Lett., 20, 359-376, (2010)
[33] Wonnacott, D., Using time skewing to eliminate idle time due to memory bandwidth and network limitations, (Proceedings 14th International Parallel and Distributed Processing Symposium, 2000, IPDPS 2000, (2000), IEEE), 171-180
[34] Ding, C.; He, Y., A ghost cell expansion method for reducing communications in solving PDE problems, (Supercomputing, ACM/IEEE 2001 Conference, (2001), IEEE), 55
[35] Lambert, J., Numerical methods for ordinary differential systems: the initial value problem, (1991), John Wiley & Sons, Inc. · Zbl 0745.65049
[36] Ringler, T.; Petersen, M.; Higdon, R. L.; Jacobsen, D.; Jones, P. W.; Maltrud, M., A multi-resolution approach to global Ocean modeling, Ocean Model., 69, 211-232, (2013)
[37] Pacanowski, R.; Dixon, K.; Rosati, A., The GFDL modular Ocean model users guide, (1993), Geophysical Fluid Dynamics Laboratory Princeton, USA, GFDL Ocean Group Tech. Rep. 2, Technical Report
[38] Kelley, C. T., Iterative methods for linear and nonlinear equations, (1995), SIAM Philadelphia, PA · Zbl 0832.65046
[39] Saad, Y., Iterative methods for sparse linear systems, The PWS Series in Computer Science, (1995), PWS Publishing Company Boston, MA · Zbl 1002.65042
[40] Pernice, M.; Walker, H. F., NITSOL: a Newton iterative solver for nonlinear systems, SIAM J. Sci. Comput., 19, 302-318, (1998) · Zbl 0916.65049
[41] Brown, P. N.; Saad, Y., Hybrid Krylov methods for nonlinear systems of equations, SIAM J. Sci. Stat. Comput., 11, 450-481, (1990) · Zbl 0708.65049
[42] Dembo, R.; Eisenstat, S. C.; Steihaug, T., Inexact Newton methods, SIAM J. Numer. Anal., 19, 400-408, (1982) · Zbl 0478.65030
[43] Eisenstat, S. C.; Walker, H. F., Choosing the forcing terms in a inexact Newton method, SIAM J. Sci. Comput., 17, 16-32, (1996) · Zbl 0845.65021
[44] Chacón, L.; Knoll, D. A.; Finn, J. M., An implicit, nonlinear reduced resistive MHD solver, J. Comput. Phys., 178, 15-36, (2002) · Zbl 1139.76328
[45] Knoll, D. A.; Rider, W. J., A multigrid preconditioned Newton-Krylov method, SIAM J. Sci. Comput., 21, 691-710, (2000) · Zbl 0952.65102
[46] Trottenberg, U.; Oosterlee, C. W.; Schüller, A., Multigrid, (2000), Academic Press
[47] Arakawa, A.; Lamb, V. R., Computational design of the basic dynamical processes of the UCLA general circulation model, (Chang, J., Methods in Computational Physics, vol. 17, (1977), Academic Press New York), 173-265
[48] Stallman, R., Using the GNU compiler collection, (2012), Free Software Foundation Cambridge, MA
[49] Gabriel, E.; Fagg, G. E.; Bosilca, G.; Angskun, T.; Dongarra, J. J.; Squyres, J. M.; Sahay, V.; Kambadur, P.; Barrett, B.; Lumsdaine, A.; Castain, R. H.; Daniel, D. J.; Graham, R. L.; Woodall, T. S., Open MPI: goals, concept, and design of a next generation MPI implementation, (Proceedings 11th European PVM/MPI Users’ Group Meeting, Budapest, Hungary, (2004)), 97-104
[50] Li, J.; Liao, W. K.; Choudhary, A.; Ross, R.; Thakur, R.; Gropp, W.; Latham, R.; Siegel, A.; Gallagher, B.; Zingale, M., Parallel netcdf: a scientific high-performance I/O interface, (Proceedings of ACM/IEEE Conference on Supercomputing, (2003), ACM Press), 39
[51] Heroux, M. A.; Bartlett, R. A.; Howle, V. E.; Hoekstra, R. J.; Hu, J. J.; Kolda, T. G.; Lehoucq, R. B.; Long, K. R.; Pawlowski, R. P.; Phipps, E. T.; Salinger, A. G.; Thornquist, H. K.; Tuminaro, R. S.; Willenbring, J. M.; Williams, A.; Stanley, K. S., An overview of the trilinos project, ACM Trans. Math. Softw., 31, 397-423, (2005) · Zbl 1136.65354
[52] Gee, M.; Siefert, C.; Hu, J.; Tuminaro, R.; Sala, M., ML 5.0 smoothed aggregation User’s guide, (2006), Sandia National Laboratories, Technical Report SAND2006-2649
[53] Marotzke, J.; Welander, P.; Willebrand, J., Instability and multiple steady states in a meridional-plane model of the thermohaline circulation, Tellus A, 40, 162-172, (1988)
[54] Willert, J.; Taitano, W. T.; Knoll, D., Leveraging Anderson acceleration for improved convergence of iterative solutions to transport systems, J. Comput. Phys., 273, 278-286, (2014) · Zbl 1351.82093
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.