×

Parallel performance of a lattice-Boltzmann/finite element cellular blood flow solver on the IBM blue gene/P architecture. (English) Zbl 1218.92030

Summary: We discuss the parallel implementation and scaling results of a hybrid lattice-Boltzmann/finite element code for suspension flow simulations. This code allows the direct numerical simulation of cellular blood flow, fully resolving the two-phase nature of blood and the deformation of the suspended phase. A brief introduction to the numerical methods employed is given followed by an outline of the code structure. Scaling results obtained on Argonne National Laboratories IBM Blue Gene/P (BG/P) are presented. Details include performance characteristics on 512 to 65,536 processor cores.

MSC:

92C35 Physiological flow
92-04 Software, source code, etc. for problems pertaining to biology
76M28 Particle methods and lattice-gas methods
92C30 Physiology (general)

Software:

MUPHY; Hemelb
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Aidun, C. K.; Lu, Y., Lattice Boltzmann simulation of solid particles suspended in fluid, J. Stat. Phys., 81, 1, 49-61 (1995) · Zbl 1106.82342
[2] Aidun, C. K.; Lu, Y.; Ding, E.-J., Direct analysis of particulate suspensions with inertia using the discrete Boltzmann equation, J. Fluid Mech., 373, 287-311 (1998) · Zbl 0933.76092
[3] Aidun, C. K.; Qi, D. W., A new method for analysis of the fluid interaction with a deformable membrane, J. Stat. Phys., 90, 1-2, 145-158 (1998) · Zbl 0918.73050
[4] Ding, E.-J.; Aidun, C. K., Extension of the lattice-Boltzmann method for direct simulation of suspended particles near contact, J. Stat. Phys., 112, 3/4, 685-708 (2003) · Zbl 1035.82024
[5] Wu, J.; Aidun, C. K., A method for direct simulation of flexible fiber suspensions using lattice-Boltzmann equation with external boundary force, Int. J. Multiphas. Flow, 36, 202-309 (2010)
[6] MacMeccan, R. M.; Clausen, J. R.; Neitzel, G. P.; Aidun, C. K., Simulating deformable particle suspensions using a coupled lattice-Boltzmann and finite-element method, J. Fluid Mech., 618, 13-39 (2009) · Zbl 1156.76455
[7] (2009), IBM, Blue Gene
[8] (2009), Argonne National Laboratory, Computing and storage
[9] Merz, H., Scaling Performance of CubePM and TVD MHD on Blue Gene, Blue Gene Applications White Paper (2006), Canadian Institute for Theoretical Astrophysics
[10] Kalyanaraman, A.; Emrich, S. J.; Schnable, P. S., Assembling genomes on large-scale parallel computers, Journal of Parallel and Distributed Computing, 67, 12, 1240-1255 (2007)
[11] M. Djurfeldt, C. Johansson, O. Ekeberg, M. Rehn, M. Lundqvist, A. Lansner, Massively parallel simulation of brain-scale neoronal network models, Tech. Rep. TRITA-NA-P0513 KTH, School of Computer Science and Communication, University of Stockholm, 2005; M. Djurfeldt, C. Johansson, O. Ekeberg, M. Rehn, M. Lundqvist, A. Lansner, Massively parallel simulation of brain-scale neoronal network models, Tech. Rep. TRITA-NA-P0513 KTH, School of Computer Science and Communication, University of Stockholm, 2005
[12] C.P. Sosa, T. Milledge, J. McAllister, G.Z. Milledge, Life sciences molecular dynamics applications on the IBM system Blue Gene solution: Performance overview, Tech. Rep., IBM, 2006; C.P. Sosa, T. Milledge, J. McAllister, G.Z. Milledge, Life sciences molecular dynamics applications on the IBM system Blue Gene solution: Performance overview, Tech. Rep., IBM, 2006
[13] X. Feng, K.W. Cameron, C.P. Sosa, B. Smith, Building the tree of life on terascale systems, in: Proceedings of the 21st IEEE International Parallel and Distributed Processing Symposium (IPDPS 07), 2007; X. Feng, K.W. Cameron, C.P. Sosa, B. Smith, Building the tree of life on terascale systems, in: Proceedings of the 21st IEEE International Parallel and Distributed Processing Symposium (IPDPS 07), 2007
[14] A. Peters, M.E. Lundberg, P.T. Lang, C.P. Sosa, High throughput computing validation for drug discovery using the DOCK program on a massively parallel system, Tech. Rep. Redpaper 4410, IBM, 2008; A. Peters, M.E. Lundberg, P.T. Lang, C.P. Sosa, High throughput computing validation for drug discovery using the DOCK program on a massively parallel system, Tech. Rep. Redpaper 4410, IBM, 2008
[15] (2008), Fluent, Fluent 6.3 Benchmarks, webpage:
[16] Ladd, A. J.C., Numerical simulations of particulate suspensions via a discretized Boltzmann equation. Part 1. Theoretical foundation, J. Fluid Mech., 271, 285-309 (1994) · Zbl 0815.76085
[17] Ladd, A. J.C., Numerical simulations of particulate suspensions via a discretized Boltzmann equation. Part 2. Numerical results, J. Fluid Mech., 271, 311-339 (1994) · Zbl 0815.76085
[18] Ladd, A. J.C.; Verberg, R., Lattice-Boltzmann simulations of particle-fluid suspensions, J. Stat. Phys., 104, 5, 1191-1251 (2001) · Zbl 1046.76037
[19] Vahala, G.; Keating, B.; Soe, M.; Yepez, J.; Vahala, L.; Carter, J.; Ziegeler, S., MHD turbulence studies using lattice Boltzmann algorithms, Commun. Comput. Phys., 4, 3, 624-646 (2008) · Zbl 1364.76190
[20] Mazzeo, M. D.; Coveney, P. V., HemeLB: A high performance parallel lattice-Boltzmann code for large scale fluid flow in complex geometries, Comput. Phys. Commun., 178, 894-914 (2008) · Zbl 1196.76008
[21] Bernaschi, M.; Melchionna, S.; Succi, S.; Fyta, M.; Kaxiras, E.; Sircar, J. K., MUPHY: A parallel MUlti PHYsics/scale code for high performance bio-fluidic simulations, Comput. Phys. Commun., 180, 9, 1495-1502 (2009)
[22] Orlandini, E.; Marenduzzo, D.; Yeomans, J. M., Shear dynamics in cholesterics, Comput. Phys. Commun., 169, 122-125 (2005)
[23] Malevanets, A.; Yeomans, J. M., A particle-based algorithm for the hydrodynamics of binary fluid mixtures, Comput. Phys. Commun., 129, 282-288 (2000) · Zbl 0976.76068
[24] Yamamoto, R.; Nakayama, Y.; Kim, K., A method to resolve hydrodynamic interactions in colloidal dispersions, Comput. Phys. Commun., 169, 301-304 (2005)
[25] Ding, E.-J.; Aidun, C. K., Cluster size distribution and scaling for spherical particles and red blood cells in pressure-driven flows at small Reynolds number, Phys. Rev. Lett., 96, 20, 204502 (2006)
[26] Chen, S.; Doolen, G. D., Lattice Boltzmann method for fluid flows, Annu. Rev. Fluid Mech., 30, 329-364 (1998) · Zbl 1398.76180
[27] Aidun, C. K.; Clausen, J. R., Lattice-Boltzmann method for complex flows, Annu. Rev. Fluid Mech., 42, 439-472 (2010) · Zbl 1345.76087
[28] Zeiser, T.; Hager, G.; Wellein, G., Vector computers in a world of commodity clusters, massively parallel systems and many-core many-threaded CPUs: Recent experience based on an advanced lattice Boltzmann flow solver, (Wolfgang, D. B.K.; Nagel, E.; Resch, M. M., High Performance Computing in Science and Engineering ’08 (2009), Springer-Verlag: Springer-Verlag Berlin, Heidelberg), 333-347
[29] Axner, L.; Bernsdorf, J.; Zeiser, T.; Lammers, P.; Linxweiler, J.; Hoekstra, A. G., Performance evaluation of a parallel sparse lattice Boltzmann solver, J. Comput. Phys., 227, 4895-4911 (2008) · Zbl 1137.76045
[30] Junk, M.; Klar, A.; Luo, L.-S., Asymptotic analysis of the lattice Boltzmann equation, J. Comput. Phys., 210, 2, 676-704 (2005) · Zbl 1079.82013
[31] Junk, M.; Yong, W., Rigorous Navier-Stokes limit of the lattice Boltzmann equation, Asymptotic Anal., 35, 2, 165-185 (2003) · Zbl 1043.76003
[32] Bhatnagar, P. L.; Gross, E. P.; Krook, M., A model for collision processes in gases. I. Small amplitude processes in charged and neutral one-component systems, Phys. Rev., 94, 3, 511-525 (1954) · Zbl 0055.23609
[33] Möller, T.; Trumbore, B., Fast, minimum storage ray-triangle intersection, Journal of Graphics Tools, 2, 1, 21-28 (1997)
[34] Popel, A. S.; Johnson, P. C., Microcirculation and hemorheology, Annu. Rev. Fluid Mech., 37, 43-69 (2005) · Zbl 1117.76078
[35] Wootton, D. M.; Ku, D. N., Fluid mechanics of vascular systems, diseases, and thrombosis, Annu. Rev. Fluid Mech., 1, 1, 299-329 (1999)
[36] ANSYS User’s Manual (2000), ANSYS Inc.: ANSYS Inc. Canonsburg, PA
[37] Bathe, K.-J., Finite Element Procedures (1996), Prentice Hall: Prentice Hall Upper Saddle River, NJ
[38] Logan, D. L., A First Course in the Finite Element Method (2002), Brooks/Cole: Brooks/Cole Pacific Grove, CA
[39] R.M. MacMeccan, Mechanistic effects of erythrocytes on platelet deposition in coronary thrombosis, Mechanical Engineering, PhD Dissertation, Georgia Institute of Technology, Atlanta, GA, 2007; R.M. MacMeccan, Mechanistic effects of erythrocytes on platelet deposition in coronary thrombosis, Mechanical Engineering, PhD Dissertation, Georgia Institute of Technology, Atlanta, GA, 2007
[40] Fung, Y. C., Biomechanics: Circulation (1996), Springer-Verlag · Zbl 0366.73091
[41] Merrill, E.; Cokelet, G.; Britten, A.; Wells, R., Non-Newtonian rheology of human blood-effect of fibrinogen deduced by “subtraction”, Circ. Res., 13, 1, 48-55 (1963)
[42] Clausen, J. R.; Aidun, C. K., Galilean invariance in the lattice-Boltzmann method and its effect on the calculation of rheological properties in suspensions, Int. J. Multiphas. Flow, 35, 307-311 (2009)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.