## KELLEY

 swMATH ID: 4829 Software Authors: Kelley, C.T. Description: Iterative methods for optimization This book gives an introduction to optimization methods for unconstrained and bound constrained minimization problems. The style of the book is probably best described by the following quote from the book’s preface: ‘dots{} we treat a small number of methods in depth, giving less detailed description of only a few [dots{}]. We aim for clarity and brevity rather than complete generality and confine our scope to algorithms that are easy to implement (by the reader!) and understand.’ par This book is partitioned into two parts. The first part, occupying approximately 100 pages, is devoted to the optimization of smooth functions. The methods studied in this first part rely on the availability and accuracy of first order, and sometimes also second order derivatives of the objective function. The first part contains five chapters. The first chapter provides basic concepts. It also introduces a parameter identification problem and a discretized optimal control problem, both of which are used to demonstrate all methods discussed in the first part. Chapter 2 studies the local convergence of Newton’s method, inexact Newton methods, and the Gauss-Newton method for the solution of nonlinear least squares problems. Both, overdetermined and underdetermined nonlinear least squares problems are considered. Chapter 3 is devoted to line-search and trust-region methods, which are used to globalize convergence, i.e., remove the restriction that the starting point of the optimization iteration is sufficiently close to a solution. par The BFGS method is studied in chapter 4. A local convergence analysis is provided and implementation details are discussed. Other quasi-Newton methods are sketched. The last chapter of the first part, chapter 5, studies projection methods for the solution of bound constrained problems. All chapters conclude with a demonstration of the methods discussed in the respective chapter using the parameter identification problem and the discretized optimal control problem introduced in chapter 1, and with a set of exercises. par The second part of the book, which is approximately 50 pages long, deals with the optimization of noisy functions. Such optimization problems arise, e.g., when the evaluation of the objective function involves computer simulations. In such cases the noise often introduces artificial minimizers. Gradient information, even if available, cannot expected to be reliable. This second part contains three chapters. The first chapter provides a discussion of noisy functions, basics concepts, and three simple examples that are later used to demonstrate the behavior of optimization algorithms. Chapter 7 introduces implicit filtering, a technique due to the author and his group. Implicit filtering methods use finite difference approximations of the gradient, which are adjusted to the noise level in the function. Direct search algorithms, including the Nelder-Mead, multidirectional search, and the Hooke-Jeves algorithms are discussed in Chapter 8. Again, the latter two chapters conclude with a numerical demonstration of the methods discussed in the respective chapter, and with a set of exercises. par The treatment of both, optimization methods for smooth and for noisy functions is a unique feature of this book. Matlab implementations of all algorithms discussed in this book are easily accessible from the author’s or the publisher’s web-page. Homepage: http://www4.ncsu.edu/~ctk/optimization_codes.html Dependencies: Matlab Keywords: unconstrained optimization; bound constraints; Newton’s method; Gauss-Newton method; quasi-Newton methods; line-search; trust-region; nonlinear least squares; projection methods; noisy functions; direct search methods; implicit filtering Related Software: minpack; Matlab; L-BFGS; IMFIL; NITSOL; DFO; CUTEr; NEWUOA; PETSc; TRON; UOBYQA; mctoolbox; NewtonLib; SifDec; fminsearch; LAPACK; SDBOX; tn; L-BFGS-B; DFBOX_IMPR Cited in: 649 Publications
all top 5

### Cited by 1,142 Authors

 24 Kelley, Carl T. 10 Bellavia, Stefania 10 Kevrekidis, Ioannis George 9 Banks, Harvey Thomas 9 Herty, Michael Matthias 8 Martínez, José Mario 8 Morini, Benedetta 8 Volkwein, Stefan 7 Antil, Harbir 7 Bardsley, Johnathan M. 7 Chacón, Luis 7 Fukushima, Masao 7 Haber, Eldad 7 Hedar, Abdel-Rahman 7 Olufsen, Mette Sofie 7 Siettos, Constantinos I. 6 Nagy, James Gerard 6 Ottesen, Johnny Tom 6 Quyen, Tran Nhan Tam 6 Vicente, Luis Nunes 5 Bai, Zhongzhi 5 Coutinho, Alvaro L. G. A. 5 Heinkenschloss, Matthias 5 Hinze, Michael 5 Kalita, Jiten C. 5 Klar, Axel 5 Pinnau, René 5 Seaïd, Mohammed 5 Yotov, Ivan 4 Bergamaschi, Luca 4 Blanco, Pablo Javier 4 Cătinaş, Emil 4 Chen, Ke 4 Galligani, Emanuele 4 Gear, Charles William 4 Karátson, János 4 Kavanagh, Kathleen Rose 4 Lapenta, Giovanni 4 Macconi, Maria 4 Martínez, Ángeles 4 Martins, Marcos A. D. 4 Miller, Cass T. 4 Sabo, Kristian 4 Scitovski, Rudolf 4 Tran, Hien T. 4 Wheeler, Mary Fanett 3 Ascher, Uri M. 3 Banda, Mapundi Kondwani 3 Birgin, Ernesto G. 3 Bru, Rafael 3 Buscaglia, Gustavo Carlos 3 Diniz-Ehrhardt, Maria Aparecida 3 Duintjer Tebbens, Jurjen 3 Dussault, Jean-Pierre 3 Elman, Howard C. 3 Fan, Jinyan 3 Gablonsky, Joerg M. 3 Gomes-Ruggiero, Márcia Ap. 3 Gondzio, Jacek 3 Gugat, Martin 3 Hare, Warren L. 3 Hoppe, Ronald H. W. 3 Huang, Jingfang 3 Jarry-Bolduc, Gabriel 3 Kees, Christopher E. 3 Klibanov, Michael V. 3 Krejić, Nataša 3 Lee, Taeyoung 3 Leiva, Jorge S. 3 Li, Zhaoyan 3 McClamroch, N. Harris 3 Mehlsen, Jesper 3 Regis, Rommel G. 3 Reynolds, Daniel R. 3 Sachs, Ekkehard W. 3 Sciandrone, Marco 3 Tůma, Miroslav 3 Xiao, Yunhai 3 Xin, Jianguo 3 Yuan, Ya-xiang 3 Zhang, Daoping 2 Ahookhosh, Masoud 2 Al-Baali, Mehiddin 2 Al-Jeiroudi, Ghussoun 2 Alves, José L. D. 2 Amini, Keyvan 2 An, Dong 2 Atkinson, Kendall Eugene 2 Bachmann, Freya 2 Baldick, Ross 2 Banerjee, Malay 2 Blauth, Sebastian 2 Brady, Renee 2 Caldwell, Timothy M. 2 Caliari, Marco 2 Calvetti, Daniela 2 Cardona, Alberto 2 Chan, Swee-Huat 2 Chan, Tony Fan-Cheong 2 Chen, Xiaojun ...and 1,042 more Authors
all top 5

### Cited in 209 Serials

 28 Journal of Computational and Applied Mathematics 28 Computational Optimization and Applications 26 Journal of Computational Physics 22 Optimization Methods & Software 19 SIAM Journal on Scientific Computing 18 SIAM Journal on Optimization 17 Applied Mathematics and Computation 16 Computer Methods in Applied Mechanics and Engineering 15 Inverse Problems 13 Journal of Optimization Theory and Applications 12 Optimization and Engineering 10 International Journal for Numerical Methods in Engineering 10 Applied Numerical Mathematics 10 Numerical Algorithms 9 SIAM Journal on Numerical Analysis 9 Numerical Linear Algebra with Applications 8 Computer Physics Communications 8 Mathematics of Computation 7 Inverse Problems in Science and Engineering 6 Automatica 6 Journal of Scientific Computing 5 Computers & Mathematics with Applications 5 Journal of Mathematical Biology 5 Mathematics and Computers in Simulation 5 Numerical Functional Analysis and Optimization 5 Journal of Global Optimization 5 Computational Statistics and Data Analysis 4 Computers and Fluids 4 International Journal for Numerical Methods in Fluids 4 BIT 4 Computing 4 Journal of Integral Equations and Applications 4 M$$^3$$AS. Mathematical Models & Methods in Applied Sciences 4 European Journal of Operational Research 4 Mathematical Programming. Series A. Series B 4 Journal of Inverse and Ill-Posed Problems 4 SIAM Journal on Applied Dynamical Systems 4 Multiscale Modeling & Simulation 4 Inverse Problems and Imaging 4 SIAM Journal on Imaging Sciences 3 IMA Journal of Numerical Analysis 3 Mathematical Biosciences 3 Bulletin of Mathematical Biology 3 Numerische Mathematik 3 SIAM Journal on Control and Optimization 3 Optimal Control Applications & Methods 3 Journal of Computational Mathematics 3 Numerical Methods for Partial Differential Equations 3 Journal of Economic Dynamics & Control 3 Applied Mathematics Letters 3 Mathematical and Computer Modelling 3 SIAM Journal on Matrix Analysis and Applications 3 International Journal of Computer Mathematics 3 SIAM Review 3 Communications in Numerical Methods in Engineering 3 CEJOR. Central European Journal of Operations Research 3 Acta Numerica 3 Frontiers in Applied Mathematics 2 International Journal for Numerical and Analytical Methods in Geomechanics 2 Journal of Mathematical Analysis and Applications 2 ACM Transactions on Mathematical Software 2 Systems & Control Letters 2 Computational Mechanics 2 Journal of Parallel and Distributed Computing 2 IEEE Transactions on Circuits and Systems. I: Fundamental Theory and Applications 2 Journal of Non-Newtonian Fluid Mechanics 2 Journal of Mathematical Imaging and Vision 2 International Journal of Bifurcation and Chaos in Applied Sciences and Engineering 2 Electronic Journal of Differential Equations (EJDE) 2 ETNA. Electronic Transactions on Numerical Analysis 2 Advances in Computational Mathematics 2 Engineering Analysis with Boundary Elements 2 Mathematical Problems in Engineering 2 Computing and Visualization in Science 2 Mathematical Methods of Operations Research 2 Computational Geosciences 2 Engineering Computations 2 Mathematical Biosciences and Engineering 2 European Series in Applied and Industrial Mathematics (ESAIM): Mathematical Modelling and Numerical Analysis 2 Pacific Journal of Optimization 2 Advances in Design and Control 2 Optimization Letters 2 Communications in Computational Physics 2 Mathematical Modelling of Natural Phenomena 2 Journal of Theoretical Biology 2 Journal of Computational Dynamics 1 Applicable Analysis 1 Archive for Rational Mechanics and Analysis 1 Biological Cybernetics 1 International Journal of Control 1 International Journal of Engineering Science 1 International Journal of Plasticity 1 Indian Journal of Pure & Applied Mathematics 1 International Journal of Solids and Structures 1 Journal of Engineering Mathematics 1 Journal of Fluid Mechanics 1 Journal of the Franklin Institute 1 Mathematical Methods in the Applied Sciences 1 Wave Motion 1 Chaos, Solitons and Fractals ...and 109 more Serials
all top 5

### Cited in 46 Fields

 376 Numerical analysis (65-XX) 189 Operations research, mathematical programming (90-XX) 116 Partial differential equations (35-XX) 89 Calculus of variations and optimal control; optimization (49-XX) 89 Fluid mechanics (76-XX) 60 Biology and other natural sciences (92-XX) 46 Computer science (68-XX) 43 Systems theory; control (93-XX) 36 Mechanics of deformable solids (74-XX) 35 Statistics (62-XX) 27 Information and communication theory, circuits (94-XX) 24 Optics, electromagnetic theory (78-XX) 23 Ordinary differential equations (34-XX) 17 Dynamical systems and ergodic theory (37-XX) 17 Geophysics (86-XX) 13 Game theory, economics, finance, and other social and behavioral sciences (91-XX) 12 Integral equations (45-XX) 12 Operator theory (47-XX) 12 Statistical mechanics, structure of matter (82-XX) 11 Classical thermodynamics, heat transfer (80-XX) 10 Probability theory and stochastic processes (60-XX) 9 Linear and multilinear algebra; matrix theory (15-XX) 7 Approximations and expansions (41-XX) 7 Mechanics of particles and systems (70-XX) 4 Combinatorics (05-XX) 4 Quantum theory (81-XX) 3 Astronomy and astrophysics (85-XX) 2 General and overarching topics; collections (00-XX) 2 Sequences, series, summability (40-XX) 2 Integral transforms, operational calculus (44-XX) 2 Functional analysis (46-XX) 2 Convex and discrete geometry (52-XX) 2 Differential geometry (53-XX) 2 Relativity and gravitational theory (83-XX) 1 History and biography (01-XX) 1 Field theory and polynomials (12-XX) 1 Commutative algebra (13-XX) 1 Group theory and generalizations (20-XX) 1 Real functions (26-XX) 1 Functions of a complex variable (30-XX) 1 Potential theory (31-XX) 1 Difference and functional equations (39-XX) 1 Harmonic analysis on Euclidean spaces (42-XX) 1 Geometry (51-XX) 1 Global analysis, analysis on manifolds (58-XX) 1 Mathematics education (97-XX)