KELLEY swMATH ID: 4829 Software Authors: Kelley, C. T. Description: Iterative methods for optimization This book gives an introduction to optimization methods for unconstrained and bound constrained minimization problems. The style of the book is probably best described by the following quote from the book’s preface: ‘dots{} we treat a small number of methods in depth, giving less detailed description of only a few [dots{}]. We aim for clarity and brevity rather than complete generality and confine our scope to algorithms that are easy to implement (by the reader!) and understand.’ par This book is partitioned into two parts. The first part, occupying approximately 100 pages, is devoted to the optimization of smooth functions. The methods studied in this first part rely on the availability and accuracy of first order, and sometimes also second order derivatives of the objective function. The first part contains five chapters. The first chapter provides basic concepts. It also introduces a parameter identification problem and a discretized optimal control problem, both of which are used to demonstrate all methods discussed in the first part. Chapter 2 studies the local convergence of Newton’s method, inexact Newton methods, and the Gauss-Newton method for the solution of nonlinear least squares problems. Both, overdetermined and underdetermined nonlinear least squares problems are considered. Chapter 3 is devoted to line-search and trust-region methods, which are used to globalize convergence, i.e., remove the restriction that the starting point of the optimization iteration is sufficiently close to a solution. par The BFGS method is studied in chapter 4. A local convergence analysis is provided and implementation details are discussed. Other quasi-Newton methods are sketched. The last chapter of the first part, chapter 5, studies projection methods for the solution of bound constrained problems. All chapters conclude with a demonstration of the methods discussed in the respective chapter using the parameter identification problem and the discretized optimal control problem introduced in chapter 1, and with a set of exercises. par The second part of the book, which is approximately 50 pages long, deals with the optimization of noisy functions. Such optimization problems arise, e.g., when the evaluation of the objective function involves computer simulations. In such cases the noise often introduces artificial minimizers. Gradient information, even if available, cannot expected to be reliable. This second part contains three chapters. The first chapter provides a discussion of noisy functions, basics concepts, and three simple examples that are later used to demonstrate the behavior of optimization algorithms. Chapter 7 introduces implicit filtering, a technique due to the author and his group. Implicit filtering methods use finite difference approximations of the gradient, which are adjusted to the noise level in the function. Direct search algorithms, including the Nelder-Mead, multidirectional search, and the Hooke-Jeves algorithms are discussed in Chapter 8. Again, the latter two chapters conclude with a numerical demonstration of the methods discussed in the respective chapter, and with a set of exercises. par The treatment of both, optimization methods for smooth and for noisy functions is a unique feature of this book. Matlab implementations of all algorithms discussed in this book are easily accessible from the author’s or the publisher’s web-page. Homepage: http://www4.ncsu.edu/~ctk/optimization_codes.html Dependencies: Matlab Keywords: unconstrained optimization; bound constraints; Newton’s method; Gauss-Newton method; quasi-Newton methods; line-search; trust-region; nonlinear least squares; projection methods; noisy functions; direct search methods; implicit filtering Related Software: minpack; Matlab; L-BFGS; IMFIL; NITSOL; DFO; CUTEr; PETSc; NEWUOA; TRON; UOBYQA; NewtonLib; SifDec; SDBOX; mctoolbox; MultiMin; fminsearch; MATLAB ODE suite; LAPACK; DFBOX_IMPR Cited in: 686 Documents all top 5 Cited by 1,221 Authors 24 Kelley, Carl T. 10 Bellavia, Stefania 10 Kevrekidis, Ioannis George 9 Banks, Harvey Thomas 9 Herty, Michael Matthias 8 Martínez, José Mario 8 Morini, Benedetta 8 Olufsen, Mette Sofie 8 Siettos, Constantinos I. 8 Volkwein, Stefan 7 Antil, Harbir 7 Bardsley, Johnathan M. 7 Chacón, Luis 7 Fukushima, Masao 7 Haber, Eldad 7 Hedar, Abdel-Rahman 6 Nagy, James Gerard 6 Ottesen, Johnny Tom 6 Pinnau, René 6 Quyen, Tran Nhan Tam 6 Vicente, Luis Nunes 5 Bai, Zhongzhi 5 Heinkenschloss, Matthias 5 Hinze, Michael 5 Kalita, Jiten C. 5 Klar, Axel 5 Seaïd, Mohammed 5 Yotov, Ivan 4 Bergamaschi, Luca 4 Blanco, Pablo Javier 4 Blauth, Sebastian 4 Cătinaş, Emil 4 Chen, Ke 4 Coutinho, Alvaro L. G. A. 4 Galligani, Emanuele 4 Gear, Charles William 4 Hare, Warren L. 4 Jarry-Bolduc, Gabriel 4 Karátson, János 4 Kavanagh, Kathleen Rose 4 Lapenta, Giovanni 4 Macconi, Maria 4 Martínez, Ángeles 4 Martins, Marcos A. D. 4 Miller, Cass T. 4 Sabo, Kristian 4 Scitovski, Rudolf 4 Tran, Hien T. 4 Wheeler, Mary Fanett 3 Ascher, Uri M. 3 Banda, Mapundi Kondwani 3 Banerjee, Malay 3 Birgin, Ernesto G. 3 Bru, Rafael 3 Buscaglia, Gustavo Carlos 3 Diniz-Ehrhardt, Maria Aparecida 3 Duintjer Tebbens, Jurjen 3 Dussault, Jean-Pierre 3 Elman, Howard C. 3 Fan, Jinyan 3 Gablonsky, Joerg M. 3 Gomes-Ruggiero, Márcia Ap. 3 Gondzio, Jacek 3 Gugat, Martin 3 Hoppe, Ronald H. W. 3 Huang, Jingfang 3 Kees, Christopher E. 3 Kimiaei, Morteza 3 Klibanov, Michael V. 3 Krejić, Nataša 3 Lee, Taeyoung 3 Leiva, Jorge S. 3 Leok, Melvin 3 Li, Zhaoyan 3 McClamroch, N. Harris 3 Mehlsen, Jesper 3 Neumaier, Arnold 3 Planiden, Chayne 3 Raydan, Marcos 3 Reynolds, Daniel R. 3 Russo, Lucia 3 Sachs, Ekkehard W. 3 Sciandrone, Marco 3 Tůma, Miroslav 3 Volpert, Vitaly A. 3 Xiao, Yunhai 3 Xin, Jianguo 3 Yuan, Ya-xiang 3 Zhang, Daoping 2 Ahookhosh, Masoud 2 Al-Baali, Mehiddin 2 Al-Jeiroudi, Ghussoun 2 Alves, José L. D. 2 Amini, Keyvan 2 An, Dong 2 Atkinson, Kendall Eugene 2 Bachmann, Freya 2 Baldick, Ross 2 Brady, Renee 2 Caldwell, Timothy M. ...and 1,121 more Authors all top 5 Cited in 222 Serials 29 Journal of Computational and Applied Mathematics 28 Computational Optimization and Applications 27 Journal of Computational Physics 23 Optimization Methods & Software 20 SIAM Journal on Scientific Computing 18 SIAM Journal on Optimization 17 Inverse Problems 17 Applied Mathematics and Computation 16 Computer Methods in Applied Mechanics and Engineering 13 Journal of Optimization Theory and Applications 12 Optimization and Engineering 10 International Journal for Numerical Methods in Engineering 10 Applied Numerical Mathematics 10 Numerical Algorithms 9 SIAM Journal on Numerical Analysis 9 Numerical Linear Algebra with Applications 8 Computer Physics Communications 8 Mathematics of Computation 7 Inverse Problems in Science and Engineering 6 Journal of Mathematical Biology 6 Automatica 6 Journal of Scientific Computing 5 Computers & Mathematics with Applications 5 Mathematics and Computers in Simulation 5 Numerical Functional Analysis and Optimization 5 Journal of Global Optimization 5 Computational Statistics and Data Analysis 5 SIAM Journal on Imaging Sciences 4 Computers and Fluids 4 International Journal for Numerical Methods in Fluids 4 Mathematical Biosciences 4 BIT 4 Computing 4 SIAM Journal on Control and Optimization 4 Journal of Integral Equations and Applications 4 M\(^3\)AS. Mathematical Models & Methods in Applied Sciences 4 European Journal of Operational Research 4 SIAM Review 4 Mathematical Programming. Series A. Series B 4 Journal of Inverse and Ill-Posed Problems 4 SIAM Journal on Applied Dynamical Systems 4 Multiscale Modeling & Simulation 4 Acta Numerica 4 Optimization Letters 4 Inverse Problems and Imaging 3 IMA Journal of Numerical Analysis 3 Journal of Mathematical Analysis and Applications 3 Bulletin of Mathematical Biology 3 ACM Transactions on Mathematical Software 3 Numerische Mathematik 3 Optimal Control Applications & Methods 3 Journal of Computational Mathematics 3 Numerical Methods for Partial Differential Equations 3 Journal of Economic Dynamics & Control 3 Applied Mathematics Letters 3 Mathematical and Computer Modelling 3 SIAM Journal on Matrix Analysis and Applications 3 International Journal of Computer Mathematics 3 Communications in Numerical Methods in Engineering 3 CEJOR. Central European Journal of Operations Research 3 European Series in Applied and Industrial Mathematics (ESAIM): Mathematical Modelling and Numerical Analysis 3 Frontiers in Applied Mathematics 2 International Journal for Numerical and Analytical Methods in Geomechanics 2 Systems & Control Letters 2 Computational Mechanics 2 Journal of Parallel and Distributed Computing 2 IEEE Transactions on Circuits and Systems. I: Fundamental Theory and Applications 2 Journal of Non-Newtonian Fluid Mechanics 2 Journal of Mathematical Imaging and Vision 2 International Journal of Bifurcation and Chaos in Applied Sciences and Engineering 2 Electronic Journal of Differential Equations (EJDE) 2 ETNA. Electronic Transactions on Numerical Analysis 2 Advances in Computational Mathematics 2 Engineering Analysis with Boundary Elements 2 Mathematical Problems in Engineering 2 Computing and Visualization in Science 2 Multibody System Dynamics 2 Mathematical Methods of Operations Research 2 Chaos 2 Communications in Nonlinear Science and Numerical Simulation 2 Computational Geosciences 2 Engineering Computations 2 International Journal of Computational Methods 2 Mathematical Biosciences and Engineering 2 Pacific Journal of Optimization 2 Advances in Design and Control 2 Communications in Computational Physics 2 Mathematical Modelling of Natural Phenomena 2 Set-Valued and Variational Analysis 2 Mathematical Programming Computation 2 Journal of Theoretical Biology 2 Journal of Computational Dynamics 1 Applicable Analysis 1 Archive for Rational Mechanics and Analysis 1 Biological Cybernetics 1 International Journal of Control 1 International Journal of Engineering Science 1 International Journal of Plasticity 1 Indian Journal of Pure & Applied Mathematics 1 International Journal of Solids and Structures ...and 122 more Serials all top 5 Cited in 45 Fields 396 Numerical analysis (65-XX) 202 Operations research, mathematical programming (90-XX) 121 Partial differential equations (35-XX) 95 Calculus of variations and optimal control; optimization (49-XX) 91 Fluid mechanics (76-XX) 63 Biology and other natural sciences (92-XX) 47 Computer science (68-XX) 47 Systems theory; control (93-XX) 39 Statistics (62-XX) 38 Mechanics of deformable solids (74-XX) 27 Information and communication theory, circuits (94-XX) 25 Ordinary differential equations (34-XX) 25 Optics, electromagnetic theory (78-XX) 19 Geophysics (86-XX) 17 Dynamical systems and ergodic theory (37-XX) 15 Game theory, economics, finance, and other social and behavioral sciences (91-XX) 14 Operator theory (47-XX) 12 Integral equations (45-XX) 12 Statistical mechanics, structure of matter (82-XX) 11 Probability theory and stochastic processes (60-XX) 11 Classical thermodynamics, heat transfer (80-XX) 9 Linear and multilinear algebra; matrix theory (15-XX) 8 Mechanics of particles and systems (70-XX) 6 Approximations and expansions (41-XX) 5 Quantum theory (81-XX) 4 Combinatorics (05-XX) 3 Astronomy and astrophysics (85-XX) 2 General and overarching topics; collections (00-XX) 2 Real functions (26-XX) 2 Difference and functional equations (39-XX) 2 Integral transforms, operational calculus (44-XX) 2 Functional analysis (46-XX) 2 Convex and discrete geometry (52-XX) 2 Differential geometry (53-XX) 2 Relativity and gravitational theory (83-XX) 1 History and biography (01-XX) 1 Field theory and polynomials (12-XX) 1 Commutative algebra (13-XX) 1 Group theory and generalizations (20-XX) 1 Functions of a complex variable (30-XX) 1 Potential theory (31-XX) 1 Sequences, series, summability (40-XX) 1 Harmonic analysis on Euclidean spaces (42-XX) 1 Geometry (51-XX) 1 Global analysis, analysis on manifolds (58-XX) Citations by Year