Deterministic global optimization with artificial neural networks embedded. (English) Zbl 1407.90263

Summary: Artificial neural networks are used in various applications for data-driven black-box modeling and subsequent optimization. Herein, we present an efficient method for deterministic global optimization of optimization problems with artificial neural networks embedded. The proposed method is based on relaxations of algorithms using McCormick relaxations in a reduced space [the second author et al., SIAM J. Optim. 20, No. 2, 573–601 (2009; Zbl 1192.65083)] employing the convex and concave envelopes of the nonlinear activation function. The optimization problem is solved using our in-house deterministic global solver. The performance of the proposed method is shown in four optimization examples: an illustrative function, a fermentation process, a compressor plant and a chemical process. The results show that computational solution time is favorable compared to a state-of-the-art global general-purpose optimization solver.


90C26 Nonconvex programming, global optimization
90C30 Nonlinear programming
90C90 Applications of mathematical programming
68T01 General topics in artificial intelligence


Zbl 1192.65083
Full Text: DOI arXiv


[1] Hornik, K.; Stinchcombe, M.; White, H., Multilayer feedforward networks are universal approximators, Neural Netw., 2, 359-366, (1989) · Zbl 1383.92015
[2] Gasteiger, J.; Zupan, J., Neural networks in chemistry, Angew. Chem. Int. Edn. Engl., 32, 503-527, (1993)
[3] Azlan Hussain, M., Review of the applications of neural networks in chemical process control—simulation and online implementation, Artif. Intell. Eng., 13, 55-68, (1999)
[4] Agatonovic-Kustrin, S.; Beresford, R., Basic concepts of artificial neural network modeling and its application in pharmaceutical research, J. Pharm. Biomed. Anal., 22, 717-727, (2000)
[5] Witek-Krowiak, A.; Chojnacka, K.; Podstawczyk, D.; Dawiec, A.; Pokomeda, K., Application of response surface methodology and artificial neural network methods in modelling and optimization of biosorption process, Bioresour. Technol., 160, 150-160, (2014)
[6] Meireles, M.; Almeida, P.; Simoes, MG, A comprehensive review for industrial applicability of artificial neural networks, IEEE Trans. Ind. Electron., 50, 585-601, (2003)
[7] Rio-Chanona, EA; Fiorelli, F.; Zhang, D.; Ahmed, NR; Jing, K.; Shah, N., An efficient model construction strategy to simulate microalgal lutein photo-production dynamic process, Biotechnol. Bioeng., 114, 2518-2527, (2017)
[8] Cheema, JJS; Sankpal, NV; Tambe, SS; Kulkarni, BD, Genetic programming assisted stochastic optimization strategies for optimization of glucose to gluconic acid fermentation, Biotechnol. Progr., 18, 1356-1365, (2002)
[9] Desai, KM; Survase, SA; Saudagar, PS; Lele, SS; Singhal, RS, Comparison of artificial neural network and response surface methodology in fermentation media optimization: case study of fermentative production of scleroglucan, Biochem. Eng. J., 41, 266-273, (2008)
[10] Nagata, Y.; Chu, KH, Optimization of a fermentation medium using neural networks and genetic algorithms, Biotechnol. Lett., 25, 1837-1842, (2003)
[11] Fahmi, I.; Cremaschi, S., Process synthesis of biodiesel production plant using artificial neural networks as the surrogate models, Comput. Chem. Eng., 46, 105-123, (2012)
[12] Nascimento, CAO; Giudici, R.; Guardani, R., Neural network based approach for optimization of industrial chemical processes, Comput. Chem. Eng., 24, 2303-2314, (2000)
[13] Nascimento, CAO; Giudici, R., Neural network based approach for optimisation applied to an industrial nylon-6,6 polymerisation process, Comput. Chem. Eng., 22, 595-s600, (1998)
[14] Chambers, M.; Mount-Campbell, CA, Process optimization via neural network metamodeling, Int. J. Prod. Econ., 79, 93-100, (2002)
[15] Henao, C.A., Maravelias, C.T.: Surrogate-based process synthesis. In: Pierucci, S., Ferraris, G.B. (eds.) 20th European Symposium on Computer Aided Process Engineering, Computer Aided Chemical Engineering, vol. 28, pp. 1129-1134. Elsevier, Milan, Italy (2010). https://doi.org/10.1016/S1570-7946(10)28189-0
[16] Henao, CA; Maravelias, CT, Surrogate-based superstructure optimization framework, AIChE J., 57, 1216-1232, (2011)
[17] Sant Anna, HR; Barreto, AG; Tavares, FW; Souza, MB, Machine learning model and optimization of a PSA unit for methane-nitrogen separation, Comput. Chem. Eng., 104, 377-391, (2017)
[18] Smith, JD; Neto, AA; Cremaschi, S.; Crunkleton, DW, CFD-based optimization of a flooded bed algae bioreactor, Ind. Eng. Chem. Res., 52, 7181-7188, (2013)
[19] Henao, C.A.: A superstructure modeling framework for process synthesis using surrogate models. Dissertation, University of Wisconsin, Madison (2012)
[20] Kajero, OT; Chen, T.; Yao, Y.; Chuang, YC; Wong, DSH, Meta-modelling in chemical process system engineering, J. Taiwan Inst. Chem. Eng., 73, 135-145, (2017)
[21] Lewandowski, J., Lemcoff, N.O., Palosaari, S.: Use of neural networks in the simulation and optimization of pressure swing adsorption processes. Chem. Eng. Technol. 21(7), 593-597 (1998). https://doi.org/10.1002/(SICI)1521-4125(199807)21:7\(<\)593::AID-CEAT593\(>\)3.0.CO;2-U
[22] Gutiérrez-Antonio, C.: Multiobjective stochastic optimization of dividing-wall distillation columns using a surrogate model based on neural networks. Chem. Biochem. Eng. Q. 29(4), 491-504 (2016). https://doi.org/10.15255/CABEQ.2014.2132
[23] Chen, CR; Ramaswamy, HS, Modeling and optimization of variable retort temperature thermal processing using coupled neural networks and genetic algorithms, J. Food Eng., 53, 209-220, (2002)
[24] Dornier, M.; Decloux, M.; Trystram, G.; Lebert, A., Interest of neural networks for the optimization of the crossflow filtration process, LWT-Food Sci. Technol., 28, 300-309, (1995)
[25] Fernandes, FAN, Optimization of Fischer-Tropsch synthesis using neural networks, Chem. Eng. Technol., 29, 449-453, (2006)
[26] Grossmann, I.E., Viswanathan, J., Vecchietti, A., Raman, R., Kalvelagen, E.: GAMS/DICOPT: A discrete continuous optimization package. GAMS Corporation Inc, Cary (2002)
[27] Drud, AS, Conopt—a large-scale GRG code, ORSA J. Comput., 6, 207-216, (1994) · Zbl 0806.90113
[28] Nandi, S.; Ghosh, S.; Tambe, SS; Kulkarni, BD, Artificial neural-network-assisted stochastic process optimization strategies, AIChE J., 47, 126-141, (2001)
[29] Tawarmalani, M.; Sahinidis, NV, A polyhedral branch-and-cut approach to global optimization, Math. Program., 103, 225-249, (2005) · Zbl 1099.90047
[30] Weerdt, E.; Chu, QP; Mulder, JA, Neural network output optimization using interval analysis, IEEE Trans. Neural Netw., 20, 638-653, (2009)
[31] Moore, R.E., Bierbaum, F.: Methods and Applications of Interval Analysis, 2 edn. SIAM Studies in Applied Mathematics. Society for Industrial and Applied Mathematics, Philadelphia (1979). https://doi.org/10.1137/1.9781611970906
[32] Misener, R.; Floudas, CA, ANTIGONE: Algorithms for continuous/integer global optimization of nonlinear equations, J. Glob. Optim., 59, 503-526, (2014) · Zbl 1301.90063
[33] Maher, S.J., Fischer, T., Gally, T., Gamrath, G., Gleixner, A., Gottwald, R.L., Hendel, G., Koch, T., Lübbecke, M.E., Miltenberger, M., Müller, B., Pfetsch, M.E., Puchert, C., Rehfeldt, D., Schenker, S., Schwarz, R., Serrano, F., Shinano, Y., Weninger, D., Witt, J.T., Witzig, J.: The SCIP optimization suite (version 4.0)
[34] Epperly, TGW; Pistikopoulos, EN, A reduced space branch and bound algorithm for global optimization, J. Glob. Optim., 11, 287-311, (1997) · Zbl 1040.90567
[35] Mitsos, A.; Chachuat, B.; Barton, PI, McCormick-based relaxations of algorithms, SIAM J. Optim., 20, 573-601, (2009) · Zbl 1192.65083
[36] Scott, JK; Stuber, MD; Barton, PI, Generalized McCormick relaxations, J. Glob. Optim., 51, 569-606, (2011) · Zbl 1232.49033
[37] Bongartz, D.; Mitsos, A., Deterministic global optimization of process flowsheets in a reduced space using McCormick relaxations, J. Glob. Optim., 20, 419, (2017) · Zbl 1386.90112
[38] Huster, WR; Bongartz, D.; Mitsos, A., Deterministic global optimization of the design of a geothermal organic rankine cycle, Energy Proc., 129, 50-57, (2017)
[39] McCormick, GP, Computability of global solutions to factorable nonconvex programs: part I—convex underestimating problems, Math. Program., 10, 147-175, (1976) · Zbl 0349.90100
[40] Bompadre, A.; Mitsos, A., Convergence rate of McCormick relaxations, J. Glob. Optim., 52, 1-28, (2012) · Zbl 1257.90077
[41] Najman, J.; Mitsos, A., Convergence analysis of multivariate McCormick relaxations, J. Glob. Optim., 66, 597-628, (2016) · Zbl 1394.90471
[42] Tsoukalas, A.; Mitsos, A., Multivariate McCormick relaxations, J. Glob. Optim., 59, 633-662, (2014) · Zbl 1312.90068
[43] Najman, J.; Bongartz, D.; Tsoukalas, A.; Mitsos, A., Erratum to multivariate McCormick relaxations, J. Glob. Optim., 68, 219-225, (2017) · Zbl 1476.90265
[44] Khan, KA; Watson, HAJ; Barton, PI, Differentiable McCormick relaxations, J. Glob. Optim., 67, 687-729, (2017) · Zbl 1365.49027
[45] Khan, KA; Wilhelm, M.; Stuber, MD; Cao, H.; Watson, HAJ; Barton, PI, Corrections to differentiable McCormick relaxations, J. Glob. Optim., 70, 705-706, (2018) · Zbl 1447.49042
[46] Bongartz, D., Mitsos, A.: Infeasible path global flowsheet optimization using McCormick relaxations. In: Espuna, A. (ed.) 27th European Symposium on Computer Aided Process Engineering, Computer Aided Chemical Engineering, vol. 40. Elsevier, San Diego (2017). https://doi.org/10.1016/B978-0-444-63965-3.50107-0 · Zbl 1386.90112
[47] Wechsung, A.; Scott, JK; Watson, HAJ; Barton, PI, Reverse propagation of McCormick relaxations, J. Glob. Optim., 63, 1-36, (2015) · Zbl 1322.49048
[48] Stuber, MD; Scott, JK; Barton, PI, Convex and concave relaxations of implicit functions, Optim. Methods Softw., 30, 424-460, (2015) · Zbl 1327.65114
[49] Bishop, C.M.: Pattern Recognition and Machine Learning. Information Science and Statistics, 8th edn. Springer, New York (2009)
[50] Bertsekas, D.P., Nedic, A., Ozdaglar, A.E.: Convex Analysis and Optimization, Athena Scientific Optimization and Computation Series, vol. 1. Athena Scientific, Belmont (2003) · Zbl 1140.90001
[51] Bongartz, D., Najman, J., Sass, S., Mitsos, A.: MAiNGO: McCormick based Algorithm for mixed integer Nonlinear Global Optimization. Technical report (2018)
[52] Chachuat, B.: MC++ (version 2.0): A toolkit for bounding factorable functions (2014)
[53] Chachuat, B.; Houska, B.; Paulen, R.; Peri’c, N.; Rajyaguru, J.; Villanueva, ME, Set-theoretic approaches in analysis, estimation and control of nonlinear systems, IFAC-PapersOnLine, 48, 981-995, (2015)
[54] Hofschuster, W., Krämer, W.: FILIB++ Interval Library (version 3.0.2) (1998)
[55] International Business Machies: IBM ilog CPLEX (version 12.1) (2009)
[56] Gleixner, AM; Berthold, T.; Müller, B.; Weltge, S., Three enhancements for optimization-based bound tightening, J. Glob. Optim., 67, 731-757, (2017) · Zbl 1369.90106
[57] Ryoo, HS; Sahinidis, NV, Global optimization of nonconvex NLPs and MINLPs with applications in process design, Comput. Chem. Eng., 19, 551-566, (1995)
[58] Locatelli, M., Schoen, F. (eds.): Global optimization: theory, algorithms, and applications. MOS-SIAM series on optimization. Mathematical Programming Society, Philadelphia, PA (2013). https://doi.org/10.1137/1.9781611972672 · Zbl 1286.90003
[59] Kraft, D.: A software package for sequential quadratic programming. Deutsche Forschungs- und Versuchsanstalt für Luft- und Raumfahrt Köln: Forschungsbericht. Wiss. Berichtswesen d. DFVLR, Köln (1988)
[60] Johnson, S.G.: The NLopt nonlinear-optimization package (version 2.4.2) (2016)
[61] Bendtsen, C., Stauning, O.: Fadbad++ (version 2.1): a flexible C++ package for automatic differentiation (2012)
[62] Najman, J., Mitsos, A.: Tighter McCormick relaxations through subgradient propagation. Optimization online. http://www.optimization-online.org/DB_FILE/2017/10/6296.pdf (2017)
[63] Ghorbanian, K.; Gholamrezaei, M., An artificial neural network approach to compressor performance prediction, Appl. Energy, 86, 1210-1221, (2009)
[64] Luyben, WL, Design and control of the cumene process, Ind. Eng. Chem. Res., 49, 719-734, (2010)
[65] Schultz, ES; Trierweiler, JO; Farenzena, M., The importance of nominal operating point selection in self-optimizing control, Ind. Eng. Chem. Res., 55, 7381-7393, (2016)
[66] Lee, U.; Burre, J.; Caspari, A.; Kleinekorte, J.; Schweidtmann, AM; Mitsos, A., Techno-economic optimization of a green-field post-combustion CO\(_2\) capture process using superstructure and rate-based models, Ind. Eng. Chem. Res., 55, 12014-12026, (2016)
[67] Helmdach, D.; Yaseneva, P.; Heer, PK; Schweidtmann, AM; Lapkin, AA, A multiobjective optimization including results of life cycle assessment in developing biorenewables-based processes, ChemSusChem, 10, 3632-3643, (2017)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.