Efficient computation of derivatives for solving optimization problems in R and Python using SWIG-generated interfaces to ADOL-C.

*(English)*Zbl 1453.90109Summary: Scripting languages are gaining acceptance because of their ease of use and value for rapid prototyping in many fields, including machine learning and statistics. In the context of algorithmic differentiation, however, the main development effort continues to be concentrated on traditional compiled languages such as Fortran and C/C\(++\), whether source transformation tools or operator overloading tools. There is therefore a need for AD tools for computing derivatives efficiently within scripting languages. ADOL-C is an operator overloading-based C\(++\) library that provides accurate first- and higher order derivatives for applications in C\(++\). SWIG is a preprocessor that uses the C/C\(++\) header files to wrap the API of a library to be callable from scripting languages such as R and Python and several other high-level programming languages. Although every language has its caveats, the overall process of making the C/C\(++\) API available via SWIG is the same for all scripting languages. After an initial effort required per language, because SWIG is an automated interface generator based on the library’s actual header files, only minimal effort is required to maintain the scripting interface in sync with upstream developments in the original C/C\(++\) library. In addition to achieving our original goal of creating an interface for R, we were able to generate an interface for Python that proved an order of magnitude faster than the previously implemented interface. This paper gives an overview of the interface generation process, the challenges we encountered with both scripting languages and some numerical results to demonstrate both usefulness and efficiency.

##### MSC:

90C15 | Stochastic programming |

68W30 | Symbolic computation and algebraic computation |

90-04 | Software, source code, etc. for problems pertaining to operations research and mathematical programming |

##### Keywords:

algorithmic differentiation; autodiff; machine learning; stochastic optimization; data analysis; ADOL-C; scripting; R; Python; SWIG##### Software:

OpenAD/F; numDeriv; SciPy; radx; DOLFIN; Numdifftools; pyadolc; Ryacas; CoDiPack; Yacas; PyCppAD; SWIG; Deriv; CppAD; Python; R; AlgoPy; FEniCS; madness; ADOL-C; Theano; SymPy; ADMB; ADIC2; ADIC; dco/c++; TAPENADE
PDF
BibTeX
XML
Cite

\textit{K. Kulshreshtha} et al., Optim. Methods Softw. 33, No. 4--6, 1173--1191 (2018; Zbl 1453.90109)

Full Text:
DOI

**OpenURL**

##### References:

[1] | ADMB—Template Model Builder, 2014. Available at |

[2] | Alnæs, M. S.; Blechta, J.; Hake, J.; Johansson, A.; Kehlet, B.; Logg, A.; Richardson, C.; Ring, J.; Rognes, M. E.; Wells, G. N., the fenics project version 1.5, Arch. Numer. Softw., 3, 9-23, (2015) |

[3] | Automatic Differentiation in R, 2014. Available at |

[4] | SWIG: An easy to use tool for integrating scripting languages with C and C++, 4th Annual Tcl/Tk Workshop, Monterey, CA, 1996. Available at |

[5] | Using SWIG to control, prototype, and debug C programs with Python, 4th International Python Conference, Livermore, CA, 1996. Available at |

[6] | Feeding a large-scale physics application to Python, 6th International Python Conference, San Jose, CA, 1997. Available at |

[7] | CppAD: A package for differentiation of C++ algorithms. Available at |

[8] | Bessac, J.; Constantinescu, E. M.; Anitescu, M., stochastic simulation of predictive space-time scenarios of wind speed using observations and physical model outputs, Ann. Appl. Stat., (2017) |

[9] | Stochastic quasi-newton: Modular implementation of new algorithmAvailable at |

[10] | Numdifftools. Available at |

[11] | Byrd, R. H.; Hansen, S. L.; Nocedal, J.; Singer, Y., A stochastic quasi-Newton method for large-scale optimization, SIAM J. Optim., 26, 1008-1031, (2016) · Zbl 1382.65166 |

[12] | Calculus—SymPy 1.0 Documentation. Available at |

[13] | Applying TAF to generate efficient derivative code of Fortran 77–95 programs, Proceedings of GAMM 2002, Augsburg, Germany, 2003 |

[14] | Ryacas—An R interface to the YACAS computer algebra system, 2014. Available at |

[15] | Griewank, A.; Walther, A., Principles and Techniques of Algorithmic Differentiation, (2008), Philadelphia, SIAM · Zbl 1159.65026 |

[16] | Griewank, A.; Kulshreshtha, K.; Walther, A., on the numerical stability of algorithmic differentiation, Computing, 94, 125-149, (2012) · Zbl 1238.65013 |

[17] | Hascoet, L.; Pascual, V., the tapenade automatic differentiation tool: principles, model, and specification, ACM Trans. Math. Softw., 39, 20:1-20:43, (2013) · Zbl 1295.65026 |

[18] | Hogan, R. J., fast reverse-mode automatic differentiation using expression templates in C++, ACM Trans. Math. Softw., 40, 26:1-26:16, (2014) · Zbl 1369.65037 |

[19] | Kalnay, E., Atmospheric Modeling, Data Assimilation and Predictability, (2003), Cambridge University Press, Cambridge |

[20] | Lamboni, M.; Iooss, B.; Popelin, A. L.; Gamboa, F., derivative-based global sensitivity measures: general links with sobol’ indices and numerical tests, Math. Comput. Simul., 87, 45-54, (2013) |

[21] | Logg, A.; Wells, G. N., DOLFIN: automated finite element computing, ACM Trans. Math. Softw., 37, 1-28, (2010) · Zbl 1364.65254 |

[22] | DOLFIN: A C++/python finite element libraryAutomated Solution of Differential Equations by the Finite Element MethodLecture Notes in Computational Science and Engineering, Vol. 84, chap. 10Springer2012 |

[23] | dco/c++—Derivative code by overloading in C++, Tech. Rep. 2011,06, Aachener Informatik-Berichte, Aachen, 2011 |

[24] | Narayanan, S. H.K.; Norris, B.; Winnicka, B., ADIC2: development of a component source transformation system for differentiating C and C++, Procedia Comput. Sci., 1, 1845-1853, (2010) |

[25] | Package numDeriv. Available at |

[26] | madness: Automatic Differentiation of Multivariate Operations, 2017. Available at , R package version 0.2.2 |

[27] | CoDiPack. Available at |

[28] | Saltelli, A.; Ratto, M.; Andres, T.; Campolongo, F.; Cariboni, J.; Gatelli, D.; Saisana, M.; Tarantola, S., Global Sensitivity Analysis: The Primer, (2008), Wiley, Chichester · Zbl 1161.00304 |

[29] | Scientific Computing Tools for Python—SciPy.org. Available at |

[30] | SWIG website. Available at |

[31] | Symbolic and Algorithmic Derivatives of Simple Expressions. Available at |

[32] | The R Project for Statistical Computing. Available at |

[33] | Theano: A Python framework for fast computation of mathematical expressions, preprint (2016). Available at arXiv e-prints abs/1605.02688. |

[34] | Utke, J.; Naumann, U.; Fagan, M.; Tallent, N.; Strout, M.; Heimbach, P.; Hill, C.; Wunsch, C., openad/F: A modular open-source tool for automatic differentiation of Fortran codes, ACM Trans. Math. Softw., 34, 18:1-18:36, (2008) · Zbl 1291.65140 |

[35] | AD in Python with Application in Science and Engineering, Eighth EuroAD Workshop, The Numerical Algorithms Group, Oxford, 2009 |

[36] | Algorithmic Differentiation in Python with PYADOLC and PYCPPAD, EuroScipy Conference, Leipzig, Germany, 2009 |

[37] | PyADOLC. Available at |

[38] | Walter, S. F.; Lehmann, L., algorithmic differentiation in python with algopy, J. Comput. Sci., 4, 334-344, (2013) |

[39] | Walter, S. F.; Schmidt, A.; Körkel, S., adjoint-based optimization of experimental designs with many control variables, J. Process Control, 24, 1504-1515, (2014) |

[40] | getting started with ADOL-CCombinatorial Scientific ComputingChapman-HallNew York2012181202 |

This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.