×

A benchmark of selected algorithmic differentiation tools on some problems in computer vision and machine learning. (English) Zbl 1453.65050

Summary: Algorithmic differentiation (AD) allows exact computation of derivatives given only an implementation of an objective function. Although many AD tools are available, a proper and efficient implementation of AD methods is not straightforward. The existing tools are often too different to allow for a general test suite. In this paper, we compare 15 ways of computing derivatives including 11 automatic differentiation tools implementing various methods and written in various languages (C++, F#, MATLAB, Julia and Python), 2 symbolic differentiation tools, finite differences and hand-derived computation.
We look at three objective functions from computer vision and machine learning. These objectives are for the most part simple, in the sense that no iterative loops are involved, and conditional statements are encapsulated in functions such as abs or logsumexp. However, it is important for the success of AD that such ‘simple’ objective functions are handled efficiently, as so many problems in computer vision and machine learning are of this form.

MSC:

65D25 Numerical differentiation
65D19 Computational issues in computer and robotic vision
65Y99 Computer aspects of numerical algorithms
68T05 Learning and adaptive systems in artificial intelligence
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] Ceres Solver. Available at
[2] Bundle Adjustment in the Large, ECCV 2010, Springer Berlin Heidelberg, 2010, pp. 29–42
[3] Agarwal, S.; Furukawa, Y.; Snavely, N.; Simon, I.; Curless, B.; Seitz, S. M.; Szeliski, R., building Rome in a day, Commun. ACM, 54, 105-112, (2011)
[4] Theano: new features and speed improvements, DLUFL NIPS Workshop, 2012
[5] Automatic differentiation in machine learning: a survey, preprint (2015). Available at arXiv:1502.05767
[6] Combining source transformation and operator overloading techniques to compute derivatives for MATLAB programs, SCAM, Washington, DC, 2002, pp. 65–72
[7] Community portal for automatic differentiation. Available at
[8] Davison, A. J.; Reid, I. D.; Molton, N. D.; Stasse, O., monoslam: real-time single camera slam, IEEE Trans. Pattern Anal. Mach. Intell., 29, 1052-1067, (2007)
[9] Dürrbaum, A.; Klier, W.; Hahn, H., comparison of automatic and symbolic differentiation in mathematical modeling and computer simulation of rigid-body systems, Multibody Syst. Dyn., 7, 331-355, (2002) · Zbl 1007.70003
[10] Gebremedhin, A. H.; Nguyen, D.; Patwary, M. M.A.; Pothen, A., colpack: software for graph coloring and related problems in scientific computing, ACM Trans. Math. Softw., 40, 1:1-1:31, (2013) · Zbl 1295.65144
[11] Griewank, A.; Walther, A., Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation, (2008), SIAM, Philadelphia, PA · Zbl 1159.65026
[12] Eigen v3, 2010. Available at
[13] Hartley, R. I.; Zisserman, A., Multiple View Geometry in Computer Vision, (2004), Cambridge University Press, Cambridge · Zbl 1072.68104
[14] Hascoët, L.; Pascual, V., the tapenade automatic differentiation tool: principles, model, and specification, ACM TOMS, 39, 20:1-20:43, (2013) · Zbl 1295.65026
[15] Hogan, R. J., fast reverse-mode automatic differentiation using expression templates in C++, ACM TOMS, 40, 26:1-26:24, (2014) · Zbl 1369.65037
[16] Autograd: Effortless gradients in numpy, ICML 2015 AutoML Workshop, 2015
[17] Narayanan, S. H.K.; Norris, B.; Winnicka, B., ADIC2: development of a component source transformation system for differentiating C and C++, Procedia. Comput. Sci., 1, 1845-1853, (2010)
[18] Patterson, M. A.; Weinstein, M.; Rao, A. V., an efficient overloaded method for computing derivatives of mathematical functions in Matlab, ACM Trans. Math. Softw., 39, 17:1-17:36, (2013) · Zbl 1295.65027
[19] Forward-mode automatic differentiation in Julia, preprint (2016). Available at arXiv:1607.07892 [cs.MS]
[20] Towards fast image-based localization on a city-scale, Outdoor and Large-Scale Real-World Scene Analysis: 15th International Workshop on Theoretical Foundations of Computer Vision, Springer, Berlin/Heidelberg, 2012, pp. 191–211
[21] Reality Skins: Creating Immersive and Tactile Virtual Environments, IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Merida, Mexico, 2016, pp. 115–124
[22] Efficient Implementation of a Higher-Order Language with Built-In AD, AD2016: Programme and Abstracts, Oxford, 2016
[23] Photo tourism: Exploring photo collections in 3D, in SIGGRAPH, Boston, 2006, pp. 835–846
[24] User-Specific Hand Modeling from Monocular Depth Sequences, in CVPR, Columbus, OH, 2014
[25] The MathWorks, Inc., Mupad, MATLAB Symbolic Math Toolbox
[26] Bundle Adjustment – A Modern Synthesis, Proceedings of the International Workshop on Vision Algorithms: Theory and Practice, ICCV, Corfu, Greece, 2000, pp. 298–372
[27] clad. Available at
[28] Getting started with ADOL-C, in Combinatorial Scientific Computing, U. Naumann and O. Schenk, eds., Chapter 7, Chapman-Hall CRC Computational Science, New York, 2012, pp. 181–202
[29] ShadowHands: High-Fidelity Remote Hand Gesture Visualization Using a Hand Tracker, Proceedings of the 2016 ACM on Interactive Surfaces and Spaces, ISS’16, Niagara Falls, Ontario, Canada, ACM, 2016, pp. 77–84
[30] Yu, D.; Deng, L., Automatic Speech Recognition: A Deep Learning Approach, (2014), Springer, London
[31] From learning models of natural image patches to whole image restoration, in ICCV, November, Barcelona, Spain, 2011, pp. 479–486
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.