logsumexpsoftmaxtests
swMATH ID:  42124 
Software Authors:  Blanchard, Pierre; Higham, Desmond J.; Higham, Nicholas J. 
Description:  Accurately computing the logsumexp and softmax functions. Evaluating the logsumexp function or the softmax function is a key step in many modern data science algorithms, notably in inference and classification. Because of the exponentials that these functions contain, the evaluation is prone to overflow and underflow, especially in lowprecision arithmetic. Software implementations commonly use alternative formulas that avoid overflow and reduce the chance of harmful underflow, employing a shift or another rewriting. Although mathematically equivalent, these variants behave differently in floatingpoint arithmetic and shifting can introduce subtractive cancellation. We give rounding error analyses of different evaluation algorithms and interpret the error bounds using condition numbers for the functions. We conclude, based on the analysis and numerical experiments, that the shifted formulas are of similar accuracy to the unshifted ones, so can safely be used, but that a divisionfree variant of softmax can suffer from loss of accuracy. 
Homepage:  https://academic.oup.com/imajna/article/41/4/2311/5893596?login=false 
Source Code:  https://github.com/higham/logsumexpsoftmaxtests 
Dependencies:  Matlab 
Keywords:  logsumexp; softmax; floatingpoint arithmetic; rounding error analysis; overflow; underflow; condition number 
Related Software:  SciPy; FEniCS; trlib; FFC; UFC; UFL; GQTPAR; SyFi; DOLFIN; FIAT; PRMLT; MNIST; R; Matlab 
Cited in:  6 Publications 
Standard Articles
1 Publication describing the Software, including 1 Publication in zbMATH  Year 

Accurately computing the logsumexp and softmax functions. Zbl 07528305 Blanchard, Pierre; Higham, Desmond J.; Higham, Nicholas J. 
2021

all
top 5
Cited by 15 Authors
Cited in 4 Serials
2  SIAM Journal on Control and Optimization 
1  IMA Journal of Numerical Analysis 
1  Computational Mathematics and Mathematical Physics 
1  Mathematical Control and Related Fields 