Drop-Activation swMATH ID: 41511 Software Authors: Liang, Senwei; Khoo, Yuehaw; Yang, Haizhao Description: Drop-activation: implicit parameter reduction and harmonious regularization. Overfitting frequently occurs in deep learning. In this paper, we propose a novel regularization method called drop-activation to reduce overfitting and improve generalization. The key idea is to drop nonlinear activation functions by setting them to be identity functions randomly during training time. During testing, we use a deterministic network with a new activation function to encode the average effect of dropping activations randomly. Our theoretical analyses support the regularization effect of drop-activation as implicit parameter reduction and verify its capability to be used together with batch normalization [ extit{S. Ioffe} and extit{C. Szegedy}, “Batch normalization: accelerating deep network training by reducing internal covariate shift”, Preprint, url{arXiv:1502.03167}]. The experimental results on CIFAR10, CIFAR100, SVHN, EMNIST, and ImageNet show that drop-activation generally improves the performance of popular neural network architectures for the image classification task. Furthermore, as a regularizer drop-activation can be used in harmony with standard training and regularization techniques such as batch normalization and AutoAugment [ extit{E. D. Cubuk} et al., “AutoAugment: learning augmentation strategies from data”, in: Proceedings of the IEEE conference on computer vision and pattern recognition, CVPR 2019. Long Beach: IEEE Computer Society. 113–123 (2019)]. The code is available at url{https://github.com/LeungSamWai/Drop-Activation}. Homepage: https://arxiv.org/abs/1811.05850 Source Code: https://github.com/LeungSamWai/Drop-Activation Dependencies: Python Keywords: deep learning; image classification; overfitting; regularization Related Software: AutoAugment; CIFAR; GitHub; AlexNet; Zoneout; mixup; EMNIST; ImageNet; MNIST Cited in: 1 Document Standard Articles 1 Publication describing the Software, including 1 Publication in zbMATH Year Drop-activation: implicit parameter reduction and harmonious regularization. Zbl 1477.68280Liang, Senwei; Khoo, Yuehaw; Yang, Haizhao 2021 Cited by 3 Authors 1 Khoo, Yuehaw 1 Liang, Senwei 1 Yang, Haizhao Cited in 1 Serial 1 Communications on Applied Mathematics and Computation Cited in 2 Fields 1 Statistics (62-XX) 1 Computer science (68-XX) Citations by Year