Zoneout swMATH ID: 36444 Software Authors: David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, Chris Pal Description: Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations. We propose zoneout, a novel method for regularizing RNNs. At each timestep, zoneout stochastically forces some hidden units to maintain their previous values. Like dropout, zoneout uses random noise to train a pseudo-ensemble, improving generalization. But by preserving instead of dropping hidden units, gradient information and state information are more readily propagated through time, as in feedforward stochastic depth networks. We perform an empirical investigation of various RNN regularizers, and find that zoneout gives significant performance improvements across tasks. We achieve competitive results with relatively simple models in character- and word-level language modelling on the Penn Treebank and Text8 datasets, and combining with recurrent batch normalization yields state-of-the-art results on permuted sequential MNIST. Homepage: https://arxiv.org/abs/1606.01305 Source Code: https://github.com/teganmaharaj/zoneout Dependencies: Mathematica Keywords: arXiv_cs.NE; arXiv_cs.CL; Machine Learning; arXiv_cs.LG; Regularizing RNNs; Preserving Hidden Activations; zoneout Related Software: Adam; Clockwork RNN; ImageNet; MNIST; IMDB; UCF101; word2vec; Skip RNN; BinaryConnect; Theano; RMSprop; Penn Treebank; SGDR; GELUs; GitHub; Drop-Activation; AutoAugment; AlexNet; mixup; EMNIST Cited in: 2 Publications Standard Articles 1 Publication describing the Software Year Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, Chris Pal 2016 all top 5 Cited by 7 Authors 1 Bengio, Yoshua 1 Chandar, Sarath 1 Cho, Kyunghyun 1 Gülçehre, Çağlar 1 Khoo, Yuehaw 1 Liang, Senwei 1 Yang, Haizhao Cited in 2 Serials 1 Neural Computation 1 Communications on Applied Mathematics and Computation Cited in 2 Fields 2 Computer science (68-XX) 1 Statistics (62-XX) Citations by Year