×

zbMATH — the first resource for mathematics

ESAE: evolutionary strategy-based architecture evolution. (English) Zbl 07239981
Pan, Linqiang (ed.) et al., Bio-inspired computing: theories and applications. 14th international conference, BIC-TA 2019, Zhengzhou, China, November 22–25, 2019. Revised selected papers. Part I. Singapore: Springer (ISBN 978-981-15-3424-9/pbk; 978-981-15-3425-6/ebook). Communications in Computer and Information Science 1159, 193-208 (2020).
Summary: Although deep neural networks (DNNs) play important roles in many fields, the architecture design of DNNs can be challenging due to the difficulty of input data representation, the huge number of parameters and the complex layer relationships. To overcome the obstacles of architecture design, we developed a new method to generate the optimal structure of DNNs, named Evolutionary Strategy-based Architecture Evolution (ESAE), consisting of a bi-level representation and a probability distribution learning approach. The bi-level representation encodes architectures in the gene and parameter levels. The probability distribution learning approach ensures the efficient convergence of the architecture searching process. By using Fashion-MNIST and CIFAR-10, the effectiveness of the proposed ESAS is verified. The evolved DNNs, starting from a trivial initial architecture with one single convolutional layer, achieved the accuracies of 94.48% and 93.49% on Fashion-MNIST and CIFAR-10, respectively, and require remarkably less hardware costs in terms of GPUs and running time, compared with the existing state-of-the-art manual screwed architectures.
For the entire collection see [Zbl 1440.68009].
MSC:
68Q07 Biologically inspired models of computation (DNA computing, membrane computing, etc.)
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Krizhevsky, A., Sutskever, I., Hinton, G.-E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84-90 (2012)
[2] Hinton, G., Deng, L., Yu, D., et al.: Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29(6), 82-97 (2012)
[3] Wu, Y., et al.: Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)
[4] Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Boston, pp. 1-9 (2015)
[5] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
[6] He, K., Zhang, X., Ren, S., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Las Vegas, pp. 770-778 (2016)
[7] Huang, G., Liu, Z., Van, D.M.L., et al.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Puerto Rico, pp. 2261-2269 (2017)
[8] Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13(1), 281-305 (2012) · Zbl 1283.68282
[9] Snoek, J., Rippel, O., Swersky, K., et al.: Scalable Bayesian optimization using deep neural networks. Statistics 37, 1861-1869 (2015)
[10] Miller, G.F., Todd, P.M., Hegde, S.U.: Designing neural networks using genetic algorithms. In: Proceedings of the Third International Conference on Genetic Algorithms, ICGA, San Francisco, pp. 379-384 (1989)
[11] Yao, X., Liu, Y.: A new evolutionary system for evolving artificial neural networks. IEEE Trans. Neural Netw. 8(3), 694-713 (1997)
[12] Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evol. Comput. 10(2), 99-127 (2002)
[13] Stanley, K.O., D’Ambrosio, D.B., Gauci, J.-A.: Hypercube-based encoding for evolving large-scale neural networks. Artif. Life 15(2), 185-212 (2009)
[14] Suganuma, M., Shirakawa, S., Nagao, T.A.: Genetic programming approach to designing convolutional neural network architectures. In: Proceedings of the Genetic and Evolutionary Computation Conference, pp. 497-504. ACM, Berlin (2017)
[15] Miikkulainen, R., et al.: Evolving deep neural networks. In: Artificial Intelligence in the Age of Neural Networks and Brain Computing, pp. 293-312. Academic Press (2017)
[16] Real, E., Aggarwal, A., Huang, Y., et al.: Aging evolution for image classifier architecture search. In: Thirty-Third AAAI Conference on Artificial Intelligence, pp. 5048-5056. AAAI, Hawaii (2019)
[17] Real, E., Moore, S., Selle, A., et al.: Large-scale evolution of image classifiers. In: 34th International Conference on Machine Learning, ICML, Sydney, pp. 2902-2911 (2017)
[18] Assunção, F., et al.: DENSER: deep evolutionary network structured representation. Genet. Program. Evolvable Mach. 20(1), 5-35 (2019)
[19] Zoph, B., Vasudevan, V., Shlens, J., et al.: Learning transferable architectures for scalable image recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Salt Lake, pp. 8697-8710 (2018)
[20] Liu, H., Simonyan, K., Yang, Y.: DARTS: differentiable architecture search. In: International Conference on Learning Representations, ICLR, New Orleans (2019)
[21] Hundt, A., Jain, V., Hager, G.-D.: sharpDARTS: faster and more accurate differentiable architecture search. arXiv preprint arXiv:1903.09900 (2019)
[22] Zhang, J.-R., Zhang, J., Lok, T.-M., et al.: A hybrid particle swarm optimization-back-propagation algorithm for feedforward neural network training. Appl. Math. Comput. 185(2), 1026-1037 (2007) · Zbl 1112.65059
[23] Wang, G.G., Deb, S., et al.: Monarch butterfly optimization. Neural Comput. Appl. 31(7), 1-20 (2015)
[24] Ge, W.G., Suash, D., Santos, C.L.D.: Earthworm optimisation algorithm: a bio-inspired metaheuristic algorithm for global optimisation problems. Int. J. Bio-Inspir. Comput. 12(1), 1-17 (2018)
[25] Wang, G.G., et al.: Elephant herding optimization. In: 2015 3rd International Symposium on Computational and Business Intelligence. IEEE, Bali Indonesia (2015)
[26] Wang, G.-G., et al.: Moth search algorithm: a bio-inspired metaheuristic algorithm for global optimization problems. Mem. Comput. 10(2), 151-164 (2016)
[27] Kiranyaz, S., Ince, T., et al.: Evolutionary artificial neural networks by multi-dimensional particle swarm optimization. Neural Netw. 22(10), 1448-1462 (2009)
[28] Das, G., Pattnaik, P.-K., Padhy, S.-K.: Artificial neural network trained by particle swarm optimization for non-linear channel equalization. Expert Syst. Appl. 41(7), 3491-3496 (2014)
[29] Salama, K.-M., Abdelbar, A.-M.: Learning neural network structures with ant colony algorithms. Swarm Intell. 9(4), 229-265 (2015)
[30] Zhou, X.-H., Zhang, M.X., et al.: Shallow and deep neural network training by water wave optimization. Swarm Evol. Comput. 50, 100561 (2019)
[31] LeCun, Y., Boser, B.E., et al.: Handwritten digit recognition with a back-propagation network. In: Advances in neural Information Processing Systems, pp. 396-404 (1990)
[32] Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017)
[33] Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, University of Toronto (2009)
[34] Gaier, A.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.