Reachable sets of classifiers and regression models: (non-)robustness analysis and robust training. (English) Zbl 07432833

Summary: Neural networks achieve outstanding accuracy in classification and regression tasks. However, understanding their behavior still remains an open challenge that requires questions to be addressed on the robustness, explainability and reliability of predictions. We answer these questions by computing reachable sets of neural networks, i.e. sets of outputs resulting from continuous sets of inputs. We provide two efficient approaches that lead to over- and under-approximations of the reachable set. This principle is highly versatile, as we show. First, we use it to analyze and enhance the robustness properties of both classifiers and regression models. This is in contrast to existing works, which are mainly focused on classification. Specifically, we verify (non-)robustness, propose a robust training procedure, and show that our approach outperforms adversarial attacks as well as state-of-the-art methods of verifying classifiers for non-norm bound perturbations. Second, we provide techniques to distinguish between reliable and non-reliable predictions for unlabeled inputs, to quantify the influence of each feature on a prediction, and compute a feature ranking.


68T05 Learning and adaptive systems in artificial intelligence
Full Text: DOI arXiv


[1] Bastani, O., Ioannou, Y., Lampropoulos, L., Vytiniotis, D., Nori, A. V., & Criminisi, A. (2016). Measuring neural net robustness with constraints. In NeurIPS, Vol. 29.
[2] Brooks, T. F., Pope, D. S., & Marcolini, A. M. (1989). Airfoil self-noise and prediction. NASA Technical Reports.
[3] Bunel, R., Turkaslan, I., Torr, P. H., Kohli, P., & Kumar, M. P. (2018). A unified view of piecewise linear neural network verification. In NeurIPS, Vol. 31, PP. 4795-4804.
[4] Dua, D., & Graff, C. (2017). UCI machine learning repository. In University of California.
[5] Ehlers, R. (2017). Formal verification of piece-wise linear feed-forward neural networks. In Automated Technology for Verification and Analysis, PP. 269-286.
[6] Fisher, R. A. (1936). The use of multiple measurements in taxonomic problems. Annals of Eugenics.
[7] Forina, M., Leardi, R., Armanino, C., & Lanteri, S. (1990). Parvus: An extendable package of programs for data exploration. Journal of Chemometrics.
[8] Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., & Vechev, M. (2018). Ai2: Safety and robustness certification of neural networks with abstract interpretation. In IEEE Symposium on Security and Privacy, PP. 3-18.
[9] Goodfellow, I. J., Shlens, J., & Szegedy, C. (2015). Explaining and harnessing adversarial examples. ICLR.
[10] Gover, E.; Krikorian, N., Determinants and the volumes of parallelotopes and zonotopes, Linear Algebra and its Applications, 433, 28-40 (2010) · Zbl 1194.52009
[11] Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., Arandjelovic, R., Mann, R., & Kohli, P. (2019). Scalable verified training for provably robust image classification. In ICCV, PP. 4841-4850.
[12] Harrison, D.; Rubinfeld, DL, Hedonic prices and the demand for clean air, Journal of Environmental Economics and Management, 5, 81-102 (1978) · Zbl 0375.90023
[13] Hein, M., & Andriushchenko, M. (2017). Formal guarantees on the robustness of a classifier against adversarial manipulation. In NeurIPS, Vol. 30.
[14] Jossinet, J., Variability of impedivity in normal and pathological breast tissue, Medical and Biological Engineering and Computing, 34, 346-350 (1996)
[15] Katz, G.; Barrett, CW; Dill, DL; Julian, K.; Kochenderfer, MJ, Reluplex: An efficient SMT solver for verifying deep neural networks, CAV, 10426, 97-117 (2017)
[16] Kühn, W., Rigorously computed orbits of dynamical systems without the wrapping effect, Computing, 61, 47-67 (1998) · Zbl 0910.65052
[17] LeCun, Y.; Cortes, C.; Burges, CJ, Mnist handwritten digit database (2010), NYU: Courant Institute, NYU
[18] Liu, C.; Arnon, T.; Lazarus, C.; Barrett, CW; Kochenderfer, MJ, Algorithms for verifying deep neural networks, Foundations and Trends in Optimization, 4, 244-404 (2019)
[19] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2018). Towards deep learning models resistant to adversarial attacks. ICLR.
[20] Mirman, M.; Gehr, T.; Vechev, M., Differential abstract interpretation for provably robust neural networks, ICML, 80, 3578-3586 (2018)
[21] Nash, W. J., Sellers, T. L., Talbot, S. R., Cawthorn, A. J., & Ford, W. B. (1994). The population biology of abalone (haliotis species) in tasmania. i. Blacklip abalone (h. rubra) from the north coast and islands of bass strait. Sea Fisheries Division, Technical Report, 48.
[22] Raghunathan, A., Steinhardt, J., & Liang, P. (2018). Semidefinite relaxations for certifying robustness to adversarial examples. In NeurIPS, Vol. 31.
[23] Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “why should i trust you?”: Explaining the predictions of any classifier. In SIGKDD, PP. 1135-1144.
[24] Ruan, W., Huang, X., & Kwiatkowska, M. (2018). Reachability analysis of deep neural networks with provable guarantees. In IJCAI, PP. 2651-2659.
[25] Singh, G., Ganvir, R., Püschel, M., & Vechev, M. (2019a). Beyond the single neuron convex barrier for neural network certification. NeurIPS, 32.
[26] Singh, G., Gehr, T., Mirman, M., Püschel, M., & Vechev, M. (2018). Fast and effective robustness certification. In NeurIPS, Vol. 31.
[27] Singh, G., Gehr, T., Püschel, M., & Vechev, M. (2019). An abstract domain for certifying neural networks. In Proceedings of the ACM Programming Languages, 3.
[28] Singh, G., Gehr, T., Püschel, M., & Vechev, M. (2019). Boosting robustness certification of neural networks. ICLR.
[29] Steinhardt, J., Koh, P. W., & Liang, P. (2017). Certified defenses for data poisoning attacks. In NeurIPS, Vol. 30, PP. 3520-3532.
[30] Street, N.; Wolberg, W.; Mangasarian, OL, Nuclear feature extraction for breast tumor diagnosis, Biomedical Image Processing and Biomedical Visualization, 1905, 861-870 (1999)
[31] Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic attribution for deep networks. ICML, 79.
[32] Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2014). Intriguing properties of neural networks. ICLR.
[33] Tjeng, V., Xiao, K. Y., & Tedrake, R. (2019). Evaluating robustness of neural networks with mixed integer programming. ICLR.
[34] Wong, E.; Kolter, JZ, Provable defenses against adversarial examples via the convex outer adversarial polytope, ICML, 80, 5283-5292 (2018)
[35] Xiang, W., Tran, H.-D., & Johnson, T. (2017). Reachable set computation and safety verification for neural networks with relu activations. CoRR.
[36] Xiao, H., Rasul, K., & Vollgraf, R. (2017). Fashion-mnist: A novel image dataset for benchmarking machine learning algorithms. Zalando SE.
[37] Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2018). Mixup: Beyond empirical risk minimization. ICLR.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.