PASCAL VOC swMATH ID: 36507 Software Authors: Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C. K. I., Winn, J., Zisserman, A Description: The PASCAL Visual Object Classes Challenge: A Retrospective. The PASCAL Visual Object Classes (VOC) challenge consists of two components: (i) a publicly available dataset of images together with ground truth annotation and standardised evaluation software; and (ii) an annual competition and workshop. There are five challenges: classification, detection, segmentation, action classification, and person layout. In this paper we provide a review of the challenge from 2008–2012. The paper is intended for two audiences: algorithm designers, researchers who want to see what the state of the art is, as measured by performance on the VOC datasets, along with the limitations and weak points of the current generation of algorithms; and, challenge designers, who want to see what we as organisers have learnt from the process and our recommendations for the organisation of future challenges. To analyse the performance of submitted algorithms on the VOC datasets we introduce a number of novel evaluation methods: a bootstrapping method for determining whether differences in the performance of two algorithms are significant or not; a normalised average precision so that performance can be compared across classes with different proportions of positive instances; a clustering method for visualising the performance across multiple algorithms so that the hard and easy images can be identified; and the use of a joint classifier over the submitted algorithms in order to measure their complementarity and combined performance. We also analyse the community’s progress through time using the methods of Hoiem et al. (Proceedings of European Conference on Computer Vision, 2012) to identify the types of occurring errors. We conclude the paper with an appraisal of the aspects of the challenge that worked well, and those that could be improved in future challenges. Homepage: http://host.robots.ox.ac.uk/pascal/VOC/ Related Software: SIFT; ImageNet; MS-COCO; LabelMe; AlexNet; Caltech-256; LIBSVM; GrabCut; DeepLab; Faster R-CNN; SURF; KITTI; pix2pix; CycleGAN; SSD; EMD; WordNet; Caffe; PCA-SIFT; BSDS Cited in: 82 Publications all top 5 Cited by 297 Authors 3 Torr, Philip H. S. 3 Yang, Ming-Hsuan 2 Avidan, Shai 2 Ma, Yi 2 Meng, Deyu 2 Tao, Dacheng 2 van de Weijer, Joost 2 Verbeek, Jakob J. 2 Waegeman, Willem 2 Zhang, Jing 2 Zhang, Tianzhu 1 Abdullah, Azizi 1 Abdusalomov, Akmalbek 1 Ahmad, M. Omair 1 Ahmad, Shahzor 1 Ahuja, Narendra 1 Ajanthan, Thalaiyasingam 1 Amerini, Irene 1 Anagnostopoulos, Aris 1 Anari, Nima 1 Arlazarov, Vladimir Viktorovich 1 Arnab, Anurag 1 Avazov, Kuldoshbay 1 Bagdanov, Andrew D. 1 Bai, Xiang 1 Bar-Hillel, Aharon 1 Baratov, Nodirbek 1 Bengio, Samy 1 Boix, Xavier 1 Borji, Ali 1 Bouguila, Nizar 1 Bunel, Rudy 1 Burgard, Wolfram 1 Caetano, Tibério S. 1 Cao, Jiezhang 1 Carreira, João 1 Cauchois, Maxime 1 Celsi, Lorenzo Ricciardi 1 Chan, Tsung-Han 1 Chen, Jie 1 Chen, Yiwen 1 Chen, Zhe 1 Cheong, LoongFah 1 Cho, Young Im 1 Choi, Kwontaeg 1 Chu, Wen-Sheng 1 Cohn, Jeffrey F. 1 Cottrell, Garrison W. 1 Daneshmand, Amir 1 De Baets, Bernard 1 Dembczyński, Krzysztof 1 Desai, Chaitanya 1 Deselaers, Thomas 1 Desmaison, Alban 1 Dong, Xuanyi 1 Droit, Benjamin 1 Duchi, John C. 1 Dy, Jennifer G. 1 Efros, Alexei A. 1 Elfiky, Noha M. 1 Facchinei, Francisco 1 Fan, Aoxiang 1 Fan, Jinlong 1 Fan, Yi 1 Fang, Fang 1 Feng, Jiashi 1 Feng, Mengyang 1 Fernando, Basura 1 Fieguth, Paul W. 1 Fowlkes, Charless C. 1 Fromont, Élisa 1 Galleguillos, Carolina 1 Gao, Shenghua 1 Gao, Wen 1 Gao, Yongsheng 1 Ghanem, Bernard 1 Ghanta, Sindhu 1 Gonfaus, Josep M. 1 Gong, Boqing 1 Gong, Maoguo 1 Gonzàlez, Jordi 1 Grauman, Kristen 1 Guo, Ge 1 Guo, Zhanbing 1 Gupta, Suyash 1 Haghtalab, Nika 1 Han, Jungong 1 Han, Junwei 1 Hazan, Tamir 1 Hebert, Martial H. 1 Heigold, Georg 1 Hoiem, Derek 1 Hu, Xiping 1 Huang, Tiejun 1 Hullermeier, Eyke 1 Ilea, Dana E. 1 Iliadis, Dimitrios 1 Imiya, Atsushi 1 Ion, Adrian 1 Itoh, Hayato ...and 197 more Authors all top 5 Cited in 28 Serials 35 International Journal of Computer Vision 8 Pattern Recognition 5 Journal of Machine Learning Research (JMLR) 3 Machine Learning 2 Information Sciences 2 International Journal of Approximate Reasoning 2 Multidimensional Systems and Signal Processing 2 Journal of Mathematical Imaging and Vision 2 Data Mining and Knowledge Discovery 2 SIAM Journal on Imaging Sciences 1 Circuits, Systems, and Signal Processing 1 Computer Aided Geometric Design 1 Neural Networks 1 Neural Computation 1 Computational Statistics and Data Analysis 1 Journal of Computer and Systems Sciences International 1 Complexity 1 INFORMS Journal on Computing 1 Mathematical Problems in Engineering 1 Abstract and Applied Analysis 1 IEEE Transactions on Image Processing 1 International Journal of Wavelets, Multiresolution and Information Processing 1 Foundations and Trends in Computer Graphics and Vision 1 Vestnik Yuzhno-Ural’skogo Gosudarstvennogo Universiteta. Seriya Matematicheskoe Modelirovanie i Programmirovanie 1 Algorithms 1 Advances in Applied Mathematics and Mechanics 1 Bayesian Analysis 1 Computer Science Review all top 5 Cited in 9 Fields 71 Computer science (68-XX) 15 Information and communication theory, circuits (94-XX) 10 Statistics (62-XX) 3 Numerical analysis (65-XX) 3 Operations research, mathematical programming (90-XX) 2 Probability theory and stochastic processes (60-XX) 1 Partial differential equations (35-XX) 1 Calculus of variations and optimal control; optimization (49-XX) 1 Game theory, economics, finance, and other social and behavioral sciences (91-XX) Citations by Year