zbMATH — the first resource for mathematics

Generalized iterative RELIEF for supervised distance metric learning. (English) Zbl 1207.68265
Summary: The RELIEF algorithm is a popular approach for feature weighting. Many extensions of the RELIEF algorithm are developed, and I-RELIEF is one of the famous extensions. In this paper, I-RELIEF is generalized for supervised distance metric learning to yield a Mahananobis distance function. The proposed approach is justified by showing that the objective function of the generalized I-RELIEF is closely related to the expected leave-one-out nearest-neighbor classification rate. In addition, the relationships among the generalized I-RELIEF, the neighbourhood components analysis, and graph embedding are also pointed out. Experimental results on various data sets all demonstrate the superiority of the proposed approach.

68T10 Pattern recognition, speech recognition
68T05 Learning and adaptive systems in artificial intelligence
AR face; UCI-ml
Full Text: DOI
[1] K. Kira, L.A. Rendell, A practical approach to feature selection, in: Proceedings of the Ninth International Conference on Machine Learning, 1992, pp. 249-256.
[2] Dietterich, T.G., Machine-learning research—four current directions, AI magazine, 18, 97-136, (1997)
[3] Kohavi, R.; John, G., Wrappers for feature selection, Artificial intelligence, 97, 273-324, (1997) · Zbl 0904.68143
[4] Wettschereck, D.; Aha, D.W.; Mohri, T., A review and empirical evaluation of feature weighting methods for a class of lazy learning algorithms, Artificial intelligence review, 11, 273-314, (1997)
[5] I. Kononenko, Estimating attributes: analysis and extensions of RELIEF, in: European Conference on Machine Learning, 1994, pp. 171-182.
[6] Sun, Y., Iterative RELIEF for feature weighting: algorithms, theories, and applications, IEEE transactions on pattern analysis and machine intelligence, 29, 6, 1035-1051, (2007)
[7] Kononenko, I.; Simec, E.; Robnik-Sikonja, M., Overcoming the myopia of inductive learning algorithms with RELIEFF, Applied intelligence, 7, 39-55, (1997)
[8] Robnik-Šikonja, M.; Kononenko, I., Theoretical and empirical analysis of relieff and rrelieff, Machine learning, 53, 23-69, (2003) · Zbl 1076.68065
[9] Dempster, A.P.; Laird, N.M.; Rubin, D.B., Maximum likelihood from incomplete data via the EM algorithm, Journal of the royal statistical society series B, 39, 1, 1-38, (1977) · Zbl 0364.62022
[10] J. Yang, Y.-P. Li, Orthogonal RELIEF algorithm for feature selection, in: Proceedings of 2006 International Conference on Intelligent Computing, Springer, Berlin/Heidelberg, 2006.
[11] Y. Sun, D. Wu, A RELIEF based feature extraction algorithm, in: Proceedings of the SIAM International Conference on Data Mining, 2008, pp. 188-195.
[12] J. Bins, B. Draper, Feature selection from huge feature sets, in: Proceedings of the Eighth IEEE International Conference on Computer Vision, vol. 2, 2001, pp. 159-165.
[13] R. Flórez-López, Reviewing RELIEF and its extensions: a new approach for estimating attributes considering high-correlated features, in: Proceedings of IEEE International Conference on Data Mining, 2002, pp. 605-608.
[14] L. Yang, R. Jin, Distance metric learning: a comprehensive survey, Technical Report 24, Department of Computer Science and Engineering, Michigan State University, 2006.
[15] Fukunaga, K.; Flick, T.E., An optimal global nearest neighbor metric, IEEE transactions on pattern analysis and machine intelligence PAMI, 6, 3, 314-318, (1984) · Zbl 0534.62041
[16] Hastie, T.; Tibshirani, R., Discriminant adaptive nearest neighbor classification, IEEE transactions on pattern analysis and machine intelligence, 18, 6, 607-616, (1996)
[17] Goldberger, J.; Roweis, S.; Hinton, G.; Salakhutdinov, R., Neighbourhood components analysis, (), 513-520
[18] L. Yang, R. Jin, R. Sukthankar, Y. Liu, An efficient algorithm for local distance metric learning, in: Proceedings of the 21st National Conference on Artificial Intelligence, 2006.
[19] Xing, E.P.; Ng, A.Y.; Jordan, M.I.; Russell, S., Distance metric learning with application to clustering with side-information, (), 505-512
[20] Weinberger, K.; Blitzer, J.; Saul, L., Distance metric learning for large margin nearest neighbor classification, (), 1473-1480
[21] Yan, S.; Liu, J.; Tang, X.; Huang, T.S., A parameter-free framework for general supervised subspace learning, IEEE transactions on information forensics and security, 2, 1, 69-76, (2007)
[22] S. Chopra, R. Hadsell, Y. LeCun, Learning a similarity metric discriminatively, with application to face verification, in: Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), 2005, pp. 539-546.
[23] J.V. Davis, B. Kulis, P. Jain, I.S. Dhillon, Information-theoretic metric learning, in: Proceedings of the 24th international conference on Machine learning, 2007, pp. 209-216.
[24] Yan, S.; Xu, D.; Zhang, B.; Zhang, H.J.; Yang, Q.; Lin, S., Graph embedding and extensions: a general framework for dimensionality reduction, IEEE transactions on pattern analysis and machine intelligence, 29, 1, 40-51, (2007)
[25] Cover, T.M.; Thomas, J.A., Elements of information theory, (1991), John & Wiley Sons, Inc New York · Zbl 0762.94001
[26] Bazaraa, M.S.; Sherali, H.D.; Shetty, C.M., Nonlinear programming theory and algorithms, (1993), John & Wiley sons, Inc New York · Zbl 0774.90075
[27] Chung, F.R.K., Spectral graph theory, (1997), American Mathematical Society New York
[28] Horn, R.A.; Johnson, C.R., Matrix analysis, (1985), Cambridge University Press New York · Zbl 0576.15001
[29] Sugiyama, M., Dimensionality reduction of multimodal labeled data by local Fisher discriminant analysis, Journal of machine learning research, 8, 1027-1061, (2007) · Zbl 1222.68312
[30] Zelnik-Manor, L.; Perona, P., Self-tuning spectral clustering, (), 1601-1608
[31] Xu, D.; Yan, S.; Tao, D.; Lin, S.; Zhang, H.J., Marginal Fisher analysis and its variants for human gait recognition and content-based image retrieval, IEEE transactions on image processing, 16, 11, 2811-2821, (2007)
[32] A. Bar-Hillel, T. Hertz, N. Shental, D. Weinshall, Learning distance functions using equivalence relations, in: Proceedings of the 20th International Conference on Machine Learning, 2003, pp. 11-18. · Zbl 1222.68140
[33] Duda, R.O.; Hart, P.E.; Stork, D.G., Pattern classification, (2001), John Wiley & Sons, Inc · Zbl 0968.68140
[34] A. Asuncion, D. Newman, UCI machine learning repository, 2007. URL \(\langle\)http://www.ics.uci.edu/∼mlearn/MLRepository.html〉.
[35] Singh, D.; Febbo, P.G.; Ross, K.; Jackson, D.G.; Manola, J.; Ladd, C.; Tamayo, P.; Renshaw, A.A.; D’Amico, A.V.; Richie, J.P.; Lander, E.S.; Loda, M.; Kantoff, P.W.; Golub, T.R.; Sellers, W.R., Gene expression correlates of clinical prostate cancer behavior, Cancer cell, 1, 203-209, (2002)
[36] Ramaswamy, S.; Tamayo, P.; Rifkin, R.; Mukherjee, S.; Yeang, C.; Angelo, M.; Ladd, C.; Reich, M.; Latulippe, E.; Mesirov, J.; Poggio, T.; Gerald, W.; Loda, M.; Lander, E.; Golub, R., Multiclass cancer diagnosis using tumor gene expression signatures, Proceedings of the national Academy of science, 98, 26, 15149-15154, (2001)
[37] Lee, K.; Ho, J.; Kriegman, D., Acquiring linear subspaces for face recognition under variable lighting, IEEE transactions pattern analysis and machine intelligence, 27, 5, 684-698, (2005)
[38] A.M. Martinez, R. Benavente, The AR face database, Technical Report 24, CVC, June 1998.
[39] Fukunaga, K., Introduction to statistical pattern recognition, (1990), Academic Press Boston · Zbl 0711.62052
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.