×

Nonlinear adaptive filtering using kernel-based algorithms with dictionary adaptation. (English) Zbl 1333.93244

Summary: Nonlinear adaptive filtering has been extensively studied in the literature, using, for example, Volterra filters or neural networks. Recently, kernel methods have been offering an interesting alternative because they provide a simple extension of linear algorithms to the nonlinear case. The main drawback of online system identification with kernel methods is that the filter complexity increases with time, a limitation resulting from the representer theorem, which states that all past input vectors are required. To overcome this drawback, a particular subset of these input vectors (called dictionary) must be selected to ensure complexity control and good performance. Up to now, all authors considered that, after being introduced into the dictionary, elements stay unchanged even if, because of nonstationarity, they become useless to predict the system output. The objective of this paper is to present an adaptation scheme of dictionary elements, which are considered here as adjustable model parameters, by deriving a gradient-based method under collinearity constraints. The main interest is to ensure a better tracking performance. To evaluate our approach, dictionary adaptation is introduced into three well-known kernel-based adaptive algorithms: kernel recursive least squares, kernel normalized least mean squares, and kernel affine projection. The performance is evaluated on nonlinear adaptive filtering of simulated and real data sets. As confirmed by experiments, our dictionary adaptation scheme allows either complexity reduction or a decrease of the instantaneous quadratic error, or both simultaneously.

MSC:

93E11 Filtering in stochastic control theory
93C40 Adaptive control/observation systems
93C10 Nonlinear systems in control theory
68T05 Learning and adaptive systems in artificial intelligence
93B30 System identification
PDFBibTeX XMLCite
Full Text: DOI HAL

References:

[1] Aihara, Adaptive filtering for stochastic risk premia in bond market, International Journal of Innovative Computing, Information and Control 8 (3(B)) pp 2203– (2012)
[2] Yu, Adaptive filtering with adaptive p power error criterion, International Journal of Innovative Computing, Information and Control 7 (4) pp 1725– (2011)
[3] Schetzen, The Volterra and Wiener Theories of Nonlinear Systems (1980) · Zbl 0501.93002
[4] Wiener, Nonlinear Problems in Random Theory (Technology Press Research Monographs) (1966)
[5] Ogunfunmi, Adaptive Nonlinear System Identification: The Volterra and Wiener Model Approaches (2007) · Zbl 1130.93034 · doi:10.1007/978-0-387-68630-1
[6] Haykin, Neural Networks and Learning Machines (2008)
[7] Smola A Schölkopf B A tutorial on support vector regression NeuroCOLT Technical Report UK 1998
[8] Suykens, Least Squares Support Vector Machines (2002) · Zbl 1017.93004 · doi:10.1142/5089
[9] Honeine P Richard C Bermudez JCM On-line nonlinear sparse approximation of functions Proceedings of IEEE International Symposium on Information Theory Nice, France 2007 956 960
[10] Mercer, Functions of positive and negative type and their connection with the theory of integral equations, Philosophical Transactions of the Royal Society of London CCIX pp 415– (1909) · JFM 40.0408.02
[11] Aronszajn, Theory of reproducing kernels, Transactions of the American Mathematical Society 68 (3) pp 337– (1950) · Zbl 0037.20701 · doi:10.1090/S0002-9947-1950-0051437-7
[12] Aizerman, Theoretical foundations of the potential function method in pattern recognition learning, Automation and Remote Control 25 pp 821– (1964)
[13] Dodd, A theory of iterative sparse solutions to strict interpolation in reproducing kernel Hilbert spaces, Technical Report 827 (2002)
[14] Dodd TJ Kadirkamanathan V Harrison RF Function estimation in Hilbert space using sequential projections Intelligent Control Systems and Signal Processing 2003 113 118
[15] Dodd, Sparse, online learning in reproducing kernel Hilbert spaces, IEEE Transactions on Signal Processing (2005)
[16] Phonphitakchai S Dodd TJ Stochastic meta descent in online kernel methods 6th International Conference on Electrical Engineering/Electronics, Computer, Telecommunications and Information Technology, 2009. ECTI-CON 2009 02 Pattaya, Thailand 2009 690 693
[17] Engel, Proceedings of the 13th European Conference on Machine Learning pp 84– (2002)
[18] Engel, Kernel recursive least squares, IEEE Transactions on Signal Processing 52 pp 2275– (2004) · Zbl 1369.68280 · doi:10.1109/TSP.2004.830985
[19] Honeine P Richard C Bermudez JCM Modélisation parcimonieuse non linéaire en ligne par une méthode á noyau reproduisant et un critère de cohérence Actes du XXI-ème Colloque GRETSI sur le Traitement du Signal et des Images Troyes, France 2007 1257 1260
[20] Richard, Online prediction of time series data with kernels, IEEE Transactions on Signal Processing 57 (3) pp 1058– (2009) · Zbl 1391.94376 · doi:10.1109/TSP.2008.2009895
[21] Mercer, Functions of positive and negative type and their connection with the theory of integral equations, Philosophical Transactions of the Royal Society of London A 209 (441-458) pp 415– (1909) · JFM 40.0408.02 · doi:10.1098/rsta.1909.0016
[22] Schölkopf B Herbrich R Smola A A generalized representer theorem COLT/EuroCOLT’01 Amsterdam, The Netherlands 2001 416 426
[23] Kimeldorf, Some results on Tchebycheffian spline functions, Journal of Mathematical Analysis and Applications 33 (1) pp 82– (1971) · Zbl 0201.39702 · doi:10.1016/0022-247X(71)90184-3
[24] Wahba, CBMS-NSF Regional Conference Series in Applied Mathematics, in: Spline Models for Observational Data (1990) · doi:10.1137/1.9781611970128
[25] Tropp, Greed is good: algorithmic results for sparse approximation, IEEE Transactions on Information Theory 50 pp 2231– (2004) · Zbl 1288.94019 · doi:10.1109/TIT.2004.834793
[26] Sun Y Gomez FJ Schmidhuber J On the size of the online kernel sparsification dictionary Proceedings of the 29th International Conference on Machine Learning (ICML 2012) Edinburgh, Scotland 2012 329 336
[27] Csató, Advances in Neural Information Processing Systems pp 444– (2001)
[28] Mallat, Matching pursuit with time-frequency dictionaries, IEEE Transactions on Signal Processing 41 pp 3397– (1993) · Zbl 0842.94004 · doi:10.1109/78.258082
[29] Cucker, On the mathematical foundations of learning, Bulletin of the American Mathematical Society 39 pp 1– (2002) · Zbl 0983.68162 · doi:10.1090/S0273-0979-01-00923-5
[30] Ozeki, An adaptive filtering algorithm using an orthogonal projection to an affine subspace and its properties, Electronic Communications of Japan 67 (A) pp 1927– (1984)
[31] Sayed, Fundamentals of Adaptive Filtering (2003)
[32] Shams Esfand Abadi, A family of set-membership affine projection adaptive filter algorithms, International Journal of Innovative Computing, Information and Control 8 (2) pp 1313– (2012) · Zbl 1204.93116
[33] Liu, Kernel affine projection algorithms, EURASIP Journal on Advances in Signal Processing 2008 (1) pp 784292+– (2008) · Zbl 1184.68411 · doi:10.1155/2008/784292
[34] Liu, Kernel Adaptive Filtering: A Comprehensive Introduction, 1. ed. (2010) · doi:10.1002/9780470608593
[35] Saide C Lengelle R Honeine P Richard C Achkar R Dictionary adaptation for online prediction of time series data with kernels 2012 IEEE Statistical Signal Processing Workshop (SSP) Ann Arbor, Michigan, USA 2012 604 607 · doi:10.1109/SSP.2012.6319772
[36] Hathaway D The sunspot cycle 2012 http://solarscience.msfc.nasa.gov/SunspotCycle.shtml
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.