×

Model selection approaches for non-linear system identification: a review. (English) Zbl 1233.93097

Summary: The identification of nonlinear systems using only observed finite datasets has become a mature research area over the last two decades. A class of linear-in-the-parameter models with universal approximation capabilities have been intensively studied and widely used due to the availability of many linear-learning algorithms and their inherent convergence conditions. This article presents a systematic overview of basic research on model selection approaches for linear-in-the-parameter models. One of the fundamental problems in nonlinear system identification is to find the minimal model with the best model generalization performance from observational data only. The important concepts in achieving good model generalization used in various nonlinear system-identification algorithms are first reviewed, including Bayesian parameter regularization and models selective criteria based on the cross validation and experimental design. A significant advance in machine learning has been the development of the support vector machine as a means for identifying kernel models based on the structural risk minimization principle. The developments on the convex optimization-based model construction algorithms including the support vector regression algorithms are outlined. Input selection algorithms and on-line system identification algorithms are also included in this review. Finally, some industrial applications of nonlinear models are discussed.

MSC:

93E12 Identification in stochastic control theory
93C10 Nonlinear systems in control theory
PDFBibTeX XMLCite
Full Text: DOI Link

References:

[1] DOI: 10.1016/S0893-6080(00)00049-6 · doi:10.1016/S0893-6080(00)00049-6
[2] DOI: 10.1049/ip-cta:20045152 · doi:10.1049/ip-cta:20045152
[3] DOI: 10.1109/81.855463 · doi:10.1109/81.855463
[4] Aguirre LA, Physical Review E 69 pp 026701–1– (2004)
[5] DOI: 10.1109/TAC.1974.1100705 · Zbl 0314.62039 · doi:10.1109/TAC.1974.1100705
[6] DOI: 10.2307/1267500 · Zbl 0286.62044 · doi:10.2307/1267500
[7] DOI: 10.1177/0021998305055547 · doi:10.1177/0021998305055547
[8] DOI: 10.1090/S0002-9947-1950-0051437-7 · doi:10.1090/S0002-9947-1950-0051437-7
[9] DOI: 10.1016/0005-1098(71)90059-8 · Zbl 0219.93004 · doi:10.1016/0005-1098(71)90059-8
[10] Aström KJ, Adaptive Control (1989)
[11] Atkinson AC, Optimum Experimental Designs (1992)
[12] DOI: 10.1016/S0005-1098(97)00198-2 · Zbl 0915.93018 · doi:10.1016/S0005-1098(97)00198-2
[13] DOI: 10.1016/j.automatica.2003.11.007 · doi:10.1016/j.automatica.2003.11.007
[14] Barron AR, SelfOrganizing Methods in Modeling (1984)
[15] Bartlett PL, Lecture Notes, CS281B/Stat241B, Statistical Learning Theory (2003)
[16] DOI: 10.1109/TCST.2004.843129 · doi:10.1109/TCST.2004.843129
[17] DOI: 10.1109/72.298224 · doi:10.1109/72.298224
[18] DOI: 10.1080/00207720512331338120 · Zbl 1068.93004 · doi:10.1080/00207720512331338120
[19] DOI: 10.1080/00207178708933894 · Zbl 0615.93070 · doi:10.1080/00207178708933894
[20] DOI: 10.1016/0888-3270(89)90012-5 · doi:10.1016/0888-3270(89)90012-5
[21] DOI: 10.1016/0005-1098(71)90063-X · Zbl 0216.27904 · doi:10.1016/0005-1098(71)90063-X
[22] Bohlin T, Practical Grey-box Process Identification: Theory and Applications (2006)
[23] Bossley KM, , Ph.D. thesis (1997)
[24] DOI: 10.1016/j.jhydrol.2004.06.021 · doi:10.1016/j.jhydrol.2004.06.021
[25] Box GEP, Time Series Analysis: Forecasting and Control (1976)
[26] DOI: 10.1093/bioinformatics/bti242 · doi:10.1093/bioinformatics/bti242
[27] Broomhead DS, Complex Systems 2 pp 321– (1988)
[28] Brown M, Neurofuzzy Adaptive Modelling and Control (1994)
[29] Brown MD, Transactions on Inst MC, Special Issue on Electrical Power Generation 24 pp 215– (2002)
[30] Brown MD, Proc. IFAC nt. Symp. on Advanced Control of Chemical Processes, AdChem 97 pp 13– (1997)
[31] Burges CJ, Proc. International Conference on Machine Learning pp 71– (1996)
[32] Chen S, Proceedings of 6th Int. Cof. Signal Processing pp 1229– (2002)
[33] DOI: 10.1080/00207178908953472 · Zbl 0686.93093 · doi:10.1080/00207178908953472
[34] DOI: 10.1137/S1064827596304010 · Zbl 0919.94002 · doi:10.1137/S1064827596304010
[35] DOI: 10.1049/ip-cta:20030253 · doi:10.1049/ip-cta:20030253
[36] DOI: 10.1109/TAC.2003.812790 · Zbl 1364.62167 · doi:10.1109/TAC.2003.812790
[37] Chen S, Proceedings of 2005 Int. Conf. Intelligent Computing pp 777– (2005)
[38] DOI: 10.1109/72.788663 · doi:10.1109/72.788663
[39] DOI: 10.1016/0165-1684(91)90030-M · doi:10.1016/0165-1684(91)90030-M
[40] DOI: 10.1109/TSMCB.2003.817107 · doi:10.1109/TSMCB.2003.817107
[41] DOI: 10.1109/TCST.2004.841652 · doi:10.1109/TCST.2004.841652
[42] Chen S, 2007 International Joint Conference on Neural Networks (IJCNN) (2007)
[43] DOI: 10.1109/19.948295 · doi:10.1109/19.948295
[44] Clarkson T, Proceedings of the ERUDIT Workshop on Application of Computational Intelligence Techniques in Telecommunication (1999)
[45] Cohn DA, Journal of Artificial Intelligence Research 4 pp 129– (1996)
[46] Connally, P, Li, K and Irwin, GW. 2005. Two Applications of Eng-genes Bases Nonlinear Identification. Proceedings of the 16th IFAC World Congress. 2005, Prague.
[47] Connally P, Neurocomputing (2007)
[48] DOI: 10.1007/BF02551274 · Zbl 0679.94019 · doi:10.1007/BF02551274
[49] Daubechies I, Ten Lectures on Wavelets (1992) · Zbl 0776.42018 · doi:10.1137/1.9781611970104
[50] Debnath L, Introduction to Hilbert Spaces with Applications (1998)
[51] DOI: 10.1109/18.382009 · Zbl 0820.62002 · doi:10.1109/18.382009
[52] DOI: 10.1162/15324430260185637 · Zbl 1037.68111 · doi:10.1162/15324430260185637
[53] DOI: 10.1049/cp:19980312 · doi:10.1049/cp:19980312
[54] DOI: 10.1017/CBO9780511543494 · doi:10.1017/CBO9780511543494
[55] DOI: 10.1109/TSP.2004.830985 · Zbl 1369.68280 · doi:10.1109/TSP.2004.830985
[56] DOI: 10.1109/CDC.2005.1583074 · doi:10.1109/CDC.2005.1583074
[57] DOI: 10.1109/TAC.2005.856656 · Zbl 1365.93521 · doi:10.1109/TAC.2005.856656
[58] DOI: 10.1023/A:1018946025316 · Zbl 0939.68098 · doi:10.1023/A:1018946025316
[59] Eykhoff P, System Identification Parameter and State Estimation (1974)
[60] Fabri SG, Functional Adaptive Control: An Intelligent Systems Approach (2001) · doi:10.1007/978-1-4471-0319-6
[61] DOI: 10.1016/0376-0421(95)00011-9 · doi:10.1016/0376-0421(95)00011-9
[62] Flood I, Handbook of Intelligent Control (1998)
[63] DOI: 10.1214/aos/1176347963 · Zbl 0765.62064 · doi:10.1214/aos/1176347963
[64] DOI: 10.2307/2287576 · doi:10.2307/2287576
[65] DOI: 10.1016/0893-6080(89)90003-8 · doi:10.1016/0893-6080(89)90003-8
[66] DOI: 10.1113/eph8602172 · doi:10.1113/eph8602172
[67] DOI: 10.1109/TFUZZ.2003.814833 · doi:10.1109/TFUZZ.2003.814833
[68] DOI: 10.1162/089976601750399263 · Zbl 0992.68119 · doi:10.1162/089976601750399263
[69] DOI: 10.1177/089443939100900304 · doi:10.1177/089443939100900304
[70] Ge SS, Stable Adaptive Neural Network Control (2001)
[71] DOI: 10.1162/neco.1992.4.1.1 · doi:10.1162/neco.1992.4.1.1
[72] DOI: 10.3166/ejc.11.335-352 · Zbl 1293.93206 · doi:10.3166/ejc.11.335-352
[73] DOI: 10.1162/089976698300017269 · doi:10.1162/089976698300017269
[74] DOI: 10.1007/BF00195855 · Zbl 0714.94029 · doi:10.1007/BF00195855
[75] DOI: 10.1080/002071799221109 · Zbl 0941.93544 · doi:10.1080/002071799221109
[76] DOI: 10.1016/j.automatica.2005.02.002 · Zbl 1086.93064 · doi:10.1016/j.automatica.2005.02.002
[77] DOI: 10.1109/TAC.2003.817940 · Zbl 1364.93828 · doi:10.1109/TAC.2003.817940
[78] Goodwin GC, Adaptive Filtering Prediction and Control (1984)
[79] DOI: 10.1016/j.conengprac.2004.11.006 · doi:10.1016/j.conengprac.2004.11.006
[80] Gray GJ, Late Breaking Papers at the Genetic Programming 1996 Conference Stanford University July 28–31, 1996 pp 32– (1996)
[81] DOI: 10.1080/00207728908910318 · Zbl 0688.93062 · doi:10.1080/00207728908910318
[82] Green PJ, Nonparametric Regression and Generalized Linear Models (1994) · doi:10.1007/978-1-4899-4473-3
[83] DOI: 10.1007/978-3-642-18242-6 · Zbl 1005.68048 · doi:10.1007/978-3-642-18242-6
[84] DOI: 10.1016/j.annemergmed.2004.09.012 · doi:10.1016/j.annemergmed.2004.09.012
[85] Hastie TJ, Generalised Additive Models (1996)
[86] Hastie TJ, The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2002)
[87] DOI: 10.1016/j.neucom.2004.04.013 · Zbl 02223927 · doi:10.1016/j.neucom.2004.04.013
[88] DOI: 10.2307/1267351 · Zbl 0202.17205 · doi:10.2307/1267351
[89] Hong WC, International Journal of Management 23 pp 375– (2006)
[90] DOI: 10.1109/TSMCB.2004.839910 · doi:10.1109/TSMCB.2004.839910
[91] DOI: 10.1049/ip-cta:20010704 · doi:10.1049/ip-cta:20010704
[92] DOI: 10.1109/91.917117 · doi:10.1109/91.917117
[93] DOI: 10.1080/00207720310001640223 · Zbl 1043.93004 · doi:10.1080/00207720310001640223
[94] DOI: 10.1080/00207720601051463 · Zbl 1129.93052 · doi:10.1080/00207720601051463
[95] Hong X, International Journal of Hybrid Intelligent Systems 1 pp 90– (2004) · Zbl 1089.68594 · doi:10.3233/HIS-2004-11-211
[96] DOI: 10.1049/ip-cta:20030311 · doi:10.1049/ip-cta:20030311
[97] DOI: 10.1016/0893-6080(89)90020-8 · Zbl 1383.92015 · doi:10.1016/0893-6080(89)90020-8
[98] DOI: 10.1109/TSMCB.2004.834428 · doi:10.1109/TSMCB.2004.834428
[99] Huang GB, Proceedings of International Joint Conference on Neural Networks (IJCNN2004), Vol. 2 pp 985– (2004)
[100] DOI: 10.1109/TNN.2005.857948 · doi:10.1109/TNN.2005.857948
[101] DOI: 10.1002/0471725250 · Zbl 0536.62025 · doi:10.1002/0471725250
[102] Iplikci S, International Journal of Robust and Non-Linear Control 16 pp 834– (2006) · Zbl 1134.93418
[103] Irwin GW, Neural Network Applications in Control (1995) · doi:10.1049/PBCE053E
[104] James W, Proceedings Fourth Berkeley Symp. Math. Statist. Prob. 1 pp 311–319– (1961)
[105] DOI: 10.1016/0005-1098(95)00119-1 · Zbl 0846.93019 · doi:10.1016/0005-1098(95)00119-1
[106] DOI: 10.1002/fuce.200400056 · doi:10.1002/fuce.200400056
[107] DOI: 10.1016/j.enconman.2004.03.012 · doi:10.1016/j.enconman.2004.03.012
[108] DOI: 10.1162/neco.1993.5.6.954 · doi:10.1162/neco.1993.5.6.954
[109] DOI: 10.1109/72.774241 · doi:10.1109/72.774241
[110] DOI: 10.1016/j.envsoft.2003.03.004 · doi:10.1016/j.envsoft.2003.03.004
[111] DOI: 10.1109/TBME.2006.870249 · doi:10.1109/TBME.2006.870249
[112] DOI: 10.1080/00207179308923037 · Zbl 0786.93016 · doi:10.1080/00207179308923037
[113] DOI: 10.1061/(ASCE)1084-0699(2006)11:3(199) · doi:10.1061/(ASCE)1084-0699(2006)11:3(199)
[114] DOI: 10.1214/aos/1015957397 · Zbl 1105.62357 · doi:10.1214/aos/1015957397
[115] DOI: 10.1007/BF02367385 · doi:10.1007/BF02367385
[116] DOI: 10.1109/TPAMI.2002.1114861 · Zbl 05112849 · doi:10.1109/TPAMI.2002.1114861
[117] DOI: 10.1080/0020772021000017317 · Zbl 1013.93055 · doi:10.1080/0020772021000017317
[118] DOI: 10.1080/0020718508961129 · Zbl 0569.93011 · doi:10.1080/0020718508961129
[119] DOI: 10.1109/72.950144 · doi:10.1109/72.950144
[120] DOI: 10.1088/0964-1726/11/1/309 · doi:10.1088/0964-1726/11/1/309
[121] DOI: 10.1080/002071798222262 · doi:10.1080/002071798222262
[122] Li, K. 2005. Eng-genes: A New Genetic Modelling Approach for Nonlinear Dynamic Systems. Proceedings of the 16th IFAC World Congress. 2005, Prague.
[123] DOI: 10.1142/S0218001406004570 · Zbl 05153453 · doi:10.1142/S0218001406004570
[124] DOI: 10.1016/j.automatica.2006.03.004 · Zbl 1117.93314 · doi:10.1016/j.automatica.2006.03.004
[125] DOI: 10.1109/TAC.2005.854581 · Zbl 1365.93239 · doi:10.1109/TAC.2005.854581
[126] DOI: 10.1016/S0967-0661(03)00171-0 · doi:10.1016/S0967-0661(03)00171-0
[127] Lin B, Higher-order-Statistics-based Radial basis Function Networks for Signal Enhancement 18 pp 823– (2007)
[128] Liu GP, Nonlinear Identification and Control-A Neural Network Approach (2001)
[129] DOI: 10.1109/5326.740668 · doi:10.1109/5326.740668
[130] Ljung L, System Identification: Theory for the User (1987)
[131] DOI: 10.1109/TAC.2005.856638 · Zbl 1365.93007 · doi:10.1109/TAC.2005.856638
[132] Ljung L, Theory and Practice of Recersive Identification (1983)
[133] DOI: 10.1016/S0967-0661(98)00086-0 · doi:10.1016/S0967-0661(98)00086-0
[134] DOI: 10.1243/0959651041568524 · doi:10.1243/0959651041568524
[135] Luo, W, Morris, AJ, Karim, MN, Martin, EB and Hong, T. 1996. Online Identification of a Ph Waste Water Neutralisation Process using Time-varying Nonlinear Arx Models. IEEE International Conference on Systems, Man, and Cybernetics, 1996. 14–17 Oct.1996. Vol. 1, pp.107–112.
[136] Ma, Q and Wang, J. 2000. Application of Bayesian Neural Networks to Protein Sequence Classification. ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining. 2000. pp.305–309.
[137] MacKay DJC, Bayesian Methods for Adaptive Models. PhD thesis (1991)
[138] Mackay DJ, Lecture notes at NIPS 1997 (1997)
[139] DOI: 10.1109/34.192463 · Zbl 0709.94650 · doi:10.1109/34.192463
[140] DOI: 10.1109/78.258082 · Zbl 0842.94004 · doi:10.1109/78.258082
[141] DOI: 10.1002/047084535X · doi:10.1002/047084535X
[142] DOI: 10.1006/mssp.1998.0180 · doi:10.1006/mssp.1998.0180
[143] DOI: 10.1080/002071700219830 · Zbl 1028.93029 · doi:10.1080/002071700219830
[144] DOI: 10.1080/002077299292209 · Zbl 1049.93523 · doi:10.1080/002077299292209
[145] DOI: 10.1016/j.automatica.2004.08.014 · Zbl 1066.93056 · doi:10.1016/j.automatica.2004.08.014
[146] Marose RA, AI Expert 5 pp 50– (1990)
[147] DOI: 10.2307/1267205 · Zbl 0205.46102 · doi:10.2307/1267205
[148] DOI: 10.1080/00207729708929425 · Zbl 0885.93036 · doi:10.1080/00207729708929425
[149] DOI: 10.1098/rsta.1909.0016 · JFM 40.0408.02 · doi:10.1098/rsta.1909.0016
[150] Miller AJ, Subset Selection in Regression (1990) · doi:10.1007/978-1-4899-2939-6
[151] Minoux M, Mathematical Programming: Theory and Algorithms (1986)
[152] DOI: 10.1109/TSMCC.2006.879384 · doi:10.1109/TSMCC.2006.879384
[153] Moody J, From Statistics to Neural Networks: Theory and Pattern Recognition Applications (1994)
[154] DOI: 10.1109/72.914517 · doi:10.1109/72.914517
[155] Murray-Smith R, PhD thesis (1994)
[156] Murray-Smith R, Multiple Model Approaches to Modelling and Control (1997)
[157] Myers RH, Response Surface Methodology: Process and Product Design using Designed Experimetns (1995)
[158] Neal RM, Bayesian Learning for Neural Networks (1996) · doi:10.1007/978-1-4612-0745-0
[159] DOI: 10.1214/aos/1176346522 · Zbl 0544.62063 · doi:10.1214/aos/1176346522
[160] DOI: 10.1162/neco.1995.7.3.606 · doi:10.1162/neco.1995.7.3.606
[161] DOI: 10.1109/72.97910 · doi:10.1109/72.97910
[162] DOI: 10.1063/1.1737818 · Zbl 1080.70009 · doi:10.1063/1.1737818
[163] DOI: 10.1016/j.neucom.2004.11.029 · Zbl 02223947 · doi:10.1016/j.neucom.2004.11.029
[164] DOI: 10.1016/S0967-0661(03)00050-9 · doi:10.1016/S0967-0661(03)00050-9
[165] DOI: 10.1109/TNN.2006.880860 · doi:10.1109/TNN.2006.880860
[166] DOI: 10.1162/neco.1991.3.2.213 · doi:10.1162/neco.1991.3.2.213
[167] Platt J, Advances in Kernel Methods Support Vector Learning pp 185– (1999)
[168] DOI: 10.1109/72.207618 · doi:10.1109/72.207618
[169] Powell MJD, Algorithms for Approximation pp 143– (1985)
[170] Priestley MB, Spectral Analysis and Time Series (1981)
[171] DOI: 10.1016/j.jprocont.2004.06.011 · doi:10.1016/j.jprocont.2004.06.011
[172] DOI: 10.1049/ip-cta:20045250 · doi:10.1049/ip-cta:20045250
[173] DOI: 10.1109/TSMCC.2004.843228 · doi:10.1109/TSMCC.2004.843228
[174] DOI: 10.1016/j.automatica.2006.12.013 · Zbl 1282.93087 · doi:10.1016/j.automatica.2006.12.013
[175] Ruano AE, Intelligent Control Systems using Computational Intelligence Techniques (2005) · doi:10.1049/PBCE070E
[176] DOI: 10.1109/72.165588 · doi:10.1109/72.165588
[177] DOI: 10.1002/aic.690450920 · doi:10.1002/aic.690450920
[178] DOI: 10.1109/72.870050 · doi:10.1109/72.870050
[179] DOI: 10.1109/TFUZZ.2004.834810 · Zbl 05452597 · doi:10.1109/TFUZZ.2004.834810
[180] DOI: 10.1016/0005-1098(95)00120-8 · Zbl 0846.93018 · doi:10.1016/0005-1098(95)00120-8
[181] Smola AJ, Learning with Kernels. Ph.D. thesis (1998)
[182] Smola AJ, NeuroCOLT Technical Report NC-TR-98-030 (1998)
[183] Söderström T, Proc. 14th IFAC Symposium on System Identification (2006)
[184] Söderström T, System Identification (1989)
[185] DOI: 10.1016/0005-1098(90)90164-D · Zbl 0714.93056 · doi:10.1016/0005-1098(90)90164-D
[186] DOI: 10.1016/j.automatica.2004.11.004 · doi:10.1016/j.automatica.2004.11.004
[187] DOI: 10.1243/09596518JSCE185 · doi:10.1243/09596518JSCE185
[188] DOI: 10.1080/00207720310001640287 · Zbl 1052.65008 · doi:10.1080/00207720310001640287
[189] Stone M, Journal of the Royal Statistical Society, Series B 36 pp 117– (1974)
[190] DOI: 10.1016/S0893-6080(00)00079-4 · Zbl 02022505 · doi:10.1016/S0893-6080(00)00079-4
[191] DOI: 10.1142/9789812776655 · doi:10.1142/9789812776655
[192] Tibshirani R, Journal of Royal Statistical Society. Series B 58 pp 267– (1996)
[193] DOI: 10.1158/0008-5472.CAN-04-3617 · doi:10.1158/0008-5472.CAN-04-3617
[194] Townsend S, Nonlinear Predictive Control: Theory and Practice (2001)
[195] DOI: 10.1049/ip-epa:20045058 · doi:10.1049/ip-epa:20045058
[196] DOI: 10.1080/10556789908805759 · Zbl 0973.90518 · doi:10.1080/10556789908805759
[197] Vapnik V, The Nature of Statictical Learning Theory (1995) · Zbl 0833.62008 · doi:10.1007/978-1-4757-2440-0
[198] Vapnik V, Statistical Learning Theory (1998)
[199] DOI: 10.1109/70.681254 · doi:10.1109/70.681254
[200] Wahba G, Society for Industrial and Applied Mathematics (1990)
[201] DOI: 10.1109/72.159070 · doi:10.1109/72.159070
[202] DOI: 10.1109/72.809087 · doi:10.1109/72.809087
[203] Xia, X and Li, K. 2007. A New Score Correlation analysis Multi-class Support Vector Machine for Microarray. 2007 International Joint Conference on Neural Networks. 2007, Florida. Orlendo
[204] DOI: 10.1109/TNN.2005.845145 · doi:10.1109/TNN.2005.845145
[205] DOI: 10.1109/72.661125 · doi:10.1109/72.661125
[206] Young PC, Recursive Estimation and Time Series Analysis (1984) · doi:10.1007/978-3-642-82336-7
[207] Zhang Q, IEEE Transactions on Neural Networks 8 (1993)
[208] DOI: 10.1080/00207720601014552 · Zbl 1111.93081 · doi:10.1080/00207720601014552
[209] DOI: 10.1016/0893-6080(95)00139-5 · Zbl 05473695 · doi:10.1016/0893-6080(95)00139-5
[210] DOI: 10.1080/002071799220065 · Zbl 0941.93511 · doi:10.1080/002071799220065
[211] DOI: 10.1080/00207720110121105 · Zbl 1021.93007 · doi:10.1080/00207720110121105
[212] DOI: 10.1016/j.jpowsour.2006.01.040 · doi:10.1016/j.jpowsour.2006.01.040
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.