×

zbMATH — the first resource for mathematics

Combining diverse on-line and off-line systems for handwritten text line recognition. (English) Zbl 1182.68198
Summary: We present a Multiple Classifier System (MCS) for on-line handwriting recognition. The MCS combines several individual recognition systems based on Hidden Markov Models (HMMs) and Bidirectional Long Short-Term Memory networks (BLSTM). Beside using two different recognition architectures (HMM and BLSTM), we use various feature sets based on on-line and off-line features to obtain diverse recognizers. Furthermore, we generate a number of different neural network recognizers by changing the initialization parameters. To combine the word sequences output by the recognizers, we incrementally align these sequences using the recognizer output voting error reduction framework. For deriving the final decision, different voting strategies are applied. The best combination ensemble has a recognition rate of 84.13%, which is significantly higher than the 83.64% achieved if only one recognition architecture (HMM or BLSTM) is used for the combination, and even remarkably higher than the 81.26% achieved by the best individual classifier. To demonstrate the high performance of the classification system, the results are compared with two widely used commercial recognizers from Microsoft and Vision Objects.

MSC:
68T10 Pattern recognition, speech recognition
Software:
IAM-OnDB; LSTM
PDF BibTeX XML Cite
Full Text: DOI
References:
[1] Plamondon, R.; Srihari, S.N., On-line and off-line handwriting recognition: a comprehensive survey, IEEE transactions on pattern analysis and machine intelligence, 22, 1, 63-84, (2000)
[2] Vinciarelli, A., A survey on off-line cursive script recognition, Pattern recognition, 35, 7, 1433-1446, (2002) · Zbl 1016.68086
[3] Bunke, H., Recognition of cursive Roman handwriting—past, present and future, (), 448-459
[4] Seiler, R.; Schenkel, M.; Eggimann, F., Off-line cursive handwriting recognition compared with on-line recognition, (), 505
[5] Tappert, C.C.; Suen, C.Y.; Wakahara, T., The state of the art in online handwriting recognition, IEEE transactions on pattern analysis and machine intelligence, 12, 8, 787-808, (1990)
[6] Steinherz, T.; Rivlin, E.; Intrator, N., Offline cursive script word recognition—a survey, International journal on document analysis and recognition, 2, 2-3, 90-110, (1999)
[7] C.Y. Suen, K. Kim, Q. Xu, J. Kim, L. Lam, Handwriting recognition—the last frontiers, in: Proceedings of the 15th International Conference on Pattern Recognition, vol. 4, 2000, pp. 4001-4010.
[8] Wilfong, G.; Sinden, F.; Ruedisueli, L., On-line recognition of handwritten symbols, IEEE transactions on pattern analysis and machine intelligence, 18, 9, 935-940, (1996)
[9] Hu, J.; Lim, S.G.; Brown, M.K., Writer independent on-line handwriting recognition using an HMM approach, Pattern recognition, 33, 1, 133-147, (2000)
[10] Bahlmann, C.; Burkhardt, H., The writer independent online handwriting recognition system frog on hand and cluster generative statistical dynamic time warping, IEEE transactions on pattern analysis and machine intelligence, 26, 3, 299-310, (2004)
[11] C. Bahlmann, B. Haasdonk, H. Burkhardt, Online handwriting recognition with support vector machines—a kernel approach, in: Proceedings of the 8th International Workshop on Frontiers in Handwriting Recognition, 2002, pp. 49-54.
[12] Sayre, K.M., Machine recognition of handwritten words: a project report, Pattern recognition, 5, 3, 213-228, (1973)
[13] Schomaker, L., Using stroke- or character-based self-organizing maps in the recognition of on-line, connected cursive script, Pattern recognition, 26, 3, 443-450, (1993)
[14] Kavallieratou, E.; Fakotakis, N.; Kokkinakis, G., An unconstrained handwriting recognition system, International journal on document analysis and recognition, 4, 4, 226-242, (2002)
[15] L.R. Rabiner, A tutorial on hidden Markov models and selected applications in speech recognition, in: Proceedings of the IEEE, vol. 77(2), 1989, pp. 257-286.
[16] S. Bercu, G. Lorette, On-line handwritten word recognition: an approach based on hidden Markov models, in: Proceedings of the 3rd International Workshop on Frontiers in Handwriting Recognition, 1993, pp. 385-390.
[17] T. Starner, J. Makhoul, R. Schwartz, G. Chou, Online cursive handwriting recognition using speech recognition techniques, in: International Conference on Acoustics, Speech and Signal Processing, vol. 5, 1994, pp. 125-128.
[18] Hu, J.Y.; Brown, M.K.; Turin, W., HMM based online handwriting recognition, IEEE transactions on pattern analysis and machine intelligence, 18, 10, 1039-1045, (1996)
[19] Liwicki, M.; Bunke, H., Handwriting recognition of whiteboard notes—studying the influence of training set size and type, International journal of pattern recognition and artificial intelligence, 21, 1, 83-98, (2007)
[20] Haykin, S., Neural networks: A comprehensive foundation, (1994), Prentice-Hall PTR Englewood Cliffs, NJ · Zbl 0828.68103
[21] Bourbakis, N.G., Handwriting recognition using a reduced character method and neural nets, (), 592-601
[22] A. Brakensiek, A. Kosmala, D. Willett, W. Wang, G. Rigoll, Performance evaluation of a new hybrid modeling technique for handwriting recognition using identical on-line and off-line data, in: Proceedings of the 5th International Conference on Document Analysis and Recognition, Bangalore, India, 1999, pp. 446-449.
[23] Marukatat, S.; Artières, T.; Dorizzi, B.; Gallinari, P., Sentence recognition through hybrid neuro-Markovian modelling, (), 731-735 · Zbl 1012.68166
[24] Schenkel, M.; Guyon, I.; Henderson, D., On-line cursive script recognition using time delay neural networks and hidden Markov models, Machine vision and applications, 8, 215-223, (1995)
[25] Jäger, S.; Manke, S.; Reichert, J.; Waibel, A., Online handwriting recognition: the npen++ recognizer, International journal on document analysis and recognition, 3, 3, 169-180, (2001)
[26] Caillault, E.; Viard-Gaudin, C.; Ahmad, A.R., MS-TDNN with global discriminant trainings, (), 856-861
[27] A.W. Senior, F. Fallside, An off-line cursive script recognition system using recurrent error propagation networks, in: International Workshop on Frontiers in Handwriting Recognition, Buffalo, NY, USA, 1993, pp. 132-141.
[28] Senior, A.W.; Robinson, A.J., An off-line cursive handwriting recognition system, IEEE transactions on pattern analysis and machine intelligence, 20, 3, 309-321, (1998)
[29] J. Schenk, G. Rigoll, Novel hybrid NN/HMM modelling techniques for on-line handwriting recognition, in: Proceedings of the 10th International Workshop on Frontiers in Handwriting Recognition, 2006, pp. 619-623.
[30] Johnson, M.T., Capacity and complexity of HMM duration modeling techniques, IEEE signal processing letters, 12, 5, 407-410, (2005)
[31] Graves, A.; Fernández, S.; Gomez, F.; Schmidhuber, J., Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks, (), 369-376
[32] Graves, A.; Schmidhuber, J., Framewise phoneme classification with bidirectional LSTM and other neural network architectures, Neural networks, 18, 5-6, 602-610, (2005)
[33] Liwicki, M.; Graves, A.; Bunke, H.; Schmidhuber, J., A novel approach to on-line handwriting recognition based on bidirectional long short-term memory networks, (), 367-371
[34] Kuncheva, L.I., Combining pattern classifiers: methods and algorithms, (2004), Wiley New York · Zbl 1066.68114
[35] Huang, Y.S.; Suen, C.Y., A method of combining multiple experts for the recognition of unconstrained handwritten numerals, IEEE transactions on pattern analysis and machine intelligence, 17, 1, 90-94, (1995)
[36] Vinciarelli, A.; Perrone, M., Combining online and offline handwriting recognition, (), 844-848
[37] Ye, X.; Cheriet, M.; Suen, C.Y., Strcombo: combination of string recognizers, Pattern recognition letters, 23, 381-394, (2002) · Zbl 1006.68114
[38] M.C.F.K. Sirlantzis, M.S. Hoque, Genetic algorithms for multi-classifier system configuration: a case study in character recognition, in: Proceedings of the 2nd Workshop on Multiple Classifier Systems, 2001, pp. 99-108. · Zbl 0980.68908
[39] Günter, S.; Bunke, H., Multiple classifier systems in off-line handwritten word recognition on the influence of training set and vocabulary size, International journal of pattern recognition and artificial intelligence, 18, 1303-1320, (2004)
[40] J. Fiscus, A post-processing system to yield reduced word error rates: recognizer output voting error reduction ROVER, in: IEEE Workshop on Automatic Speech Recognition and Understanding, 1997, pp. 347-352.
[41] Bertolami, R.; Bunke, H., Multiple handwritten text recognition systems derived from specific integration of a language model, (), 521-524
[42] O. Velek, S. Jäger, M. Nakagawa, Accumulated-recognition-rate normalization for combining multiple on/off-line Japanese character classifiers tested on a large database, in: Proceedings of the 4th Workshop on Multiple Classifier Systems, 2003, pp. 196-205. · Zbl 1040.68767
[43] Liwicki, M.; Bunke, H., Combining on-line and off-line systems for handwriting recognition, (), 372-376
[44] Pittman, J.A., Handwriting recognition: tablet PC text input, Computer, 40, 9, 49-54, (2007)
[45] Knerr, S.; Anisimov, V.; Baret, O.; Gorsky, N.; Price, D.; Simon, J., The A2ia INTERCHEQUE system: courtesy amount and legal amount recognition for French checks, International journal of pattern recognition and artificial intelligence, 11, 4, 505-548, (1997)
[46] S. Knerr, E. Augustin, A neural network-hidden Markov model hybrid for cursive word recognition, in: International Conference on Pattern Recognition, vol. 2, 1998, pp. 1518-1520.
[47] M. Liwicki, H. Bunke, Combining on-line and off-line bidirectional long short-term memory networks for handwritten text line recognition, in: Proceedings of the 11th International Conference on Frontiers in Handwriting Recognition, 2008, pp. 31-36.
[48] Dempster, A.P.; Laird, N.M.; Rubin, D.B., Maximum likelihood from incomplete data via the EM algorithm, Journal of royal statistical society, 39, 1, 1-38, (1977) · Zbl 0364.62022
[49] G.D. Forney, The Viterbi algorithm, in: Proceedings of the IEEE, vol. 61, 1973, pp. 268-278.
[50] Günter, S.; Bunke, H., HMM-based handwritten word recognition: on the optimization of the number of states, training iterations and Gaussian components, Pattern recognition, 37, 2069-2079, (2004)
[51] Rojas, R., Neural networks—A systematic introduction, (1996), Springer Berlin · Zbl 0861.68072
[52] Fernández, S.; Graves, A.; Schmidhuber, J., Sequence labelling in structured domains with hierarchical recurrent neural networks, (), 774-779
[53] Schuster, M.; Paliwal, K.K., Bidirectional recurrent neural networks, IEEE transactions on signal processing, 45, 2673-2681, (1997)
[54] Wang, L.; Jiang, T., On the complexity of multiple sequence alignment, Journal of computational biology, 1, 4, 337-348, (1994)
[55] Wagner, R.; Fischer, M., The string-to-string correction problem, Journal of the ACM, 21, 168-173, (1974) · Zbl 0278.68032
[56] Liwicki, M.; Bunke, H., IAM-ondb—an on-line English sentence database acquired from handwritten text on a whiteboard, (), 956-961
[57] E. Bauer, R. Kohavi, An empirical comparison of voting classification algorithms: bagging, boosting, and variants, Machine Learning 36 (1-2) (1999).
[58] Ho, T.K., The random subspace method for constructing decision forests, IEEE transactions on pattern analysis and machine intelligence, 20, 832-844, (1998)
[59] Bertolami, R.; Zimmermann, M.; Bunke, H., Rejection strategies for offline handwritten text line recognition, Pattern recognition letters, 27, 16, 2005-2012, (2006)
[60] Gorski, N., Optimizing error-reject trade off in recognition systems, (), 1092-1096
[61] Pitrelli, J.; Perrone, M.P., Confidence-scoring post-processing for off-line handwritten-character recognition verification, (), 278-282
[62] U.-V. Marti, H. Bunke, Using a statistical language model to improve the performance of an HMM-based cursive handwriting recognition system, in: Int. J. Pattern Recognition Artif. Intell. 21 (2001) 65-90.
[63] J. Schenk, S. Schwrzler, G. Rigoll, Discrete single vs. multiple stream HMMs: a comparative evaluation of their use in on-line handwriting recognition of whiteboard notes, in: Proceedings of the 11th International Conference on Frontiers in Handwriting Recognition, 2008, pp. 550-555.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.