×

Tensor decomposition and non-linear manifold modeling for 3D head pose estimation. (English) Zbl 1477.68349

Summary: Head pose estimation is a challenging computer vision problem with important applications in different scenarios such as human-computer interaction or face recognition. In this paper, we present a 3D head pose estimation algorithm based on non-linear manifold learning. A key feature of the proposed approach is that it allows modeling the underlying 3D manifold that results from the combination of rotation angles. To do so, we use tensor decomposition to generate separate subspaces for each variation factor and show that each of them has a clear structure that can be modeled with cosine functions from a unique shared parameter per angle. Such representation provides a deep understanding of data behavior. We show that the proposed framework can be applied to a wide variety of input features and can be used for different purposes. Firstly, we test our system on a publicly available database, which consists of 2D images and we show that the cosine functions can be used to synthesize rotated versions from an object from which we see only a 2D image at a specific angle. Further, we perform 3D head pose estimation experiments using other two types of features: automatic landmarks and histogram-based 3D descriptors. We evaluate our approach on two publicly available databases, and demonstrate that angle estimations can be performed by optimizing the combination of these cosine functions to achieve state-of-the-art performance.

MSC:

68T45 Machine vision and scene understanding
68T05 Learning and adaptive systems in artificial intelligence
PDFBibTeX XMLCite
Full Text: DOI HAL

References:

[1] Ahn, B., Park, J., & Kweon, I. S. (2014). Real-time head orientation from a monocular camera using deep neural network. In Asian conference on computer vision (pp. 82-96). Springer.
[2] Bakry, A., & Elgammal, A. (2014). Untangling object-view manifold for multiview recognition and pose estimation. In European conference on computer vision (pp. 434-449). Springer.
[3] Balasubramanian, V. N., Ye, J., & Panchanathan, S. (2007). Biased manifold embedding: A framework for person-independent head pose estimation. In Computer vision and pattern recognition (CVPR) (pp. 1-7). IEEE.
[4] Baltrušaitis, T., Robinson, P., & Morency, L. P. (2012). 3D constrained local model for rigid and non-rigid facial tracking. In Computer vision and pattern recognition (CVPR) (pp. 2610-2617). IEEE.
[5] Barros, J. M. D., Mirbach, B., Garcia, F., Varanasi, K., & Stricker, D. (2018). Fusion of keypoint tracking and facial landmark detection for real-time head pose estimation. In Winter conference on applications of computer vision (WACV) (pp. 2028-2037). IEEE.
[6] BenAbdelkader, C. (2010). Robust head pose estimation using supervised manifold learning. In European conference on computer vision (pp. 518-531). Springer.
[7] Bergqvist, G., & Larsson, E. G. (2010). The higher-order singular value decomposition: Theory and an application [lecture notes]. IEEE Signal Processing Magazine, 27(3), 151-154. · doi:10.1109/MSP.2010.936030
[8] Borghi, G., Fabbri, M., Vezzani, R., Calderara, S., & Cucchiara, R. (2019). Face-from-depth for head pose estimation on depth images. IEEE Transactions on Pattern Analysis and Machine Intelligence(in press).
[9] Borghi, G., Venturelli, M., Vezzani, R., & Cucchiara, R. (2017). Poseidon: Face-from-depth for driver pose estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4661-4670).
[10] Breitenstein, M. D., Kuettel, D., Weise, T., Van Gool L, & Pfister, H. (2008). Real-time face pose estimation from single range images. In Computer vision and pattern recognition (pp. 1-8). IEEE.
[11] Byrd, R. H., Nocedal, J., & Schnabel, R. B. (1994). Representations of quasi-newton matrices and their use in limited memory methods. Mathematical Programming, 63(1-3), 129-156. · Zbl 0809.90116 · doi:10.1007/BF01582063
[12] Chen, J., Wu, J., Richter, K., Konrad, J., & Ishwar, P. (2016). Estimating head pose orientation using extremely low resolution images. In Southwest symposium on image analysis and interpretation (SSIAI) (pp. 65-68). IEEE
[13] Comon, P. (2014). Tensors: A brief introduction. Signal Processing Magazine, 31(3), 44-53. · doi:10.1109/MSP.2014.2298533
[14] De Lathauwer, L., De Moor, B., & Vandewalle, J. (2000). A multilinear singular value decomposition. SIAM Journal on Matrix Analysis and Applications, 21(4), 1253-1278. · Zbl 0962.15005 · doi:10.1137/S0895479896305696
[15] Derkach, D., Ruiz, A., & Sukno, F. M. (2017). Head pose estimation based on 3-D facial landmarks localization and regression. In 12th IEEE international conference on automatic face and gesture recognition (FG 2017) (pp. 820-827). IEEE.
[16] Derkach, D., Ruiz, A., & Sukno, F. M. (2018). 3D head pose estimation using tensor decomposition and non-linear manifold modeling. In: International conference on 3D Vision (3DV) (pp. 505-513). IEEE.
[17] Fanelli, G., Dantone, M., Gall, J., Fossati, A., & Van Gool, L. (2013). Random forests for real time 3D face analysis. International Journal of Computer Vision, 101(3), 437-458. · doi:10.1007/s11263-012-0549-0
[18] Fanelli, G., Weise, T., Gall, J., & Van Gool, L. (2011). Real time head pose estimation from consumer depth cameras. In Joint pattern recognition symposium (pp. 101-110). Springer.
[19] Frome, A., Huber, D., Kolluri, R., Bulow, T., & Malik, J. (2004). Recognizing objects in range data using regional point descriptors. In European conference on computer vision (pp. 224-237). Springer. · Zbl 1098.68766
[20] Fu, Y., & Huang, T. S. (2006). Graph embedded analysis for head pose estimation. In International conference on automatic face and gesture recognition (pp. 6-8). IEEE.
[21] Ghiass, R. S., Arandjelović, O., & Laurendeau, D. (2015). Highly accurate and fully automatic head pose estimation from a low quality consumer-level rgb-d sensor. In Proceedings of the 2nd workshop on computational models of social interactions: Human-Computer-Media communication (pp. 25-34). ACM.
[22] Gu, J., Yang, X., De Mello, S., & Kautz, J. (2017). Dynamic facial analysis: From bayesian filtering to recurrent neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1548-1557).
[23] Johnson, A., & Hebert, M. (1999). Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(5), 433-449. · doi:10.1109/34.765655
[24] Kolda, T. G., & Bader, B. W. (2009). Tensor decompositions and applications. SIAM Review, 51(3), 455-500. · Zbl 1173.65029 · doi:10.1137/07070111X
[25] Lathuiliére, S., Juge, R., Mesejo, P., Muñoz-Salinas, R., & Horaud, R. (2017). Deep mixture of linear inverse regressions applied to head-pose estimation. In Conference on computer vision and pattern recognition (vol. 3, pp. 4817-4825).
[26] Lathuiliére, S., Mesejo, P., Alameda-Pineda, X., & Horaud, R. (2019). A comprehensive analysis of deep regression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1-1 (in press).
[27] Lee, D., Yang, M. H., & Oh, S. (2015). Fast and accurate head pose estimation via random projection forests. In International conference on computer vision (pp. 1958-1966). IEEE.
[28] Lee, D., Yang, M. H., & Oh, S. (2017). Head and body orientation estimation using convolutional random projection forests. In IEEE transactions on pattern analysis and machine intelligence (pp. 1-14)
[29] Li, D., & Pedrycz, W. (2014). A central profile-based 3D face pose estimation. Pattern Recognition, 47(2), 525-534. · doi:10.1016/j.patcog.2013.07.019
[30] Li, S., Ngan, K. N., Paramesran, R., & Sheng, L. (2016). Real-time head pose tracking with online face template reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(9), 1922-1928. · doi:10.1109/TPAMI.2015.2500221
[31] Liu, X., Liang, W., Wang, Y., Li, S., & Pei, M. (2016). 3D head pose estimation with convolutional neural network trained on synthetic images. In International conference on image processing (ICIP) (pp. 1289-1293). IEEE.
[32] Liu, X., Lu, H., & Li, W. (2010). Multi-manifold modeling for head pose estimation. In International conference on image processing (ICIP) (pp. 3277-3280). IEEE.
[33] Lüsi, I., Escalera, S., & Anbarjafari, G. (2016a). Human head pose estimation on SASE database using random hough regression forests. Video Analytics (pp. 137-150). Springer: Face and Facial Expression Recognition and Audience Measurement.
[34] Lüsi, I., Escarela, S., & Anbarjafari, G. (2016b). SASE: RGB-depth database for human head pose estimation. In European conference on computer vision (pp. 325-336). Springer.
[35] Lüsi, I., Jacques Junior, J. C. S., Gorbova, J., Baró X, Escalera, S., Demirel, H., Allik, J., Ozcinar, C., & Anbarjafari, G. (2017). Joint challenge on dominant and complementary emotion recognition using micro emotion features and head-pose estimation: Databases. In International conference on automatic face and gesture recognition (pp. 809-813). IEEE.
[36] Martin, M., Van De Camp, F., & Stiefelhagen, R. (2014). Real time head model creation and head pose estimation on consumer depth cameras. In International conference on 3D vision (3DV) (vol. 1, pp. 641-648). IEEE.
[37] Meyer, G. P., Gupta, S., Frosio, I., Reddy, D., & Kautz, J. (2015). Robust model-based 3D head pose estimation. In Proceedings of the IEEE international conference on computer vision (pp. 3649-3657). IEEE.
[38] Murphy-Chutorian, E., & Trivedi, M. M. (2009). Head pose estimation in computer vision: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(4), 607-626. · doi:10.1109/TPAMI.2008.106
[39] Nene, S. A., Nayar, S. K., Murase, H., et al. (1996). Columbia object image library (coil-20).
[40] Padeleris, P., Zabulis, X., & Argyros, A. A. (2012). Head pose estimation on depth data based on particle swarm optimization. In Computer society conference on computer vision and pattern recognition workshops (CVPRW) (pp. 42-49). IEEE.
[41] Papazov, C., Marks, T. K., & Jones, M. (2015). Real-time 3D head pose and facial landmark estimation from depth images using triangular surface patch features. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4722-4730).
[42] Patacchiola, M., & Cangelosi, A. (2017). Head pose estimation in the wild using convolutional neural networks and adaptive gradient methods. Pattern Recognition, 71, 132-143. · doi:10.1016/j.patcog.2017.06.009
[43] Peng, X., Huang, J., Hu, Q., Zhang, S., & Metaxas, D. N. (2014). Head pose estimation by instance parameterization. In International conference on pattern recognition (ICPR) (pp. 1800-1805). IEEE.
[44] Raytchev, B., Yoda, I., & Sakaue, K. (2004). Head pose estimation by nonlinear manifold learning. In International conference on pattern recognition (ICPR) (vol. 4, pp. 462-466). IEEE.
[45] Ruiz, N., Chong, E., & Rehg, J. M. (2018). Fine-grained head pose estimation without keypoints. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 2074-2083).
[46] Rusu, R. B., Blodow, N., & Beetz, M. (2009). Fast point feature histograms (fpfh) for 3d registration. In International conference on robotics and automation, Citeseer (pp. 3212-3217).
[47] Schmidt, M. (2012). Minfunc: Unconstrained differentiable multivariate optimization in matlab. Software available at http://www.cs.ubc.ca/schmidtm/Software/minFunc.htm.
[48] Seemann, E., Nickel, K., & Stiefelhagen, R. (2004). Head pose estimation using stereo vision for human – robot interaction. In International conference on automatic face and gesture recognition (pp. 626-631). IEEE.
[49] Sukno, F., Waddington, J., & Whelan, P. (2012). Comparing 3D descriptors for local search of craniofacial landmarks. In International symposium on visual computing (pp. 92-103). Springer.
[50] Sukno, F., Waddington, J., & Whelan, P. (2013). Rotationally invariant 3D shape contexts using asymmetry patterns. International conference on computer graphics theory and applications (pp. 7-17).
[51] Sukno, F. M., Waddington, J. L., & Whelan, P. F. (2015). 3-D facial landmark localization with asymmetry patterns and shape regression from incomplete local features. IEEE Transactions on Cybernetics, 45(9), 1717-1730. · doi:10.1109/TCYB.2014.2359056
[52] Sun, Y., & Yin, L. (2008). Automatic pose estimation of 3D facial models. In International conference on pattern recognition (pp. 1-4.).
[53] Sundararajan, K., & Woodard, D. L. (2015). Head pose estimation in the wild using approximate view manifolds. In International conference on computer vision and pattern recognition workshops (pp. 50-58). IEEE.
[54] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1-9).
[55] Takallou, H. M., & Kasaei, S. (2014). Head pose estimation and face recognition using a non-linear tensor-based model. IET Computer Vision, 8(1), 54-65. · doi:10.1049/iet-cvi.2012.0217
[56] Tan, D. J., Tombari, F., & Navab, N. (2018). Real-time accurate 3d head tracking and pose estimation with consumer rgb-d cameras. International Journal of Computer Vision, 126(2-4), 158-183. · doi:10.1007/s11263-017-0988-8
[57] Tenenbaum, J. B., & Freeman, W. T. (1997). Separating style and content. In Advances in neural information processing systems (pp. 662-668).
[58] Tenenbaum, J. B., & Freeman, W. T. (2000). Separating style and content with bilinear models. Neural Computation, 12(6), 1247-1283. · doi:10.1162/089976600300015349
[59] Tombari, F., Salti, S., & Di Stefano, L. (2010). Unique signatures of histograms for local surface description. In European conference on computer vision (pp. 356-369). Springer.
[60] Tulyakov, S., Vieriu, R. L., Semeniuta, S., & Sebe, N. (2014). Robust real-time extreme head pose estimation. In International conference on pattern recognition (ICPR) (pp. 2263-2268). IEEE.
[61] Vasilescu, M. A. O., & Terzopoulos, D. (2002). Multilinear analysis of image ensembles: Tensorfaces. In European conference on computer vision (pp. 447-460). Springer. · Zbl 1034.68693
[62] Wang, B., Liang, W., Wang, Y., & Liang, Y. (2013). Head pose estimation with combined 2D SIFT and 3D HOG features. In International conference on image and graphics (ICIG) (pp. 650-655). IEEE.
[63] Wang, C., Guo, Y., & Song, X. (2017a). Head pose estimation via manifold learning. InTech: In Manifolds-current research areas. · doi:10.5772/65903
[64] Wang, C., & Song, X. (2014). Robust head pose estimation via supervised manifold learning. Neural Networks, 53, 15-25. · Zbl 1308.68101 · doi:10.1016/j.neunet.2014.01.009
[65] Wang, K., Wu, Y., & Ji, Q. (2018). Head pose estimation on low-quality images. In International conference on automatic face and gesture recognition (FG 2018) (pp. 540-547). IEEE.
[66] Wang, M., Panagakis, Y., Snape, P., Zafeiriou, S., et al. (2017b). Learning the multilinear structure of visual data. In: Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4592-4600).
[67] Wang, Y., Liang, W., Shen, J., Jia, Y., & Yu, L. F. (2019). A deep coarse-to-fine network for head pose estimation from synthetic data. Pattern Recognition, 94, 196-206. · doi:10.1016/j.patcog.2019.05.026
[68] Xu, Y., Hao, R., Yin, W., & Su, Z. (2015). Parallel matrix factorization for low-rank tensor completion. Inverse Problems and Imaging, 9(2), 601-624. · Zbl 1359.15021 · doi:10.3934/ipi.2015.9.601
[69] Yu, Y., Mora, K. A. F., & Odobez, J. M. (2017). Robust and accurate 3D head pose estimation through 3dmm and online head model reconstruction. In International conference on automatic face and gesture recognition (FG 2017) (pp. 711-718). IEEE.
[70] Zhang, H., El-Gaaly, T., Elgammal, A., & Jiang, Z. (2015). Factorization of view-object manifolds for joint object recognition and pose estimation. Computer Vision and Image Understanding, 139, 89-103. · doi:10.1016/j.cviu.2015.03.014
[71] Zhao, Q., Zhang, L., & Cichocki, A. (2015). Bayesian cp factorization of incomplete tensors with automatic rank determination. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9), 1751-1763. · doi:10.1109/TPAMI.2015.2392756
[72] Zhu, Y., Xue, Z., & Li, C. (2014). Automatic head pose estimation with synchronized sub manifold embedding and random regression forests. International Journal of Signal Processing, Image Processing and Pattern Recognition, 7(3), 123-134. · doi:10.14257/ijsip.2014.7.3.11
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.