×

A variational approach to video registration with subspace constraints. (English) Zbl 1304.68175

Summary: This paper addresses the problem of non-rigid video registration, or the computation of optical flow from a reference frame to each of the subsequent images in a sequence, when the camera views deformable objects. We exploit the high correlation between 2D trajectories of different points on the same non-rigid surface by assuming that the displacement of any point throughout the sequence can be expressed in a compact way as a linear combination of a low-rank motion basis. This subspace constraint effectively acts as a trajectory regularization term leading to temporally consistent optical flow. We formulate it as a robust soft constraint within a variational framework by penalizing flow fields that lie outside the low-rank manifold. The resulting energy functional can be decoupled into the optimization of the brightness constancy and spatial regularization terms, leading to an efficient optimization scheme. Additionally, we propose a novel optimization scheme for the case of vector valued images, based on the dualization of the data term. This allows us to extend our approach to deal with colour images which results in significant improvements on the registration results. Finally, we provide a new benchmark dataset, based on motion capture data of a flag waving in the wind, with dense ground truth optical flow for evaluation of multi-frame optical flow algorithms for non-rigid surfaces. Our experiments show that our proposed approach outperforms state of the art optical flow and dense non-rigid registration algorithms.

MSC:

68T45 Machine vision and scene understanding
68T10 Pattern recognition, speech recognition
68U10 Computing methodologies for image processing

Software:

DTAM
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Akhter, I., Sheikh, Y., Khan, S., & Kanade, T. (2008). Nonrigid structure from motion in trajectory space. In Neural Information Processing Systems, pp. 41–48.
[2] Akhter, I., Sheikh, Y., Khan, S., & Kanade, T. (2011). Trajectory space: A dual representation for nonrigid structure from motion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(7), 1442–1456. · doi:10.1109/TPAMI.2010.201
[3] Alvarez L., Esclarín J., Lefébure M., Sánchez J. (1999). A PDE model for computing the optical flow. In Proceedings of the XVI Congreso de Ecuaciones Diferenciales y Aplicaciones (pp. 1349–1356). Las Palmas de Gran Canaria, Spain, Sept. 1999.
[4] Alvarez, L., Weickert, J., & Sánchez, J. (Aug. 2000). Reliable estimation of dense optical flow fields with large displacements. International Journal of Computer Vision, 39(1), 41–56. · Zbl 1060.68635
[5] Aubert, G., Deriche, R., & Kornprobst, P. (1999). Computing optical flow via variational techniques. SIAM Journal on Applied Mathematics, 60(1), 156–182. · Zbl 0942.35057 · doi:10.1137/S0036139998340170
[6] Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M., & Szeliski, R. (2011). A database and evaluation methodology for optical flow. International Journal of Computer Vision, 92, 1–31. · Zbl 06023231 · doi:10.1007/s11263-010-0390-2
[7] Bartoli, A., Gay-Bellile, V., Castellani, U., Peyras, J., Olsen, S. I., & Sayd, P. (2008). Coarse-to-fine low-rank structure-from-motion. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8.
[8] Blomgren, P., & Chan, T. (1998). Color TV: Total variation methods for restoration of vector-valued images. IEEE Transactioons on Image Processing, 7(3):304–309, Special issue on partial differential equations and geometry-driven diffusion in image processing and analysis.
[9] Brand, M. (2001). Morphable models from video. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 456–463.
[10] Bregler, C., Hertzmann, A., & Biermann, H. (2000). Recovering non-rigid 3D shape from image streams. In IEEE Conference on Computer Vision and Pattern Recognition, Vol. 2, pp. 690–696.
[11] Brox, T., Bruhn, A., Papenberg, N., & Weickert, J. (2004). High accuracy optical flow estimation based on a theory for warping. In T. Pajdla & J. Matas (Eds.), European Conference on Computer Vision–ECCV 2004 (pp. 25–36). Part IV, volume 3024 of LNCS Berlin: Springer. · Zbl 1098.68736
[12] Brox, T., & Malik, J. (2011). Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE Transactions on Pattern Aanalysis and Machine Intelligencea, 33(3), 500–513.
[13] Chambolle, A. (2004). An algorithm for total variation minimization and applications. Journal of Mathematical Imaging and Vision, 20, 89–97. · Zbl 1366.94048 · doi:10.1023/B:JMIV.0000011320.81911.38
[14] Chambolle, A., & Pock, T. (2011). A first-order primal-dual algorithm for convex problems with applications to imaging. Journal of Mathematical Imaging and Vision, 40(1), 120–145. · Zbl 1255.68217
[15] Deriche, R., Kornprobst, P., & Aubert, G. (1995). Optical-flow estimation while preserving its discontinuities: A variational approach. In Proceedings of the Second Asian Conference on Computer Vision (Vol. 2, pp. 290–295). Singapore, Dec. 1995.
[16] Esser, E., Zhang, X., & Chan, T. F. (2010). A general framework for a class of first order primal-dual algorithms for convex optimization in imaging science. SIAM Journal on Imaging Sciences, 3(4), 1015–1046. · Zbl 1206.90117 · doi:10.1137/09076934X
[17] Garg, R., Pizarro, L., Rueckert, D., & Agapito, L. (2010). Dense multi-frame optic flow for non-rigid objects using subspace constraints. In Asian Conference on Computer Vision, pp. 460–473.
[18] Garg, R., Roussos, A., & Agapito, L. (2011). Robust trajectory-space TV-L1 optical flow for non-rigid sequences. In 8th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR), pp. 300–314.
[19] Horn, B., & Schunck, B. (1981). Determining optical flow. Artificial Intelligence, 17, 185–203. · doi:10.1016/0004-3702(81)90024-2
[20] Irani, M. (2002). Multi-frame correspondence estimation using subspace constraints. International Journal of Computer Vision, 48(3), 173–194. · Zbl 1012.68758
[21] Kumar, A., Tannenbaum, A. R., & Balas, G. J. (Apr. 1996). Optic flow: A curve evolution approach. IEEE Transactions on Image Processing, 5(4), 598–610.
[22] Liu, C., Yuen, J., & Torralba, A. (2011). SIFT flow: Dense correspondence across scenes and its applications. In IEEE Transational on Pattern Analysis and Machins Intelligence, 33(5), 978–994.
[23] Lucas, B., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. In International Joint Conference on Artificial Intelligence.
[24] Newcombe, R., Lovegrove, S., & Davison, A. (2011). DTAM: Dense tracking and mapping in real-time. In International Conference on Computer Vision, pp. 2320–2327.
[25] Nir, T., Bruckstein, A. M., & Kimmel, R. (February 2008). Over-parameterized variational optical flow. International Journal of Computer Vision, 76, 205–216.
[26] Papadakis, N., Corpetti, T., & Mémin, E. (2007). Dynamically consistent optical flow estimation. In ICCV (pp. 1–7). Rio de Janeiro, Brazil, October 2007.
[27] Papenberg, N., Bruhn, A., Brox, T., Didas, S., & Weickert, J. (Apr. 2006). Highly accurate optic flow computation with theoretically justified warping. International Journal of Computer Vision, 67(2), 141–158.
[28] Pizarro, D., & Bartoli, A. (2010). Feature-based deformable surface detection with self-occlusion reasoning. In International symposium on 3D data processing, visualization and transmission, 3DPVT’10. · Zbl 1235.68283
[29] Pock, T., & Chambolle, A. (2011). Diagonal preconditioning for first order primal-dual algorithms in convex optimization. In International Conference on Computer Vision, pp. 1762–1769. · Zbl 1255.68217
[30] Pock, T., Cremers, D., Bischof, H., & Chambolle, A. (2010). Global solutions of variational models with convex regularization. SIAM Journal on Imaging Sciences, 3(4), 1122–1145. · Zbl 1202.49031
[31] Rakêt, L. L., Roholm, L., Nielsen, M., & Lauze, F. (2011). TV-L1 optical flow for vector valued images. In 8th International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR), pp. 329–343.
[32] Ricco, S., & Tomasi, C. (2012). Dense lagrangian motion estimation with occlusions. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1800–1807.
[33] Rockafellar, R. T. (1997). Convex analysis. Princeton Landmarks in Mathematics. Princeton University Press, Princeton, NJ, 1997. Reprint of the 1970 original, Princeton Paperbacks.
[34] Rudin, L., Osher, S., & Fatemi, E. (1992). Nonlinear total variation based noise removal algorithms. Physica D, 60, 259–268. · Zbl 0780.49028 · doi:10.1016/0167-2789(92)90242-F
[35] Sapiro, G. (1997). Color snakes. Computer Vision and Image Understanding, 68(2), 247–253.
[36] Schnörr, C. (1994). Segmentation of visual motion by minimizing convex non-quadratic functionals. In Proceedings of the twelfth international conference on pattern recognition (Vol. A, pp. 661–663). Jerusalem, Israel, Oct. 1994. IEEE Computer Society Press.
[37] Shi, J., & Tomasi, C. (1994). Good features to track. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 593–600.
[38] Steinbruecker, F., Pock, T., & Cremers, D. (2009). Large displacement optical flow computation without warping. In International Conference on Computer Vision, pp. 1609–1614.
[39] Stuehmer, J., Gumhold, S., & Cremers, D. (2010). Real-time dense geometry from a handheld camera. In Pattern recognition (Proc. DAGM) (pp. 11–20), September 2010.
[40] Sun, D., Roth, S., Lewis, J. P., & Black, M. (2008). Learning optical flow. In European Conference on Computer Vision, pp. 83–97.
[41] Tian, Y., & Narasimhan, S. (2010). A globally optimal data-driven approach for image distortion estimation. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 1277–1284.
[42] Torresani, L., & Bregler, C. (2002). Space-time tracking. In European Conference on Computer Vision, pp. 801–812. · Zbl 1034.68685
[43] Torresani, L., Hertzmann, A., & Bregler, C. (2008). Non-rigid structure-from-motion: Estimating shape and motion with hierarchical priors. PAMI, 30(5), 878–892.
[44] Torresani, L., Yang, D., Alexander, E., & Bregler, C. (2001). Tracking and modeling non-rigid objects with rank constraints. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 493–500.
[45] Tschumperlé, D., & Deriche, R. (2005). Vector-valued image regularization with PDE’s: A common framework for different applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(4), 506–517. · Zbl 05111099 · doi:10.1109/TPAMI.2005.87
[46] Uras, S., Girosi, F., Verri, A., & Torre, V. (1988). A computational approach to motion perception. Biological Cybernetics, 60, 79–87. · doi:10.1007/BF00202895
[47] Varol, A., Salzmann, M., Tola, E., & Fua, P. (2009). Template-free monocular reconstruction of deformable surfaces. In International Conference on Computer Vision, pp. 1811–1818.
[48] Volz, S., Bruhn, A., Valgaerts, L., & Zimmer, H. (2011). Modeling temporal coherence for optical flow. In International Conference on Computer Vision, pp. 1116–1123.
[49] Wedel, A., Cremers, D., Pock, T., & Bischof, H. (2009). Structure- and motion-adaptive regularization for high accuracy optic flow. In International Conference on Computer Vision, pp. 1663–1668.
[50] Wedel, A., Pock, T., Braun, J., Franke, U., & Cremers, D. (2008). Duality TV-L1 flow with fundamental matrix prior. In Image and Vision Computing New Zealand, pp. 1–6.
[51] Wedel, A., Pock, T., Zach, C., Bischof, H., & Cremers, D. (2009). An improved algorithm for TV-L1 optical flow. In Statistical and geometrical approaches to visual motion analysis, LNCS (pp. 23–45). Springer, Berlin.
[52] Weickert, J. (1998). On discontinuity-preserving optic flow. In S. Orphanoudakis, P. Trahanias, J. Crowley, & N. Katevas (Eds.), Proceedings of the computer vision and mobile robotics workshop (pp. 115–122). Santorini, Greece, Sept 1998.
[53] Weickert, J., & Schnörr, C. (Dec. 2001). A theoretical framework for convex regularizers in PDE-based computation of image motion. International Journal of Computer Vision, 45(3), 245–264. · Zbl 0987.68600
[54] Weickert, J., & Schnörr, C. (May 2001). Variational optic flow computation with a spatio-temporal smoothness constraint. Journal of Mathematical Imaging and Vision, 14(3), 245–255. · Zbl 0988.68821
[55] Werlberger, M., Trobin, W., Pock, T., Wedel, A., Cremers, D., & Bischof, H. (2009). Anisotropic Huber-L1 optical flow. In British Machine Vision Conference, Vol. 34, pp. 1–11.
[56] White, R., Crane, K., Forsyth, D. (2007). Capturing and animating occluded cloth. In ACM Transactions on Graphics.
[57] Zach, C., Pock, T., & Bischof, H. (2007). A duality based approach for realtime TV-L1 optical flow. In Pattern recognition (Proc. DAGM), pp. 214–223.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.