×

Shape-from-template with curves. (English) Zbl 1477.68354

Summary: Shape-from-Template (SfT) is the problem of using a shape template to infer the shape of a deformable object observed in an image. The usual case of SfT is ‘Surface’ SfT, where the shape is a 2D surface embedded in 3D, and the image is a 2D perspective projection. We introduce ‘Curve’ SfT, comprising two new cases of SfT where the shape is a 1D curve. The first new case is when the curve is embedded in 2D and the image a 1D perspective projection. The second new case is when the curve is embedded in 3D and the image a 2D perspective projection. We present a thorough theoretical study of these new cases for isometric deformations, which are a good approximation of ropes, cables and wires. Unlike Surface SfT, we show that Curve SfT is only ever solvable up to discrete ambiguities. We present the necessary and sufficient conditions for solvability with critical point analysis. We further show that unlike Surface SfT, Curve SfT cannot be solved locally using exact non-holonomic Partial Differential Equations. Our main technical contributions are two-fold. First, we give a stable, global reconstruction method that models the problem as a discrete Hidden Markov Model. This can generate all candidate solutions. Second, we give a non-convex refinement method using a novel angle-based deformation parameterization. We present quantitative and qualitative results showing that real curve shaped objects such as a necklace can be successfully reconstructed with Curve SfT.

MSC:

68T45 Machine vision and scene understanding
35Q68 PDEs in connection with computer science
68U05 Computer graphics; computational geometry (digital and algorithmic aspects)

Software:

SIFT; UGM; SeDuMi; YALMIP
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Agisoft. (2013). Agisoft Lens Version 0.4.1 beta 64 bit (build 1718). http://www.agisoft.com.
[2] Agisoft. (2014). Agisoft PhotoScan Version 1.2.3 build 2331 (64 bit). http://www.agisoft.com.
[3] Bartoli, A., & Özgür, E. (2016). A perspective on non-isometric shape-from-template. In ISMAR.
[4] Bartoli, A.; Gérard, Y.; Chadebecq, F.; Collins, T.; Pizarro, D., Shape-from-template, Transactions on Pattern Analysis and Machine Intelligence, 37, 10, 2099-2118 (2015) · doi:10.1109/TPAMI.2015.2392759
[5] Berthilsson, R.; Åström, K.; Heyden, A., Reconstruction of general curves, using factorization and bundle adjustment, International Journal of Computer Vision, 41, 3, 171-182 (2001) · Zbl 1012.68712 · doi:10.1023/A:1011104020586
[6] Blanz, V., & Vetter, T. (1999). Morphable model for the synthesis of 3D faces. In SIGGRAPH.
[7] Blender. (2017). Blender 2.78a. https://www.blender.org.
[8] Brunet, F.; Hartley, R.; Bartoli, A., Monocular template-based 3D surface reconstruction: Convex inextensible and nonconvex isometric methods, Computer Vision and Image Understanding, 125, 138-154 (2014) · doi:10.1016/j.cviu.2014.04.003
[9] Casillas-Perez, D., & Pizarro, D. (2017). Solutions of quadratic first-order odes applied to computer vision problems. ArXiv:1710.04265.
[10] Chhatkuli, A.; Pizarro, D.; Bartoli, A.; Collins, T., A stable analytical framework for isometric shape-from-template by surface integration, IEEE Transactions on Pattern Analysis and Machine Intelligence, 39, 5, 833-850 (2017) · doi:10.1109/TPAMI.2016.2562622
[11] Collins, T., & Bartoli, A. (2014). Using isometry to classify correct/incorrect 3D-2D correspondences. In ECCV.
[12] Collins, T., & Bartoli, A. (2015). Realtime shape-from-template: System and applications. In ISMAR.
[13] Collins, T., Mesejo, P., & Bartoli, A. (2014). An analysis of errors in graph-based keypoint matching and proposed solutions. In ECCV.
[14] Collins, T., Bartoli, A., Bourdel, N., & Canis, M. (2016). Dense, robust and real-time 3D tracking of deformable organs in monocular laparoscopy. In MICCAI.
[15] David 3D Scanner. (2014). http://www.david-3d.com/en/products/david4.
[16] Eliashberg, Y., & Mishachev, N. M. (2002). Introduction to the h-principle. Number Grad. Stud. Math. 48. New York: American Mathematical Society. · Zbl 1008.58001
[17] Faugeras, O.; Papadopoulo, T., A theory of the motion fields of curves, International Journal of Computer Vision, 10, 2, 125-156 (1993) · doi:10.1007/BF01420734
[18] Gallardo, M., Pizarro, D., Bartoli, A., & Collins, T. (2015). Shape-from-template in flatland. In CVPR.
[19] Gallardo, M., Collins, T., & Bartoli, A. (2016). Using shading and a 3D template to reconstruct complex surface deformations. In BMVC.
[20] Haouchine, N., Dequidt, J., Berger, M. O., & Cotin, S. (2014). Single view augmentation of 3D elastic objects. In ISMAR.
[21] Hartley, Ri; Zisserman, A., Multiple view geometry in computer vision (2003), Cambridge: Cambridge University Press, Cambridge
[22] Kahl, F., & August, J. (2003). Multiview reconstruction of space curves. In ICCV.
[23] Liu-Yin, Q., Yu, R., Agapito, L., Fitzgibbon, A., & Russell, C. (2016). Better together: Joint reasoning for non-rigid 3D reconstruction with specularities and shading. In BMVC.
[24] Löfberg, J. (2004). YALMIP: A toolbox for modeling and optimization in MATLAB. In International symposium on computer-aided control system design.
[25] Lowe, Dg, Distinctive image features from scale-invariant keypoints, International Journal of Computer Vision, 60, 2, 91-110 (2004) · doi:10.1023/B:VISI.0000029664.99615.94
[26] Mai, F., & Hung, Y. S. (2010). 3D curves reconstruction from multiple images. In Digital image computing: techniques and applications.
[27] Malti, A.; Bartoli, A., Combining conformal deformation and cook-torrance shading for 3D reconstruction in laparoscopy, IEEE Transactions on Biological Engineering, 61, 6, 1684-1692 (2014) · doi:10.1109/TBME.2014.2300237
[28] Malti, A., Bartoli, A., & Collins, T. (2011). A pixel-based approach to template-based monocular 3D reconstruction of deformable surfaces. In Proceedings of the IEEE international workshop on dynamic shape capture and analysis at ICCV.
[29] Malti, A., Hartley, R., Bartoli, A., & Kim, J. (2013). Monocular template-based 3D reconstruction of extensible surfaces with local linear elasticity. In CVPR.
[30] Malti, A., Bartoli, A., & Hartley, R. I. (2015). A linear least-squares solution to elastic shape-from-template. In CVPR.
[31] Martinsson, H., Gaspard, F., Bartoli, A., & Lavest, J. (2007). Energy-based reconstruction of 3D curves for quality control. In EMMCVPR.
[32] Ngo, T. D., Park, S., Jorstad, A. A., Crivellaro, A., Yoo, C., & Fua, P. (2015). Dense image registration and deformable surface reconstruction in presence of occlusions and minimal texture. In ICCV.
[33] Ngo, Td; Östlund, J.; Fua, P., Template-based monocular 3D shape recovery using laplacian meshes, IEEE Transactions on Pattern Analysis and Machine Intelligence, 38, 1, 172-187 (2016) · doi:10.1109/TPAMI.2015.2435739
[34] Özgür, E.; Bartoli, A., Particle-SfT: A provably-convergent, fast shape-from-template algorithm, International Journal of Computer Vision, 123, 2, 184-205 (2017) · Zbl 1458.68243 · doi:10.1007/s11263-016-0968-4
[35] Parashar, S., Pizarro, D., Bartoli, A., & Collins, T. (2015). As-rigid-as-possible volumetric shape-from-template. In ICCV.
[36] Perriollat, M.; Hartley, R.; Bartoli, A., Monocular template-based reconstruction of inextensible surfaces, International Journal of Computer Vision, 95, 2, 124-137 (2011) · Zbl 1235.68281 · doi:10.1007/s11263-010-0352-8
[37] Pizarro, D.; Bartoli, A., Feature-based deformable surface detection with self-occlusion reasoning, International Journal of Computer Vision, 97, 1, 54-70 (2012) · Zbl 1235.68283 · doi:10.1007/s11263-011-0452-0
[38] Rabiner, Lr, A tutorial on hidden markov models and selected applications in speech recognition, Proceedings of the IEEE, 77, 2, 257-286 (1989) · doi:10.1109/5.18626
[39] Robert, L., & Faugeras, O. (1991). Curve-based stereo: Figural continuity and curvature. In CVPR.
[40] Salzmann, M., & Fua, P. (2009). Reconstructing sharply folding surfaces: A convex formulation. In CVPR.
[41] Salzmann, M.; Fua, P., Linear local models for monocular reconstruction of deformable surfaces, IEEE Transactions on Pattern Analysis and Machine Intelligence, 33, 5, 931-944 (2011) · doi:10.1109/TPAMI.2010.158
[42] Salzmann, M.; Pilet, J.; Ilic, S.; Fua, P., Surface deformation models for nonrigid 3D shape recovery, IEEE Transactions on Pattern Analysis and Machine Intelligence, 29, 8, 1481-1487 (2007) · doi:10.1109/TPAMI.2007.1080
[43] Sbert, C.; Solé, Af, 3D curves reconstruction based on deformable models, Journal of Mathematical Imaging and Vision, 18, 211-223 (2003) · Zbl 1051.68129 · doi:10.1023/A:1022821409482
[44] Schmidt, M. (2007). UGM: A matlab toolbox for probabilistic undirected graphical models. http://www.cs.ubc.ca/ schmidtm/Software.
[45] Sorkine, O., & Alexa, M. (2007). As-rigid-as-possible surface modeling. In Proceedings of the fifth Eurographics symposium on geometry processing (pp. 109-116).
[46] Sturm, J., Using SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones, Optimization Methods and Software, 11-12, 625-653 (1999) · Zbl 0973.90526 · doi:10.1080/10556789908805766
[47] TurboSquid. (2016). http://www.turbosquid.com.
[48] Vicente, S., & Agapito, L. (2013). Balloon shapes: Reconstructing and deforming objects with volume from images. In 3DV, June.
[49] Warehouse, D. (2016). https://3dwarehouse.sketchup.com.
[50] Wu, C. (2011). VisualSFM: A visual structure from motion system. http://ccwu.me/vsfm.
[51] Yu, R., Russell, C., Campbell, N. D. F., & Agapito, L. (2015). Direct, dense, and deformable: Template-based non-rigid 3D reconstruction from RGB video. In ICCV.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.