Differentiable deformation graph-based neural non-rigid registration. (English) Zbl 1515.65057

Feng, Wanquan; Cai, Hongrui; Hou, Junhui; Deng, Bailin; Zhang, Juyong

Commun. Math. Stat. 11, No. 1, 151-167 (2023).

Summary: The traditional pipeline for non-rigid registration is to iteratively update the correspondence and alignment such that the transformed source surface aligns well with the target surface. Among the pipeline, the correspondence construction and iterative manner are key to the results, while existing strategies might result in local optima. In this paper, we adopt the widely used deformation graph-based representation, while replacing some key modules with neural learning-based strategies. Specifically, we design a neural network to predict the correspondence and its reliability confidence rather than the strategies like nearest neighbor search and pair rejection. Besides, we adopt the GRU-based recurrent network for iterative refinement, which is more robust than the traditional strategy. The model is trained in a self-supervised manner and thus can be used for arbitrary datasets without ground-truth. Extensive experiments demonstrate that our proposed method outperforms the state-of-the-art methods by a large margin.

MSC:

65D19	Computational issues in computer and robotic vision
68U05	Computer graphics; computational geometry (digital and algorithmic aspects)

Keywords:

differentiable deformation graph; non-rigid registration

Software:

Adam; D-Faust; PWC-Net

References:

[1]	Amberg, B., Romdhani, S., Vetter, T.: Optimal step nonrigid ICP algorithms for surface registration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2007)
[2]	Aoki, Y., Goforth, H., Srivatsan, R.A., Lucey, S.: Pointnetlk: robust & efficient point cloud registration using pointnet. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7163-7172 (2019)
[3]	Aubry, M., Schlickewei, U., Cremers, D.: The wave kernel signature: A quantum mechanical approach to shape analysis. In: IEEE International Conference on Computer Vision (ICCV), pp. 1626-1633 (2011)
[4]	Aygün, M., Lähner, Z., Cremers, D.: Unsupervised dense shape correspondence using heat kernels. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 573-582 (2020)
[5]	Besl, PJ; McKay, ND, A method for registration of 3-d shapes, IEEE Trans. Pattern Anal. Mach. Intell., 14, 2, 239-256 (1992) · doi:10.1109/34.121791
[6]	Bogo, F., Romero, J., Pons-Moll, G., Black, M.J.: Dynamic FAUST: Registering human bodies in motion. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
[7]	Boykov, Y., Veksler, O., Zabih, R.: Markov random fields with efficient approximations. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 648-655 (1998)
[8]	Bozic, A., Palafox, P.R., Zollhöfer, M., Dai, A., Thies, J., Nießner, M.: Neural non-rigid tracking. In: Annual Conference on Neural Information Processing Systems (NeurIPS) (2020)
[9]	Bronstein, M.M., Kokkinos, I.: Scale-invariant heat kernel signatures for non-rigid shape recognition. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1704-1711 (2010)
[10]	Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Generalized multidimensional scaling: a framework for isometry-invariant partial surface matching. In: Proceedings of the National Academy of Sciences, 1168-1172 (2006) · Zbl 1160.65306
[11]	Bronstein, A.M., Bronstein, M.M., Kimmel, R.: Numerical Geometry of Non-Rigid Shapes. Monographs in Computer Science, (2009) · Zbl 1178.68608
[12]	Chen, J.; Ma, J.; Yang, C.; Ma, L.; Zheng, S., Non-rigid point set registration via coherent spatial mapping, Signal Process, 106, 62-72 (2015) · doi:10.1016/j.sigpro.2014.07.004
[13]	Cho, K., van Merrienboer, B., Bahdanau, D., Bengio, Y.: On the properties of neural machine translation: encoder-decoder approaches. In: Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103-111 (2014)
[14]	Chui, H.; Rangarajan, A., A new point matching algorithm for non-rigid registration, Comput. Vis. Image Underst., 89, 2-3, 114-141 (2003) · Zbl 1053.68123 · doi:10.1016/S1077-3142(03)00009-2
[15]	Coifman, R.R., Lafon, S., Lee, A.B., Maggioni, M., Nadler, B., Warner, F., Zucker, S.W.: Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps. In: Proceedings of the National Academy of Sciences, pp. 7426-7431 (2005) · Zbl 1405.42043
[16]	Feng, W., Zhang, J., Cai, H., Xu, H., Hou, J., Bao, H.: Recurrent multi-view alignment network for unsupervised surface registration. CoRR abs/2011.12104 (2020)
[17]	Golyanik, V., Taetz, B., Reis, G., Stricker, D.: Extended coherent point drift algorithm with correspondence priors and optimal subsampling. In: Winter Conference on Applications of Computer Vision (WACV), pp. 1-9 (2016)
[18]	Groueix, T., Fisher, M., Kim, V.G., Russell, B.C., Aubry, M.: 3d-coded: 3d correspondences by deep deformation. In: Computer Vision-ECCV 2018-15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part II, vol. 11206 (2018)
[19]	Guo, K., Xu, F., Wang, Y., Liu, Y., Dai, Q.: Robust non-rigid motion tracking and surface reconstruction using L0 regularization. In: IEEE International Conference on Computer Vision (ICCV), pp. 3083-3091 (2015)
[20]	Halimi, O., Litany, O., Rodolà, E., Bronstein, A.M., Kimmel, R.: Unsupervised learning of dense shape correspondence. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4370-4379 (2019)
[21]	Hirose, O., A Bayesian formulation of coherent point drift, IEEE Trans. Pattern Anal. Mach. Intell., 99, 1-1 (2020)
[22]	Hurlburt, NE; Jaffey, S., A spectral optical flow method for determining velocities from digital imagery, Earth Sci. Inform., 8, 4, 959-965 (2015) · doi:10.1007/s12145-015-0224-4
[23]	Jian, B.; Vemuri, BC, Robust point set registration using gaussian mixture models, IEEE Trans. Pattern Anal. Mach. Intell., 33, 8, 1633-1645 (2011) · doi:10.1109/TPAMI.2010.223
[24]	Kim, VG; Lipman, Y.; Funkhouser, TA, Blended intrinsic maps, ACM Trans. Graph., 30, 79 (2011) · doi:10.1145/2010324.1964974
[25]	Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: International Conference on Learning Representations (ICLR) (2015)
[26]	Kittenplon, Y., Eldar, Y.C., Raviv, D.: Flowstep3d: model unrolling for self-supervised scene flow estimation. CoRR abs/2011.10147 (2020)
[27]	Kolesov, I.; Lee, J.; Sharp, G.; Vela, PA; Tannenbaum, AR, A stochastic approach to diffeomorphic point set registration with landmark constraints, IEEE Trans. Pattern Anal. Mach. Intell., 38, 2, 238-251 (2016) · doi:10.1109/TPAMI.2015.2448102
[28]	Li, H.; Sumner, RW; Pauly, M., Global correspondence optimization for non-rigid registration of depth scans, Comput. Graph. Forum, 27, 5, 1421-1430 (2008) · doi:10.1111/j.1467-8659.2008.01282.x
[29]	Li, H.; Adams, B.; Guibas, LJ; Pauly, M., Robust single-view geometry and motion reconstruction, ACM Trans. Graph., 28, 175 (2009) · doi:10.1145/1618452.1618521
[30]	Li, C., Simon, T., Saragih, J.M., Póczos, B., Sheikh, Y.: LBS autoencoder: self-supervised fitting of articulated meshes to point clouds. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11967-11976 (2019)
[31]	Lipman, Y.; Funkhouser, TA, Möbius voting for surface correspondence, ACM Trans. Graph., 28, 1-12 (2009) · doi:10.1145/1531326.1531378
[32]	Litany, O., Remez, T., Rodolà, E., Bronstein, A.M., Bronstein, M.M.: Deep functional maps: structured prediction for dense shape correspondence. In: IEEE International Conference on Computer Vision (ICCV) (2017)
[33]	Liu, P., Lyu, M.R., King, I., Xu, J.: Selflow: self-supervised learning of optical flow. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4571-4580 (2019)
[34]	Liu, X., Qi, C.R., Guibas, L.J.: Flownet3d: learning scene flow in 3d point clouds. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 529-537 (2019)
[35]	Ma, J.; Zhao, J.; Yuille, AL, Non-rigid point set registration by preserving global and local structures, IEEE Trans. Image Process, 25, 1, 53-64 (2016) · Zbl 1408.94463 · doi:10.1109/TIP.2015.2467217
[36]	Mémoli, F.; Sapiro, G., A theoretical and computational framework for isometry invariant recognition of point cloud data, Found. Comput. Math., 45, 313-347 (2005) · Zbl 1101.53022 · doi:10.1007/s10208-004-0145-y
[37]	Mittal, H., Okorn, B., Held, D.: Just go with the flow: self-supervised scene flow estimation. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11174-11182 (2020)
[38]	Myronenko, A.; Song, XB, Point set registration: coherent point drift, IEEE Trans. Pattern Anal. Mach. Intell., 32, 12, 2262-2275 (2010) · doi:10.1109/TPAMI.2010.46
[39]	Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohli, P., Shotton, J., Hodges, S., Fitzgibbon, A.W.: Kinectfusion: real-time dense surface mapping and tracking. In: IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 127-136 (2011)
[40]	Newcombe, R.A., Fox, D., Seitz, S.M.: Dynamicfusion: reconstruction and tracking of non-rigid scenes in real-time. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 343-352 (2015)
[41]	Nguyen, A.; Ben-Chen, M.; Welnicka, K.; Ye, Y.; Guibas, LJ, An optimization approach to improving collections of shape maps, Comput. Graph. Forum, 72, 1481-1491 (2011) · doi:10.1111/j.1467-8659.2011.02022.x
[42]	Qian, C., Sun, X., Wei, Y., Tang, X., Sun, J.: Realtime and robust hand tracking from depth. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1106-1113 (2014)
[43]	Ovsjanikov, M.; Ben-Chen, M.; Solomon, J.; Butscher, A.; Guibas, LJ, Functional maps: a flexible representation of maps between shapes, ACM Trans. Graph., 31, 1-11 (2012) · doi:10.1145/2185520.2185526
[44]	Rodolà, E., Möller, M., Cremers, D.: Point-wise map recovery and refinement from functional correspondence. In: International Symposium on Vision Modeling and Visualization, pp. 25-32 (2015)
[45]	Rustamov, R.M.: Laplace-Beltrami eigenfunctions for deformation invariant shape representation. In: Eurographics Symposium on Geometry Processing, pp. 225-233 (2007)
[46]	Shimada, S., Golyanik, V., Tretschk, E., Stricker, D., Theobalt, C.: Dispvoxnets: non-rigid point set alignment with supervised learning proxies. In: International Conference on 3D Vision, pp. 27-36 (2019)
[47]	Smith, L.N., Topin, N.: Super-convergence: very fast training of residual networks using large learning rates. CoRR abs/1708.07120 (2017)
[48]	Sumner, RW; Schmid, J.; Pauly, M., Embedded deformation for shape manipulation, ACM Trans. Graph., 26, 3, 80 (2007) · doi:10.1145/1276377.1276478
[49]	Sun, J.; Ovsjanikov, M.; Guibas, LJ, A concise and provably informative multi-scale signature based on heat diffusion, Comput. Graph. Forum, 324, 1383-1392 (2009) · doi:10.1111/j.1467-8659.2009.01515.x
[50]	Sun, D., Yang, X., Liu, M., Kautz, J.: PWC-net: CNNS for optical flow using pyramid, warping, and cost volume. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8934-8943 (2018)
[51]	Teed, Z., Deng, J.: RAFT: recurrent all-pairs field transforms for optical flow. In: European Conference on Computer Vision (ECCV), pp. 402-419 (2020)
[52]	Teed, Z., Deng, J.: RAFT-3D: scene flow using rigid-motion embeddings. CoRR abs/2012.00726 (2020)
[53]	Tombari, F., Salti, S., di Stefano, L.: Unique signatures of histograms for local surface description. In: European Conference on Computer Vision (ECCV), pp. 356-369 (2010)
[54]	Tombari, F., Salti, S., di Stefano, L.: Unique signatures of histograms for local surface description. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) European Conference on Computer Vision (ECCV). Lecture Notes in Computer Science, vol. 6313, pp. 356-369 (2010)
[55]	Vedula, S.; Baker, S.; Rander, P.; Collins, RT; Kanade, T., Three-dimensional scene flow, IEEE Trans. Pattern Anal. Mach. Intell., 27, 3, 475-480 (2005) · doi:10.1109/TPAMI.2005.63
[56]	Wang, L., Fang, Y.: Coherent point drift networks: unsupervised learning of non-rigid point set registration. CoRR abs/1906.03039 (2019)
[57]	Wang, Y., Solomon, J.: Deep closest point: learning representations for point cloud registration. In: IEEE International Conference on Computer Vision (ICCV), pp. 3522-3531 (2019)
[58]	Wang, L., Chen, J., Li, X., Fang, Y.: Non-rigid point set registration networks. CoRR abs/1904.01428 (2019)
[59]	Wang, Y.; Sun, Y.; Liu, Z.; Sarma, SE; Bronstein, MM; Solomon, JM, Dynamic graph CNN for learning on point clouds, ACM Trans. Graph., 38, 5, 146-114612 (2019) · doi:10.1145/3326362
[60]	Yang, Y.; Ong, SH; Foong, KWC, A robust global and local mixture distance based non-rigid point set registration, Pattern Recognit., 48, 1, 156-173 (2015) · doi:10.1016/j.patcog.2014.06.017
[61]	Yao, Y., Deng, B., Xu, W., Zhang, J.: Quasi-newton solver for robust non-rigid registration. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7597-7606 (2020)
[62]	Yu, T., Zheng, Z., Guo, K., Zhao, J., Dai, Q., Li, H., Pons-Moll, G., Liu, Y.: Doublefusion: real-time capture of human performances with inner body shapes from a single depth sensor. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
[63]	Zeng, Y., Qian, Y., Zhu, Z., Hou, J., Yuan, H., He, Y.: Corrnet3d: unsupervised end-to-end learning of dense correspondence for 3d point clouds. CoRR abs/2012.15638 (2020)