Self-attention implicit function networks for 3D dental data completion. (English) Zbl 07424069

Summary: While complete dental models are crucial for digital dentistry, current technologies mostly focus on the 3D dental crown but overlook the dental gum that is important for applications in orthodontics and prosthodontics. To reconstruct the complete dental models with visually realistic geometry from the given crown data, we propose to combine the implicit function representation with the self-attention mechanism. Recent studies have shown that the implicit function is an effective 3D representation for shape completion. However, existing methods fail in dealing with dental models with complex shapes and details, because the convolution and linear operations adopted in their networks are inefficient for modeling long-range dependencies or hard to maintain detailed geometry of the shapes. Therefore, we propose to introduce self-attention to the implicit function network for the first time and use it to effectively capture non-local features at different levels. Extensive ablation studies were conducted to validate the efficiency of our method. Quantitative and qualitative comparisons demonstrate that the feature extracted by our network is more expressive and thus leads to better dental model completion and reconstruction results.


65Dxx Numerical approximation and computational geometry (primarily algorithms)
Full Text: DOI


[1] Bagautdinov, T.; Wu, C.; Saragih, J.; Fua, P.; Sheikh, Y., Modeling facial geometry using compositional vaes, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)), 3877-3886
[2] Baque, P.; Remelli, E.; Fleuret, F.; Fua, P., Geodesic convolutional shape optimization (2018), arXiv preprint
[3] Ben-Hamu, H.; Maron, H.; Kezurer, I.; Avineri, G.; Lipman, Y., Multi-chart generative surface modeling, ACM Trans. Graph., 37, 1-15 (2018)
[4] Carr, J. C.; Beatson, R. K.; Cherrie, J. B.; Mitchell, T. J.; Fright, W. R.; McCallum, B. C.; Evans, T. R., Reconstruction and representation of 3d objects with radial basis functions, (Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (2001)), 67-76
[5] Chang, A. X.; Funkhouser, T.; Guibas, L.; Hanrahan, P.; Huang, Q.; Li, Z.; Savarese, S.; Savva, M.; Song, S.; Su, H., Shapenet: an information-rich 3d model repository (2015), arXiv preprint
[6] Chen, Z.; Zhang, H., Learning implicit fields for generative shape modeling, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019)), 5939-5948
[7] Cheng, J.; Dong, L.; Lapata, M., Long short-term memory-networks for machine reading (2016), arXiv preprint
[8] Chibane, J.; Alldieck, T.; Pons-Moll, G., Implicit functions in feature space for 3d shape reconstruction and completion, (Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2020)), 6970-6981
[9] Choy, C. B.; Xu, D.; Gwak, J.; Chen, K.; Savarese, S., 3d-r2n2: a unified approach for single and multi-view 3d object reconstruction, (European Conference on Computer Vision (2016), Springer), 628-644
[10] Cignoni, P.; Callieri, M.; Corsini, M.; Dellepiane, M.; Ganovelli, F.; Ranzuglia, G., Meshlab: an open-source mesh processing tool, (Eurographics Italian Chapter Conference, Salerno (2008)), 129-136
[11] Cui, Z.; Li, C.; Wang, W., Toothnet: automatic tooth instance segmentation and identification from cone beam ct images, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019)), 6368-6377
[12] Gregor, K.; Danihelka, I.; Graves, A.; Rezende, D. J.; Wierstra, D., Draw: a recurrent neural network for image generation (2015), arXiv preprint
[13] Groueix, T.; Fisher, M.; Kim, V. G.; Russell, B. C.; Aubry, M., A papier-mâché approach to learning 3d surface generation, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)), 216-224
[14] Guo, K.; Zou, D.; Chen, X., 3d mesh labeling via deep convolutional neural networks, ACM Trans. Graph., 35, 1-12 (2015)
[15] Han, X.; Laga, H.; Bennamoun, M., Image-based 3d object reconstruction: state-of-the-art and trends in the deep learning era, (IEEE Transactions on Pattern Analysis and Machine Intelligence (2019))
[16] Häne, C.; Tulsiani, S.; Malik, J., Hierarchical surface prediction for 3d object reconstruction, (2017 International Conference on 3D Vision (3DV), IEEE (2017)), 412-420
[17] Le, H. S.; Tuan, T. M.; Fujita, H.; Dey, N.; Ashour, A. S.; Ngoc, V. T.N.; Anh, L. Q.; Chu, D. T., Dental diagnosis from x-ray images: an expert system based on fuzzy computing, Biomed. Signal Process. Control, 39, 64-73 (2017)
[18] Lian, C.; Wang, L.; Wu, T. H.; Wang, F.; Yap, P. T.; Ko, C. C.; Shen, D., Deep multi-scale mesh feature learning for automated labeling of raw dental surfaces from 3d intraoral scanners, IEEE Trans. Med. Imaging, 39, 2440-2450 (2020)
[19] Litany, O.; Bronstein, A.; Bronstein, M.; Makadia, A., Deformable shape completion with graph convolutional autoencoders, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)), 1886-1895
[20] Liu, Q. M.; Jia, R. S.; Zhao, C. Y.; Liu, X. Y.; Sun, H. M.; Zhang, X. L., Face super-resolution reconstruction based on self-attention residual network, IEEE Access, 8, 4110-4121 (2019)
[21] Liu, S.; Giles, L.; Ororbia, A., Learning a hierarchical latent-variable model of 3d shapes, (2018 International Conference on 3D Vision (3DV), IEEE (2018)), 542-551
[22] Lorensen, W. E.; Cline, H. E., Marching cubes: a high resolution 3d surface construction algorithm, ACM SIGGRAPH Comput. Graph., 21, 163-169 (1987)
[23] Maturana, D.; Scherer, S., Voxnet: a 3d convolutional neural network for real-time object recognition, (2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE (2015)), 922-928
[24] Mescheder, L.; Oechsle, M.; Niemeyer, M.; Nowozin, S.; Geiger, A., Occupancy networks: learning 3d reconstruction in function space, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019)), 4460-4470
[25] Na’Am, J.; Harlan, J.; Madenda, S.; Santony, J.; Suharinto, C., Detection of proximal caries at the molar teeth using edge enhancement algorithm, Int. J. Electr. Comput. Eng., 8, 3259 (2018)
[26] Park, J. J.; Florence, P.; Straub, J.; Newcombe, R.; Lovegrove, S., Deepsdf: learning continuous signed distance functions for shape representation, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2019)), 165-174
[27] Parmar, N.; Vaswani, A.; Uszkoreit, J.; Kaiser, Ł.; Shazeer, N.; Ku, A.; Tran, D., Image transformer (2018), arXiv preprint
[28] Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L., Pytorch: an imperative style, high-performance deep learning library, (Advances in Neural Information Processing Systems (2019)), 8026-8037
[29] Peng, S.; Niemeyer, M.; Mescheder, L.; Pollefeys, M.; Geiger, A., Convolutional occupancy networks (2020), arXiv preprint
[30] Pishchulin, L.; Wuhrer, S.; Helten, T.; Theobalt, C.; Schiele, B., Building statistical shape spaces for 3d human modeling, Pattern Recognit. (2017)
[31] Qi, C. R.; Su, H.; Mo, K.; Guibas, L. J., Pointnet: deep learning on point sets for 3d classification and segmentation, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)), 652-660
[32] Qi, C. R.; Yi, L.; Su, H.; Guibas, L. J., Pointnet++: deep hierarchical feature learning on point sets in a metric space, (Advances in Neural Information Processing Systems (2017)), 5099-5108
[33] Riegler, G.; Osman Ulusoy, A.; Geiger, A., Octnet: learning deep 3d representations at high resolutions, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)), 3577-3586
[34] Rubner, Y.; Tomasi, C.; Guibas, L. J., The Earth mover’s distance as a metric for image retrieval, Int. J. Comput. Vis., 40, 99-121 (2000) · Zbl 1012.68705
[35] Shen, C.; O’Brien, J. F.; Shewchuk, J. R., Interpolating and approximating implicit surfaces from polygon soup, (ACM SIGGRAPH 2004 Papers (2004)), 896-904
[36] Sinha, A.; Bai, J.; Ramani, K., Deep learning 3d shape surfaces using geometry images, (European Conference on Computer Vision (2016), Springer), 223-240
[37] Tatarchenko, M.; Dosovitskiy, A.; Brox, T., Octree generating networks: efficient convolutional architectures for high-resolution 3d outputs, (Proceedings of the IEEE International Conference on Computer Vision (2017)), 2088-2096
[38] Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; Polosukhin, I., Attention is all you need, (Advances in Neural Information Processing Systems (2017)), 5998-6008
[39] Wang, X.; Girshick, R.; Gupta, A.; He, K., Non-local neural networks, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)), 7794-7803
[40] Wang, Y.; Sun, Y.; Liu, Z.; Sarma, S. E.; Bronstein, M. M.; Solomon, J. M., Dynamic graph cnn for learning on point clouds, ACM Trans. Graph., 38, 1-12 (2019)
[41] Wei, G.; Cui, Z.; Liu, Y.; Chen, N.; Chen, R.; Li, G.; Wang, W., Tanet: towards fully automatic tooth arrangement, (European Conference on Computer Vision (2020), Springer), 481-497
[42] Wu, Z.; Song, S.; Khosla, A.; Yu, F.; Zhang, L.; Tang, X.; Xiao, J., 3d shapenets: a deep representation for volumetric shapes, (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015)), 1912-1920
[43] Xu, K.; Ba, J.; Kiros, R.; Cho, K.; Courville, A.; Salakhudinov, R.; Zemel, R.; Bengio, Y., Show, attend and tell: neural image caption generation with visual attention, (International Conference on Machine Learning (2015)), 2048-2057
[44] Xu, X.; Liu, C.; Zheng, Y., 3d tooth segmentation and labeling using deep convolutional neural networks, IEEE Trans. Vis. Comput. Graph., 25, 2336-2348 (2018)
[45] Yang, L.; Shi, Z.; Wu, Y.; Li, X.; Zhou, K.; Fu, H.; Zheng, Y., Iorthopredictor: model-guided deep prediction of teeth alignment, ACM Trans. Graph., 39, 1-15 (2020)
[46] Yuan, W.; Khot, T.; Held, D.; Mertz, C.; Hebert, M., Pcn: point completion network, (2018 International Conference on 3D Vision (3DV), IEEE (2018)), 728-737
[47] Zhang, H.; Goodfellow, I.; Metaxas, D.; Odena, A., Self-attention generative adversarial networks, (International Conference on Machine Learning, PMLR (2019)), 7354-7363
[48] Zhang, J.; Li, C.; Song, Q.; Gao, L.; Lai, Y. K., Automatic 3d tooth segmentation using convolutional neural networks in harmonic parameter space, Graph. Models, 101071 (2020)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.