×

Label transfer between images and 3D shapes via local correspondence encoding. (English) Zbl 1508.68400

Summary: We present a generic and accurate method to transfer part-level labels from annotated 3D shapes to images or annotated images to 3D shapes. The label transfer problem is formulated as a Conditional random field (CRF) model, in which a novel local correspondence encoding term encoding the probability of label assignment of each pixel as a label histogram serves as the likelihood of the labels. With the aid of local correspondence encoding, highly accurate part-level label transfer results can be easily achieved. Our method works well for the bi-directional label transfer between 3D shapes and images, suitable to image-guided 3D shape segmentation, 3D shape-guided image segmentation, and has also found various applications. We thoroughly evaluate our method and demonstrate its superiority over the state-of-the-art methods through experiments on benchmark datasets.

MSC:

68U07 Computer science aspects of computer-aided design
68U10 Computing methodologies for image processing
94A08 Image processing (compression, reconstruction, etc.) in information and communication theory
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Aberman, K.; Liao, J.; Shi, M.; Lischinski, D.; Chen, B.; Cohen-Or, D., Neural best-buddies: sparse cross-domain correspondence, ACM Trans. Graph., 37, Article 69 pp. (2018)
[2] Attene, M.; Falcidieno, B.; Spagnuolo, M., Hierarchical mesh segmentation based on fitting primitives, Vis. Comput., 22, 181-193 (2006)
[3] Aubry, M.; Maturana, D.; Efros, A. A.; Russell, B. C.; Sivic, J., Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of cad models, (The IEEE Conference on Computer Vision and Pattern Recognition. The IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2014))
[4] Boykov, Y. Y.; Jolly, M. P., Interactive graph cuts for optimal boundary amp; region segmentation of objects in n-d images, (Proceedings Eighth IEEE International Conference on Computer Vision. Proceedings Eighth IEEE International Conference on Computer Vision, ICCV 2001, vol. 1 (2001)), 105-112
[5] Chang, A. X.; Funkhouser, T.; Guibas, L.; Hanrahan, P.; Huang, Q.; Li, Z.; Savarese, S.; Savva, M.; Song, S.; Su, H.; Xiao, J.; Yi, L.; Yu, F., ShapeNet: An Information-Rich 3D Model Repository (2015), Stanford University — Princeton University — Toyota Technological Institute at Chicago, Technical Report
[6] Chen, X.; Golovinskiy, A.; Funkhouser, T., A benchmark for 3D mesh segmentation, ACM Trans. Graph. (TOG), 28, Article 73 pp. (2009)
[7] Dalal, N.; Triggs, B., Histograms of oriented gradients for human detection, (2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR’05, vol. 1 (2005)), 886-893
[8] Golovinskiy, A.; Funkhouser, T., Randomized cuts for 3D mesh analysis, ACM Trans. Graph., 27, Article 145 pp. (2008)
[9] Gould, S.; Fulton, R.; Koller, D., Decomposing a scene into geometric and semantically consistent regions, (2009 IEEE 12th International Conference on Computer Vision (2009)), 1-8
[10] Guo, K.; Zou, D.; Chen, X., 3D mesh labeling via deep convolutional neural networks, ACM Trans. Graph., 35, Article 3 pp. (2015)
[11] He, K.; Gkioxari, G.; Dollár, P.; Girshick, R., Mask R-CNN, (2017 IEEE International Conference on Computer Vision. 2017 IEEE International Conference on Computer Vision, ICCV (2017), IEEE), 2980-2988
[12] He, X.; Zemel, R. S., Learning hybrid models for image annotation with partially labeled data, (Koller, D.; Schuurmans, D.; Bengio, Y.; Bottou, L., Advances in Neural Information Processing Systems, vol. 21 (2009), Curran Associates, Inc.), 625-632
[13] Hilaga, M.; Shinagawa, Y.; Kohmura, T.; Kunii, T. L., Topology matching for fully automatic similarity estimation of 3D shapes, (ACM Siggraph 2001 (2001), ACM: ACM New York, NY, USA), 203-212
[14] Huang, Q.; Wang, H.; Koltun, V., Single-view reconstruction via joint analysis of image and shape collections, ACM Trans. Graph., 34, Article 87 pp. (2015)
[15] Kalogerakis, E.; Averkiou, M.; Maji, S.; Chaudhuri, S., 3D shape segmentation with projective convolutional networks, (Proc. IEEE Computer Vision and Pattern Recognition. Proc. IEEE Computer Vision and Pattern Recognition, CVPR (2017))
[16] Kalogerakis, E.; Hertzmann, A.; Singh, K., Learning 3D mesh segmentation and labeling, ACM Trans. Graph. (TOG), 29, 102 (2010)
[17] Ladický, L.; Russell, C.; Kohli, P.; Torr, P. H.S., Associative hierarchical CRFs for object class image segmentation, (2009 IEEE 12th International Conference on Computer Vision (2009)), 739-746
[18] Lai, Y. K.; Hu, S. M.; Martin, R. R.; Rosin, P. L., Fast mesh segmentation using random walks, (Proceedings of the 2008 ACM Symposium on Solid and Physical Modeling (2008), ACM: ACM New York, NY, USA), 183-191
[19] Lee, J.; Funkhouser, T. A., Sketch-based search and composition of 3D models, (SBM (2008)), 97-104
[20] Li, Y.; Su, H.; Qi, C. R.; Fish, N.; Cohen-Or, D.; Guibas, L. J., Joint embeddings of shapes and images via cnn image purification, ACM Trans. Graph., 34, Article 234 pp. (2015)
[21] Li, Y.; Sun, J.; Tang, C. K.; Shum, H. Y., Lazy snapping, ACM Trans. Graph., 23, 303-308 (2004)
[22] Liu, C.; Yuen, J.; Torralba, A., Sift flow: dense correspondence across scenes and its applications, IEEE Trans. Pattern Anal. Mach. Intell., 33, 978-994 (2011)
[23] Long, J.; Shelhamer, E.; Darrell, T., Fully convolutional networks for semantic segmentation, (The IEEE Conference on Computer Vision and Pattern Recognition. The IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2015))
[24] Malisiewicz, T.; Gupta, A.; Efros, A. A., Ensemble of exemplar-SVMs for object detection and beyond, (2011 International Conference on Computer Vision (2011)), 89-96
[25] Munoz, D.; Bagnell, J. A.; Hebert, M., Stacked Hierarchical Labeling, 57-70 (2010), Springer: Springer Berlin Heidelberg, Berlin, Heidelberg
[26] Noh, H.; Hong, S.; Han, B., Learning deconvolution network for semantic segmentation, (The IEEE International Conference on Computer Vision. The IEEE International Conference on Computer Vision, ICCV (2015))
[27] Rother, C.; Kolmogorov, V.; Blake, A., “GrabCut”: interactive foreground extraction using iterated graph cuts, ACM Trans. Graph., 23, 309-314 (2004)
[28] Shamir, A., A survey on mesh segmentation techniques, (Computer Graphics Forum (2008), Wiley Online Library), 1539-1556 · Zbl 1162.68769
[29] Shapira, L.; Shamir, A.; Cohen-Or, D., Consistent mesh partitioning and skeletonisation using the shape diameter function, Vis. Comput., 24, 249-259 (2008)
[30] Shu, Z.; Qi, C.; Xin, S.; Hu, C.; Wang, L.; Zhang, Y.; Liu, L., Unsupervised 3D shape segmentation and co-segmentation via deep learning, Geometric Modeling and Processing 2016. Geometric Modeling and Processing 2016, Comput. Aided Geom. Des., 43, 39-52 (2016) · Zbl 1417.65094
[31] Su, H.; Huang, Q.; Mitra, N. J.; Li, Y.; Guibas, L., Estimating image depth using shape collections, ACM Trans. Graph., 33, Article 37 pp. (2014)
[32] Tighe, J.; Lazebnik, S., SuperParsing: Scalable Nonparametric Image Parsing with Superpixels, 352-365 (2010), Springer: Springer Berlin Heidelberg, Berlin, Heidelberg
[33] Torralba, A.; Murphy, K. P.; Freeman, W. T., Sharing visual features for multiclass and multiview object detection, IEEE Trans. Pattern Anal. Mach. Intell., 29, 854-869 (2007)
[34] Wang, Y.; Asafi, S.; van Kaick, O.; Zhang, H.; Cohen-Or, D.; Chen, B., Active co-analysis of a set of shapes, ACM Trans. Graph., 31, Article 165 pp. (2012)
[35] Wang, Y.; Gong, M.; Wang, T.; Cohen-Or, D.; Zhang, H.; Chen, B., Projective analysis for 3D shape segmentation, ACM Trans. Graph., 32, Article 192 pp. (2013)
[36] Wang, T. Y.; Su, H.; Huang, Q.; Huang, J.; Guibas, L.; Mitra, N. J., Unsupervised texture transfer from images to model collections, ACM Trans. Graph., 35, Article 177 pp. (2016)
[37] Yi, L.; Guibas, L.; Hertzmann, A.; Kim, V. G.; Su, H.; Yumer, E., Learning hierarchical shape segmentation and labeling from online repositories, ACM Trans. Graph., 36, Article 70 pp. (2017)
[38] Yi, L.; Kim, V. G.; Ceylan, D.; Shen, I. C.; Yan, M.; Su, H.; Lu, C.; Huang, Q.; Sheffer, A.; Guibas, L., A scalable active framework for region annotation in 3D shape collections, ACM Trans. Graph., 35, Article 210 pp. (2016)
[39] Zhou, T.; Krahenbuhl, P.; Aubry, M.; Huang, Q.; Efros, A. A., Learning dense correspondence via 3D-guided cycle consistency, (The IEEE Conference on Computer Vision and Pattern Recognition. The IEEE Conference on Computer Vision and Pattern Recognition, CVPR (2016))
[40] Zhou, H.; Zheng, J.; Wei, L., Texture aware image segmentation using graph cuts and active contours, Pattern Recognit., 46, 1719-1733 (2013) · Zbl 1264.68209
[41] Zhu, J.; Guo, Y.; Ma, H., A data-driven approach for furniture and indoor scene colorization, IEEE Trans. Vis. Comput. Graph., 24, 2473-2486 (2018)
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.