×

A nonlocal feature-driven exemplar-based approach for image inpainting. (English) Zbl 1497.68540

Summary: We present a nonlocal variational image completion technique which admits simultaneous inpainting of multiple structures and textures in a unified framework. The recovery of geometric structures is achieved by using general convolution operators as a measure of behavior within an image. These are combined with a nonlocal exemplar-based approach to exploit the self-similarity of an image in the selected feature domains and to ensure the inpainting of textures. We also introduce an anisotropic patch distance metric to allow for better control of the feature selection within an image and present a nonlocal energy functional based on this metric. Finally, we derive an optimization algorithm for the proposed variational model and examine its validity experimentally with various test images.

MSC:

68U10 Computing methodologies for image processing
65D18 Numerical aspects of computer graphics, image analysis, and computational geometry
65K10 Numerical optimization and variational techniques

Software:

PatchMatch; FAIR.m
PDFBibTeX XMLCite
Full Text: DOI arXiv

References:

[1] P. Arias, G. Facciolo, V. Caselles, and G. Sapiro, A variational framework for exemplar-based image inpainting, Int. J. Comput. Vis., 93 (2011), pp. 319-347, https://doi.org/10.1007/s11263-010-0418-7. · Zbl 1235.94015
[2] J. Aujol, S. Ladjal, and S. Masnou, Exemplar-based inpainting from a variational point of view, SIAM J. Math. Anal., 42 (2010), pp. 1246-1285, https://doi.org/10.1137/080743883. · Zbl 1210.49002
[3] C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, and J. Verdera, Filling-in by joint interpolation of vector fields and gray levels, IEEE Trans. Image Process., 10 (2001), pp. 1200-1211, https://doi.org/10.1109/83.935036. · Zbl 1037.68771
[4] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman, PatchMatch: A randomized correspondence algorithm for structural image editing, ACM Trans. Graphics, 28 (2009), https://doi.org/10.1145/1531326.1531330.
[5] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, Image inpainting, in Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, New York, ACM Press/Addison-Wesley, 2000, pp. 417-424, https://doi.org/10.1145/344779.344972.
[6] A. Bertozzi, S. Esedoḡlu, and A. Gillette, Analysis of a two-scale Cahn-Hilliard model for binary image inpainting, Multiscale Model. Simul., 6 (2007), pp. 913-936, https://doi.org/10.1137/060660631. · Zbl 1149.35309
[7] R. Bornard, E. Lecan, L. Laborelli, and J.-H. Chenot, Missing data correction in still images and image sequences, in Proceedings of the 10th ACM International Conference on Multimedia, New York, Association for Computing Machinery, 2002, pp. 355-361, https://doi.org/10.1145/641007.641084.
[8] A. Buades, B. Coll, and J.-M. Morel, A review of image denoising algorithms, with a new one, Multiscale Model. Simul., 4 (2005), pp. 490-530, https://doi.org/10.1137/040616024. · Zbl 1108.94004
[9] M. Burger, L. He, and C.-B. Schönlieb, Cahn-Hilliard inpainting and a generalization for grayvalue images, SIAM J. Imaging Sci., 2 (2009), pp. 1129-1167, https://doi.org/10.1137/080728548. · Zbl 1180.49007
[10] F. Cao, Y. Gousseau, S. Masnou, and P. Pérez, Geometrically guided exemplar-based inpainting, SIAM J. Imaging Sci., 4 (2011), pp. 1143-1179, https://doi.org/10.1137/110823572. · Zbl 1235.94017
[11] T. F. Chan and J. J. Shen, Image Processing and Analysis: Variational, PDE, Wavelet, and Stochastic Methods, SIAM, Philadelphia, 2005. · Zbl 1095.68127
[12] A. Criminisi, P. Perez, and K. Toyama, Object removal by exemplar-based inpainting, in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 2, 2003, https://doi.org/10.1109/CVPR.2003.1211538.
[13] A. Criminisi, P. Perez, and K. Toyama, Region filling and object removal by exemplar-based image inpainting, IEEE Trans. Image Process., 13 (2004), pp. 1200-1212, https://doi.org/10.1109/TIP.2004.833105.
[14] L. Demanent, B. Song, and T. F. Chan, Image Inpainting by Correspondence Maps: A Deterministic Approach, Tech. report, UCLA, 2003.
[15] I. Drori, D. Cohen-Or, and H. Yeshurun, Fragment-based image completion, in ACM SIGGRAPH 2003 Papers, New York, Association for Computing Machinery, 2003, pp. 303-312, https://doi.org/10.1145/1201775.882267.
[16] Q. Du, M. Gunzburger, R. B. Lehoucq, and K. Zhou, A nonlocal vector calculus, nonlocal volume-constrained problems, and nonlocal balance laws, Math. Models Methods Appl. Sci., 23 (2013), pp. 493-540, https://doi.org/10.1142/S0218202512500546. · Zbl 1266.26020
[17] V. Fedorov, P. Arias, G. Facciolo, and C. Ballester, Affine invariant self-similarity for exemplar-based inpainting, in VISAPP, 2016, https://hal-enpc.archives-ouvertes.fr/hal-01240846.
[18] V. Fedorov, G. Facciolo, and P. Arias, Variational framework for non-local inpainting, IPOL J. Image Process. Online, 5 (2015), pp. 362-386, https://doi.org/10.5201/ipol.2015.136.
[19] G. Gilboa and S. Osher, Nonlocal linear image regularization and supervised segmentation, Multiscale Model. Simul., 6 (2007), pp. 595-630, https://doi.org/10.1137/060669358. · Zbl 1140.68517
[20] G. Gilboa and S. Osher, Nonlocal operators with applications to image processing, Multiscale Model. Simul., 7 (2008), pp. 1005-1028, https://doi.org/10.1137/070698592. · Zbl 1181.35006
[21] S. Iizuka, E. Simo-Serra, and H. Ishikawa, Globally and locally consistent image completion, ACM Trans. Graphics, 36 (2017), pp. 1-14, https://doi.org/10.1145/3072959.3073659.
[22] E. T. Jaynes, Information theory and statistical mechanics, Phys. Rev., 106 (1957), pp. 620-630, https://doi.org/10.1103/PhysRev.106.620. · Zbl 0084.43701
[23] N. Kawai, T. Sato, and N. Yokoya, Image inpainting considering brightness change and spatial locality of textures and its evaluation, in Advances in Image and Video Technology, T. Wada, F. Huang, and S. Lin, eds., Springer, Berlin, 2009, pp. 271-282, https://doi.org/10.1007/978-3-540-92957-4_24.
[24] O. Le Meur, M. Ebdelli, and C. Guillemot, Hierarchical super-resolution-based inpainting, IEEE Trans. Image Process., 22 (2013), pp. 3779-3790, https://doi.org/10.1109/TIP.2013.2261308. · Zbl 1373.94223
[25] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro, Image inpainting for irregular holes using partial convolutions, in Proceedings of the European Conference on Computer Vision, 2018.
[26] Y. Liu and V. Caselles, Exemplar-based image inpainting using multiscale graph cuts, IEEE Trans. Image Process., 22 (2012), pp. 1699-1711, https://doi.org/10.1109/TIP.2012.2218828. · Zbl 1373.94263
[27] S. Masnou and J.-M. Morel, Level lines based disocclusion, in Proceedings of the International Conference on Image Processing, IEEE, 1998, pp. 259-263.
[28] J. Modersitzki, FAIR: Flexible Algorithms for Image Registration, Fundam. Algorithms 6, SIAM, Philadelphia, 2009. · Zbl 1183.68695
[29] D. Mumford and J. Shah, Optimal approximations by piecewise smooth functions and associated variational problems, Comm. Pure Appl. Math., 42 (1989), pp. 577-685, https://doi.org/10.1002/cpa.3160420503. · Zbl 0691.49036
[30] K. Nazeri, E. Ng, T. Joseph, F. Qureshi, and M. Ebrahimi, EdgeConnect: Structure guided image inpainting using edge prediction, in IEEE International Conference on Computer Vision Workshops, 2019.
[31] A. Newson, A. Almansa, M. Fradet, Y. Gousseau, and P. Pérez, Video inpainting of complex scenes, SIAM J. Imaging Sci., 7 (2014), pp. 1993-2019, https://doi.org/10.1137/140954933. · Zbl 1320.68204
[32] A. Newson, A. Almansa, Y. Gousseau, and P. Pérez, Non-local patch-based image inpainting, IPOL J. Image Process. Online, 7 (2017), pp. 373-385, https://doi.org/10.5201/ipol.2017.189.
[33] E. D. Nezza, G. Palatucci, and E. Valdinoci, Hitchhiker’s guide to the fractional Sobolev spaces, Bull. Sci. Math., 136 (2012), pp. 521-573, https://doi.org/10.1016/j.bulsci.2011.12.004. · Zbl 1252.46023
[34] S. Osher, Z. Shi, and W. Zhu, Low dimensional manifold model for image processing, SIAM J. Imaging Sci., 10 (2017), pp. 1669-1690, https://doi.org/10.1137/16M1058686. · Zbl 1401.49042
[35] K. Papafitsoros, C. B. Schoenlieb, and B. Sengul, Combined first and second order total variation inpainting using split Bregman, IPOL J. Image Process. Online, 3 (2013), pp. 112-136, https://doi.org/10.5201/ipol.2013.40.
[36] K. Papafitsoros and C.-B. Schönlieb, A combined first and second order variational approach for image reconstruction, J. Math. Imaging Vision, 48 (2014), pp. 308-338, https://doi.org/10.1007/s10851-013-0445-4. · Zbl 1362.94009
[37] G. Peyré, S. Bougleux, and L. Cohen, Non-local regularization of inverse problems, Inverse Problems Imaging, 5 (2011), pp. 511-530, https://doi.org/10.3934/ipi.2011.5.511. · Zbl 1223.68116
[38] L. I. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based noise removal algorithms, Phys. D, 60 (1992), pp. 259-268, https://doi.org/10.1016/0167-2789(92)90242-F. · Zbl 0780.49028
[39] J. Shen and T. F. Chan, Mathematical models for local nontexture inpaintings, SIAM J. Appl. Math., 62 (2002), pp. 1019-1043, https://doi.org/10.1137/S0036139900368844. · Zbl 1050.68157
[40] J. Shen, S. H. Kang, and T. F. Chan, Euler’s elastica and curvature-based inpainting, SIAM J. Appl. Math., 63 (2003), pp. 564-592, https://doi.org/10.1137/S0036139901390088. · Zbl 1028.68185
[41] Y. Wang, X. Tao, X. Qi, X. Shen, and J. Jia, Image inpainting via generative multi-column convolutional neural networks, in Advances in Neural Information Processing Systems, 2018, pp. 331-340.
[42] Y. Wexler, E. Shechtman, and M. Irani, Space-time completion of video, IEEE Trans. Pattern Anal. Mach. Intell., 29 (2007), pp. 463-476, https://doi.org/10.1109/TPAMI.2007.60.
[43] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, Generative image inpainting with contextual attention, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.