A texture synthesis model based on semi-discrete optimal transport in patch space.

*(English)*Zbl 1423.62121##### MSC:

62M40 | Random fields; image analysis |

65D18 | Numerical aspects of computer graphics, image analysis, and computational geometry |

65K10 | Numerical optimization and variational techniques |

68U10 | Computing methodologies for image processing |

##### Keywords:

optimal transport; texture synthesis; patch distribution; nearest neighbor assignments; Gaussian random fields##### Software:

Steerable pyramid
PDF
BibTeX
XML
Cite

\textit{B. Galerne} et al., SIAM J. Imaging Sci. 11, No. 4, 2456--2493 (2018; Zbl 1423.62121)

Full Text:
DOI

##### References:

[1] | C. Aguerrebere, Y. Gousseau, and G. Tartavel, Exemplar-based texture synthesis: The Efros-Leung algorithm, IPOL J. Image Process. Online, 2013 (2013), pp. 213–231, . |

[2] | F. Aurenhammer, F. Hoffmann, and B. Aronov, Minkowski-type theorems and least-squares clustering, Algorithmica, 20 (1998), pp. 61–76. · Zbl 0895.68135 |

[3] | G. Berger and R. Memisevic, Incorporating long-range consistency in CNN-based texture generation, in Proceedings of the 5th International Conference on Learning Representations, 2017. |

[4] | J. Bruna and S. Mallat, Audio Texture Synthesis with Scattering Moments, preprint, arXiv:1311.0407, 2013. |

[5] | R. Chellappa, Two-dimensional discrete Gaussian Markov random field models for image processing, in Progress in Pattern Recognition 2, Elsevier, Amsterdam, 1985, pp. 79–112. · Zbl 0634.68092 |

[6] | R. Chellappa and S. Chatterjee, Classification of textures using Gaussian Markov random fields, IEEE Trans. Acoust. Speech Signal Process., 33 (1985), pp. 959–963. |

[7] | R. Chellappa and A. Jain, Markov Random Fields: Theory and Application, Academic Press, Boston, 1992. |

[8] | R. Chellappa and R. Kashyap, Texture synthesis using 2D noncausal autoregressive models, IEEE Trans. Acoust. Speech Signal Process., 33 (1985), pp. 194–203. |

[9] | G. Cross and A. Jain, Markov random field texture models, IEEE Trans. Pattern Anal. Mach. Intell., 5 (1983), pp. 25–39. |

[10] | H. Derin and H. Elliott, Modeling and segmentation of noisy and textured images using Gibbs random fields, IEEE Trans. Pattern Anal. Mach. Intell., 9 (1987), pp. 39–55. |

[11] | Y. Dong, S. Lefebvre, X. Tong, and G. Drettakis, Lazy solid texture synthesis, Comput. Graph. Forum, 27 (2008), pp. 1165–1174. |

[12] | A. Efros and W. Freeman, Image quilting for texture synthesis and transfer, ACM Trans. Graph, (2001), pp. 341–346. |

[13] | A. A. Efros and T. K. Leung, Texture synthesis by non-parametric sampling, in Proceedings of the IEEE ICCV, 1999, IEEE Computer Society, Los Alamitos, CA, pp. 1033–1038. |

[14] | B. Galerne, Y. Gousseau, and J.-M. Morel, Random phase textures: Theory and synthesis, IEEE Trans. Image Process., 20 (2011), pp. 257–267. · Zbl 1372.94086 |

[15] | B. Galerne, A. Leclaire, and L. Moisan, A texton for fast and flexible Gaussian texture synthesis, in Proceedings of EUSIPCO, IEEE, Piscataway, NJ, 2014, pp. 1686–1690. |

[16] | B. Galerne, A. Leclaire, and L. Moisan, Texton noise, Comput. Graph. Forum, 36 (2017), pp. 205–218. |

[17] | B. Galerne, A. Leclaire, and J. Rabin, Semi-discrete optimal transport in patch space for enriching Gaussian textures, in Proceedings of Geometric Science of Information (GSI), Springer, Cham, Switzerland, 2017, pp. 1686–1690. · Zbl 1426.60003 |

[18] | L. Gatys, A. S. Ecker, and M. Bethge, Texture synthesis using convolutional neural networks, in Advances in Neural Information Processing Systems, Curran, Red Hook, NY, 2015, pp. 262–270. |

[19] | S. Geman and C. Graffigne, Markov random field image models and their applications to computer vision, in Proceedings of the International Congress of Mathematicians, Vol. 1, AMS, Providence, RI, 1986, pp. 1496–1517. · Zbl 0665.68067 |

[20] | A. Genevay, M. Cuturi, G. Peyré, and F. Bach, Stochastic optimization for large-scale optimal transport, in Advances in Neural Information Processing Systems, Curran, Red Hook, NY, 2016, pp. 3432–3440. |

[21] | J. Gutierrez, B. Galerne, J. Rabin, and T. Hurtut, Optimal patch assignment for statistically constrained texture synthesis, in Proceedings of SSVM, Springer, Cham, Switzerland, 2017. |

[22] | C. Han, E. Risser, R. Ramamoorthi, and E. Grinspun, Multiscale texture synthesis, ACM Trans. Graph., 27 (2008), 51. |

[23] | J. Han, K. Zhou, L.-Y. Wei, M. Gong, H. Bao, X. Zhang, and B. Guo, Fast example-based surface texture synthesis via discrete optimization, Vis. Comput., 22 (2006), pp. 918–925. |

[24] | D. J. Heeger and J. R. Bergen, Pyramid-based texture analysis/synthesis, in Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, 1995, pp. 229–238. |

[25] | B. Julesz, Visual pattern discrimination, IRE Trans. Inform. Theory, 8 (1962), pp. 84–92. |

[26] | B. Julesz, Textons, the elements of texture perception, and their interactions, Nature, 290 (1981), p. 91–97. |

[27] | A. Kaspar, B. Neubert, D. Lischinski, M. Pauly, and J. Kopf, Self tuning texture optimization, Comput. Graph. Forum, 34 (2015), pp. 349–359. |

[28] | J. Kitagawa, Q. Mérigot, and B. Thibert, A Newton algorithm for semi-discrete optimal transport, J. Eur. Math Soc. (JEMS), (2017). |

[29] | J. Kopf, C. Fu, D. Cohen-Or, O. Deussen, D. Lischinski, and T. Wong, Solid texture synthesis from 2D exemplars, ACM Trans. Graph., 26 (2007), 2. |

[30] | V. Kwatra, I. Essa, A. Bobick, and N. Kwatra, Texture optimization for example-based synthesis, ACM Trans. Graph., 24 (2005), pp. 795–802. |

[31] | V. Kwatra, A. Schödl, I. Essa, G. Turk, and A. Bobick, Graphcut textures: Image and video synthesis using graph cuts, ACM Trans. Graph., 22 (2003), pp. 277–286. |

[32] | C. Lantuéjoul, Geostatistical Simulation: Models and Algorithms, Springer, Berlin, 2002. · Zbl 0990.86007 |

[33] | S. Lefebvre and H. Hoppe, Parallel controllable texture synthesis, ACM Trans. Graph., 24 (2005), pp. 777–786. |

[34] | E. Levina and P. Bickel, Texture synthesis and nonparametric resampling of random fields, Ann. Statist., 34 (2006), pp. 1751–1773. · Zbl 1246.62194 |

[35] | B. Lévy, A numerical algorithm for L2 semi-discrete optimal transport in \(3\)D, ESAIM Math. Model. Numer. Anal., 49 (2015), pp. 1693–1715. · Zbl 1331.49037 |

[36] | J. Lewis, Texture synthesis for digital painting, in Proceedings of the Conference on Computer Graphics and Interactive Techniques, SIGGRAPH, ACM, New York, 1984, pp. 245–252. |

[37] | J. Lewis, Algorithms for solid noise synthesis, in Proceedings of the Conference on Computer Graphics and Interactive Techniques, SIGGRAPH, ACM, New York, 1989, pp. 263–270. |

[38] | C. Li and M. Wand, Combining Markov random fields and convolutional neural networks for image synthesis, in Proceedings of the IEEE CVPR, 2016, pp. 2479–2486. |

[39] | M. Li, C.and Wand, Precomputed real-time texture synthesis with Markovian generative adversarial networks, in European Conference on Computer Vision, Springer, Cham, Switzerland, 2016, pp. 702–716. |

[40] | Y. Li, C. Fang, J. Yang, Z. Wang, X. Lu, and M. Yang, Universal style transfer via feature transforms, in Proceedings of NIPS, Curran, Red Hook, NY, 2017, pp. 386–396. |

[41] | L. Liang, C. Liu, Y.-Q. Xu, B. Guo, and H.-Y. Shum, Real-time texture synthesis by patch-based sampling, ACM Trans. Graph., 20 (2001), pp. 127–150. |

[42] | A. Lippman, Maximum Entropy Method for Expert System Construction, Ph.D. thesis, Brown University, Providence, RI, 1986. |

[43] | G. Liu, Y. Gousseau, and G. Xia, Texture synthesis through convolutional neural networks and spectrum constraints, in International Conference on Pattern Recognition (ICPR), IEEE, Piscataway, NJ, 2016, pp. 3234–3239. |

[44] | Y. Lu, S.-C. Zhu, and Y. N. Wu, Learning frame models using CNN filters, in 30th AAAI Conference on Artificial Intelligence, AAAI, Palo Alto, CA, 2016, pp. 1902–1910. |

[45] | G. McLachlan and T. Krishnan, The EM Algorithm and Extensions, Vol. 382, Wiley, New York, 2007. · Zbl 1165.62019 |

[46] | Q. Mérigot, A multiscale approach to optimal transport, Comput. Graph. Forum, 30 (2011), pp. 1583–1592. · Zbl 1431.76040 |

[47] | E. Moulines and F. Bach, Non-asymptotic analysis of stochastic approximation algorithms for machine learning, in Advances in Neural Information Processing Systems, Curran, Red Hook, NY, 2011, pp. 451–459. |

[48] | D. Mumford and A. Desolneux, Pattern Theory: The Stochastic Analysis of Real-World Signals, A K Peters, Natick, MA, 2010. · Zbl 1210.94002 |

[49] | R. Paget and I. D. Longstaff, Texture synthesis via a noncausal nonparametric multiscale Markov random field, IEEE Trans. Image Process., 7 (1998), pp. 925–931. |

[50] | G. Peyré, Sparse modeling of textures, J. Math. Imaging Vision, 34 (2009), pp. 17–31. |

[51] | K. Popat and R. Picard, Novel cluster-based probability model for texture synthesis, classification, and compression, in Visual Communications and Image Processing, SPIE Proc. 2094, SPIE, Bellingham, WA, 1993, pp. 756–769. |

[52] | J. Portilla and E. Simoncelli, Representation and Synthesis of Visual Texture, . · Zbl 1012.68698 |

[53] | J. Portilla and E. Simoncelli, A parametric texture model based on joint statistics of complex wavelet coefficients, Int. J. Comput. Vis., 40 (2000), pp. 49–70. · Zbl 1012.68698 |

[54] | L. Raad, A. Davy, A. Desolneux, and J.-M. Morel, A survey of exemplar-based texture synthesis, Ann. Math. Sci. Appl., 3 (2018), pp. 89–148. · Zbl 1425.94022 |

[55] | L. Raad, A. Desolneux, and J. Morel, A conditional multiscale locally Gaussian texture synthesis algorithm, J. Math. Imaging Vision, 56 (2016), pp. 260–279. · Zbl 1409.94504 |

[56] | J. Rabin, J. Delon, and Y. Gousseau, A statistical approach to the matching of local features, SIAM J. Imaging Sci., 2 (2009), pp. 931–958. · Zbl 1175.62069 |

[57] | J. Rabin, J. Delon, and Y. Gousseau, Removing artefacts from color and contrast modifications, IEEE Trans. Image Process., 20 (2011), pp. 3073–3085. · Zbl 1372.94215 |

[58] | J. Rabin, G. Peyré, J. Delon, and M. Bernot, Wasserstein barycenter and its application to texture mixing, in Proceedings of SSVM, Springer, Berlin, 2012, pp. 435–446. |

[59] | F. Santambrogio, Optimal transport for applied mathematicians, Birkhäuser, New York, 2015. · Zbl 1401.49002 |

[60] | O. Sendik and D. Cohen-Or, Deep correlations for texture synthesis, ACM Trans. Graph., 36 (2017), 161. |

[61] | D. Simakov, Y. Caspi, E. Shechtman, and M. Irani, Summarizing visual data using bidirectional similarity, in Proceedings of the IEEE CVPR, IEEE, Piscataway, NJ, 2008. |

[62] | K. Simonyan and A. Zisserman, Very deep convolutional networks for large-scale image recognition, preprint, arXiv:1409.1556, 2014. |

[63] | S. Tabti, Modélisation des Images par Patchs pour leur Restauration et leur Interprétation. Applications à l’Imagerie SAR., Ph.D. thesis, Télécom ParisTech, Paris, 2016. |

[64] | G. Tartavel, Y. Gousseau, and G. Peyré, Variational texture synthesis with sparsity and spectrum constraints, J. Math. Imaging Vision, 52 (2015), pp. 124–144. · Zbl 1332.62371 |

[65] | G. Tartavel, G. Peyré, and Y. Gousseau, Wasserstein loss for image synthesis and restoration, SIAM J. Imaging Sci., 9 (2016), pp. 1726–1755. · Zbl 1358.90102 |

[66] | D. Ulyanov, V. Lebedev, A. Vedaldi, and V. Lempitsky, Texture networks: Feed-forward synthesis of textures and stylized images, in Proceedings of the International Conference on Machine Learning, Curran, Red Hook, NY, 2016, pp. 1349–1357. |

[67] | J. J. Van Wijk, Spot noise texture synthesis for data visualization, in ACM SIGGRAPH Comput. Graph., 25 (1991), pp. 309–318. |

[68] | M. Varma and A. Zisserman, Texture classification: Are filter banks necessary?, in Proceedings of the IEEE CVPR, Vol. 2, IEEE Computer Society, Los Alamitos, CA, 2003, pp. 691–698. |

[69] | C. Villani, Topics in Optimal Transportation, AMS, Providence, RI, 2003. · Zbl 1106.90001 |

[70] | L.-Y. Wei, S. Lefebvre, V. Kwatra, and G. Turk, State of the art in example-based texture synthesis, in Eurographics, State of the Art Reports, Eurographics Associations, Aire-la-Ville, Switzerland, 2009, pp. 93–117. |

[71] | L.-Y. Wei and M. Levoy, Fast texture synthesis using tree-structured vector quantization, in Proceedings of SIGGRAPH ’00, ACM, New York, 2000, pp. 479–488. |

[72] | G.-S. Xia, S. Ferradans, G. Peyré, and J.-F. Aujol, Synthesizing and mixing stationary Gaussian texture models, SIAM J. Imaging Sci., 7 (2014), pp. 476–508. · Zbl 1391.94108 |

[73] | G. Yu, G. Sapiro, and S. Mallat, Solving inverse problems with piecewise linear estimators: From Gaussian mixture models to structured sparsity, IEEE Trans. Image Process., 21 (2012), pp. 2481–2499. · Zbl 1373.94471 |

[74] | S. Zhu, Y. Wu, and D. Mumford, Filters, random fields and maximum entropy (FRAME): Towards a unified theory for texture modeling, Int. J. Comput. Vis., 27 (1998), pp. 107–126. |

[75] | D. Zoran and Y. Weiss, From learning models of natural image patches to whole image restoration, in Proceedings of the IEEE ICCV, IEEE, Piscataway, NJ, 2011, pp. 479–486. |

This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. It attempts to reflect the references listed in the original paper as accurately as possible without claiming the completeness or perfect precision of the matching.