×

Structural similarity preserving GAN for infrared and visible image fusion. (English) Zbl 1474.68404

Summary: Compared with a single image, in a complex environment, image fusion can utilize the complementary information provided by multiple sensors to significantly improve the image clarity and the information, more accurate, reliable, comprehensive access to target and scene information. It is widely used in military and civil fields, such as remote sensing, medicine, security and other fields. In this paper, we propose an end-to-end fusion framework based on structural similarity preserving GAN (SSP-GAN) to learn a mapping of the fusion tasks for visible and infrared images. Specifically, on the one hand, for making the fusion image natural and conforming to visual habits, structure similarity is introduced to guide the generator network produce abundant texture structure information. On the other hand, to fully take advantage of shallow detail information and deep semantic information for achieving feature reuse, we redesign the network architecture of multi-modal image fusion meticulously. Finally, a wide range of experiments on real infrared and visible TNO dataset and RoadScene dataset prove the superior performance of the proposed approach in terms of accuracy and visual. In particular, compared with the best results of other seven algorithms, our model has improved entropy, edge information transfer factor, multi-scale structural similarity and other evaluation metrics, respectively, by 3.05%, 2.4% and 0.7% on TNO dataset. And our model has also improved by 0.7%, 2.82% and 1.1% on RoadScene dataset.

MSC:

68T45 Machine vision and scene understanding
68U10 Computing methodologies for image processing

Software:

FusionGAN; IFCNN
PDFBibTeX XMLCite
Full Text: DOI

References:

[1] Akbarpour, T., Shamsi, M., Daneshvar, S. and Pooreisa, M., Medical image fusion based on nonsubsampled shearlet transform and principal component averaging, Int. J. Wavelets Multiresolut. Inf. Process.17(4) (2019) 1950023. · Zbl 1418.92066
[2] Aslantas, V. and Bendes, E., A new image quality metric for image fusion: The sum of the correlations of differences, AEU-Int. J. Electron. Commun.69(12) (2015) 1890-1896.
[3] Dong, X., Yan, Y., Tan, M., Yang, Y. and Tsang, I. W., Late fusion via subspace search with consistency preservation, IEEE Trans. Image Process.28(1) (2018) 518-528. · Zbl 1409.90240
[4] Eskicioglu, A. M. and Fisher, P. S., Image quality measures and their performance, IEEE Trans. Commun.43(12) (1995) 2959-2965.
[5] Guariglia, E., Entropy and fractal antennas, Entropy18(3) (2016) 84.
[6] Guariglia, E., Spectral analysis of the weierstrass-mandelbrot function, in Proc. 2nd Int. Multidisciplinary Conf. Computer and Energy Science (SpliTech) (IEEE, Split, 2017), pp. 1-6.
[7] Guariglia, E., Harmonic sierpinski gasket and applications, Entropy20(9) (2018) 714.
[8] Guariglia, E. and Silvestrov, S., Fractional-wavelet analysis of positive definite distributions and wavelets on \(\mathcal{\mathcal{D}}^\prime(\mathbb{C})\), Engineering Mathematics II (Springer, Berlin, 2016), pp. 337-353. · Zbl 1365.65294
[9] R. C. Guido, Practical and useful tips on discrete wavelet transforms [sp tips & tricks], IEEE Signal Process. Mag.32(3) (2015) 162-166.
[10] Hu, Y. and Zhang, B., A bio-inspired two-scale complementarity evaluation method for image fusion, Int. J. Wavelets Multiresolut. Inf. Process.15(6) (2017) 1750053. · Zbl 06815468
[11] Huang, M., Liu, Z., Ye, L., Zhou, X. and Wang, Y., Saliency detection via multi-level integration and multi-scale fusion neural networks, Neurocomputing364 (2019) 310-321.
[12] Jian, M. and Jung, C., Interactive image segmentation using adaptive constraint propagation, IEEE Trans. Image Process.25(3) (2016) 1301-1311. · Zbl 1408.94285
[13] Kong, W., Miao, Q. and Lei, Y., Multimodal sensor medical image fusion based on local difference in non-subsampled domain, IEEE Trans. Instrum. Meas.68(4) (2018) 938-951.
[14] Lekic, V. and Babic, Z., Automotive radar and camera fusion using generative adversarial networks, Comput. Vis. Image Underst.184 (2019) 1-8.
[15] Lewis, J. J., O’Callaghan, R. J., Nikolov, S. G., Bull, D. R. and Canagarajah, N., Pixel-and region-based image fusion with complex wavelets, Inf. Fusion8(2) (2007) 119-130.
[16] Li, S., Kang, X. and Hu, J., Image fusion with guided filtering, IEEE Trans. Image Process.22(7) (2013) 2864-2875.
[17] Li, Y., Lv, Z., Zhao, J. and Pan, Z., Improving performance of medical image fusion using histogram, dictionary learning and sparse representation, Multimedia Tools Appl.78(24) (2019) 34459-34482.
[18] Li, H. and Wu, X.-J., Densefuse: A fusion approach to infrared and visible images, IEEE Trans. Image Process.28(5) (2018) 2614-2623.
[19] H. Li and X. J. Wu, Infrared and visible image fusion using latent low-rank representation, preprint (2018), arXiv:1804.08992.
[20] Li, H., Wu, X.-J. and Kittler, J., Infrared and visible image fusion using a deep learning framework, in Proc. 24th Int. Conf. Pattern Recognition (ICPR) (IEEE, Beijing, 2018), pp. 2705-2710.
[21] Li, J., Yang, J., Gong, C. and Liu, Q., Saliency fusion via sparse and double low rank decomposition, Pattern Recognit. Lett.107 (2018) 114-122.
[22] Liang, Y., Mao, Y., Xia, J., Xiang, Y. and Liu, J., Scale-invariant structure saliency selection for fast image fusion, Neurocomputing356 (2019) 119-130.
[23] Liu, Y., Chen, X., Cheng, J., Peng, H. and Wang, Z., Infrared and visible image fusion with convolutional neural networks, Int. J. Wavelets Multiresolut. Inf. Process.16(3) (2018) 1850018. · Zbl 1392.62196
[24] Liu, Y., Chen, X., Peng, H. and Wang, Z., Multi-focus image fusion with a deep convolutional neural network, Inf. Fusion36 (2017) 191-207.
[25] Liu, X., Liu, Q. and Wang, Y., Remote sensing image fusion based on two-stream fusion network, Inf. Fusion55 (2020) 1-15.
[26] Liu, X., Zhang, H., Cheung, Y.-M., You, X. and Tang, Y. Y., Efficient single image dehazing and denoising: An efficient multi-scale correlated wavelet approach, Comput. Vis. Image Underst.162 (2017) 23-33.
[27] Liu, Y., Liu, S. and Wang, Z., A general framework for image fusion based on multi-scale transform and sparse representation, Inf. Fusion24 (2015) 147-164.
[28] Ma, J., Liang, P., Yu, W., Chen, C., Guo, X., Wu, J. and Jiang, J., Infrared and visible image fusion via detail preserving adversarial learning, Inf. Fusion54 (2020) 85-98.
[29] Ma, J., Ma, Y. and Li, C., Infrared and visible image fusion methods and applications: A survey, Inf. Fusion45 (2019) 153-178.
[30] Ma, J., Yu, W., Liang, P., Li, C. and Jiang, J., Fusiongan: A generative adversarial network for infrared and visible image fusion, Inf. Fusion48 (2019) 11-26.
[31] Ma, K., Zeng, K. and Wang, Z., Perceptual quality assessment for multi-exposure image fusion, IEEE Trans. Image Process.24(11) (2015) 3345-3356. · Zbl 1408.94465
[32] Ma, J., Zhou, Z., Wang, B. and Zong, H., Infrared and visible image fusion based on visual saliency map and weighted least square optimization, Infrared Phys. Technol.82 (2017) 8-17.
[33] Mallat, S. G., Multifrequency channel decompositions of images and wavelet models, IEEE Trans. Acoust. Speech Signal Process.37(12) (1989) 2091-2110.
[34] Mao, Q., Wang, S., Wang, S., Zhang, X. and Ma, S., Enhanced image decoding via edge-preserving generative adversarial networks, in Proc. IEEE Int. Conf. Multimedia and Expo (ICME) (IEEE, San Diego, CA, 2018), pp. 1-6.
[35] Naidu, V., Image fusion technique using multi-resolution singular value decomposition, Def. Sci. J.61(5) (2011) 479-484.
[36] Qu, G., Zhang, D. and Yan, P., Information measure for performance of image fusion, Electron. Lett.38(7) (2002) 313-315.
[37] Sun, Z., Hu, Z. P., Chiong, R., Wang, M. and Zhao, S., An adaptive weighted fusion model with two subspaces for facial expression recognition, Signal Image Video Process.12(5) (2018) 835-843.
[38] Wang, Z., Bovik, A. C., Sheikh, H. R., Simoncelli, E. P.et al., Image quality assessment: From error visibility to structural similarity, IEEE Trans. Image Process.13(4) (2004) 600-612.
[39] Xia, K. J., Yin, H. S. and Wang, J. Q., A novel improved deep convolutional neural network model for medical image fusion, Cluster Comput.22(1) (2019) 1515-1527.
[40] Xydeas, C. and Petrovic, V., Objective image fusion performance measure, Electron. Lett.36(4) (2000) 308-309.
[41] Yang, L., Su, H., Zhong, C., Meng, Z., Luo, H., Li, X., Tang, Y. Y. and Lu, Y., Hyperspectral image classification using wavelet transform-based smooth ordering, Int. J. Wavelets Multiresolut. Inf. Process.17(6) (2019) 1950050. · Zbl 1434.62140
[42] Yang, B., Zhong, J., Li, Y. and Chen, Z., Multi-focus image fusion and super-resolution with convolutional neural network, Int. J. Wavelets Multiresolut. Inf. Process.15(4) (2017) 1750037. · Zbl 1369.62146
[43] Zhang, K., Huang, Y. and Zhao, C., Remote sensing image fusion via RPCA and adaptive PCNN in NSST domain, Int. J. Wavelets Multiresolut. Inf. Process.16(5) (2018) 1850037. · Zbl 1401.60131
[44] Zhang, Y., Liu, Y., Sun, P., Yan, H., Zhao, X. and Zhang, L., IFCNN: A general image fusion framework based on convolutional neural network, Inf. Fusion54 (2020) 99-118.
[45] Zheng, X., Tang, Y. Y. and Zhou, J., A framework of adaptive multiscale wavelet decomposition for signals on undirected graphs, IEEE Trans. Signal Process.67(7) (2019) 1696-1711. · Zbl 1458.94160
[46] Zhou, Z., Wang, B., Li, S. and Dong, M., Perceptual fusion of infrared and visible images through a hybrid multi-scale decomposition with gaussian and bilateral filters, Inf. Fusion30 (2016) 15-26.
[47] Zhu, Z., Yin, H., Chai, Y., Li, Y. and Qi, G., A novel multi-modality image fusion method based on image decomposition and sparse representation, Inf. Sci.432 (2018) 516-529.
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.