^{1}

^{*}

^{1}

^{1}

^{1}

^{1}

^{1}

Considering the continuous advancement in the field of imaging sensor, a host of other new issues have emerged. A major problem is how to find focus areas more accurately for multi-focus image fusion. The multi-focus image fusion extracts the focused information from the source images to construct a global in-focus image which includes more information than any of the source images. In this paper, a novel multi-focus image fusion based on Laplacian operator and region optimization is proposed. The evaluation of image saliency based on Laplacian operator can easily distinguish the focus region and out of focus region. And the decision map obtained by Laplacian operator processing has less the residual information than other methods. For getting precise decision map, focus area and edge optimization based on regional connectivity and edge detection have been taken. Finally, the original images are fused through the decision map. Experimental results indicate that the proposed algorithm outperforms the other series of algorithms in terms of both subjective and objective evaluations.

Image fusion is one of the most important techniques used to extract and integrate as much information as possible for image analysis, such as surveillance, target tracking, target detection and face recognition [

In the transform domain, the multi-scale decomposition is very similar to the human visual system and computer vision process from coarse to fine understanding of things, and no block effect in the fusion process [

The method based on spatial domain mainly deals with the image fusion according to the spatial feature information of image pixels [

From a large number of literatures, one of the key problems of spatial image fusion algorithm is how to measure the sharpness of blocks or regions or the saliency level of regions. In order to solve these problems, new multi-focus image fusion method, a spatial domain method, have been proposed based on Laplacian operator and region optimization. The saliency level of regions is the main part of the paper. The method of evaluating saliency level of image includes Tenengrad gradient function [

In order to prove the superiority of the proposed fusion method, three sets of images are selected for multi-focus image fusion, as shown in Figures 1(a)-(c). The images on the top row are mainly focused on the foreground while the images on the bottom row are mainly focused on the background. To better evaluate the performance of the fusion method, the proposed method is compared with several current mainstream multi-focus image fusion methods based on DWT [

In the quality evaluation of no reference image [

The Laplacian operator is an important algorithm in the image processing, which is a marginal point detection operator that is independent of the edge direction. The Laplacian operator is a kind of second order differential operator. A continuous two-element function f (x, y), whose Laplacian operation is defined as

∇ 2 f = ∂ 2 f / ∂ x 2 + ∂ 2 f / ∂ y 2 (1)

For digital images, the Laplacian operation can be simplified as

g ( i , j ) = 4 f ( i , j ) − f ( i + 1 , j ) − f ( i − 1 , j ) − f ( i , j + 1 ) − f ( i , j − 1 ) (2)

At the same time the above formula can be expressed as a convolution form, that is

g ( i , j ) = ∑ r = − k k ∑ s = − l l f ( i − r , j − s ) H ( r , s ) (3)

In the above formula, i , j = 0 , 1 , 2 , ⋯ , N − 1 ; k = 1, l = 1, H(r, s) can take a lot of values, one of which is

H 1 = [ 0 1 0 1 − 4 1 0 1 0 ]

Experiments show that the higher the image saliency is, the greater the sum of the mean of the corresponding matrix is after being processed by the Laplacian operator. Therefore, the image saliency (D(f)) based on the Laplacian gradient function is defined as follows:

D ( f ) = ∑ y ∑ x | g ( x , y ) | ( g ( x , y ) > T ) (4)

Among them, g (x, y) is the convolution of Laplacian operators at pixel points (x, y).

By using the value of D(f), it is easy to divide images with different clarity. Next, it is applied to the saliency decision of different regions of images. According to the above, the region saliency of an image can be defined as:

D I ( i , j ) = D ( I ( i − n : i + n , j − n : j + n ) ) (5)

Among them, D is the function of saliency method based on Laplacian gradient operator. D_{I} is the matrix of saliency of image I. And ( 2 n + 1 ) × ( 2 n + 1 ) is the scale of processing template.

In the multi-focus image processing, we can get significant matrices (D_{I}_{1}, D_{I}_{2}) of different focus images, obtain a decision matrix (M_{decision}) by comparing.

M decision = ( D I 1 ≥ D I 2 ) (6)

For various reasons, there are some noise and erroneous judgment in the decision map. It will affect the quality of image fusion. As for erroneous judgment, it will be mentioned later in the article.

In the first obtained decision map, there are often some noise and misjudged areas need to be corrected. In most methods, morphological processing is usually used to solve this problem. But this method often leads to the destruction of the boundary. H. Hariharan et al. [

M DF-decision = Delete Larea ( M decision ) (7)

As for Delete_{Larea}, it needs to be mentioned that its function is to delete smaller connected areas which include most noise and misjudged areas.

At this stage, there is an important problem to be solved. The erroneous judgment adhered to the focus edge is not removed by the above method. When using the Laplacian method to deal with the edges of multi-focus images, there is often edge information interference, in the case of

[ 1 1 1 0 0 1 1 0 0 0 1 1 1 0 0 1 1 0 0 0 1 0 0 0 0 ] → f A = [ 1 1 1 0 0 1 1 1 0 0 1 1 0 0 0 1 1 0 0 0 1 0 0 0 0 ] → g B = [ 1 1 1 0 0 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 1 0 0 0 0 ] } → h C = [ 1 1 1 0 0 1 1 0 0 0 1 1 1 0 0 1 1 0 0 0 1 0 0 0 0 ] (8)

Image fusion is carried out according to the final decision map (D_{final}). Then, the fused image f (x, y) could be expressed as:

f ( x , y ) = D final × f 1 ( x , y ) + ( 1 − D final ) × f 2 ( x , y ) (9)

It means the fused image is composed by the focus regions in the image f_{1} (x, y) and f_{2} (x, y). Though these steps, a fused image fully focused could be obtained.

For more than two images to be fused, it is necessary to change the form of the decision map. It will storage the serial number of the most significant image in the corresponding region.

f ( x , y ) = f D final ( x , y ) ( x , y ) (10)

The performance of the fusion algorithm can be evaluated subjectively and objectively. Since the evaluation is highly dependent on human visual characteristics, it is difficult to distinguish between the fused images when they are approximately similar. Therefore, one subjective evaluation method and four objective evaluation methods are adopted in this article.

1) Subjective evaluation method

a) Comparison of residual maps

The residual map can display the difference between two images in the image. We can observe the effect of image fusion by observing residual maps of different methods. The residual map I_{r} between the source image and the fused image is defined as follows:

I r = I origin − I fusion + max ( I origin ) / 2 (11)

2) Objective evaluation methods

a) Mutual information (MI)

The greater the sum of the mutual information between the fusion image and the source image, the richer the information obtained from the source image of the fused image, and the better the fusion effect. The MI between the source image and the fused image is defined as follows:

M I = ∑ k = 0 L ∑ i = 0 L p A F ( i , k ) log 2 p A F ( i , k ) p A ( i ) p F ( k ) + ∑ k = 0 L ∑ j = 0 L p B F ( j , k ) log 2 p B F ( j , k ) p A ( j ) p F ( k ) (12)

Among them, p_{A}, p_{B} and p_{F} are the normalized gray histogram of A, B and F. p_{AF} (i, k) and p_{BF} (j, k) are united gray histograms between the fused image and the source image. L is the number of intensity levels.

b) Peak signal to noise ratio (PSNR)

PSNR is the most common and widely used objective measure of image quality. The larger the PSNR, the less the distortion is represented. The PSNR is calculated as follows:

P S N R = 10 ⋅ log 10 ( M A X I 2 / ( 1 m n ∑ i = 1 m − 1 ∑ j = 1 n − 1 ‖ I ( i , j ) − K ( i , j ) ‖ 2 ) ) (13)

where A represents one of the pre-processed images, F represents the processed image, and MAX_{I} is the maximum value that represents the color of the image point.

c) Spatial frequencies (SF)

SF reflects the change of the pixel gray level of the image in space. To some extent, SF can reflect the clarity of images. SF is defined as follows:

S F = 1 M × N ∑ i = 1 M ∑ j = 2 N [ I ( i , j ) − I ( i , j − 1 ) ] 2 + 1 M × N ∑ i = 2 M ∑ j = 1 N [ I ( i , j ) − I ( i − 1 , j ) ] 2 (14)

where I (I, j) represents the image, and M and N represent the number of rows and columns of the image.

d) Edge intensity (EI)

EI is a measure of the local change intensity of the image in the normal direction along the edge, and also reflects the image sharpness to some extent. Its formula is expressed as:

E I = 1 M × N ∑ i = 1 M ∑ j = 1 N I x 2 ( i , j ) + I y 2 ( i , j ) (15)

where I_{x} (i, j) and I_{y} (i, j) represent horizontal gradient and longitudinal gradient of the image.

In order to test the accuracy of the above method, the experiment uses MATLAB language programming to achieve the above algorithm. Experimental pictures use Lena images. The image size is 512 × 512 pixels. Then, the four focus images are generated by blurring each with a Gaussian radius of 2.5, 5, 7.5, and 10, respectively. Five images of Lena, Lena 2.5, Lena 5, Lena 7.5, and Lena 10 are shown in Figures 4(a)-(e).

The five images were tested using the image saliency assessment method based on the Laplacian gradient. Get the corresponding D(f). The data is shown

that this method is very sensitive to fuzziness. Contrast experiments are performed using a group of multi-focus images in

From Figures 5(a)-(d), one can clearly see that the performance of these fusion methods showed difference when fused with the same multi-focus image. From a detailed observation, the fused image obtained by Tenengrad and SMD is not clear and there are a large number of residuals in

In turn, the focusing connectivity of the same focal plane and a focus edge optimization method based on edge detection are used to deal with the initial decision map. We can see that obvious interference have been removed in decision map in

The whole process can be summarized below. First, we choose a set of multi-focus images (

the multi-focus images are fused according to the final decision map. In the final decision map (

The fused image and corresponding residual map of Backgammon Clock, and Lab using different method are shown in Figures 8(a)-(j), Figures 9(a)-(j) and Figures 10(a)-(j). From Figures 8(a)-(e), we see that the five algorithms can produce fused images separately, but it is very difficult to distinguish the differences between some fusion results only by visual observation. To better evaluate the visual quality of the fused image, it is a good method to compare their residual map shown in Figures 8(f)-(j). Comparing the residual maps of these methods, the results are obvious. The residual maps obtained by DWT, NSCT, OPT and LP has more residual information, but the proposed method has less.

What would be resulted from

In the last part, the residual maps are used to compare the different image fusion methods. In order to further verify the performance of the proposed method, the objective quality evaluation is carried out. Objective evaluation indicators have been introduced above, including MI, PSNR, SF and EI. The evaluation results are shown in

From the data in

This paper presents an improved algorithm for multi-focus image fusion based

Image | Index | DWT | NSCT | OPT | LP | PROP |
---|---|---|---|---|---|---|

Backgammon | EI | 44.3606 | 40.5569 | 32.1144 | 44.3887 | 45.2426 |

MI | 4.6911 | 4.7151 | 4.7771 | 4.8213 | 4.4933 | |

SF | 18.5980 | 17.6258 | 13.1015 | 18.9101 | 18.9419 | |

SHA | 7.3057 | 7.2822 | 7.1955 | 7.3080 | 7.3120 | |

Clock | EI | 65.1337 | 59.5038 | 52.0855 | 65.6575 | 66.0519 |

MI | 4.3841 | 4.2744 | 4.4965 | 4.5743 | 5.6305 | |

SF | 18.3283 | 16.8511 | 14.1644 | 18.3780 | 18.6850 | |

SHA | 7.3570 | 7.3807 | 7.3141 | 7.3891 | 7.4122 | |

Lab | EI | 69.9127 | 58.0424 | 55.7379 | 70.2203 | 67.6791 |

MI | 4.0034 | 4.1135 | 4.0143 | 4.2268 | 5.7650 | |

SF | 21.2634 | 18.7092 | 16.5812 | 21.7725 | 21.8918 | |

SHA | 7.3377 | 7.2548 | 7.2360 | 7.3711 | 7.2868 |

on Laplacian operator and region optimization. There are two innovations in the algorithm which are the evaluation of image saliency based on Laplacian gradient and focus area and edge optimization based on the connectedness of the focused region and edge detection.

The evaluation of image saliency based on Laplacian gradient has performed well in distinguishing image clarity. It facilitates the extraction of precise focus areas. At the same time, focus area and edge optimization can make the focus area more accurate. From subjective and objective evaluation, it can be seen that the proposed algorithm is effective for multi-focus image fusion and it performs better than other four representative fusion algorithms. Many experiments have been done, and the algorithm still needs to be improved in the edge detection. Accurate edge detection can bring better fusion results.

This work is partially supported by the Hubei Provincial Department of Education, National Natural Science Foundation of China (11571041) and Natural Science Foundation of Hubei Province (2013CFA053).

Wang, C., Yuan, R., Sun, Y.Q., Jiang, Y.X., Chen, C.S. and Lin, X.L. (2018) A New Method of Multi-Focus Image Fusion Using Laplacian Operator and Region Optimization. Journal of Computer and Communications, 6, 106-118. https://doi.org/10.4236/jcc.2018.65009