ABSTRACT

Considering the continuous advancement in the field of imaging sensor, a host of other new issues have emerged. A major problem is how to find focus areas more accurately for multi-focus image fusion. The multi-focus image fusion extracts the focused information from the source images to construct a global in-focus image which includes more information than any of the source images. In this paper, a novel multi-focus image fusion based on Laplacian operator and region optimization is proposed. The evaluation of image saliency based on Laplacian operator can easily distinguish the focus region and out of focus region. And the decision map obtained by Laplacian operator processing has less the residual information than other methods. For getting precise decision map, focus area and edge optimization based on regional connectivity and edge detection have been taken. Finally, the original images are fused through the decision map. Experimental results indicate that the proposed algorithm outperforms the other series of algorithms in terms of both subjective and objective evaluations.

Keywords:

Image Fusion, Laplacian Operator, Multi-Focus, Region Optimization

1. Introduction

Image fusion is one of the most important techniques used to extract and integrate as much information as possible for image analysis, such as surveillance, target tracking, target detection and face recognition [1] [2] . Image fusion is often applied to multi-focus image processing. Due to the limited focus range of the optical lens, the optical lens will blur the object outside the focused region in the process of optical imaging [3] . To get the full focus image, multi-focus image fusion is an effective technique to solve this problem. Multi-focus image fusion is to integrate the focus area from images with different depth focus. So far, many multi-focus image fusion algorithms have been proposed. All methods can be divided into two categories: spatial domain fusion and transform domain fusion [4] .

In the transform domain, the multi-scale decomposition is very similar to the human visual system and computer vision process from coarse to fine understanding of things, and no block effect in the fusion process [5] . In multi-focus image fusion algorithm and the image fusion field, they are considered by researchers. This class of algorithms is more widely in current research. At present, the research on multiscale image fusion methods is mainly focused on the image multiscale analysis tools and the fusion rules. In recent years, researchers have proposed many tools for multiscale analysis of images, including pyramid transform, wavelet transform and other multiscale geometric analysis methods.

The method based on spatial domain mainly deals with the image fusion according to the spatial feature information of image pixels [6] . As a single pixel cannot represent the image space feature information, the block method is generally used. This method has a better effect on the area rich image. However, the processing of the flat area is likely to cause misjudgment, the size of the block is difficult to select, and the image edge will appear discontinuous small pieces, resulting in serious block effect. In view of the shortcomings of the image fusion algorithm based on block segmentation, some scholars have proposed an improved scheme. Among them, V. Aslantas and R. Kurban proposed differential evolution algorithm to determine the size of segmented image blocks, and achieved some results [6] . To a certain extent, it solved the problem that the size of image blocks was difficult to select. A. Goshtasby and others calculate the corresponding blocks of the fused image by calculating the weighted sum of the sub blocks, and introduce the weighting factors to each corresponding block in the source image [7] ; H. Hariharan et al. defined the focal connectivity of the same focal plane, and segmented the fused source image according to the connectivity [8] . In addition to the above several spatial domain fusion algorithms, many scholars have proposed the fusion method based on the focus region detection in recent years.

From a large number of literatures, one of the key problems of spatial image fusion algorithm is how to measure the sharpness of blocks or regions or the saliency level of regions. In order to solve these problems, new multi-focus image fusion method, a spatial domain method, have been proposed based on Laplacian operator and region optimization. The saliency level of regions is the main part of the paper. The method of evaluating saliency level of image includes Tenengrad gradient function [9] , Laplacian gradient [10] function, sum modulus difference (SMD) [11] function, energy gradient [12] function, and so on. The image was processed by the better method of evaluating saliency level of image, and then the general focusing region was obtained. Then, the focusing region is optimized according to the focusing connectivity of the focal plane and the edge detection. Finally, the multi-focus image fusion is finished by using the final decision map.

2. Materials and Methology

2.1. Materials

In order to prove the superiority of the proposed fusion method, three sets of images are selected for multi-focus image fusion, as shown in Figures 1(a)-(c). The images on the top row are mainly focused on the foreground while the images on the bottom row are mainly focused on the background. To better evaluate the performance of the fusion method, the proposed method is compared with several current mainstream multi-focus image fusion methods based on DWT [13] , NSCT [14] , OPT [15] and LP [16] . All experiments are carried out in MATLAB2016a.

2.2. The Evaluation of Image Saliency

In the quality evaluation of no reference image [17] , the saliency of image is an important index to evaluate the quality of image. It can be better suited to human subjective feelings. If the image is not high in significance, the image is blurred. In this paper, the Laplacian gradient [10] is used.

The Laplacian operator is an important algorithm in the image processing, which is a marginal point detection operator that is independent of the edge direction. The Laplacian operator is a kind of second order differential operator. A continuous two-element function f (x, y), whose Laplacian operation is defined as

${\nabla }^{2}f={\partial }^{2}f/\partial x^2+{\partial }^{2}f/\partial y^2$ (1)

For digital images, the Laplacian operation can be simplified as

$g\left(i,j\right)=4f\left(i,j\right)-f\left(i+1,j\right)-f\left(i-1,j\right)-f\left(i,j+1\right)-f\left(i,j-1\right)$ (2)

At the same time the above formula can be expressed as a convolution form, that is

$g\left(i,j\right)={\displaystyle \underset{r=-k}{\overset{k}{\sum }}{\displaystyle \underset{s=-l}{\overset{l}{\sum }}f\left(i-r,j-s\right)H\left(r,s\right)}}$ (3)

In the above formula, $i,j=0,1,2,\cdots ,N-1$ ; k = 1, l = 1, H(r, s) can take a lot of values, one of which is

$H_1=\left[\begin{array}{ccc}0& 1& 0\\ 1& -4& 1\\ 0& 1& 0\end{array}\right]$

Experiments show that the higher the image saliency is, the greater the sum of the mean of the corresponding matrix is after being processed by the Laplacian operator. Therefore, the image saliency (D(f)) based on the Laplacian gradient function is defined as follows:

$D\left(f\right)={\displaystyle {\sum }_y{\displaystyle {\sum }_x\left|g\left(x,y\right)\right|}}\left(g\left(x,y\right)>T\right)$ (4)

Among them, g (x, y) is the convolution of Laplacian operators at pixel points (x, y).

By using the value of D(f), it is easy to divide images with different clarity. Next, it is applied to the saliency decision of different regions of images. According to the above, the region saliency of an image can be defined as:

$D_I\left(i,j\right)=D\left(I\left(i-n:i+n,j-n:j+n\right)\right)$ (5)

Among them, D is the function of saliency method based on Laplacian gradient operator. D_I is the matrix of saliency of image I. And $\left(2n+1\right)\times \left(2n+1\right)$ is the scale of processing template.

In the multi-focus image processing, we can get significant matrices (D_I1, D_I2) of different focus images, obtain a decision matrix (M_decision) by comparing.

$M_{\text{decision}}=\left(D_{I_1}\ge D_{I_2}\right)$ (6)

For various reasons, there are some noise and erroneous judgment in the decision map. It will affect the quality of image fusion. As for erroneous judgment, it will be mentioned later in the article.

2.3. Region Optimization

In the first obtained decision map, there are often some noise and misjudged areas need to be corrected. In most methods, morphological processing is usually used to solve this problem. But this method often leads to the destruction of the boundary. H. Hariharan et al. [18] defined the focusing connectivity of the same focal plane. Most of the noise and misjudged areas can be corrected, according to it.

$M_{\text{DF-decision}}={\text{Delete}}_{\text{Larea}}\left(M_{\text{decision}}\right)$ (7)

As for Delete_Larea, it needs to be mentioned that its function is to delete smaller connected areas which include most noise and misjudged areas.

At this stage, there is an important problem to be solved. The erroneous judgment adhered to the focus edge is not removed by the above method. When using the Laplacian method to deal with the edges of multi-focus images, there is often edge information interference, in the case of Figure 2. Because the black part is more than the white part in their corresponding templates, point A and the points around it will turn black. We can understand it from Formula 2. f uses 3 × 3 this module for processing. And it will also be false for other reasons. Therefore, we put forward a focus edge optimization method based on edge detection. Edge detection is used to find the edges of the original images. Using a module scans the edges to modify the area in the module. The g is an edge detection function. As shown in Formula 8, the h is the function that if it is found that one side of the edge is dominated by an element, all this side is modified to the element. Among them, A is a decision map, B is an edge map, and C is an optimized decision map.

$\left[\begin{array}{ccccc}\text{1}& 1& 1& \text{0}& 0\\ \text{1}& \text{1}& \boxed{\text{0}}& \text{0}& \text{0}\\ \text{1}& \text{1}& \boxed{\text{1}}& \text{0}& \text{0}\\ \text{1}& \text{1}& \text{0}& \text{0}& \text{0}\\ \text{1}& \text{0}& \text{0}& \text{0}& \text{0}\end{array}\right]\begin{array}{c}\stackrel{f}{\to }A=\left[\begin{array}{ccccc}\text{1}& 1& 1& \text{0}& 0\\ \text{1}& \text{1}& \boxed{1}& \text{0}& \text{0}\\ \text{1}& \text{1}& \boxed{\text{0}}& \text{0}& \text{0}\\ \text{1}& \text{1}& \text{0}& \text{0}& \text{0}\\ \text{1}& \text{0}& \text{0}& \text{0}& \text{0}\end{array}\right]\\ \stackrel{g}{\to }B=\left[\begin{array}{ccccc}\text{1}& 1& 1& \text{0}& \text{0}\\ 0& \text{1}& \text{0}& \text{0}& \text{0}\\ 0& 0& 1& \text{0}& \text{0}\\ 0& \text{1}& \text{0}& \text{0}& \text{0}\\ \text{1}& \text{0}& \text{0}& \text{0}& \text{0}\end{array}\right]\end{array}\}\stackrel{h}{\to }C=\left[\begin{array}{ccccc}\text{1}& \text{1}& 1& \text{0}& \text{0}\\ \text{1}& \text{1}& \boxed{\text{0}}& \text{0}& \text{0}\\ \text{1}& \text{1}& \boxed{\text{1}}& \text{0}& \text{0}\\ \text{1}& \text{1}& \text{0}& \text{0}& \text{0}\\ \text{1}& \text{0}& \text{0}& \text{0}& \text{0}\end{array}\right]$ (8)

2.4. Multi-Focus Image Fusion

Image fusion is carried out according to the final decision map (D_final). Then, the fused image f (x, y) could be expressed as:

$f\left(x,y\right)=D_{\text{final}}\times f_1\left(x,y\right)+\left(1-D_{\text{final}}\right)\times f_2\left(x,y\right)$ (9)

It means the fused image is composed by the focus regions in the image f₁ (x, y) and f₂ (x, y). Though these steps, a fused image fully focused could be obtained.

For more than two images to be fused, it is necessary to change the form of the decision map. It will storage the serial number of the most significant image in the corresponding region. Figure 3 is a schematic map of a decision map in the process of fusion of four multi-focus images. When image fusion is made, each point of the image is assigned according to the index value. The fused image f (x, y) could be expressed as:

$f\left(x,y\right)=f_{D_{\text{final}}\left(x,y\right)}\left(x,y\right)$ (10)

2.5. Evaluation Index System

The performance of the fusion algorithm can be evaluated subjectively and objectively. Since the evaluation is highly dependent on human visual characteristics, it is difficult to distinguish between the fused images when they are approximately similar. Therefore, one subjective evaluation method and four objective evaluation methods are adopted in this article.

1) Subjective evaluation method

a) Comparison of residual maps

The residual map can display the difference between two images in the image. We can observe the effect of image fusion by observing residual maps of different methods. The residual map I_r between the source image and the fused image is defined as follows:

$I_r=I_{\text{origin}}-I_{\text{fusion}}+\max \left(I_{\text{origin}}\right)/2$ (11)

2) Objective evaluation methods

a) Mutual information (MI)

The greater the sum of the mutual information between the fusion image and the source image, the richer the information obtained from the source image of the fused image, and the better the fusion effect. The MI between the source image and the fused image is defined as follows:

$MI={\displaystyle \underset{k=0}{\overset{L}{\sum }}{\displaystyle \underset{i=0}{\overset{L}{\sum }}{p}_{AF}\left(i,k\right){\log }_2\frac{p_{AF}\left(i,k\right)}{p_A\left(i\right)p_F\left(k\right)}}}\text{+}{\displaystyle \underset{k=0}{\overset{L}{\sum }}{\displaystyle \underset{j=0}{\overset{L}{\sum }}{p}_{BF}\left(j,k\right){\log }_2\frac{p_{BF}\left(j,k\right)}{p_A\left(j\right)p_F\left(k\right)}}}$ (12)

Among them, p_A, p_B and p_F are the normalized gray histogram of A, B and F. p_AF (i, k) and p_BF (j, k) are united gray histograms between the fused image and the source image. L is the number of intensity levels.

b) Peak signal to noise ratio (PSNR)

PSNR is the most common and widely used objective measure of image quality. The larger the PSNR, the less the distortion is represented. The PSNR is calculated as follows:

$PSNR=10\cdot {\log }_{10}\left(MA{X}_I^2/\left(\frac{1}{mn}{\displaystyle \underset{i=1}{\overset{m-1}{\sum }}{\displaystyle \underset{j=1}{\overset{n-1}{\sum }}{\Vert I\left(i,j\right)-K\left(i,j\right)\Vert }^2}}\right)\right)$ (13)

where A represents one of the pre-processed images, F represents the processed image, and MAX_I is the maximum value that represents the color of the image point.

c) Spatial frequencies (SF)

SF reflects the change of the pixel gray level of the image in space. To some extent, SF can reflect the clarity of images. SF is defined as follows:

$SF=\sqrt{\frac{1}{M\times N}{\displaystyle \underset{i=1}{\overset{M}{\sum }}{\displaystyle \underset{j=2}{\overset{N}{\sum }}{\left[I\left(i,j\right)-I\left(i,j-1\right)\right]}^2}}+\frac{1}{M\times N}{\displaystyle \underset{i=2}{\overset{M}{\sum }}{\displaystyle \underset{j=1}{\overset{N}{\sum }}{\left[I\left(i,j\right)-I\left(i-1,j\right)\right]}^2}}}$ (14)

where I (I, j) represents the image, and M and N represent the number of rows and columns of the image.

d) Edge intensity (EI)

EI is a measure of the local change intensity of the image in the normal direction along the edge, and also reflects the image sharpness to some extent. Its formula is expressed as:

$EI=\frac{1}{M\times N}{\displaystyle \underset{i=1}{\overset{M}{\sum }}{\displaystyle \underset{j=1}{\overset{N}{\sum }}\sqrt{I_x^2\left(i,j\right)+I_y^2\left(i,j\right)}}}$ (15)

where I_x (i, j) and I_y (i, j) represent horizontal gradient and longitudinal gradient of the image.

3. Results

3.1. Laplacian Gradient

In order to test the accuracy of the above method, the experiment uses MATLAB language programming to achieve the above algorithm. Experimental pictures use Lena images. The image size is 512 × 512 pixels. Then, the four focus images are generated by blurring each with a Gaussian radius of 2.5, 5, 7.5, and 10, respectively. Five images of Lena, Lena 2.5, Lena 5, Lena 7.5, and Lena 10 are shown in Figures 4(a)-(e).

The five images were tested using the image saliency assessment method based on the Laplacian gradient. Get the corresponding D(f). The data is shown

that this method is very sensitive to fuzziness. Contrast experiments are performed using a group of multi-focus images in Figure 1(a). The results are shown in Figure 5(a).

From Figures 5(a)-(d), one can clearly see that the performance of these fusion methods showed difference when fused with the same multi-focus image. From a detailed observation, the fused image obtained by Tenengrad and SMD is not clear and there are a large number of residuals in Figure 5(c) and Figure 5(d). Besides, the edge of the object is fuzzy from the decision map of Tenengrad, SMD and energy gradient. At the same time, compared with the actual situation, it can be clearly seen that the decision map obtained by these fusion methods appeared to have more obvious false information. However, it can be easily observed that the fused image acquired by the image saliency assessment method based on the Laplacian gradient is more ideal in the subjective effect because the residual information is also less than other methods which means that the method transfer almost all focus information to the fused image. On the other hand, good preprocessing is very convenient for the later operation, especially in edge optimization.

3.2. Region Optimization

In turn, the focusing connectivity of the same focal plane and a focus edge optimization method based on edge detection are used to deal with the initial decision map. We can see that obvious interference have been removed in decision map in Figure 6(e). It is not difficult to find that the edge is smoother in the decision map after edge optimization in Figure 6(f). And it shows more clearly in Figures 7(a)-(d).

3.3. Multi-Focus Image Fusion

The whole process can be summarized below. First, we choose a set of multi-focus images (Figure 6(a) and Figure 6(b)) for processing to obtain the corresponding saliency maps (Figure 6(c)). And then we can get an initial decision map (Figure 6(d)) through them. Next, the focusing connectivity of the same focal plane is used to remove most of the noise and misjudge areas. The edge correction method is used to optimize the decision map (Figure 6(e)). Finally,

the multi-focus images are fused according to the final decision map. In the final decision map (Figure 6(f)), it is clear from Figure 7(d) that the edge is more smooth and consistent with the actual situation. Finally, we got a globally clear image (Figure 6(g)).

4. Discussions

4.1. Subjective Evaluation

The fused image and corresponding residual map of Backgammon Clock, and Lab using different method are shown in Figures 8(a)-(j), Figures 9(a)-(j) and Figures 10(a)-(j). From Figures 8(a)-(e), we see that the five algorithms can produce fused images separately, but it is very difficult to distinguish the differences between some fusion results only by visual observation. To better evaluate the visual quality of the fused image, it is a good method to compare their residual map shown in Figures 8(f)-(j). Comparing the residual maps of these methods, the results are obvious. The residual maps obtained by DWT, NSCT, OPT and LP has more residual information, but the proposed method has less.

What would be resulted from Figure 8(c) and Figure 9(c) is that the images are not enough clear using OPT. Many fusion errors exit at the right edge of a Gobang box in Figure 8(a) and Figure 8(b) and the surface of the back clock in Figure 9(a), Figure 9(b) and Figure 9(d). From the residual maps, there are more residual information on the surface of clock in Figure 10(f), Figure 10(h) and Figure 10(i). Compared with other methods, the method proposed in this paper has better subjective performance.

4.2. Objective Evaluation

In the last part, the residual maps are used to compare the different image fusion methods. In order to further verify the performance of the proposed method, the objective quality evaluation is carried out. Objective evaluation indicators have been introduced above, including MI, PSNR, SF and EI. The evaluation results are shown in Table 1.

From the data in Table 1, the evaluation results are obvious. When the origin image is “Backgammon”, most values of four indexes of the proposed method are observably higher than those of the other methods. It has the same situation for “Clock” and “Lab”. By the meaning of several evaluation methods, it can be shown that the fusion image obtained by the proposed method contains more information, also shows that the fusion image has higher definition.

5. Conclusions

This paper presents an improved algorithm for multi-focus image fusion based

on Laplacian operator and region optimization. There are two innovations in the algorithm which are the evaluation of image saliency based on Laplacian gradient and focus area and edge optimization based on the connectedness of the focused region and edge detection.

The evaluation of image saliency based on Laplacian gradient has performed well in distinguishing image clarity. It facilitates the extraction of precise focus areas. At the same time, focus area and edge optimization can make the focus area more accurate. From subjective and objective evaluation, it can be seen that the proposed algorithm is effective for multi-focus image fusion and it performs better than other four representative fusion algorithms. Many experiments have been done, and the algorithm still needs to be improved in the edge detection. Accurate edge detection can bring better fusion results.

Acknowledgements

This work is partially supported by the Hubei Provincial Department of Education, National Natural Science Foundation of China (11571041) and Natural Science Foundation of Hubei Province (2013CFA053).

Cite this paper

Wang, C., Yuan, R., Sun, Y.Q., Jiang, Y.X., Chen, C.S. and Lin, X.L. (2018) A New Method of Multi-Focus Image Fusion Using Laplacian Operator and Region Optimization. Journal of Computer and Communications, 6, 106-118. https://doi.org/10.4236/jcc.2018.65009

References