### Paper Menu >>

### Journal Menu >>

Optics and Photonics Journal, 2013, 3, 76-78 doi:10.4236/opj.2013.32B019 Published Online June 2013 (http://www.scirp.org/journal/opj) Copyright © 2013 S ciRes. OPJ Image Fusion Real-time System Based on FPGA and Multi-DSP* Feng Qu, Bochao Liu, Jian Zhao, Qiang Sun Changchun Institute of Optics, Fine Mechanics and Physics Chinese Academy of Sciences, Changchun, China Email: ciompqf@sohu.com, liubochao@ciomp.ac.cn, zhaojian6789@126.com, sunq@ciomp.an.cn Received 2013 ABSTRACT In order to solve complex algorithm that is difficult to achieve real-time processing of Multiband image fusion within large amount of data, a real-time image fusion system based on FPGA and multi-DSP is designed. Five-band image acquisition, ima ge regi stra tio n, image f usi on and disp la y outp ut ca n be d one within t he sys te m whic h uses FPG A as the main processor and the other three DSP as an algorithm processor. Makin g ful l use of Flexible and hig h-speed characte- ristics of FPG A, while an image fusion algorithm based on multi-wavelet trans for m is opti mized a nd ap plied to the s ys- tem. The final experimental results show that the frame rate of 15 Hz, with a resolution of 1392 × 1040 of the five-band image can be used by the system to complete processing within 41ms. Keywords: Multi-band; Real-time Image Fusio n; Multi-wav e le t Transform; Image Registration 1. Introduction Image fusion which is a branch of the data fusion con- solidates two or more source images come from comple- mentary band sensors, and is an image processing me- thod which makes the useful information integrated into a unified image for observation or further more accu- rately full recognition, analysis and judgment to the tar- get or scene through some fusion system [1-3]. Due of very limited data infor matio n conve yed of a s ingle band , which is often difficult to meet the demand, Multisensor image fusion has the information that single source im- age can’t be captured, that is Co mplementary betwe en o f multisource image. So multi-source image fusion can obtain information from multiple viewpoints of expand of se nsing range of time and space, to improve the accu- racy and robustness of observation. Seen from the definition of image fusion and co mplete fusion mainly includes two parts, namely the image source acq uisition and image fusion system, the core task of image fusion system is the implementation of image fusion algorithm. With the fusion algorithm improve- ments, as well as the amount of data increases, the image fusion system developed becomes with considerable dif- ficulty. Therefore, how to design a reasonable real-time image fusion hardware system is a prerequisite, and a suitable image fusion algorithm image fusion system is critical. In this paper, in accordance with the requirements of the actual project, an image fusion system based on FPGA and multi-DSP is designed, and an image fusion algo- rithm based on multi-wavelet transform is optimized, and the frame rate of 15 Hz, with a resolution of 1392 × 1040 of the fi v e-ba nd image is processed in real-time at last. 2. Structure of System Image fusion system hardware design is mainly three types: single DSP or DSP parallel processing program; FPGA + DSP processing program; a large FPGA proc- essing program [4-6]. With the combination of these strengths and weaknesses of the program, we have adopted the FPGA + multi-DSP processing program as our i mage fusio n s yste m. T his way by t he F PGA to co m- plete the pre -processing part and the corresponding logic control, complete the core algorithm in the DSP part, has the advantage of more flexible and can be formulated to give full pla y to the FPG A a nd D SP r espe ctive st rengt hs. also with the difficulty is how to coordinate communica- tion and data transfer between the FPGA and DSP, to ensure that the in entire treatmen t can be quickly and efficiently. Therefore, we designed the system structure shown in Figure 1, Which FPGA chosen Alter Corpora- tion Cyclong II series EP2C70F896, and DSP chosen TMS320C6416 produced by TI. In this system, three-way camera data is collected di- rectly by the FPGA, and after preprocessing, the data is *Supported by the program of academy-locality cooperation of the Chinese Academy of Sciences (2011CJT0004) , the Jilin province science and technology development plan item (2 0090557 and F. QU ET AL. Copyright © 2013 S ciRes. OPJ 77 FP GA DSP A DSP B DSP C Visible light camera Near Infrared Camera UV camera R G B FIFO F I F O FIFO F I F O Figure 1 . Schematic diagram of the image fusion s yst em based on FP G A and DSP. transferred to the DSP A and DSP B, with image regis- tration work in them. Visible image is chosen as refer- ence image because of more detail texture characteristics, and near-infrared image is registered in DSP A, while UV i mage is do ne in D SP B , then five-band image fusion work is completed by DSP C. Finally, the fused image is transferred to FPGA and output by FPG A. In order to improve the efficiency of the transmission, as well as to coord inate the timing of the various parts of the asynchronous, we ta ke advantage of the FPGA inter- nal structure generated FIFO for data exchange between camera data acquisition and FPGA and DSP. The entire process of using the pipeline work, as long as the time occupied by the image registration and image fusion processing must not exceed 66.7 ms to complete real-time processing of multi-band image. 3. Multi -w avelet Transform Algorithm Multi-wavelet is the development of wavelet theory, and which is the wavelet generated by two or more scaling function. Multi-wavelet can have many good properties, such as symmetry, short orthogonality and higher order vanishing moments, so multi-wavelet have more advan- ages than a single wavelet [7]. 3.1. Definition of Continuous Mul ti -wavelet Transform Assuming multi-wavelet as Ψ = （ψ1，…，ψr）T ∈ L2(R)r, and for a, b ∈ R, a ≠ 0, and by dilation and translation, the orthonormal basis on L2(R)r is generated as below equation (1): 1 2 ,() ||() ab xb xa a ψ −− = Ψ (1) For any f = (f1，…，fr) ∈L2(R)r, f in Ψ on t he con- tinuous multi-wavelet transform is , ,,, 1 ( ),()() r wav ab ababi i Tfff xxdx ψ = =Ψ= ∑ ∫ (2) 3.2. Mul ti -wavelet Decomposition and Reconstruction Algorithm Let f ∈VJ , then we have equation (3): 00 0 ,, 1 , ,,, 11 () () ()() rll JkJk l kZ rl lll J kJkjkjk lkZlJ jJkZ fxC x C xdx ϕ ϕψ = ∈ = ∈=≤≤∈ = = + ∑∑ ∑∑∑ ∑∑ (3) where J < J0, and: ,, () () ll jk jk Cfxx dx ϕ = ∫ (4) ,, () () ll jk jk dfxx dx ψ = ∫ (5) 11 ,,,,, , ( ,),(,), rl rl jkjk jk jkjkjk Cc cDdd= = following de- composition formula is: 1, ,2 1, ,2 jknjkn n jknjkn n C HC D GC ++ ++ = = ∑ ∑ (6) And the reconstruction formula is : ,1,2 1,2 () jnkj knkj kn n C HCGD ++ ++ = + ∑ (7) Can be seen from Equation 6 and 7, just make sur e C_ (j, k), multi-wavelet decomposition and reconstruction can be done by multiple Mallat algorithm. When the multi-wavelet signal is processed, the fast Mallat de- composition and reconstruction algorithm based on wavelet coefficients also is availab le. 3.3. Mul ti -wavelet Transform of the Image When multi-wavelet transform two-dimensional image, the followin g steps should be. a) The first preprocessing all rows, and then the data after the pre-processing of all columns (rows). If the crit- ical sampling method is used, the amount of data is un- changed; while the repeated-row prep ro cessi ng metho d is used, The data quantity is 4 times of the original. F. QU ET AL. Copyright © 2013 S ciRes. OPJ 78 (a) (b) (c) (d) (e) (f) (g) (h) Figure 2. Ex periment results: (a) Or iginal U V image; (b) Orig inal near infra red image; (c) Original R image; (d) Ori ginal G i ma g e ; (e) Original B image; (f) UV image of r e gistration; (g) Near infrared image of registration; (h) Image of f usion. b) When after pretreatment completed, 2D multi-wavelet- transform is applied. In the computation, the first line of multi-wavelet transform, then the column multiple wave- let transform. c) At last, post -processing is used to complete image fusion, which is the inverse transform of the pretreat- ment. 4. Conclusions Using FPGA as the main processor, and the DSP pro- cessor as arithmetic operations, a real-time image fusion system based on FPGA and multi-DSP is designed, which can effectively utilize the FPGA flexible high- speed characteristics, and take full advantage of the DSP powerful computing function. In order to coordinate the asynchronous timing problems between the various modules, and improve the efficiency of data exchange, FIFO generated in FPGA is used to complete a five- band i ma ge d at a a cq ui si tion. T he n the image fus io n a l go- rithm based on multi-wavelet tr ansform is optimized and transp lant ed. I n the final e xperiment, the image fusion of the five-band image with a frame rate of 15Hz, and a resolution of 1392 × 1040, is successfully completed in 41ms by this system. The experimental results are shown in Figure 2. 5. Acknowledgements F. J. Q would li ke to thank Dr. Jian Wang at our depart- ment for many helpful suggestions and discussions; this work is partly supported by the program of academy- locality cooperation of the Chinese Academy of Sicences and the Jilin province science and technology develop- ment plan item. REFERENCES [1] X. Suxia, “Real-Time Fusion System Hardware Design of Visual/IR Image Based on DSP and FPGA,” Journal of Beijing Technology and Business University (Natural Science Edition), Vol. 26, No. 6, 2008, pp. 44-47. [2] Z. Jianbin, H. Bo, Y. Rong and Z. Yong, “A Improved Pyramidal Decompositions Algorithm for Image Fusion and the Implementation on DSP,” Journal of Wuhan University of Science and Engineering, Vol. 22, No. 1, 2009. pp. 17-20. [3] Q. Feng, L. Ying, W. Jian, D. Keyan, et al., “Infrared dual-Band Real -Time Image Fusion System,” Optics and Precision Engineering, Vol. 18, No. 7, 2010, pp.1684-16 90. [4] H. Keming, W. Guocheng and W. Yang, “Multi-Source Image Fusion System Based on D SP,” Ordnance Industry Automation, Vol. 31 , No. 2, 2012, pp. 61-63. [5] S. Yajun, Z. Zhenfu and L. Zhongling, “Analysis and Application Prospect of Image Fusion Technology Based on Multicolor FPA,” Infrared and Laser Engineering, Vol. 3 8, No. 4, 2009, pp. 725-730. [6] G. Zhiqiang and L. Y i, “ Design of An Image Fusion Sys- tem Based on DSP Parallel Processing,” Journal of WUT (Information & Management Engineering), Vol. 28 No. 11, 2006, pp. 107-109. [7] C. K. Chui and J. A. Lian, “A Study of Orthonormal Mul- tiwavelets,” Applied Numerical Mathematics, Vol. 20, 1996, pp. 273-298. |