3D MRI/IR Imaging Fusion: A New Medically Useful Computer Tool

Marcos L. Brioschi, M.D., Ph.D, Ionildo Sanches, M.Sc., Fabricius Traple, M.S. InfraredMed® – Infrared Medical Diagnosis

ABSTRACT

The measurement of temperature variation along the surface of the body provided by infrared imaging (IR) is becoming a valuable auxiliary tool for the early detection of many diseases in medicine. However, IR is essentially a 2D technique and its image does not provide useful anatomical information associated with it. However, multimodal image registration and fusion may overcome this difficulty and provide additional information for diagnosis purposes.

In this paper, a new method of registering and merging 2D IR and 3D MRI is presented. Registration of the images acquired from the two modalities is necessary since they are acquired with different image systems. First, the body volume of interest is scanned by a MRI system and a set of 2D IR of it at orthogonal angles is acquired. Next, it is necessary to register these two different sets of images. This is done by creating 2D MRI projections from the reconstructed 3D MRI volume and registering it with the IR. Once registered, the IR image is then projected over the 3D MRI. The program developed to assess the proposed method to combine MRI and IR resulted in a new tool for fusing two different image modalities, which can help medical doctors.

INTRODUCTION

Infrared imaging (IR), or simply thermography, is an extremely sensitive diagnostic test that measures and records surface temperature and natural heat distribution in the body. The applications of thermography in medicine for the diagnosis of various disease processes have expanded over the past decade coincident with technological advances in thermal imaging1-4, 14-15.

The medical use of IR is not new. However, with the recently improved sensitivity (around 0.02 degree Celsius) of the new generation of infrared sensors, IR is beginning to be a safe, efficient and reliable method for the study of some human diseases1-3.

A main advantage of IR as an auxiliary tool in medical diagnosis includes its relative low cost compared to other medical technologies. This noninvasive technique is ideal as a diagnostic method since it is completely harmless, without damaging radiation, and is devoid of needles or other noxious devices. Medical IR makes use of the long wave infrared thermal radiation energy emitted by the skin body. Nonetheless, this technique has some limitations and thus it is still not used systematically for clinical diagnosis worldwide.

IR is currently used to assess diseases such as breast and thyroid tumors, cerebral and peripheral vascular disease, rheumatic disease, musculoskeletal disorders, inflammations, and also during cardiac bypass surgery1-4, 14-15. In the particular case of Peripheral Vascular Disease (PVD) – a disease that compromises main blood supply to inferior limbs – IR is used to evaluate the extent of the damage to the limb5. PVD is characterized by constriction or occlusion of the main arteries that supply blood to the limbs. If not diagnosed in time, PVD can lead to partial or total amputation of the limbs, mainly in the lower body. In these cases, the preservation of the knee joint in patients submitted to amputation due to critical ischemia is associated to a better rehabilitation, mobility and quality-of-life post surgery.

Unfortunately, due to the absence of a safe, precise, and low-cost method to evaluate the level of local microcirculation compromise, many surgeons still prefer to amputate above the knee joint as a safety procedure. An inadequate amputation of the affected area leads to a difficult healing and to a high incidence of re-amputation. McCollum et al.6 have demonstrated that it is possible to determine more accurately the level of amputation in ischemic limbs with the use of IR.

In the US, 50% of the inferior limb amputations occur in diabetics with chronic ischemia. The amputation incidence increases with age. People with diabetes are 10 to 15 times more likely to have an amputation of the inferior limbs than non-diabetic individuals7.

Ohsawa et al.5 have used IR to determine the amputation level of 35 lower limbs in 27 diabetic patients with critical ischemia. Of those, 15 were re-amputated to superior levels. Studies determined that the presence of subcooling in the remaining part of the limb was one of the predominant factors for re-amputation. They concluded that skin thermography is an effective technique in the evaluation of amputation level which can help to prevent re-amputation in many cases. A recent study shows that this technique can be used to determine more accurately the adequate amputation level in ischemic limbs when combined with other data.

In spite of showing the details of circulatory physiology, IR lacks information about the local anatomy. This additional information can be obtained, for instance, by combining thermal data with magnetic resonance imaging (MRI) or computer tomography (CT). This combination enhances the clinical analysis of the patient, by merging together anatomical and physiological information into one image dataset14-15. Even though IR does not determine the exact site of a vascular occlusion, it is able to measure the relative temperature variation over the skin surface that results from it. IR indirectly estimates the extent of the changes in the cutaneous microcirculatory blood supply and therefore the amputation level with precision. The data obtained by merging both imaging techniques (IR & MRI), allows the determination of the extent of anatomic and physiological compromise, thus leading to a better and more adequate surgical intervention14-15 .

Techniques of IR include: (1) detection and image pickup techniques; (2) image displaying techniques; (3) techniques of qualitative and quantitative analysis; (4) measurement techniques for IR performance evaluation; (5) characterization techniques of studied surfaces and experimental conditions, and (6) experimental techniques particular to each application.

Recently, techniques for image registration and fusion have become an important tool in image analysis and visualization, with special application to multimodal medical imaging as CT, MRI, and PET scan8-10.

This article presents a new registration method that allows the fusion and 3D visualization of combined multimodal medical images, with particular interest in combining IR and MRI, originally developed by Brioschi et al. for the first time presented during the 3rd Asian-Pacific Federation of Thermology Meeting in 200214 and also at InfraMation 200415.

A computer tool that allows the correlation between the external 2D thermal maps and the 3D MRI was developed. Furthermore, the method developed can include data from Computer Tomography (CT) and Angio-CT (CTA), as well as from Magnetic Resonance Angiography (MRA).

MATERIALS AND METHODS

Infrared thermal images were acquired using an Agema Thermovision 470 IR camera and ThermaCAM® P65HS (FLIR Systems). Four thermographic images were acquired from the right lower leg and head of an asymptomatic volunteer. The camera view-angles were 90 degree clockwise shift, that is, anterior (zerodegree), lateral (90 degree), posterior (180 degree) and medial (270 degree). The room temperature was set and stabilized at 20°C and all the staff that took part in the process of thermographic image acquisition stayed in the room for approximately 20 minutes before image acquisition started. This procedure was taken to reduce to a minimum any temperature oscillations within the room during the image acquisition. The acquisition time of each IR image was around 60 seconds.

In the same day (less than 8 hours from the thermal images acquisition), an MRI experiment was performed and the right leg and head of the volunteer was MRI scanned. The MRI dataset was acquired using a Siemens 1.5T clinical scanner and stored in DICOM standard file. A T1 pulse sequence was selected to acquire a volume covering the region of interest. The size of the acquisition volume was 256x256 pixels and 100 slices, with 8 bits-per-pixel, 1.0 mm in-plane resolution and 2.0 mm slice thickness.

For the registration and fusion processes to be successful and a 3D model to be created in a satisfactory manner, the following five steps, for each of the four orthogonal views, are followed (fig. 1):

  1. Determine the object external contour for each MRI slice using any thresholding technique;
  2. Create a corresponding 2D projection of the reconstructed 3D MRI volume using a similar technique of the classical Range Image11-12 for each one of the orthogonal IR views;
  3. Register the resulting 2D MRI projection with the corresponding thermal images (same view angle);
  4. Back-project the registered thermal images onto the surface of the 3D MRI volume for the appropriate view-angle by superimposing each line on the thermal image onto the contour of the object for the corresponding MRI slice;
  5. Perform visualization of the resulting 3D object to show surface temperature distribution together with the internal structures.

Therefore, to overcome this major problem, we created a 2D MRI projection from the 3D MRI volume. Figure 2 illustrates this process. This is done by taking one MRI slice and selecting a line of pixel within the MRI fieldof-view (FOV) but outside of the object been imagined. In the sketch shown in figure 2(a), for the sake of simplicity it was draw this pixels line outside of the FOV.

Figure 2. (a) Sketch of the process of creating the 2D MRI image projection. (b) 2D projection obtained from the MRI slices.

Next, for each pixel in this line, it is counted by the number of pixels from it to the object outer contour. This number of pixels is then set to the corresponding pixel in the line (see the arrows in figure 2(a)). This is repeated for each pixel in the line and for each MRI slice, taking care to select always the same pixel line position in each slice.

At the end of this process, the data of each pixel line is put together in a new 2D image matrix. This image matrix is called depth image, an analogy to the somehow similar procedure well known as Range Image11.

It is interesting to note that the 2D MRI projection preserves the external anatomical shape of the real object imagined by the MRI system quite well. To visualize the 2D MRI projection, it is necessary to normalize the pixels values to the range of an 8 bits gray-scale image. It was found that histogram equalization gives a better image contrast and enhances some features, especially edges, making the identification of reference points for registration easer and more accurate. It was also suggested that before counting the pixel between the line and the object the tomographic MRI slices passes through any contour algorithm detection, which can be found elsewhere13. This procedure is repeated for the four orthogonal projections.

The registration process of the thermographic image over the 2D MRI projection is illustrated in Figure 3. Six reference points were manually selected in the leg IR image and four for the head IR image. The corresponding points in the 2D MRI projection are then carefully and manually marked using the computer mouse. The affine transform technique8-10 is then used to perform the necessary adjustment in the tomographic image to merge the selected image reference points.

Once the image registration has been accomplished, both images will have the same size. That is, the number of lines in the IR is equal to the number of slices of the MRI volume. Thus, the next step to be performed is the projection of the IR image information onto the MRI volume, on a slice per slice bases. This process is done so that each line on the thermal image is projected onto the respective object contour on the MRI slice.

The 3D visualization is done using the graphic library OpenGL. The voxels on the outer surface of the 3D image show the thermal data, while preserving all the MRI morphological information. After the 3D reconstruction, the user can visualize and interact with the application making geometric transformations such as rotation, translation and scaling, selecting cutting planes and changing the transparency of the surface.

Name Amount of slices

MRIHead 322 VHPHead 301 MRIBreast 192 BrainWeb 181 CTHead 113 MRILegs 100 MRIArm 300

Table 1. MRI and CT images.

Figure 3. (a) 2D MRI projection and (b) the corresponding thermal image with six markers to the leg and four to the head used to perform the image registration.

RESULTS

The software has been validated using seven sets of medical images from different modalities (MRI and CT) (Table 1).

Among them, the set of MR images of a leg and head described previously is used to illustrate the process. All four 2D projections created from the MR data set are illustrated on figure 4. These images are already normalized and equalized. The corresponding thermal images can be seen on figure 5.

Figure 4. 2D MRI projections created from the 3D MRI volume. The projections were created at zero (front view), 90, 180 and 270 degrees, respectively.

Figure 5. Thermal images of the leg and head at 0º, 90°, 180º and 270° view angles.

The result from the projection of all four thermal images onto a MR slice is illustrated on figures 6 and 7 for one MR slice. The process is repeated for the entire MR volume.

Figure 6. (a) Original MR slice and (b) MR slice with the Figure 7. Head 2D MRI anatomic detail slice covered with projected thermal contour (arrows). accurate surface temperature data.

After the 3D reconstruction, the user can rotate the object in any direction (3 axes – x, y and z – in the clockwise and anticlockwise directions), also slice it, make scale operations (amplification and reduction) and transparency voxels function (fig. 8).

Figure 8. Leg volume visualization at four different cutting levels and head visualization at pons axial cut. The MRI information is clearly seen together with surface temperature in the two cases.

DISCUSSION

A new image tool that combines IR thermography and MRI is proposed. The merging of different images information is done by creating a 2D MRI projection from the 3D MRI volume and using it as reference position for registering the two different images modalities. The 2Ds IR and MRI projection images are registered using the affine transform technique. Once registered, the transformed IR image is back-projected over the 3D MRI volume.

Preliminary studies showed that only four orthogonal projections are necessary for the data fusion and that a greater number of projections generate redundant information with no significant improvement in the quality of the final results. This also speeds up the entire process, from image acquisition to processing and reconstruction.

The IR image acquisition is a critical process since it is necessary to have a well- defined reference for each image plane, so the 2D MRI projection plane is created at the same view. In our experiments, we used the front plane as reference and the other three planes were taken at 90, 180 and 270 degrees from this reference. In principle, any set of four orthogonal angles can be used.

Manual registration of the 2D images is intrinsically a very difficult task since the reference points are not clearly defined on either set of 2D images, and further investigation is necessary to develop a fully automatic multimodal registration process. However, since the information displayed on thermal images is not spatially confined to small regions, but distributed over reasonably large areas with slow changes in intensity, any small inaccuracy on the registration process has little or no significant impact on the quality of the final results and particularly the concluding report analysis.

An initial prototype of the computer tool was created and tested using MatLab® version 6.5. Once the key processes had been fully operational, an interface was developed using OpenGL and the processes implemented in it.

The validation of the process was done using MRA, MRI, and CT images. However, this study goes as far as the development of the method and the creation of the computer tool. More clinical studies need to be carried out to establish a correlation between surface thermal information to internal clinical findings, mostly specifics diseases.

Applications of this research include improving MRI diagnoses, like breast cancer angiogenesis and peripheral vascular occlusion diseases detection. Using breast imaging fusion software tools, the clinician will be able to render regions of interest from the X-ray mammogram within the visual reference frame of the 3D MRI and IR imaging acquisition.

This new technology developed will greatly enhance the usefulness of IR imaging in medicine, as it will provide quantifiable data. Some of the further improvements on the interface would include tools for the segmentation and visualization of different structures, quantitative analysis of the 3D volume, and the use of other modalities such as CT.

CONCLUSIONS

In the present project, the aim was to build a computer tool for the fusion of multimodal images, with particular interest on thermal and MR images. The result is a tool to aid professionals in the medical area.

For the first time, multimodal 3D IR/MRI fusion has been used, incorporating anatomical and physiological information, to help define the lesion limits and extension, as well as their biological functional thermal behavior.

The software developed is reasonably easy to use and the results obtained in tests with image groups showed that combination of IR and MRI provides important information in the evaluation of some diseases.

With the use of multimodal fusion, this tool may assist medical professionals in understanding some diseases by considering their anatomical and physiological aspects simultaneously and without losing image resolution characteristics. 3D will help to describe and classify the IR and MRI findings. It also might be useful in diagnosis and surgery planning and execution and also radiotherapic proceedings as a simultaneous anatomic and functional`s monitoring.

REFERENCES

  1. Jones B.F. A Reappraisal of the Use of Infrared Thermal Image Analysis in Medicine, IEEE Transactions on Medical Imaging, v. 17, n. 6, pp. 1019-1027, December, 1998.
  2. Jones B.F., Plassmann, P. Digital infrared thermal imaging of human skin, IEEE Engineering in Medicine and Biology Magazine, v. 21, Issue: 6, pp. 41-48, 2002.
    1. Brioschi M.L., Macedo J.F., Macedo R.A.C. Skin Thermometry: New Concepts, Jornal Vascular Brasileiro,
    2. v. 2, n. 2, pp. 151-160, June, 2003. http://www.jvascbr.com.br/03-02-02/03-02-02-151/03-02-02-151.htm
  3. Aksenov P., Clark, I., Grant, D. et al. 3D Thermography for the Quantification of Heat Generation resulting from Inflammation, Proc. 8th 3D Modelling symposium, Paris, France, 2003.
  4. Ohsawa S., Inamori Y., Fukuda K., Hirotuji M. Lower Limb Amputation for Diabetic Foot, Archives of Orthopaedic and Trauma Surgery, v. 121, n. 4, pp. 186-190, 2001.
  5. McCollum P.T., Spence V.A., Walker W.F. Amputation for Peripheral Vascular Disease: The Case for Level Selection, Br Journal Surgery, v. 75, n. 12, pp. 1193-1195, December, 1988.
    1. Carrington A.L., Abbott C.A., Griffiths J., Jackson N., Johnson S.R., Kulkarni J., Van Ross E.R.E., Boulton
    2. A.J.M. A Foot Care Program for Diabetic Unilateral Lower-Limb Amputees, Diabetes Care, v. 24, pp. 216-221, 2001.
  6. Maintz J.B.A., Viergever M.A. A survey of medical image registration, Medical Image Analysis, 2(1):1-36, 1998.
  7. Zitová B., Flusser J. Image registration Methods: A Survey, Image and Vision Computing, v. 21, pp. 9771000, Oct., 2003.
  8. Hill D.L.G., Batchelor P.G., Holden M., Hawkes D.J. Medical image registration, Physics in Medicine and Biology, v. 46, R1-R45, 2001.
  9. Jain R., Kasturi R., Schunck B. G. Machine Vision, McGraw-Hill, 1995.
  10. Gariba M.A., Sanches I.J., Pardal F.C., Bichinho G.L. A method for registration and visualization of 3D MR images and 2D thermographic images for clinical application, CARS 2005 Proceedings, Berlin, 2005.
  11. Gonzalez R. C., Woods R. E. Digital Image Processing, Second Edition, Prentice Hall, New Jersey, 2002.
  12. Brioschi M. L., Colman D., Mello Neto, H. Fusing IR and magnetic resonance (MR) image. Journal of Korean Medical Thermology, Seoul, v. 2, n. 1, p. 57-58, 2002.
  13. Brioschi M.L., Vargas J.V., Malafaia O. Review of recent developments in thermographic applications in health care. InfraMation2004 Proceedings, Las Vegas, 2004.

ACKNOWLEDGEMENTS

The authors wish to thank the Brazilian National Counsel of Technological and Scientific Development (CNPQ) and FLIR Systems for sponsoring this research project, as well as Dr. Heraldo de Mello Filho (X-Leme Medical Clinics) and Munir Antônio Gariba, Gerson Linck Bichinho and Humberto Remigio Gamba, for their valuable help.

ABOUT THE AUTHOR

Dr. Brioschi is a post-doctoral fellow at the Pain Center, Department of Neurology, University of Sao Paulo Medical School. He graduated from medical school in 1996 and worked in general surgery at the Cajuru University Hospital from 1997 to 1999. His research interest was infrared imaging where he achieved Masters and PhD degrees. He is the director of InfraredMed – Medical Diagnosis by Infrared Imaging, president of the Brazilian Thermology Society, member of the American Academy of Thermology and member of the Brazilian Society of Radiology and Imaging Diagnosis. He is the leader of the Infrared Imaging Research Group of National Counsel of Technological and Scientific Development (CNPq), Brazil. He is also the medical director of InfraredMed®

CONTACT INFORMATION

Rua da Paz, 195 – cj 115 – MAB Medical Arts Building Curitiba – Paraná – Brazil CEP 80.060-160 Phone 0055-41-3362-0623

infraremed@infraredmed.org www.infraredmed.org