Diagnose Scoliosis: RGB-D Images in Deep Learning

What it does

Our approach is a new deep learning-based workflow to synthesize patients' back X-ray images from corresponding back RGB-D images which is radiation-free. With our approach, children's exposure to radiation during the diagnosis of scoliosis can be eliminated.

Your inspiration

Several studies have shown that every year, thousands of children suffer from scoliosis. They have no choice but to regularly take X-ray images for diagnosis. However, radiation given out by X-ray machines is harmful to children's health. To free more children from the harm of radiation, we decide to solve the problem. Moreover, with the rise of deep learning in the computer vision area, more and more medical images problem have been adopting deep learning approaches. Several papers purposed to synthesize all kinds of medical images using deep learning approaches. There 2 factors inspire us to synthesize X-ray images from RGB-D images.

How it works

Our approach has 3 stages. In stage 1, children's back RGB-D images and back X-ray images were collected and labeled with some anatomical landmarks. In stage 2, we implemented and trained a deep learning model to detect those anatomical landmarks on RGB-D images. RGB-D images and labeled landmarks served as the input and ground truth of the model. In stage 3, we implemented and trained another deep learning model to synthesize the X-ray images from RGB-D images collected in stage 1 and landmarks detected in stage 2. X-ray images collected in stage 1 served as the ground truth of the model. Therefore, with the trained models, in practice, we only need to take a RGB-D image from a child and feed the image to the first model. The first model will detect anatomical landmarks. Then, the RGB-D image and detected landmarks will be fed to the second model to synthesize the corresponding X-ray images. By our approach, we can obtain an X-ray image without radiation.

Design process

RGB-D images can be obtained using a radiation-free depth camera, which can solve the radiation problems induced by X-ray machines. RGB-D images contain the surface geometry of patients' back, which makes it theoretically possible to synthesize X-ray images from corresponding RGB-D images. However, directly synthesizing X-ray images from RGB-D images is hard because the model is unable to learn the shape of the spine curves. In the diagnosis of scoliosis, the shape of the spine curve is the most important factor that the model should not make any mistakes. Therefore, an intermediate step, landmark detection, was added. Firstly, a deep learning model was implemented and trained to do landmark detection on RGB-D images. The locations of these anatomical landmarks contain essential information of the shape of the spine curve. Then, we fed the RGB-D images and the detected landmarks to the second deep learning model. This model will synthesize X-ray images with the correct spine curve shape.

How it is different

X-ray images are necessary to diagnose scoliosis. Therefore, existing projects focus more on how to improve the accuracy of diagnosis on X-ray images using deep learning approaches. However, none of them can solve the problem induced by radiation given out by X-ray machines because X-ray images are also necessary for these projects as input. Our approach, starting from a fresh new perspective, aims to synthesize the X-ray images from the RGB-D images using deep learning approaches. We do not improve the accuracy of diagnosis, instead. We make the material needed for the diagnosis of scoliosis radiation-free. In short, our project focuses on a problem that is usually ignored by others. However, this problem does exist and has become more and more severe. Therefore, the focus of our project is different and significant.

Future plans

Currently, we only trained the models with 520 collected training samples because of the time limit, which was not enough. In the future, with more data available, our models can have better performance. Moreover, more deep learning backbones can be tried in the future. This project was greatly supported by Dr. Kenneth Wong, Department of Computer Science and Dr. Grace Zhang, Department of Orthopaedics and Traumatology at The University of Hong Kong.

Awards