Augmented or auGANmented ?
Or how to use GANs to extend datasets
We all have the same problem, don’t we ? It’s okay to admit that sometimes you need help from the others. And that’s why you are here. Well it sounds little bit like intro to the AA session, but the truth is that the goal of this post is totally different, so let’s dive into it.
If you had a chance to play with different models, you may find out that the crucial point to train reliable model is to have quality data. In a real world scenario, when it’s not sufficient to use existing datasets, you are forced to gather all the data manually and only after that you are able to start work on your model. Seems laborious, huh ? So, take a seat and put a seatbelt on, because when you are working in the area of medical imaging, the situation is 10 times worse. Why ? Well, the answer can be more obvious then you thought.
The first problem is with the data itself. Let’s say you want to create model to locate brain tumors on MRI images. Obviously you need to find a sufficient number of people, who have this diagnosis and are willing to provide their screenings. The second problem is that even if you have collected some data, you need an expertise to manually locate and label tumors on these images. This step seems to be even harder than the previous one, because you must find a doctor who has the time and patience for this, not to mention that not every doctor is technically capable to use programs for annotation, so you have to teach him first how to use it. The third problem (I personally think that this one is the worst in terms of the ML model) is the lack of diversity in gathered and annotated data. But, don’t be sad my fellow reader there is still hope to address these problems.
So, let’s say you were lucky and you collected a few medical images from various patients and you also found a doctor, who was willing to help you with annotations. Now your only problem is to extend this dataset to train model properly. And here we can focus on data augmentation.
Data augmentation can be explained as essential technique to teach the network the desired invariance and robustness properties when only a few training samples are available [1]. For medical image tasks, different combinations of affine transformations are commonly used to teach the network the desired invariance. For example for brain tumor segmentation scaling, rotation and flipping have been applied to increase number of samples in existing dataset. However these typical data augmentation approaches fail to increase the diversity of the training data.
Luckily there is an exhaustive research focused on usage of GANs to solve this problem and in this article I will mention the most interesting ones.
Mok Tony et al. [1] proposed Coarse-to-fine Boundary-aware Generative Adversarial Networks (CB-GANs). In proposed method, the noise vector in traditional GANs was replaced by a label map of 2D axial slice from a 3D MRI as a conditioning variable. Additionaly their architecture was made of two generators. The coarse generator aimed to sketch the primitive shape of MRI from a label map at a lower resolution and the fine generator aimed to correct the defects and completed the details of the MRI from the coarse generator. The team also focused on preserving accurate tumor boundaries, so their architecture was modified even further. Specifically, two different branches were added to the final layer of fine generator in order to output the MRI of a brain with a tumor and the boundaries of the complete tumor. As a final step, the boundary and texture image from the branches were merged together to output the final image. One last modification was performed in the discriminator itself. They adopted multi-discriminators with the same architecture but with different scales of input.
Shin, Hoo-Chang et al. [2] proposed a method which can be used to extend existing medical datasets and also can be used as anonymization tool to share the data outside of the institution. They used publicly available datasets (ADNI and BRATS) to demonstrate MRI image synthesis and to prove efficiency of their solution. The method works on 3D data and consists of two separate steps. The first step is that the generator (pix2pix architecture) learns to segment brain labels from a T1-weighted MRI input gathered from ADNI dataset(tumors aren’t presented in these images) and after that the same model was used to segment brain labels also on BRATS dataset. The second step is to generate synthetic brain MRI from the segmentation masks. As a preprocessing step, they concatenated tumor labels(gathered from BRATS dataset), where interesting part was introducing variability by adjusting those labels (e.g., changing tumor size, moving the tumor’s location and so on) with segmented brain labels obtained in previous step. Moreover, they even used this method to generate multi-parametric MRI (T1/T2/T1c/Flair).
Guibas, John et al. [3] published an article where, a novel two-stage pipeline for generating synthetic medical images from a pair of generative adversarial networks was tested in practice on retinal fundi images. Here Stage-I GAN was trained on retinal vessel segmentations from the DRIVE database. It was designed to produce segmentation masks that represent the variable geometries of the dataset. On the other hand, Stage-II GAN task was to translate the masks produced in Stage-I to photorealistic images. Stage-I GAN was based on the deep convolutional generative adversarial network (DCGAN) architecture, which demonstrated competitive results and brought improving training stability in comparison to the standard GAN architecture. Stage-II GAN was also built on the TensorFlow platform and its architecture was based on an image-to-image translation network similar to pix2pix architecture.
I hope that this quick review of existing methods involving GANs to extend existing medical datasets helps you to overcome general issues connected with medical imaging, so you can come up with perfectly trained model to help patients around the world.
Oh, and don’t forget to check original articles, where you can find details about mentioned methods. Also some clap or follow will warm my heart, so if you like this article don’t forget to express it. That’s all from me now, see you soon.
References
[1] http://arxiv.org/abs/1805.11291
[2] http://arxiv.org/abs/1807.10225
