Yıl: 2021 Cilt: 29 Sayı: 2 Sayfa Aralığı: 616 - 631 Metin Dili: İngilizce DOI: 10.3906/elk-2002-175 İndeks Tarihi: 07-06-2022

Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans

Öz:
Brain tumors have been one of the most common life-threatening diseases for all mankind. There have been huge efforts dedicated to the development of medical imaging techniques and radiomics to diagnose tumor patients quickly and efficiently. One of the main aims is to ensure that preoperative overall survival time (OS) prediction is accurate. Recently, deep learning (DL) algorithms, and particularly convolutional neural networks (CNNs) achieved promising performances in almost all computer vision fields. CNNs demand large training datasets and high computational costs. However, curating large annotated medical datasets are difficult and resource-intensive. The performances of single learners are also unsatisfactory for small datasets. Thus, this study was conducted to improve the performance of CNN models on small volumetric datasets through developing a DL-based ensemble method for OS classification of brain tumor patients using multimodal magnetic resonance images (MRI). First, we proposed multiview CNNs: OS classifiers based on representing the 3D MRI data as a set of 2D slices along all three planes (axial, sagittal, and coronal) and process them using 2D CNNs. Subsequently, the predicted probabilities by the multiview CNN models were fused using standard machine learning algorithms. The proposed approach was experimentally evaluated on 163 patients obtained from the BraTS’17 training dataset. Our best model achieved an AUC and accuracy values of 0.93 and 92.9%, respectively, on classifying patients with brain tumors into two OS groups, outperforming current state-of-the-art results. In addition, the FLAIR MRI modality yielded the best classification accuracy compared to other MRI modalities. Similarly, axial projections had the best classification performance compared to coronal and sagittal projections. Our findings may provide valuable insights for physicians in advancing treatment planning via noninvasive and accurate prediction of survival using only MRIs at the time of diagnosis.
Anahtar Kelime:

Belge Türü: Makale Makale Türü: Araştırma Makalesi Erişim Türü: Erişime Açık
  • [1] Neervoort FW, Van Ouwerkerk WJR, Folkersma H, Kaspers GJL, Vandertop WP. Surgical morbidity and mortality of pediatric brain tumors: a single center audit. Child’s Nervous System 2010; 26 (11): 1583-1592.
  • [2] He Z, Mitteer RA, Mou Y, Fan Y. Multimodality Targeting of Glioma Cells. In: Brem S, Abdullah KG (editors). Glioblastoma. Philadelphia, PA, USA: Elsevier, 2016, pp. 55–72.
  • [3] Ostrum QT, Gittleman H, Liao P, Rouse C, Chen Y et al. CBTRUS statistical report: primary brain and central nervous system tumors diagnosed in the United States in 2007–2011. Neuro‐oncology 2014; 16 (s4): iv1-iv63. doi: 10.1093/neuonc/nou223
  • [4] Van Meir EG, Hadjipanayis CG, Norden AD, Shu HK, Wen PY et al. Exciting new advances in neuro‐oncology: the avenue to a cure for malignant glioma. CA: A Cancer Journal for Clinicians 2010; 60 (3): 166-193.
  • [5] Ostrom QT, Gittleman H, Truitt G, Boscia A, Kruchko C et al. CBTRUS statistical report: primary brain and other central nervous system tumors diagnosed in the United States in 2011–2015. Neuro-oncology 2018; 20 (s4): iv1-iv86. doi: 10.1093/neuonc/noy131
  • [6] Castells X, García-Gómez JM, Navarro A, Acebes JJ, Godino Ó et al. Automated brain tumor biopsy prediction using single-labeling cDNA microarrays-based gene expression profiling. Diagnostic Molecular Pathology 2009; 18 (4): 206-218. doi: 10.1097/PDM.0b013e31818f071b
  • [7] Christakis NA, Smith JL, Parkes CM, Lamont EB. Extent and determinants of error in doctors’ prognoses in terminally ill patients: prospective cohort study Commentary: Why do doctors overestimate? Commentary: Prognoses should be based on proved indices not intuition. The British Medical Journal 2000; 320 (7233): 469-473. doi: 10.1136/bmj.320.7233.469
  • [8] Peleg R, Biderman A. A memorable patient: the scars of the Jewish holocaust. The British Medical Journal 2000; 320 (7233): 473-473. doi: 10.1136/bmj.320.7233.473
  • [9] Legaz-Aparicio AG, Verdú-Monedero R, Larrey-Ruiz J, Morales-Sánchez J, López-Mir F et al. Efficient variational approach to multimodal registration of anatomical and functional intra-patient tumorous brain data. International Journal of Neural Systems 2017; 27 (06): 1750014. doi: 10.1142/S0129065717500149
  • [10] Gillies RJ, Kinahan PE, Hricak H. Radiomics: images are more than pictures, they are data. Radiology 2016; 278 (2): 563-577. doi: 10.1148/radiol.2015151169
  • [11] Shukla G, Alexander GS, Bakas S, Nikam R, Talekar K et al. Advanced magnetic resonance imaging in glioblastoma: a review. Chinese Clinical Oncology 2017; 6 (4): 40. doi: 10.21037/cco.2017.06.28
  • [12] Liu Y, Zhang X, Feng N, Yin L, He Y et al. The effect of glioblastoma heterogeneity on survival stratification: a mul- timodal MR imaging texture analysis. Acta Radiologica 2018; 59 (10): 1239-1246. doi: 10.1177/0284185118756951
  • [13] Zhang L, Wang H, Li Q, Zhao MH, Zhan QM. Big data and medical research in China. The British Medical Journal 2018; 360: j5910. doi: 10.1136/bmj.j5910
  • [14] Wang K, Wang Y, Fan X, Wang J, Li G et al. Radiological features combined with IDH1 status for predicting the survival outcome of glioblastoma patients. Neuro-oncology 2015; 18 (4): 589-597. doi: 10.1093/neuonc/nov239
  • [15] Du W, Zhang M, Ying W, Perc M, Tang K et al. The networked evolutionary algorithm: a network science perspective. Applied Mathematics and Computation 2018; 338: 33-43. doi: 10.1016/j.amc.2018.06.002
  • [16] Erkaymaz O, Ozer M, Perc M. Performance of small-world feedforward neural networks for the diagnosis of diabetes. Applied Mathematics and Computation 2017; 311: 22-28. doi: 10.1016/j.amc.2017.05.010
  • [17] Bakas S, Reyes M, Jakab A, Bauer S, Rempfler M et al. Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge. arXiv 2018; arXiv:1811.02629 [cs.CV].
  • [18] Li Q, Bai H, Chen Y, Sun Q, Liu L et al. A fully-automatic multiparametric radiomics model: towards reproducible and prognostic imaging signature for prediction of overall survival in glioblastoma multiforme. Scientific Reports 2017; 7 (1): 1-9. doi: 10.1038/s41598-017-14753-7
  • [19] Shboul ZA, Alam M, Vidyaratne L, Pei L, Elbakary MI et al. Feature-guided deep radiomics for glioblastoma patient survival prediction. Frontiers in Neuroscience 2019; 1: 13. doi: 10.3389/fnins.2019.00966
  • [20] Li Y, Shen L. Deep learning based multimodal brain tumor diagnosis. In: Crimi A, Bakas S, Kuijf H, Menze B, Reyes M (editors). Brainlesion: Glioma, Multiple Sclerosis, Stroke and Traumatic Brain Injuries. BrainLes 2017, Lecture Notes in Computer Science. Cham, Switzerland: Springer, 2018, pp. 1-20.
  • [21] Chato L, Latifi S. Machine learning and deep learning techniques to predict overall survival of brain tumor patients using MRI images. In: 2017 IEEE 17th International Conference on Bioinformatics and Bioengineering (BIBE); Washington, DC, USA; 2017. pp. 9-14.
  • [22] Lao J, Chen Y, Li ZC, Li Q, Zhang J et al. A deep learning-based radiomics model for prediction of survival in glioblastoma multiforme. Scientific Reports 2017; 7 (1): 1-8. doi: 10.1038/s41598-017-10649-8
  • [23] Sun L, Zhang S, Chen H, Luo L. Brain Tumor Segmentation and Survival Prediction Using Multimodal MRI Scans with Deep Learning. Frontiers in Neuroscience 2019; 13: 810. doi: 10.3389/fnins.2019.00810
  • [24] Thickstun J, Harchaoui Z, Foster DP, Kakade SM. Invariances and data augmentation for supervised music transcription. In: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP); Calgary, AB, Canada; 2018. pp. 2241-2245
  • 25] CireşAn D, Meier U, Masci J, Schmidhuber J. Multi-column deep neural network for traffic sign classification. Neural Networks 2012; 32: 333-338. doi: 10.1016/j.neunet.2012.02.023
  • [26] Li R, Zhang W, Suk HI, Wang L, Li J et al. Deep learning based imaging data completion for improved brain disease diagnosis. In: Golland P, Hata N, Barillot C, Hornegger J, Howe R (editors). Medical Image Computing and Computer-Assisted Intervention – MICCAI 2014. Lecture Notes in Computer Science, Vol. 8675. Cham, Switzerland: Springer, 2014, pp. 305-312.
  • [27] Perc M, Ozer M, Hojnik J. Social and juristic challenges of artificial intelligence. Palgrave Communications 2019; 5 (1): 1-7. doi: 10.1057/s41599-019-0278-x
  • [28] Nie D, Lu J, Zhang H, Adeli E, Wang J et al. Multi-channel 3D deep feature learning for survival time prediction of brain tumor patients using multi-modal neuroimages. Scientific Reports 2019; 9 (1): 1-4.
  • [29] Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Commu- nications of the ACM 2017; 60 (6): 84-90.
  • [30] Gupta V, Demirer M, Bigelow M, Little KJ, Candemir S et al. Performance of a deep neural network algorithm based on a small medical image dataset: incremental impact of 3D-to-2D reformation combined with novel data augmentation, photometric conversion, or transfer learning. Journal of Digital Imaging 2020; 33 (2): 431-438.
  • [31] Su H, Maji S, Kalogerakis E, Learned-Miller E. Multi-view convolutional neural networks for 3d shape recognition. In: Proceedings of the IEEE international Conference on Computer Vision; Santiago, Chile; 2015. pp. 945-953.
  • [32] Bien N, Rajpurkar P, Ball RL, Irvin J, Park A et al. Deep-learning-assisted diagnosis for knee magnetic resonance imaging: development and retrospective validation of MRNet. PLoS Medicine 2018; 15 (11): e1002699.
  • [33] Choi Y, Ahn KJ, Nam Y, Jang J, Shin NY et al. Analysis of heterogeneity of peritumoral T2 hyperintensity in patients with pretreatment glioblastoma: prognostic value of MRI-based radiomics. European Journal of Radiology 2019; 120: 108642.
  • [34] Winzeck S, Mocking SJ, Bezerra R, Bouts MJ, McIntosh EC et al. Ensemble of convolutional neural networks improves automated segmentation of acute ischemic lesions using multiparametric diffusion-weighted MRI. American Journal of Neuroradiology 2019; 40 (6): 938-945. doi: 10.3174/ajnr.A6077
  • [35] Zhang B, Qi S, Monkam P, Li C, Yang F et al. Ensemble learners of multiple deep CNNs for pulmonary nodules classification using CT images. IEEE Access 2019; 7: 110358-110371. doi: 10.1109/ACCESS.2019.2933670
  • [36] Xiao Y, Wu J, Lin Z, Zhao X. A deep learning-based multi-model ensemble method for cancer prediction. Computer Methods and Programs in Biomedicine 2018; 153: 1-9. doi: 10.1016/j.cmpb.2017.09.005
  • [37] Menze BH, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K et al. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging 2014; 34 (10): 1993-2024. doi: 10.1109/TMI.2014.2377694
  • [38] Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M et al. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific Data 2017; 4: 170117. doi: 10.1038/sdata.2017.117
  • [39] Mossa AA, Yibre AM, Çevik U. Multi-view CNN with MLP for diagnosing tuberculosis patients using CT scans and clinically relevant metadata. CLEF Working Notes 2019; 2380: 9-12.
  • [40] LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE 1998; 86 (11): 2278-2324.
  • [41] Kingma DP, Ba J. Adam: A method for stochastic optimization. arXiv 2014; arXiv:1412.6980 [cs.LG].
  • [42] Paszke A, Gross S, Chintala S, Chanan G, Yang E et al. Automatic differentiation in PyTorch. In: 31st Conference on Neural Information Processing Systems; Long Beach, CA, US; 2017. pp. 1-20.
  • [43] Chato L, Chow E, Latifi S. Wavelet transform to improve accuracy of a prediction model for overall survival time of brain tumor patients based on MRI images. In: 2018 IEEE International Conference on Healthcare Informatics (ICHI); New York, NY, USA; 2018. pp. 441-442.
  • 44] Sanghani P, Ang BT, King NK, Ren H. Overall survival prediction in glioblastoma multiforme patients from volumetric, shape and texture features using machine learning. Surgical Oncology 2018; 27 (4) :709-714. doi: 10.1016/j.suronc.2018.09.002
  • [45] Chaddad A, Daniel P, Desrosiers C, Toews M, Abdulkarim B. Novel radiomic features based on joint intensity matrices for predicting glioblastoma patient survival time. IEEE journal of Biomedical and Health Informatics 2018; 23 (2): 795-804. doi: 10.1109/JBHI.2018.2825027
  • [46] Macyszyn L, Akbari H, Pisapia JM, Da X, Attiah M et al. Imaging patterns predict patient survival and molecular subtype in glioblastoma via machine learning techniques. Neuro-oncology 2015; 18 (3): 417-425. doi: 10.1093/neuonc/nov127
  • [47] Chen X, Fang M, Dong D, Liu L, Xu X et al. Development and validation of a MRI-based radiomics prognostic classifier in patients with primary glioblastoma multiforme. Academic Radiology 2019; 26 (10): 1292-1300. doi: 10.1016/j.acra.2018.12.016
  • [48] Upadhaya T, Morvan Y, Stindel E, Le Reste PJ, Hatt M. A framework for multimodal imaging-based prognostic model building: preliminary study on multimodal MRI in glioblastoma multiforme. Innovation and Research in BioMedical Engineering 2015; 36 (6): 345-350. doi: 10.1016/j.irbm.2015.08.001
  • [49] Nie D, Zhang H, Adeli E, Liu L, Shen D. (2016) 3D deep learning for multi-modal imaging-guided survival time prediction of brain tumor patients. In: International Conference On Medical Image Computing and Computer- Assisted Intervention; Athens, Greece; 2016. pp. 212-220.
APA Mossa A, Çevik U (2021). Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans. , 616 - 631. 10.3906/elk-2002-175
Chicago Mossa Abdela Ahmed,Çevik Ulus Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans. (2021): 616 - 631. 10.3906/elk-2002-175
MLA Mossa Abdela Ahmed,Çevik Ulus Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans. , 2021, ss.616 - 631. 10.3906/elk-2002-175
AMA Mossa A,Çevik U Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans. . 2021; 616 - 631. 10.3906/elk-2002-175
Vancouver Mossa A,Çevik U Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans. . 2021; 616 - 631. 10.3906/elk-2002-175
IEEE Mossa A,Çevik U "Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans." , ss.616 - 631, 2021. 10.3906/elk-2002-175
ISNAD Mossa, Abdela Ahmed - Çevik, Ulus. "Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans". (2021), 616-631. https://doi.org/10.3906/elk-2002-175
APA Mossa A, Çevik U (2021). Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans. Turkish Journal of Electrical Engineering and Computer Sciences, 29(2), 616 - 631. 10.3906/elk-2002-175
Chicago Mossa Abdela Ahmed,Çevik Ulus Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans. Turkish Journal of Electrical Engineering and Computer Sciences 29, no.2 (2021): 616 - 631. 10.3906/elk-2002-175
MLA Mossa Abdela Ahmed,Çevik Ulus Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans. Turkish Journal of Electrical Engineering and Computer Sciences, vol.29, no.2, 2021, ss.616 - 631. 10.3906/elk-2002-175
AMA Mossa A,Çevik U Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans. Turkish Journal of Electrical Engineering and Computer Sciences. 2021; 29(2): 616 - 631. 10.3906/elk-2002-175
Vancouver Mossa A,Çevik U Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans. Turkish Journal of Electrical Engineering and Computer Sciences. 2021; 29(2): 616 - 631. 10.3906/elk-2002-175
IEEE Mossa A,Çevik U "Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans." Turkish Journal of Electrical Engineering and Computer Sciences, 29, ss.616 - 631, 2021. 10.3906/elk-2002-175
ISNAD Mossa, Abdela Ahmed - Çevik, Ulus. "Ensemble learning of multiview CNN models for survival time prediction of brain tumor patients using multimodal MRI scans". Turkish Journal of Electrical Engineering and Computer Sciences 29/2 (2021), 616-631. https://doi.org/10.3906/elk-2002-175