A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study
Yıl: 2022 Cilt: 14 Sayı: 3 Sayfa Aralığı: 137 - 146 Metin Dili: İngilizce DOI: 10.5336/biostatic.2022-89951 İndeks Tarihi: 13-05-2023
A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study
Öz: Objective: In studies conducted to detect a disease, making a false negative decision in cases such as detecting a deadly disease (Case I), or making a false positive decision in cases where diseases with high treatment costs (Case II) can lead to dangerous results. In this study, a new definition of the area under the curve (AUC) is proposed using a safety threshold value t for the diagnostic test to provide flexible decisions in critical cases. The alternative cut-off point for test diagnosis is evaluated by a simulation study in terms of sensitivity/specificity and relative efficiency. Materials and Methods: A simulation study was performed using different AUC values to obtain the cut-off point c shifted towards c-t for Case I and c+t for Case II. The normal distribution is used for the diseased (X) and non-diseased (Y) data. When obtaining the shift amount t, the gamma probability, which is the desired percentage of increase or decrease in the sensitivity/specificity value, is taken into account. Results: The results of our study showed that the relative efficiency is not significantly affected by working with the safety threshold t value when the test is less accurate and has a low AUC value. Conclusion: In this study, alternative cut-off points are obtained using the shift amount t determined by a predefined gamma probability. It is suggested that in critical situations, using the extra safety threshold t, determining the actual disease margin and safety standards for subjects can provide a more tolerant decision, especially in tests with low discrimination power.
Anahtar Kelime: Ekstra Güvenlik Eşik Değeri ile Karar Vermede Eğri Altında Kalan Alan İçin Yeni Bir Bakış Açısı: Bir Simülasyon Çalışması
Öz: Amaç: Bir hastalığı saptamak için yapılan çalışmalarda, ölümcül bir hastalığın saptanması gibi durumlarda yanlış negatif karar verilmesi (Durum I), veya tedavi maliyetinin yüksek olduğu hastalıklarda yanlış pozitif karar verilmesi (Durum II) tehlikeli sonuçlara yol açabilir. Bu çalışmada, kritik durumlarda esnek karar sağlamak için tanı testi için bir güvenlik eşik değeri t kulla- nılarak eğri altındaki alanın (AUC) yeni bir tanımı önerilmiştir. Test tanısı için alternatif kesme noktası, duyarlılık/özgüllük ve göreli etkinlik açısından bir simülasyon çalışması ile değerlendi- rilmiştir. Gereçler ve Yöntemler: Durum I için c-t'ye ve Durum II için c+t'ye kaydırılan kesme noktası c'yi elde etmek için farklı AUC değerleri kullanılarak bir simülasyon çalışması yapılmıştır. Hasta (X) ve hastalıklı olmayan (Y) için normal dağılım kulla- nılmıştır. Kaydırma miktarı t elde edilirken, duyarlılık/özgüllük değerinde yüzde olarak istenen artış veya azalış miktarı olan ga- ma olasılığı dikkate alınır. Bulgular: Çalışmamızın sonuçları, testin daha az doğru ve düşük AUC değerine sahip olduğu durum- larda, güvenlik eşiği t değeriyle çalışmanın göreli etkinliğinin önemli ölçüde etkilenmediğini göstermiştir. Sonuç: Bu çalışma- da, önceden tanımlanmış bir gama olasılığı ile belirlenen kaydır- ma miktarı t kullanılarak alternatif kesme noktaları elde edilmiş- tir. Kritik durumlarda, ekstra güvenlik eşiği t kullanılarak, denek- ler için gerçek hastalık sınırı ve güvenlik standartlarının belir- lenmesi, özellikle ayırt etme gücü düşük testlerde daha toleranslı bir karar sağlayabileceği önerilmektedir.
Anahtar Kelime: Belge Türü: Makale Makale Türü: Araştırma Makalesi Erişim Türü: Erişime Açık
- 1. Altman DG, Bland JM. Diagnostic tests 3: receiver operating characteristic plots. BMJ. 1994;309(6948):188. [Crossref] [PubMed] [PMC]
- 2. Metz CE. Basic principles of ROC analysis. Semin Nucl Med. 1978;8(4):283-98. [Crossref] [PubMed]
- 3. Obuchowski NA, Lieber ML, Wians FH Jr. ROC curves in clinical chemistry: uses, misuses, and possible solutions. Clin Chem. 2004;50(7):1118- 25. [Crossref] [PubMed]
- 4. Swets JA. Measuring the accuracy of diagnostic systems. Science. 1988;240(4857):1285-93. [Crossref] [PubMed]
- 5. Swets JA, Dawes RM, Monahan J. Better decisions through science. Sci Am. 2000;283(4):82-7. [Crossref] [PubMed]
- 6. Park SH, Goo JM, Jo CH. Receiver operating characteristic (ROC) curve: practical review for radiologists. Korean J Radiol. 2004;5(1):11- 8. [Crossref] [PubMed] [PMC]
- 7. Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem. 1993;39(4):561- 77. [Crossref] [PubMed]
- 8. Swets JA. ROC analysis applied to the evaluation of medical imaging techniques. Invest Radiol. 1979;14(2):109-21. [Crossref] [PubMed]
- 9. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29- 36. [Crossref] [PubMed]
- 10. Hajian-Tilaki K. Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation. Caspian J Intern Med. 2013;4(2):627- 35. [PubMed] [PMC]
- 11. Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psy- chology. 1975;12(4):387-415. [Crossref]
- 12. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonpara- metric approach. Biometrics. 1988;44(3):837-45. [Crossref] [PubMed]
- 13. Faraggi D, Reiser B. Estimation of the area under the ROC curve. Stat Med. 2002;21(20):3093-106. [Crossref] [PubMed]
- 14. Schisterman EF, Faraggi D, Reiser B, Hu J. Youden Index and the optimal threshold for markers with mass at zero. Stat Med. 2008;27(2):297- 315. [Crossref] [PubMed] [PMC]
- 15. Liu X. Classification accuracy and cut point selection. Stat Med. 2012;31(23):2676-86. [Crossref] [PubMed]
- 16. López-Ratón M, Rodríguez-Álvarez MX, Cadarso-Suárez C, Gude-Sampedro F. Optimal Cutpoints: An R Package for Selecting Optimal Cutpoints in Diagnostic Tests. J Stat Soft. 2014;61(8):1-36. [Crossref]
- 17. Habibzadeh F, Habibzadeh P, Yadollahie M. On determining the most appropriate test cut-off value: the case of tests with continuous results. Biochem Med (Zagreb). 2016;26(3):297-307. [Crossref] [PubMed] [PMC]
- 18. Weng CG, Poon J. A new evaluation measure for imbalanced datasets. Proceedings of the 7th Australasian Data Mining Conference. Australia: 27-28 November 2008 V87. (AusDM'08). p.27-32. [Link]
- 19. Yu W, Chang YI, Park E. A modified area under the ROC curve and its application to marker selection and classification. J Korean Stat Soc. 2014;43(2):161-75. [Crossref]
- 20. Yu W, Park T. Two simple algorithms on linear combination of multiple biomarkers to maximize partial area under the ROC curve. Computational Statis- tics & Data Analysis. 2015;88:15-27. [Crossref]
- 21. Hlatky MA, Pryor DB, Harrell FE Jr, Califf RM, Mark DB, Rosati RA. Factors affecting sensitivity and specificity of exercise electrocardiography. Multivari- able analysis. Am J Med. 1984;77(1):64-71. [Crossref] [PubMed]
- 22. Brenner H, Gefeller O. Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence. Stat Med. 1997;16(9):981- 91. [Crossref] [PubMed]
- 23. Moons KG, van Es GA, Deckers JW, Habbema JD, Grobbee DE. Limitations of sensitivity, specificity, likelihood ratio, and bayes' theorem in assessing diagnostic probabilities: a clinical example. Epidemiology. 1997;8(1):12-7. [Crossref] [PubMed]
- 24. Ridker PM, Cook NR, Lee IM, Gordon D, Gaziano JM, Manson JE, et al. A randomized trial of low-dose aspirin in the primary prevention of cardiovascu- lar disease in women. N Engl J Med. 2005;352(13):1293-304. [Crossref] [PubMed]
- 25. Cook NR. Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation. 2007;115(7):928-35. [Crossref] [PubMed]
- 26. Akobeng AK. Understanding diagnostic tests 3: Receiver operating characteristic curves. Acta Paediatr. 2007;96(5):644-7. [Crossref] [PubMed]
- 27. Greiner M, Pfeiffer D, Smith RD. Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests. Prev Vet Med. 2000;45(1-2):23-41. [Crossref] [PubMed]
- 28. Fluss R, Faraggi D, Reiser B. Estimation of the Youden Index and its associated cutoff point. Biom J. 2005;47(4):458-72. [Crossref] [PubMed]
- 29. Schisterman EF, Perkins NJ, Liu A, Bondell H. Optimal cut-point and its corresponding Youden Index to discriminate individuals using pooled blood sam- ples. Epidemiology. 2005;16(1):73-81. [Crossref] [PubMed]
- 30. Lippitt CD, Rogan J, Toledano J, Sangermano F, Eastman JR, Ronald J, et al. Incorporating anthropogenic variables into a species distribution model to map gypsy moth risk. Ecological Modelling. 2008;210:339-50. [Crossref]
- 31. Karaismailoglu E, Konar NM, Goksuluk D, Karaagaoglu AE. Factors effecting the model performance measures area under the ROC curve, net reclassi- fication improvement and integrated discrimination improvement. Communications in Statistics-Simulation and Computation. 2019;48(9):2586- 98. [Crossref]
- 32. Thompson ML, Zucchini W. On the statistical analysis of ROC curves. Stat Med. 1989;8(10):1277-90. [Crossref] [PubMed]
- 33. McClish DK. Analyzing a portion of the ROC curve. Med Decis Making. 1989;9(3):190-5. [Crossref] [PubMed]
- 34. Perkins NJ, Schisterman EF. The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. Am J Epidemiol. 2006;163(7):670-5. [Crossref] [PubMed] [PMC]
| APA | GÜRLER S, ege oruc o, Goksuluk D, SUNER A (2022). A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study. , 137 - 146. 10.5336/biostatic.2022-89951 |
| Chicago | GÜRLER SELMA,ege oruc ozlem,Goksuluk Dincer,SUNER ASLI A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study. (2022): 137 - 146. 10.5336/biostatic.2022-89951 |
| MLA | GÜRLER SELMA,ege oruc ozlem,Goksuluk Dincer,SUNER ASLI A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study. , 2022, ss.137 - 146. 10.5336/biostatic.2022-89951 |
| AMA | GÜRLER S,ege oruc o,Goksuluk D,SUNER A A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study. . 2022; 137 - 146. 10.5336/biostatic.2022-89951 |
| Vancouver | GÜRLER S,ege oruc o,Goksuluk D,SUNER A A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study. . 2022; 137 - 146. 10.5336/biostatic.2022-89951 |
| IEEE | GÜRLER S,ege oruc o,Goksuluk D,SUNER A "A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study." , ss.137 - 146, 2022. 10.5336/biostatic.2022-89951 |
| ISNAD | GÜRLER, SELMA vd. "A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study". (2022), 137-146. https://doi.org/10.5336/biostatic.2022-89951 |
| APA | GÜRLER S, ege oruc o, Goksuluk D, SUNER A (2022). A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study. Türkiye Klinikleri Biyoistatistik Dergisi, 14(3), 137 - 146. 10.5336/biostatic.2022-89951 |
| Chicago | GÜRLER SELMA,ege oruc ozlem,Goksuluk Dincer,SUNER ASLI A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study. Türkiye Klinikleri Biyoistatistik Dergisi 14, no.3 (2022): 137 - 146. 10.5336/biostatic.2022-89951 |
| MLA | GÜRLER SELMA,ege oruc ozlem,Goksuluk Dincer,SUNER ASLI A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study. Türkiye Klinikleri Biyoistatistik Dergisi, vol.14, no.3, 2022, ss.137 - 146. 10.5336/biostatic.2022-89951 |
| AMA | GÜRLER S,ege oruc o,Goksuluk D,SUNER A A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study. Türkiye Klinikleri Biyoistatistik Dergisi. 2022; 14(3): 137 - 146. 10.5336/biostatic.2022-89951 |
| Vancouver | GÜRLER S,ege oruc o,Goksuluk D,SUNER A A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study. Türkiye Klinikleri Biyoistatistik Dergisi. 2022; 14(3): 137 - 146. 10.5336/biostatic.2022-89951 |
| IEEE | GÜRLER S,ege oruc o,Goksuluk D,SUNER A "A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study." Türkiye Klinikleri Biyoistatistik Dergisi, 14, ss.137 - 146, 2022. 10.5336/biostatic.2022-89951 |
| ISNAD | GÜRLER, SELMA vd. "A New Perspective for Area Under the Curve in Decision Making with Extra Safety Threshold Value: A Simulation Study". Türkiye Klinikleri Biyoistatistik Dergisi 14/3 (2022), 137-146. https://doi.org/10.5336/biostatic.2022-89951 |
