Yıl: 2022 Cilt: 2022 Sayı: 19 Sayfa Aralığı: 343 - 354 Metin Dili: İngilizce DOI: 10.46778/goputeb.1123902 İndeks Tarihi: 02-11-2022

Effect of Routing Methods on the Performance of Multi-Stage Tests

Öz:
In recent decades, thanks to the great advances and growing opportunities in the technology world, computer-based testing has become a popular alternative to the traditional fixed-item paper-pencil tests. Specifically, multi-stage tests (MST) which is a kind of CBT and an algorithm-based approach become a viable alternative to traditional fixed-item tests with important measurement advantages they provided. This study aimed to examine the effect of different routing rules and scoring methods under different ability distributions in MSTs. For this purpose, three different routing rule, three different ability estimation methods, and two different ability distributions were manipulated in a simulation design. Although there was no clear best method in the studied conditions, it was seen that the Kullback-Leibler was the most efficient routing method and worked best with the EAP scoring method in most of the conditions. Furthermore, EAP and BM provided higher measurement efficiency than the ML method. Recommendations for using those routing methods were provided and suggestions were made for further research.
Anahtar Kelime: Multi-Stage Testing Routing Ability Estimation

Yönlendirme Yöntemlerinin Çok Aşamalı Testler Üzerindeki Etkisi

Öz:
Özellikle son yıllarda, teknoloji dünyasındaki gelişmeler ve artan olanaklarla birlikte, bilgisayara dayalı testlerin popülerliği artmış ve bu testler geleneksel kâğıt-kalem testlerin yerine uygulanabilir bir alternatif halini almıştır. Bilgisayara dayalı testlerin bir türü olan ve algoritmaya dayalı bir yaklaşım olan Çok Aşamalı Testler de sağladıkları önemli avantajlarla birlikte kağıt-kalem testlerinin önemli bir alternatifi haline gelmiştir. Bu çalışmanın amacı, Çok Aşamalı Testlerde yönlendirme yöntemlerinin test performansına etkisinin farklı koşullar altında incelenmesidir. Bu amaçla bir simülasyon çalışması tasarlanmış, üç farklı yönlendirme kuralı, üç farklı yetenek kestirim yöntemi ve iki farklı yetenek dağılımı manipüle edilmiştir. Analizler sonucunda hem normal hem de uniform dağılım için, birçok koşulda Kullback-Leibler'in en etkili yönlendirme yöntemi olduğu ve koşulların çoğunda EAP puanlama yöntemiyle en iyi şekilde çalıştığı görülmüştür. Ayrıca, EAP ve BM yetenek kestirim yöntemleri, ML yönteminden daha yüksek ölçüm hassasiyeti sağlamıştır. En düşük ölçüm hassasiyeti ise, tesadüfi yönlendirme yönteminde elde edilmiştir. Bu yönlendirme yöntemlerinin kullanımına ve ileriki araştırmalara yönelik bazı önerilerde bulunulmuştur.
Anahtar Kelime:

Belge Türü: Makale Makale Türü: Araştırma Makalesi Erişim Türü: Erişime Açık
  • Diao, Q., & Reckase, M. (2009). Comparison of ability estimation and item selection methods in multidimensional computerized adaptive testing. In D. J. Weiss (Ed.), Proceedings of the 2009 GMAC Conference on Computerized Adaptive Testing. Retrieved [13.03.2021] from https://www.psych.umn.edu/psylabs/CATCentral
  • Haberman, S. J. & von Davier, A. A. (2014). Considerations on parameter estimation, scoring, and linking in multistage testing. In D. Yan, A. A. von-Davier & C. Lewis (Eds.), Computerized multistage testing (p. 229 – 246). CRC Press; Taylor&Francis Group.
  • Harwell, M., Stone, C. A., Hsu, T., & Kirisci, L. (1996). Monte Carlo studies in item response theory. Applied Psychological Measurement, 20(2), 101–125.
  • Hendrickson, A. (2007). An NCME instructional module on multistage testing. Educational Measurement: Issues and Practice, Summer 2007, 44-52.
  • Kim, S., Moses, T., & Yoo, H. (2015). A comparison of IRT proficiency estimation methods under adaptive multistage testing. Journal of Educational Measurement, 52(1), 70-79.
  • Lord, F. M. (1980). Applications of item response theory to practical testing problems. Mahwah, New Jersey: Routledge
  • Magis, D., Yan, D. & von-Davier, A. (Eds.). (2017). Computerized adaptive and multistage testing with R: Using packages catR and mstR. Springer.
  • Magis, D., Yan, D. & von Davier, A., A. (2018). Package ‘mstR’: Procedures to generate patterns under multistage testing. Retrieved from https://cran.microsoft.com/snapshot/2018-09-29/web/packages/mstR/mstR.pdf
  • OECD (2013). Technical report of the survey of adult skills (PIAAC). OECD Publishing: Paris, France.
  • Sarı, H. İ. (2016). Examining content control in adaptive tests: Computerized adaptive testing vs. computerized multistage testing. Unpublished doctoral dissertation. University of Florida, USA.
  • Sarı, H. İ., & Raborn, A. (2018). What information works best?: A comparison of routing methods. Applied Psychological Measurement, 42(6), 499–515. DOI: 10.1177/0146621617752990
  • Svetina, D., Liaw, Y. L., Rutkowski L. & Rutkowski, D. (2019). Routing strategies and optimizing design for multistage testing in international large-scale assessments. Journal of Educational Measurement, 56(1), 192-213.
  • Wainer, H. (2000). Introduction and history. In H. Wainer (Ed.), Computerized adaptive testing: A primer (2nd Edition), (p. 1–22). Lawrence Erlbaum Associates.
  • Wang, S., Lin, H., Chang, H. H., Douglas, J. (2016). Hybrid computerized adaptive testing: From group sequential design to fully sequential design. Journal of Educational Measurement, 53, 45-62.
  • Wang, K. (2017). A fair comparison of the performance of computerized adaptive testing and multistage adaptive testing. Unpublished doctoral dissertation. Michigan State University, USA.
  • Weissman, A., Belov, D.I., & Armstrong, R.D. (2007). Information-based versus number-correct routing in multistage classification tests (Research Report RR-07–05). Law School Admissions Council.
  • Yamamoto, K., Shin, H. J. and Khorramdel, L. (2019), Introduction of multistage adaptive testing design in PISA 2018.(OECD Education Working Papers, No. 209). OECD Publishing: Paris. DOI: 10.1787/b9435d4b-en.
  • Yan, D., Lewis, C & von-Davier, A. A. (2014). Multistage test design and scoring with small sample. In D. Yan, A. A. von-Davier & C. Lewis (Eds.), Computerized multistage testing (p. 303–324). CRC Press; Taylor&Francis Group.
  • Zenisky, A., Hambleton, R. K. & Luecht, R. M. (2010). Multistage testing: Issues, design, and research. In W. J. van der Linden & C. A.W. Glas (Eds.), Elements of adaptive testing (p. 355-372). Springer.
APA Erdem Kara B (2022). Effect of Routing Methods on the Performance of Multi-Stage Tests. , 343 - 354. 10.46778/goputeb.1123902
Chicago Erdem Kara Başak Effect of Routing Methods on the Performance of Multi-Stage Tests. (2022): 343 - 354. 10.46778/goputeb.1123902
MLA Erdem Kara Başak Effect of Routing Methods on the Performance of Multi-Stage Tests. , 2022, ss.343 - 354. 10.46778/goputeb.1123902
AMA Erdem Kara B Effect of Routing Methods on the Performance of Multi-Stage Tests. . 2022; 343 - 354. 10.46778/goputeb.1123902
Vancouver Erdem Kara B Effect of Routing Methods on the Performance of Multi-Stage Tests. . 2022; 343 - 354. 10.46778/goputeb.1123902
IEEE Erdem Kara B "Effect of Routing Methods on the Performance of Multi-Stage Tests." , ss.343 - 354, 2022. 10.46778/goputeb.1123902
ISNAD Erdem Kara, Başak. "Effect of Routing Methods on the Performance of Multi-Stage Tests". (2022), 343-354. https://doi.org/10.46778/goputeb.1123902
APA Erdem Kara B (2022). Effect of Routing Methods on the Performance of Multi-Stage Tests. Uluslararası Türk Eğitim Bilimleri Dergisi, 2022(19), 343 - 354. 10.46778/goputeb.1123902
Chicago Erdem Kara Başak Effect of Routing Methods on the Performance of Multi-Stage Tests. Uluslararası Türk Eğitim Bilimleri Dergisi 2022, no.19 (2022): 343 - 354. 10.46778/goputeb.1123902
MLA Erdem Kara Başak Effect of Routing Methods on the Performance of Multi-Stage Tests. Uluslararası Türk Eğitim Bilimleri Dergisi, vol.2022, no.19, 2022, ss.343 - 354. 10.46778/goputeb.1123902
AMA Erdem Kara B Effect of Routing Methods on the Performance of Multi-Stage Tests. Uluslararası Türk Eğitim Bilimleri Dergisi. 2022; 2022(19): 343 - 354. 10.46778/goputeb.1123902
Vancouver Erdem Kara B Effect of Routing Methods on the Performance of Multi-Stage Tests. Uluslararası Türk Eğitim Bilimleri Dergisi. 2022; 2022(19): 343 - 354. 10.46778/goputeb.1123902
IEEE Erdem Kara B "Effect of Routing Methods on the Performance of Multi-Stage Tests." Uluslararası Türk Eğitim Bilimleri Dergisi, 2022, ss.343 - 354, 2022. 10.46778/goputeb.1123902
ISNAD Erdem Kara, Başak. "Effect of Routing Methods on the Performance of Multi-Stage Tests". Uluslararası Türk Eğitim Bilimleri Dergisi 2022/19 (2022), 343-354. https://doi.org/10.46778/goputeb.1123902