Ability Estimation with Polytomous Items in Computerized Multistage Tests
Yıl: 2023 Cilt: 14 Sayı: 3 Sayfa Aralığı: 171 - 184 Metin Dili: İngilizce DOI: 10.21031/epod.1056079 İndeks Tarihi: 08-10-2023
Ability Estimation with Polytomous Items in Computerized Multistage Tests
Öz: The aim of the study is to examine how the ability estimations of individuals change under different conditions in tests consisting of polytomous items in an computerized multistage test environment. The research is a simulation study. In the study, 108 (3x3x6x2=108) conditions were examined consisting of three categories (3, 4 and 5), three test lengths (10, 20 and 30), six panel designs (1-2, 1-2-2, 1-3, 1-3-3, 1-4 and 1-4-4) and two routing methods (Maximum Fisher Information (MFI) and Random). Simulations and analyses were carried out in the mstR package in R program, with a pool of 200 items, 1000 people and 100 replications (e.g., iterations). As the outcomes of the research, mean absolute bias, RMSE and correlation values were calculated. It was found that as the number of categories and test length increase, the mean absolute bias and RMSE values decrease, while the correlation values increase. In terms of routing methods, although MFI and random methods have similar tendencies, MFI gives better results. There is a similarity between the panel designs in terms of results.
Anahtar Kelime: Belge Türü: Makale Makale Türü: Araştırma Makalesi Erişim Türü: Erişime Açık
- Chen, L-Y. (2010). An investigation of the optimal test design for multi-stage test using the generalized partial credit model. [Doctoral dissertation, The University of Texas]. UT Electronic Theses and Dissertations. https://repositories.lib.utexas.edu/handle/2152/ETD-UT-2010-12-344
- Choi, Y. J., & Asilkalkan, A. (2019). R packages for item response theory analysis: Descriptions and features. Measurement: Interdisciplinary Research and Perspectives, 17(3), 168-175. https://doi.org/10.1080/15366367.2019.1586404
- Dodd, B. G., De Ayala, R. J., & Koch, W. R. (1995). Computerized adaptive testing with polytomous items. Applied Psychological Measurement, 19(1), 5-22. https://doi.org/10.1177/014662169501900103
- Donoghue, J. R. (1994). An empirical examination of the IRT information of polytomously scored reading items under the generalized partial credit model. Journal of Educational Measurement, 31(4), 295-311. https://doi.org/10.1111/j.1745-3984.1994.tb00448.x
- Embretson, S. E., & Reise, S. P. (2013). Item response theory. Psychology Press.
- Han, K. C. T. (2022). User’s Manual: MSTGen. Retrieved from https://www.umass.edu/remp/software/simcata/mstgen/MSTGen_Manual.pdf
- Han, K. T. (2007). WinGen: Windows software that generates item response theory parameters and item responses. Applied Psychological Measurement, 31(5), 457-459. https://doi.org/10.1177/0146621607299271
- Hendrickson, A. (2007). An NCME instructional module on multistage testing. Educational Measurement: Issues and Practice, 26(2), 44-52. https://doi.org/10.1111/j.1745-3992.2007.00093.x
- ILOG. (2006). ILOG CPLEX 10.0 [User’s manual]. Paris, France: ILOG S.A. Retrieved from https://www.lix.polytechnique.fr/~liberti/teaching/xct/cplex/usrcplex.pdf
- Kim, J., Chung, H., & Dodd, B. G. (2010, May). Comparing routing methods in the multistage test based on the partial credit model [Conference presentation]. In AERA, Denver, CO.
- Kim, J., Chung, H., Park, R., & Dodd, B. G. (2013). A comparison of panel designs with routing methods in the multistage test with the partial credit model. Behavior Research Methods, 45, 1087–1098. https://doi.org/10.3758/s13428-013-0316-3
- Luecht, R. M. (2000, April). Implementing the computer-adaptive sequential testing (CAST) framework to mass produce high quality computer-adaptive and mastery tests. [Conference presentation]. In NCME, New Orleans, LA. https://eric.ed.gov/?id=ED442823
- Luecht, R. M., & Nungester, R. J. (1998). Some practical examples of computer adaptive sequential testing. Journal of Educational Measurement, 35(3), 229-249. https://doi.org/10.1111/j.1745- 3984.1998.tb00537.x
- Luo, F., Ding, S., Wang, X., & Xiong, J. (2016). Application study on online multistage intelligent adaptive testing for cognitive diagnosis. Quantitative Psychology Research, 265-275. https://doi:10.1007/978-3-319- 38759-8_20
- MacGregor, D., Yen, S. J., & Yu, X. (2022). Using multistage testing to enhance measurement of an english language proficiency test. Language Assessment Quarterly, 19(1), 54-75. https://doi.org/10.1080/15434303.2021.1988953
- Macken-Ruiz, C. L. (2008). A comparison of multi-stage and computerized adaptive tests based on the generalized partial credit model [Doctoral dissertation, The University of Texas]. ProQuest Dissertations Publishing. https://www.proquest.com/docview/304482829?pq-origsite=gscholar&fromopenview=true
- Magis, D., Yan, D., von Davier, A., & Magis, M. D. (2018). Package ‘mstR’. Retrieved from https://cran.r- project.org/web/packages/mstR/mstR.pdf
- Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. ETS Research Report Series, 1992(1), i–30. https://doi.org/10.1002/j.2333-8504.1992.tb01436.x
- Oztürk, N. B. (2019). How the Length and Characteristics of Routing Module Affect Ability Estimation in ca- MST?. Universal Journal of Educational Research, 7(1), 164-170. https://doi.org/10.13189/ujer.2019.070121
- R Core Team. (2018). R: A language and environment for statistical computing: R foundation for statistical computing.
- Ridho, A. (2022, January). Sociocultural Literacy Assessment: Validation of Multistage Generalized Partial Credit Testing Design. In International Conference on Madrasah Reform 2021 (ICMR 2021) (pp. 382-386). Atlantis Press. https://doi.org/10.2991/assehr.k.220104.056
- Rutkowski, L., Liaw, Y. L., Svetina, D., & Rutkowski, D. (2022). Multistage testing in heterogeneous populations: Some design and implementation considerations. Applied Psychological Measurement, 46(6), 494-508. https://doi.org/10.1177/01466216221108123
- Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 34 (17). https://psycnet.apa.org/record/1972-04809-001
- Sari, H. I., & Raborn, A. (2018). What information works best?: A comparison of routing methods. Applied Psychological Measurement, 42(6), 499-515. https://doi.org/10.1177/0146621617752990
- Sari, H.I., Yahsi Sarı, H., & Huggins Manley, A.C. (2016). Computer adaptive multistage testing: Practical issues, challenges and principles. Journal of Measurement and Evaluation in Education and Psychology, 7(2), 388- 406. https://doi.org/10.21031/epod.280183
- Svetina, D., Liaw, Y. L., Rutkowski, L., & Rutkowski, D. (2019). Routing strategies and optimizing design for multistage testing in ınternational large scale assessments. Journal of Educational Measurement, 56(1), 192-213. https://doi.org/10.1111/jedm.12206
- Weiss, D. J. (1982). Improving measurement quality and efficiency with adaptive testing. Applied Psychological Measurement, 6(4), 473-492. https://doi.org/10.1177/014662168200600408
- Weiss, D. J. (1983). Latent trait theory and adaptive testing. In Weiss D. J. (Ed.), New horizons in testing (pp. 5- 7). Academic Press.
- Weissman, A., Belov, D. I., Armstrong, R. D. (2007). Information-based versus number-correct routing in multistage classification tests. LSAC Research Report Series. No. 07-05. Law School Admission Council. https://www.researchgate.net/publication/237288650_Information-Based_Versus_Number- Correct_Routing_in_Multistage_Classification_Tests
- Zenisky A., Hambleton R.K.,& Luecht R.M. (2009) Multistage testing: Issues, designs, and research. In: van der Linden W., Glas C. (eds) Elements of Adaptive Testing. Springer.
- Zenisky, A. L. (2004). Evaluating the effects of several multi-stage testing design variables on selected psychometric outcomes for certification and licensure assessment (Publication No. 5710) [Doctoral dissertation, University of Massachusetts Amherst]. UMass Amherst Libraries. https://scholarworks.umass.edu/dissertations_1/5710
| APA | Yahşi Sarı H, Kelecioğlu H (2023). Ability Estimation with Polytomous Items in Computerized Multistage Tests. , 171 - 184. 10.21031/epod.1056079 |
| Chicago | Yahşi Sarı Hasibe,Kelecioğlu Hülya Ability Estimation with Polytomous Items in Computerized Multistage Tests. (2023): 171 - 184. 10.21031/epod.1056079 |
| MLA | Yahşi Sarı Hasibe,Kelecioğlu Hülya Ability Estimation with Polytomous Items in Computerized Multistage Tests. , 2023, ss.171 - 184. 10.21031/epod.1056079 |
| AMA | Yahşi Sarı H,Kelecioğlu H Ability Estimation with Polytomous Items in Computerized Multistage Tests. . 2023; 171 - 184. 10.21031/epod.1056079 |
| Vancouver | Yahşi Sarı H,Kelecioğlu H Ability Estimation with Polytomous Items in Computerized Multistage Tests. . 2023; 171 - 184. 10.21031/epod.1056079 |
| IEEE | Yahşi Sarı H,Kelecioğlu H "Ability Estimation with Polytomous Items in Computerized Multistage Tests." , ss.171 - 184, 2023. 10.21031/epod.1056079 |
| ISNAD | Yahşi Sarı, Hasibe - Kelecioğlu, Hülya. "Ability Estimation with Polytomous Items in Computerized Multistage Tests". (2023), 171-184. https://doi.org/10.21031/epod.1056079 |
| APA | Yahşi Sarı H, Kelecioğlu H (2023). Ability Estimation with Polytomous Items in Computerized Multistage Tests. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 14(3), 171 - 184. 10.21031/epod.1056079 |
| Chicago | Yahşi Sarı Hasibe,Kelecioğlu Hülya Ability Estimation with Polytomous Items in Computerized Multistage Tests. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi 14, no.3 (2023): 171 - 184. 10.21031/epod.1056079 |
| MLA | Yahşi Sarı Hasibe,Kelecioğlu Hülya Ability Estimation with Polytomous Items in Computerized Multistage Tests. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, vol.14, no.3, 2023, ss.171 - 184. 10.21031/epod.1056079 |
| AMA | Yahşi Sarı H,Kelecioğlu H Ability Estimation with Polytomous Items in Computerized Multistage Tests. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi. 2023; 14(3): 171 - 184. 10.21031/epod.1056079 |
| Vancouver | Yahşi Sarı H,Kelecioğlu H Ability Estimation with Polytomous Items in Computerized Multistage Tests. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi. 2023; 14(3): 171 - 184. 10.21031/epod.1056079 |
| IEEE | Yahşi Sarı H,Kelecioğlu H "Ability Estimation with Polytomous Items in Computerized Multistage Tests." Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 14, ss.171 - 184, 2023. 10.21031/epod.1056079 |
| ISNAD | Yahşi Sarı, Hasibe - Kelecioğlu, Hülya. "Ability Estimation with Polytomous Items in Computerized Multistage Tests". Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi 14/3 (2023), 171-184. https://doi.org/10.21031/epod.1056079 |
