Computerized Testlet Instrument to Assess Students' Conceptual Understanding of Chemistry on the Topic of Stoichiometry: Psychometric Analysis
Main Article Content
Abstract
Measuring students’ conceptual understanding is a challenge for teachers. Teachers usually use open-ended instruments to measure students’ conceptual understanding in depth. However, this instrument is considered highly subjective in the assessment and requires a relatively long time to check the answers. Computerised Testlet instruments developed by combining the advantages of open-ended and closed-ended instruments are deemed the solution to these problems. This study, therefore, aims to analyse the quality of the Computerised Testlet instrument psychometrically. The Rasch analysis model with Winsteps software was used in this study. A total of 10 Testlet items with three supporting questions were developed on the topic of stoichiometry. A total of 413 students (N = 413: 236 female, 177 male) from three different schools took tests on stoichiometry using the Testlet instrument, which was provided through a web developed previously (computerised). Rasch analysis model helped psychometric analysis regarding reliability, linear validity of students’ knowledge, and difficulty of items. This study reveals that Computerised Testlet able to identify students’ thinking processes. Besides, the psychometric analysis results showed that using Testlet to measure conceptual knowledge had good unidimensionality and reliability; it could be used to measure stoichiometric conceptual knowledge. Some questions needed to be revised as they proved unable to distinguish between high and low students’ conceptual knowledge; in general, the instrument could be used and provided a good function in analysing conceptual knowledge on the topic of stoichiometry
Article Details
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
Adams, C. M., & Callahan, C. M. (1995). The reliability and validity of a performance task for evaluating science process skills. Gifted Child Quarterly, 39(1), 14–20. https://doi.org/10.1177/001698629503900103
Ahmad, J., & Siew, N. M. (2021). Curiosity towards stem education: A questionnaire for primary school students. Journal of Baltic Science Education, 20(2), 289–304. https://doi.org/10.33225/jbse/21.20.289
Ajayi, O. V. (2017). Effect of hands-on activities on senior secondary chemistry students achievement and retention in stoichiometry in zone C of benue state. SSRN Electronic Journal, 5(8), 839–842. https://doi.org/10.2139/ssrn.2992803
Aydin, Y. Ç., & Uzuntiryaki, E. (2009). Development and psychometric evaluation of the high school chemistry self-efficacy scale. Educational and Psychological Measurement, 69(5), 868–880. https://doi.org/10.1177/0013164409332213
Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74. https://doi.org/10.1080/0969595980050102
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. R. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health. https://doi.org/10.3389/fpubh.2018.00149
Bolarinwa, O. (2015). Principles and methods of validity and reliability testing of questionnaires used in social and health science researches. Nigerian Postgraduate Medical Journal, 22(4), 195. https://doi.org/10.4103/1117-1936.173959
Brookhart, S. M. (2005). Developing measurement theory for classroom assessment purposes and uses. Educational Measurement: Issues and Practice, 22(4), 5–12. https://doi.org/10.1111/j.1745-3992.2003.tb00139.x
Cacciatore, K. L., & Sevian, H. (2006). Teaching lab report writing through inquiry: A green chemistry stoichiometry experiment for general chemistry. Journal of Chemical Education, 83(7), 1039–1041. https://doi.org/10.1021/ed083p1039
Chang, Y., & Wang, J. (2010). Examining testlet effects on the PIRLS 2006 assessment. Paper presented at the 4th IEA International Research Conference.
Chen, S. Y., & Liu, S. Y. (2020). Using augmented reality to experiment with elements in a chemistry course. Computers in Human Behavior, 111(October 2019), 106418. https://doi.org/10.1016/j.chb.2020.106418
Currell, K., & Jeukendrup, A. (2008). Validity, reliability and sensitivity of measures of sporting performance LK-. Sports Medicine TA - TT -, 38(4), 297–316. https://rug.on.worldcat.org/oclc/367041412
Deng, Y., & Wang, H. (2017). Research on evaluation of Chinese students’ competence in written scientific argumentation in the context of chemistry. Chemistry Education Research and Practice, 18(1), 127–150. https://doi.org/10.1039/c6rp00076b
DeMars, C. E. (2012). Confirming testlet effects. Applied Psychological Measurement, 36(2), 104–121.
DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). Thousand Oaks, CA: Sage.
Fahmina, S. S., Masykuri, M., Ramadhani, D. G., & Yamtinah, S. (2019). Content validity uses Rasch model on computerised testlet instrument to measure chemical literacy capabilities. AIP Conference Proceedings, 2194(December). https://doi.org/10.1063/1.5139755
Feyzíoglu, B., Demirdag, B., Akyildiz, M., & Altun, E. (2012). Developing a Science process skills test for secondary students: Validity and reliability study. Educational Sciences: Theory & Practice, 12(3), 1899-1906.
Finney, S. J. (2007). Book review: Exploratory and confirmatory factor analysis: understanding concepts and applications. Applied Psychological Measurement, 31(3), 245–248. https://doi.org/10.1177/0146621606290168
Garner, J. P. (2005). Stereotypies and other abnormal repetitive behaviors: Potential impact on validity, reliability, and replicability of scientific outcomes. ILAR Journal, 46(2), 106–117. https://doi.org/10.1093/ilar.46.2.106
Griffin, P., & Care, E. (Eds.) (2015). Assessment and teaching of 21st century skills: Methods and approach. Springer. https://doi.org/10.1007/978-94-017-9395-7
Gurel, D. K., Eryilmaz, A., & McDermott, L. C. (2015). A review and comparison of diagnostic instruments to identify students’ misconceptions in science. Eurasia Journal of Mathematics, Science and Technology Education, 11(5), 989–1008. https://doi.org/10.12973/eurasia.2015.1369a
Hambleton, R. K., & Jones, R. W. (1993). An NCME instructional module on comparison of classical test theory and item response theory and their applications to test development items: Instructional topics in educational measurement. Educational Measurement: Issues and Practice, 12(3), 38–47.
He, P., Liu, X., Zheng, C., & Jia, M. (2016). Using Rasch measurement to validate an instrument for measuring the quality of classroom teaching in secondary chemistry lessons. Chemistry Education Research and Practice, 17(2), 381–393. https://doi.org/10.1039/c6rp00004e
Hecht, M. Siegle, T. Weirich, S., Hecht, M., & Siegle, T. (2017). A model for the estimation of testlet response time to optimize test assembly in paper-and-pencil large-scale assessments. Journal for Educational Research Online Journal for Educational Research Online 9(1), 32–51. https://doi.org/10.25656/01:12965
Heredia, K., & Lewis, J. E. (2012). A psychometric evaluation of the Colorado learning attitudes about science survey for use in chemistry. Journal of Chemical Education, 89(4), 436–441. https://doi.org/10.1021/ed100590t
Huang, H. Y., & Wang, W. C. (2012). Higher order testlet response models for hierarchical latent traits and testlet-based items. Educational and Psychological Measurement, 73(3), 491–511. https://doi.org/10.1177/0013164412454431
Jin, Y., Rodriguez, C. A., Shah, L., & Rushton, G. T. (2020). Examining the psychometric [roperties of the redox concept inventory: A Rasch approach. Journal of Chemical Education, 97(12), 4235–4244. https://doi.org/10.1021/acs.jchemed.0c00479
Kapsalis, V. (2009). Implementation of an assessment system incorporating web-based parameterized questions. International Journal of Emerging Technologies in Learning, 4(3), 20–28. https://doi.org/10.3991/ijet.v4i3.884
Kelting-Gibson, L., Gallavan, N. P., St. Arnauld, E., Black, G., Cayson, A., Davis, J., Evans, K. D., Johnson, P. P., Levandowski, B., Mosley, K., Rickey, D., Shulsky, D. D., Thomas, D., Williamson, A. M., & Wolfgang, J. I. (2014). Four facets of classroom assessments: obstacles, obligations, outcomes, and opportunities. Action in Teacher Education, 36(5–6), 363–376. https://doi.org/10.1080/01626620.2014.977688
Khlaisang, J., & Koraneekij, P. (2019). Open online assessment management system platform and instrument to enhance the information, media, and ICT literacy skills of 21st century learners. International Journal of Emerging Technologies in Learning, 14(7), 111–127. https://doi.org/10.3991/ijet.v14i07.9953
Kieftenbeld, V., & Natesan, P. (2012). Recovery of graded response model parameters: A comparison of marginal maximum likelihood and Markov chain Monte Carlo estimation. Applied Psychological Measurement, 36(5), 399–419. https://doi.org/10.1177/0146621612446170
Klassen, S. (2006). Contextual assessment in science education: Background, issues, and policy. Science Education, 90(5), 820–851. https://doi.org/10.1002/sce.20150
Korayem, M. H., Hoshiar, A. K., & Ghofrani, M. (2017). Comprehensive modelling and simulation of cylindrical nanoparticles manipulation by using a virtual reality environment. Journal of Molecular Graphics and Modelling, 75, 266–276. https://doi.org/10.1016/j.jmgm.2017.06.006
Le Maire, N. V., Verpoorten, D. P., Fauconnier, M. L. S., & Colaux-Castillo, C. G. (2018). Clash of chemists: A gamified blog to master the concept of limiting reagent stoichiometry. Journal of Chemical Education, 95(3), 410–415. https://doi.org/10.1021/acs.jchemed.7b00256
Linacre, J. M. (2011). A user’s guide to Winsteps & Ministep Rasch Model computer program. Winsteps.
Liu, Y., & Hannig, J. (2017). Generalized fiducial inference for logistic graded response models. Psychometrika, 82(4), 1097-1125. https://doi.org/10.1007/s11336-017-9554-0
Lu, S., & Bi, H. (2016). Development of a measurement instrument to assess students’ electrolyte conceptual understanding. Chemistry Education Research and Practice, 17(4), 1030–1040. https://doi.org/10.1039/c6rp00137h
Luecht, R., Brumfield, T., & Breithaupt, K. (2006). A testlet assembly design for adaptive multistage tests. Applied Measurement in Education, 19(3), 189–202. https://doi.org/10.1207/s15324818ame1903_2
Luo, M., Wang, Z., Sun, D., & Wan, Z. H. (2018). Evaluating scientific reasoning ability : the design and validation of an assessment with a focus on reasoning and the use of evidence. Journal of Baltic Science Education, 19(2), 261–275. https://files.eric.ed.gov/fulltext/EJ1271010.pdf
Lutviana, E., Rahardjo, S. B., Susanti, E., Yamtinah, S., Mulyani, S., & Saputro, S. (2019). The computer-assisted testlet assessment instrument to measure students ’ learning difficulties in chemical bonding. Journal of Physics: Conference Series, 1156(012019), 1–5. https://doi.org/10.1088/1742-6596/1156/1/012019
Mahaffy, P. (2006). Moving Chemistry education into 3D: A tetrahedral metaphor for understanding chemistry union carbide award for chemical education 1. Journal of Chemical Education, 83(1), 49–55.
Marzano, R. J., Pickering, D., & McTighe, J. (1993). Assessing student outcomes: performance assessment using the dimensions of learning model. Association for Supervision and Curriculum Develpment. https://doi.org/ED419696
Nedungadi, S., Paek, S. H., & Brown, C. E. (2019). Utilizing Rasch analysis to establish the psychometric properties of a concept inventory on concepts important for developing proficiency in organic reaction mechanisms. Chemistry Teacher International, 0(0), 1–10. https://doi.org/10.1515/cti-2019-0004
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGrawHill.
Paek, I., Assessment, H., & Wilson, M. (2009). Random parameter structure and the testlet model : Extension of the Rasch testlet model. Journal of Applied Measurement, 10(4), 394–407.
Pentecost, T. C., & Barbera, J. (2013). Measuring learning gains in chemical education: A comparison of two methods. Journal of Chemical Education, 90(7), 839–845. https://doi.org/10.1021/ed400018v
Rachmatullah, A., Diana, S., & Ha, M. (2017). The effects of curriculum, gender and students’ favorite science subject on Indonesian high-school students’ conceptions of learning science. Journal of Baltic Science Education, 16(5), 797–812.
Rachmatullah, A., Diana, S., & Ha, M. (2018). Identifying Indonesian upper-secondary school students’ orientations to learn science and gender effect through the use of structural equation modeling. Journal of Baltic Science Education, 17(4), 633–648. https://doi.org/10.33225/jbse/18.17.633
Sabah, S., & Hammouri, H. (2009). Validation of a scale of attitudes toward science across countries using Rasch model: Findings from TIMSS. Journal of Baltic Science Education, 12(5), 692–702. https://doi.org/10.33225/jbse/13.12.692
Scantlebury K., Boone W., Kahle J. B., & Fraser B. J. (2001). Design, validation, and use of an evaluation instrument for monitoring systemic reform. Journal of Research in Science Teaching, 38, 646–662. https://doi.org/10.1002/tea.1024
Shidiq, A. S., Yamtinah, S., & Masykuri, M. (2019). Identifying and addressing students’ learning difficulties in hydrolysis using testlet instrument. AIP Conference Proceedings, 2194(020117), 1–8. https://doi.org/10.1063/1.5139849
Shidiq, A. S., Yamtinah, S., & Masykuri, M. (2016). Assessing science process skills using testlet instrument. Assessment for Improving Students’ Performance, 231–234.
Sumintono, B., & Widhiarso, W. (2014). Aplikasi model rasch untuk penelitian ilmu-ilmu sosial. Trim Komunikata Publishing House.
Syang, A., & Dale, N. B. (1993). Computerised adaptive testing in computer science: Assessing Student programming abilities. ACM SIGCSE Bulletin, 25(1), 53–56. https://doi.org/10.1145/169073.169109
Tangio, J. S. (2019). Issn 1648-3898 Issn 2538-7138 Analytic approach of response pattern of diagnostic test items in evaluating students ’ conceptual understanding of characteristics of particle of matter. Journal of Baltic Science Education, 19(5) 824–841. http://www.scientiasocialis.lt/jbse/?q=node/905
Tarhan, L., & Acar-Sesen, B. (2013). Problem based learning in acids and bases: learning achievements and students’ beliefs. Journal of Baltic Science Education, 12(5), 565-578. https://.doi.org/10.33225/jbse/13.12.565
Taskin, V., Bernholt, S., & Parchmann, I. (2015). An inventory for measuring student teachers’ knowledge of chemical representations: design, validation, and psychometric analysis. Chemistry Education Research and Practice, 16(3), 460–477. https://doi.org/10.1039/c4rp00214h
Taylor, A., & Watson, S. B. (2000). The effects of traditional classroom assessment on science learning and understanding of the processes of science. Journal of Elementary Science Education, 12(1), 19–32.
Trail, C., & Howe, A. C. (1994). The mole concept: Developing an instrument to assess conceptual understanding, Journal of Chemical Education, 71(8), 653–655.
Van Bramer, S., Fisher, M., Cooper, M. A., Elzerman, A. W., Lee, C. M., Zeile, J. V., Jones, L. L., Dorhout, P. K., Kingsbury, C., Schelble, S., Zielinski, T. J., & Schwenz, R. W. (2001). Symposium Report: Teaching chemistry in the new century. Journal of Chemical Education, 78(9), 1167–1174. https://doi.org/10.1021/ed078p1171
Wainer, H., Bradlow, E. T., & Wang, X. (2013). Testlet response theory and its applications. In Testlet Response Theory and Its Applications. Cambridge University Press. https://doi.org/10.1017/CBO9780511618765.002
Wainer, H., Bradlow, E. T., & Wang, X. (2007). Testlet response theory and its applications. Cambridge University Press. https://doi.org/10.1017/CBO9780511618765
Wainer, H., & Kiely, G. L. (1987). Item clusters and computerized adaptive testing: A case for testlets. Journal of Educational Measurement, 24(3), 185–201.
Wang, W. C., & Wilson, M. (2005). The Rasch testlet model. Applied Psychological Measurement, 29(2), 126–149. https://doi.org/10.1177/0146621604271053
Wei, S., Liu, X., Wang, Z., & Wang, X. (2012). Using rasch measurement to develop a computer modeling-based instrument to assess students’ conceptual understanding of matter. Journal of Chemical Education, 89(3), 335–345. https://doi.org/10.1021/ed100852t
Xiaohui, W., Bradlow, E. T., & Wainer, H. (2002). Model for Testlets : A general bayesian model for testlets: theory and applications (Issue 98). Educational Testing Service.
Yamtinah, S., Indriyanti, N. Y. Y., Saputro, S., Mulyani, S., Ulfa, M., Mahardiani, L., Satriana, T., & Shidiq, A. S. (2019). The identification and analysis of students’ misconception in chemical equilibrium using computerised two-tier multiple-choice instrument. Journal of Physics: Conference Series, 1157(4), 042015. https://doi.org/10.1088/1742- 6596/1157/4/042015
Yamtinah, S., Masykuri, M., Ashadi, & Shidiq, A. S. (2017). Gender differences in students’ attitudes toward science: An analysis of students’ science process skill using testlet instrument. AIP Conference Proceedings, 1868030003. https://doi.org/10.1063/1.4995102
Yamtinah, S., Saputro, S., Mulyani, S., & Shidiq, A. S. (2021). Computerized testlet instrument for assessing students’ chemical literacy in high school. Journal of Hunan University Natural Sciences, 48(2), 156–164.
Yamtinah, S., Saputro, S., Mulyani, S., Ulfa, M., Lutviana, E., & Shidiq, A. S. (2019). Do students have enough scientific literacy? A computerised testlet instrument for measuring students’ scientific literacy. AIP Conference Proceedings, 2194, 020143. https://doi.org/10.1063/1.5139875
Zhu, X., & Stone, C. A. (2012). Bayesian comparison of alternative graded response models for performance assessment applications. Educational and Psychologica, 72(5), 774– 799. https://doi.org/10.1177/0013164411434638