Uncovering students’ views through cognitive interviews: Validation procedure for statistical literacy test items
Keywords:
Statistical literacy, Test items, Validation, Cognitive interviewAbstract
High school students should be prepared and reasonably expected to be critical data-based information readers. Their critical responses are a strong predictor of statistical literacy (SL) and thus a cohesive picture of it is needed to better inform educational stakeholders. In response to this notion, it is essential to use quality instruments to measure students’ SL. This study aims to describe the careful and systematic processes of developing and validating an alternative assessment to measure high school students’ SL. Each item in the assessment assessed one of the SL skills (interpreting, communicating, evaluating, and decision-making) and the item features contain the three SL components. The evidence for reliability and validity was collected through a series of cognitive interviews (cognitive interviews I and II). Cognitive interview I involved two 9-grade and two 12-grade students with average levels of knowledge in each grade. In comparison, cognitive interview II involved six 9-grade and six 12-grade students with different levels of knowledge (2 low, 2 medium, and 2 high level students). Furthermore, cognitive interview II was administered after the students solved the test items under test condition. Results confirm that ten items of the SL instrument were valid to be used for data collection. Implication of this study includes a comprehensive process of validating SL test items that contribute to the assessment theory, particularly on SL assessment in Indonesia.
References
Bailey, N. G. & McCulloch, A. W. (2023). Describing critical statistical literacy habits of mind. The Journal of Mathematical Behavior, 70, 101063. https://doi.org/10.1016/j.jmathb.2023.101063
Beatty, P. C. & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287–311. https://doi.org/10.1093/poq/nfm006
Ben-Zvi, D. & Garfield, J. B. (2004). Statistical literacy, reasoning, and thinking: Goals, definitions, and challenges. In D. Ben-Zvi, & J. B. Garfield (Eds.), The challenge of developing statistical literacy, reasoning and thinking (pp. 3–16). Kluwer Academic Publishers. https://doi.org/10.1007/1-4020-2278-6_1
Budgett, S. & Pfannkuch, M. (2010). Assessing students’ statistical literacy. In P. Bidgood, N. Hunt & F. Jolliffe (Eds.), Assessment methods in statistical education: An international perspective (pp. 103–121). https://doi.org/10.1002/9780470710470.ch9
Budgett, S. & Renelle, A. (2023). Statistical literacy–the golden rules a review of Tim Harford’s The Data Detective: Ten easy rules to make sense of statistics. The Mathematics Enthusiast, 20(1), 256-265. https://doi.org/10.54870/1551-3440.1612
Budgett, S. & Rose, D. (2017). Developing statistical literacy in the final school year. Statistics Education Research Journal, 16(1), 139–162. https://doi.org/10.52041/serj.v16i1.221
Büscher, C. (2022). Design principles for developing statistical literacy in middle schools. Statistics Education Research Journal, 21(1), 1-16. https://doi.org/10.52041/serj.v21i1.80
Callingham, R. & Watson, J. (2017). The development of statistical literacy at school. Statistics Education Research Journal, 16(1), 181–201. https://doi.org/10.52041/serj.v16i1.223
Conrad, F. & Blair, J. (1996). From impressions to data: Increasing the objectivity of cognitive interviews. In Proceedings of the Section on Survey Research Methods, Annual Meetings of the American Statistical Association (Vol. 1, No. 10). American Statistical Association. https://www.bls.gov/osmr/research-papers/1996/pdf/st960080.pdf
Cui, Y., Chen, F., Lutsyk, A., Leighton, J. P. & Cutumisu, M. (2023). Data literacy assessments: a systematic literature review. Assessment in Education: Principles, Policy & Practice, 30(1), 76-96. https://doi.org/10.1080/0969594X.2023.2182737
Desimone, L. M. & Le Floch, K. C. (2004). Are we asking the right questions? Using cognitive interviews to improve surveys in education research. Educational Evaluation and Policy Analysis, 26(1), 1–22. https://doi.org/10.3102%2F01623737026001001
Franklin, C., Kader, G., Mewborn, D., Moreno, J., Peck, R., Perry, M. & Scheaffer, R. (2005). Guidelines for Assessment and Instruction in Statistics Education (GAISE) report. American Statistical Association. https://www.amstat.org/asa/files/pdfs/GAISE/GAISEPreK-12_Full.pdf
Gal, I. (2002). Adults’ statistical literacy: Meanings, components, responsibilities. International Statistical Review, 70(1), 1–25. https://doi.org/10.2307/1403713
Gal, I. (2019). Understanding statistical literacy: About knowledge of contexts and models. In J. M. Contreras, M. M. Gea, M. M. López-Martín & E. Molina-Portillo (Eds.), Actas del Tercer Congreso Internacional Virtual de Educación Estadística. https://www.ugr.es/local/fqm126/civeest.html
Guler, M., Gursoy, K. & Guven, B. (2016). Critical views of 8th Grade students toward statistical data in newspaper articles: Analysis in light of statistical literacy. Cogent Education, 3(1), 1268773. https://doi.org/10.1080/0020739X.2018.1431849
Helenius, R., D’Amelio, A., Campos, P. & Macfeely, S. (2020). ISLP country coordinators as ambassadors of statistical literacy and innovations. Statistics Education Research Journal, 19(1), 120-136. https://doi.org/10.52041/serj.v19i1.125
Jureckova, M. & Csachova, L. (2020). Statistical literacy of Slovak lower secondary school students. Technium Social Sciences Journal, 9, 163-173. http://dx.doi.org/10.47577/tssj.v9i1.966
Koga, S. (2022). Lessons aimed at demonstrating statistical literacy skills: A case study of Japanese high school lessons on reading statistical reports. In S. A. Peters, L. Zapata-Cardona, F. Bonafini, & A. Fan (Eds.), Bridging the gap: Empowering & educating today’s learners in statistics. International Association for Statistical Education. https://doi.org/10.52041/iase.icots11.T7B1
Kurnia, A. B., Lowrie, T., & Patahuddin, S. M. (2024). The development of high school students’ statistical literacy across grade level. Mathematics Education Research Journal, 36(Suppl 1), 7-35. https://doi.org/10.1007/s13394-023-00449-x
Magaldi, D., & Berler, M. (2020). Semi-structured interviews. In: Zeigler-Hill, V., Shackelford, T.K. (Eds), Encyclopedia of personality and individual differences (pp. 4825–4830). Springer, Cham. https://doi.org/10.1007/978-3-319-24612-3_857
Mooney, E. S. (2002). A framework for characterizing middle school students’ statistical thinking. Mathematical Thinking and Learning, 4(1), 23–63. https://doi.org/10.1207/S15327833MTL0401_2
OECD. (2004). Learning for tomorrow’s world: First results from PISA 2003. OECD Publishing.
OECD. (2014, February). PISA 2012 results: What students know and can do—student performance in mathematics, reading and science (Vol. I, rev. ed.). OECD Publishing. https://doi.org/10.1787/9789264208780-en
OECD. (2023). PISA 2022 results (volume I): The state of learning and equity in education. OECD Publishing. https://doi.org/10.1787/53f23881-en
Ralston, N. C., Li, M. & Taylor, C. (2018). The development and initial validation of an assessment of algebraic thinking for students in the elementary grades. Educational Assessment, 23(3), 211–227. https://doi.org/10.1080/10627197.2018.1483191
Reinhart, A., Evans, C., Luby, A., Orellana, J., Meyer, M., Wieczorek, J., Elliot, P., Burckhardt, P. & Nugent, R. (2022). Think-aloud interviews: A tool for exploring student statistical reasoning. Journal of Statistics and Data Science Education, 30(2), 100-113. https://doi.org/10.1080/26939169.2022.2063209
Sabbag, A., Garfield, J. & Zieffler, A. (2018). Assessing statistical literacy and statistical reasoning: The REALI instrument. Statistics Education Research Journal, 17(2), 141-160. https://doi.org/10.52041/serj.v17i2.163
Schield, M. (2017). GAISE 2016 promotes statistical literacy. Statistics Education Research Journal, 16(1), 50-54. http://iase-web.org/Publications.php?p=SERJ
Shafer, K. & Lohse, B. (2005). How to conduct a cognitive interview: A nutrition education example. National Institute of Food and Agriculture, U.S. Department of Agriculture.
Sharma, S. (2017). Definitions and models of statistical literacy: A literature review. Open Review of Educational Research, 4:1, 118-133. https://doi.org/10.1080/23265507.2017.1354313
Sharma, S. (2013). Developing statistical literacy with Year 9 students: a collaborative research project. Research in Mathematics Education, 15(2), 203-204. https://doi.org/10.1080/14794802.2013.797742
Sutherland, M., Fainstein, D., Lesner, T., Kimmel, G. L., Clarke, B. & Doabler, C. T. (2022). Teaching statistical literacy and data analysis to students with mathematics difficulties. Teaching Exceptional Children, 1-11. https://doi.org/10.1177/00400599221118647
Tiruneh, D. T., De Cock, M., Weldeslassie, A. G., Elen, J. & Janssen, R. (2017). Measuring critical thinking in physics: Development and validation of a critical thinking test in electricity and magnetism. International Journal of Science and Mathematics Education, 15(4), 663–682. https://doi.org/10.1007/s10763-016-9723-0
Wallman, K. K. (1993). Enhancing statistical literacy: Enriching our society. Journal of the American Statistical Association, 88(421), 1–8. https://doi.org/10.1080/01621459.1993.10594283
Watson, J. & Callingham, R. (2003). Statistical literacy: A complex hierarchical construct. Statistics Education Research Journal, 2(2), 3–46. https://doi.org/10.52041/serj.v2i2.553
Weathington, B. L., Cunningham, C. J. & Pittenger, D. J. (2010). Research methods for the behavioral and social sciences. Wiley.
Willis, G. B. (1999). Cognitive interviewing: A ‘how to’ guide. Research Triangle Institute.
Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage.
Ziegler, L. & Garfield, J. (2018). Developing a statistical literacy assessment for the modern introductory statistics course. Statistics Education Research Journal, 17(2), 161–178. https://doi.org/10.52041/serj.v17i2.164
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Achmad Badrun Kurnia
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Abstract views: 0
,
PDF Downloads: 0
