From Black-Box Automated Scoring to Engaging Feedback:

A Docimological-Pedagogical Pipeline for Assessment with Local LLMs

Autori

  • Marco Giganti Università degli Studi di Bergamo
  • Marco Lazzari

Parole chiave:

valutazione automatizzata delle risposte brevi, feedback formativo, valutazione ibrida uomo-macchina

Abstract

The use of artificial intelligence in assessment raises questions about how automated judgements relate to learning goals and how students can use the feedback they receive. This paper presents the first stage of a broader research project on the assessment of written responses in Italian provided by local large language models (LLMs). We conducted a purposive narrative review of ten technical publications, retrieved through ACL Anthology, arXiv and publishers’ websites before 1 October 2026, utilizing literature on feedback and assessment practices. The synthesis shows that scoring performance depends on the task, the human reference and the evaluation setting, while evidence concerning feedback use remains limited. These findings inform a proposed five-layer pipeline in which explicit criteria and independently rated responses provide the basis for semantic analysis, local-model evaluations and teacher review. The pedagogical rationale is to support judgements whose grounds can be verified and feedback that learners can use in their revision. As the pipeline has not yet been tested, we outline a validation program addressing scoring agreement, the quality of reviewed decisions, teacher workload and students’ interpretation and use of feedback.

Riferimenti bibliografici

Aggarwal, D., Bhattacharyya, P., & Raman, B. (2024). “I understand why I got this grade”: Automatic short answer grading with feedback (Version 1) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2407.12818

Andersen, N., Mang, J., Goldhammer, F., & Zehner, F. (2025). Algorithmic fairness in automatic short an-swer scoring. International Journal of Artificial Intelligence in Education, 35, 3128–3165. https://doi.org/10.1007/s40593-025-00495-5

Baethge, C., Goldbeck-Wood, S., & Mertens, S. (2019). SANRA—a scale for the quality assessment of nar-rative review articles. Research Integrity and Peer Review, 4, Article 5. https://doi.org/10.1186/s41073-019-0064-8

Barabanti, P., Giganti, M., & Viganò, R. (in press). Teachers’ engagement with generative AI: Evidence from TALIS 2024. In Proceedings of TEEM 2026. Springer.

Carless, D., & Boud, D. (2018). The development of student feedback literacy: Enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315–1325. https://doi.org/10.1080/02602938.2018.1463354

Emirtekin, E. (2025). Large language model-powered automated assessment: A systematic review. Applied Sciences, 15(10), Article 5683. https://doi.org/10.3390/app15105683

Gaggioli, A., Casaburi, G., Ercolani, L., Collovà, F., Torre, P., & Davide, F. (2025). Assessing the reliability and validity of large language models for automated assessment of student essays in higher education [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2508.02442

Google. (n.d.). Gemma 4 model overview. Google AI for Developers. Retrieved October 1, 2026, from https://ai.google.dev/gemma/docs/core

Mayfield, E., & Black, A. W. (2020). Should you fine-tune BERT for automated essay scoring? In J. Burstein, E. Kochmar, C. Leacock, N. Madnani, I. Pilán, H. Yannakoudakis, & T. Zesch (Eds.), Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications (pp. 151–162). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.bea-1.15

Menini, S., Tonelli, S., De Gasperis, G., & Vittorini, P. (2019). Automated short answer grading: A simple solution for a difficult task. In R. Bernardi, R. Navigli, & G. Semeraro (Eds.), Proceedings of the Sixth Italian Conference on Computational Linguistics (CLiC-it 2019) (pp. 257–263). CEUR Workshop Proceedings. https://aclanthology.org/2019.clicit-1.39/

Misgna, H., On, B.-W., Lee, I., & Choi, G. S. (2025). A survey on deep learning-based automated essay scor-ing and feedback generation. Artificial Intelligence Review, 58, Article 36. https://doi.org/10.1007/s10462-024-11017-5

Perkins, M., Roe, J., & Furze, L. (2024). The AI Assessment Scale revisited: A framework for educational as-sessment (Version 1) [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2412.09029

Shibata, T., & Uto, M. (2022). Analytic automated essay scoring based on deep neural networks integrating multidimensional item response theory. In N. Calzolari, C.-R. Huang, H. Kim, J. Pustejovsky, L. Wanner, K.-S. Choi, P.-M. Ryu, H.-H. Chen, L. Donatelli, H. Ji, S. Kurohashi, P. Paggio, N. Xue, S. Kim, Y. Hahm, Z. He, T. K. Lee, E. Santus, F. Bond, & S.-H. Na (Eds.), Proceedings of the 29th In-ternational Conference on Computational Linguistics (pp. 2917–2926). International Committee on Computational Linguistics. https://aclanthology.org/2022.coling-1.257/

Taghipour, K., & Ng, H. T. (2016). A neural approach to automated essay scoring. In J. Su, K. Duh, & X. Carreras (Eds.), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (pp. 1882–1891). Association for Computational Linguistics. https://doi.org/10.18653/v1/D16-1193

Uto, M., & Uchida, Y. (2020). Automated short-answer grading using deep neural networks and item response theory. In I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Millán (Eds.), Artificial intelligence in education (pp. 334–339). Springer. https://doi.org/10.1007/978-3-030-52240-7_61

##submission.downloads##

Pubblicato

2026-11-19

Come citare

Giganti, M., & Lazzari, M. (2026). From Black-Box Automated Scoring to Engaging Feedback:: A Docimological-Pedagogical Pipeline for Assessment with Local LLMs. Journal of Inclusive Methodology and Technology in Learning and Teaching, 6(3). Recuperato da https://www.inclusiveteaching.it/index.php/inclusiveteaching/article/view/811