Reliable Question Answering for Learning: When Should Educational AI Answer or Abstain?

Autori

  • Ludovico Boratto
  • Francesco Congiu
  • Gianni Fenu
  • Mirko Marras University of Cagliari
  • Giacomo Medda

Parole chiave:

Retrieval-Augmented Generation, Educational Artificial Intelligence, Question Answering, Large Language Models, Learning Support

Abstract

Retrieval-augmented generation (RAG) is increasingly relevant to educational applications in which learners use conversational artificial intelligence (AI) to access course materials, institutional documents, textbooks, and other instructor-selected resources. In these settings, a useful response should not only be plausible or factually correct: it should also be supported by the materials that define the learning context. Conversely, when those materials do not provide sufficient evidence, refraining from answering may be more appropriate than generating a fluent but unsupported explanation. In this study, we examine this relationship through an evidence-aware evaluation framework that distinguishes the availability of supporting information from the subsequent behavior of the language model. Using three question-answering benchmarks as a controlled testing environment, we evaluate six open-weight language models under a common retrieval and abstention-aware generation protocol. Results show that providing more retrieved material has asymmetric effects. When supporting evidence is available, larger contexts generally reduce unnecessary abstention; when support is absent, additional context can instead increase unsupported and hallucinated responses. Within evidence-supported contexts, unnecessary abstention is also associated with lower evidence redundancy, while the position of sparse evidence affects several models and datasets. These findings suggest that the design of educational RAG systems should not focus exclusively on retrieving more information or maximizing answer coverage. More importantly, retrieval and generation should be coordinated so that learner-facing responses remain aligned with the resources made available to the system.

Riferimenti bibliografici

An, S., Ma, Z., Lin, Z., Zheng, N., Lou, J.-G., & Chen, W. (2024). Make your LLM fully utilize the context. Advances in Neural Information Processing Systems, 37, 62160–62188.

Berant, J., Chou, A., Frostig, R., & Liang, P. (2013). Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (pp. 1533–1544).

Es, S., James, J., Espinosa Anke, L., & Schockaert, S. (2024). RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations (pp. 150–158). Association for Computational Linguistics.

Fan, W., Ding, Y., Ning, L., Wang, S., Li, H., Yin, D., Chua, T.-S., & Li, Q. (2024). A survey on RAG meeting LLMs: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 6491–6501).

Farahani, M., & Johansson, R. (2024). Deciphering the interplay of parametric and non-parametric memory in retrieval-augmented language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 16966–16977).

He, J., Pan, K., Dong, X., Song, Z., Liu, Y., Sun, Q., Liang, Y., Wang, H., Zhang, E., & Zhang, J. (2024). Never lost in the middle: Mastering long-context question answering with position-agnostic decompositional training. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (pp. 13628–13642).

Jin, Z., Cao, P., Chen, Y., Liu, K., Jiang, X., Xu, J., Li, Q., & Zhao, J. (2024). Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models. In Proceedings of LREC-COLING 2024 (pp. 16867–16878).

Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017). TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (pp. 1601–1611).

Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., ... & Petrov, S. (2019). Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7, 453-466.

Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173.

Madhusudhan, N., Madhusudhan, S. T., Yadav, V., & Hashemi, M. (2025). Do LLMs know when to NOT answer? Investigating abstention abilities of large language models. In Proceedings of the 31st International Conference on Computational Linguistics (pp. 9329–9345).

Peng, X., Choubey, P. K., Xiong, C., & Wu, C.-S. (2025). Unanswerability evaluation for retrieval augmented generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (pp. 8452–8472).

Saad-Falcon, J., Khattab, O., Potts, C., & Zaharia, M. (2024). ARES: An automated evaluation framework for retrieval-augmented generation systems. In Proceedings of NAACL 2024 (pp. 338–354).

Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y., & Xu, W. (2024). Knowledge conflicts for LLMs: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (pp. 8541–8565).

Zhang, H., Diao, S., Lin, Y., Fung, Y., Lian, Q., Wang, X., Chen, Y., Ji, H., & Zhang, T. (2024). R-Tuning: Instructing large language models to say “I don't know.” In Proceedings of NAACL 2024 (pp. 7113–7139).

##submission.downloads##

Pubblicato

2026-11-19

Come citare

Boratto, L., Congiu, F., Fenu, G., Marras, M., & Medda, G. (2026). Reliable Question Answering for Learning: When Should Educational AI Answer or Abstain?. Journal of Inclusive Methodology and Technology in Learning and Teaching, 6(3). Recuperato da https://www.inclusiveteaching.it/index.php/inclusiveteaching/article/view/878

Puoi leggere altri articoli dello stesso autore/i