Evaluating AI dialogues in MOOCs: Learner perceptions of usefulness, cognitive engagement and usability
Abstract
This paper describes a study that evaluated learner perceptions of artificial intelligence (AI)-supported dialogues integrated into six leadership-oriented massive open online courses (MOOCs). These structured interactions, delivered via Coursera Coach, were designed to prompt reflection, support personal insight, and reinforce course concepts. A mixed-methods approach was employed, combining survey data (n = 510) with thematic analysis of 285 open-ended responses. Three core dimensions were assessed: perceived usefulness, cognitive engagement, and usability. Results showed that 79 per cent of learners rated the dialogues as ‘very useful’, while 73 per cent found them ‘very easy to use’, indicating strong acceptance and accessibility. Cognitive engagement outcomes were also positive: 43 per cent reported that the dialogues challenged them to think in new ways, and 53 per cent stated they helped consolidate existing knowledge. Courses employing two AI dialogues (at the start and end) saw increased learner ratings in both usefulness and cognitive engagement over time, compared to single mid-course dialogues. Learner responses suggest that AI dialogues can offer a useful space for motivation, reflection, and applied thinking; however, feedback also identified limitations in naturalness of AI responses, onboarding clarity, and personalisation. These insights indicate that while conversational AI holds promise for supporting learner engagement at scale, its effectiveness depends on thoughtful implementation and continued refinement. This article is also included in The Business & Management Collection which can be accessed at https://hstalks.com/business/.
The full article is available to subscribers to this journal (subscription is free).
Author's Biography
Catia Costa is Online Education Enhancement Manager at the University of London Worldwide. She specialises in lifelong learning, digital pedagogy and learning design for online and distance education. Catia completed her undergraduate degree and master’s degree at the University of Lisbon, Faculty of Psychology and Educational Sciences, Portugal. She later earned a Postgraduate Certificate in Learning Technologies from the University of Leicester, UK. She is currently pursuing a PhD in education, examining how human cognitive architecture influences instructional design in digital learning environments. Her research explores the intersection of cognitive science and educational technology, with a particular interest in how artificial intelligence (AI) can be used to support cognitive stimulation. While technological tools evolve rapidly, Catia’s work emphasises the importance of grounding innovation in the brain’s relatively stable architecture, developed over millennia. She sees this as a valuable design opportunity: to harness the potential of AI not by replacing pedagogy, but by deepening our understanding of how people learn. With over a decade of experience in online and distance education, Catia designs learning environments that are not only engaging but grounded in cognitive science, with particular attention to how information is processed, retrieved and retained in long-term memory.
Citation
Costa, Catia (2026, March 1). Evaluating AI dialogues in MOOCs: Learner perceptions of usefulness, cognitive engagement and usability. In the Advances in Online Education: A Peer-Reviewed Journal, Volume 4, Issue 3. https://doi.org/10.69554/NYTR9943.
Publications LLP