(CHI'25) Designing LLM-Powered Multimodal Instructions to Support Rich Hands-on Skills Remote Learning: A Case Study with Massage Instructors and Learners

Abstract

Although remote learning is widely used for delivering and capturing knowledge, it has limitations in teaching hands-on skills that require nuanced instructions and demonstrations of precise actions, such as massage. Furthermore,scheduling conficts between instructors and learners often limit the availability of real-time feedback, reducing learning efciency. To address these challenges, we developed a synthesis tool utilizing an LLM-powered Virtual Teaching Assistant (VTA). This tool integrates multimodal instructions that convey precise data, such as stroke patterns and pressure control, while providing real-time feedback for learners and summarizing their performance for instructors. Our case study with instructors and learners demonstrated the efectiveness of these multimodal instructions and the VTA in enhancing massage teaching and learning. We then discuss the tools’ use in other hands-on skills instruction and cognitive process diferences in various courses.