LumiNote explores how Large Language Models (LLMs) can assist instructors in Virtual Reality (VR) environments to teach complex topics like stage lighting, transforming spoken instructions into interactive learning aids.

In plain terms

Teaching complex skills like stage lighting involves abstract concepts, technical details, and making them understandable for learners. Virtual Reality (VR) helps remove physical limits, but existing VR systems don't offer much support for live instruction from teachers. Researchers developed LumiNote, a VR system that uses a Large Language Model (LLM), which is a type of AI that understands and generates human language. Instructors can speak their teaching intentions, and the LLM translates these into helpful tools within VR, including visual notes, interactive demonstrations, and text-based explanations. An exploratory study with 3 instructors and 24 students showed that LLMs were most useful for general, "expressive" teaching goals, though expert intervention was still needed for very specific tasks. While instructors appreciated refining the AI's suggestions, externalized expert reasoning sometimes didn't align with novice comprehension.