Using Augmented Reality and Artificial Intelligence to Improve Teaching and Learning Spatial Transformations in STEM Disciplines

AR- Classroom

AR- Classroom is an NSF-funded collaborative project. We use augmented reality (AR) and artificial intelligence (AI) to improve student's understanding of spatial transforms. Specifically, we use AI to generate an AR overlay for physical manipulatives (such as a LEGO airplane), allowing students to view the associated rotation matrices and translation vectors in real-time. Students can then use this real-time overlay to confirm their mathematical intuition about spatial transformations. Students physically interact with objects and see the impact of those actions on math calculations on AR display. This interactivity can facilitate an embodied approach to learning math.

Detailed Content Link

AR- Classroom, Workshop 1 for Virtual Rotation

AR- Classroom, Workshop 2 for Physical Rotation

Integrating Conversational AI with AR

We incorporated a Conversational AI-powered chatbot to help students with mathematical concepts, the augmented reality environment, and the functionalities of the app. We used both text and voice input to communicate with the Chatbot.

User Testing

While developing the application we have let students interact with many development builds to improve the interface and usability of the application. In the figures below we have photos of our most recent usability test with middle school-aged students. We have conducted multiple usability tests and learning experiments among undergraduate students of Texas A&M Univesity. These experiments provided promising results on the effectiveness of AR-Classroom and the contribution of Conversational AI supporting AR-Classroom.

Mixed Reality Game for learning 3D Rotation

This Mixed Reality game is divided into five levels to introduce students to different aspects of 3D rotation - Roll, Pitch, and Yaw etc.

Outreach Activity

I attended several events to showcase Augmented Reality research projects of the BIM-SIM lab to represent the Department of Architecture under the supervision of Dr. Wei Yan.

Elevating Futures event at the Brazos Valley Expo Center

Publications

AR-Classroom: Integrating Conversational Artificial Intelligence with Augmented Reality Technology for Learning Spatial Transformations and Their Matrix Representation. Accepted in IEEE FIE 2024
"Learning 3D Matrix Algebra Using Virtual and Physical Manipulatives: Statistical Analysis of Quantitative Data Evaluating the Efficacy of the AR-Classroom." International Conference on Human-Computer Interaction. Cham: Springer Nature Switzerland, 2024.
"Learning 3D matrix algebra using virtual and physical manipulatives: qualitative analysis of the efficacy of the AR-classroom." International Conference on Human-Computer Interaction. Cham: Springer Nature Switzerland, 2024.
"AR-Classroom: Investigating User-App-Interactions to Enhance Usability of AR Technology for Learning Two and Three Dimensional Rotations", International Conference on Human-Computer Interaction, 249-256, 2023, Springer
"AR-classroom: augmented reality technology for learning 3D spatial transformations and their matrix representation", 2023 IEEE Frontiers in Education Conference (FIE), 1-8, 2023, IEEE
Ar-classroom: Usability of ar educational technology for learning rotations using three-dimensional matrix algebra", 2023 IEEE Frontiers in Education Conference (FIE), 1-8, 2023, IEEE