XR Medical Simulations
Immersive XR medical training simulation on Meta Quest 3, delivering step-by-step procedural learning with spatial anchors, voice commands, and rich multimedia.
Client
Softura (for University of Michigan)
Platform
Meta Quest 3
Engine
Unity (XR Interaction Toolkit)
Procedure/Course Flow — immersive, step-by-step training experience with module navigation, media playback, and interactive XR content.
Project Overview
XR Medical Simulations was developed for Softura, for their client, the University of Michigan (UofM), as an immersive XR medical training simulation on the Meta Quest 3. The project aimed to deliver a step-by-step training experience for medical students and residents, combining interactive 3D content, rich multimedia, and XR interactions to enhance learning and procedural skill acquisition.
Goals & Challenges
The project aimed to deliver a scalable, data-driven XR training system with immersive interactions, integrated media, and robust client-side functionality while addressing platform, media and SDK complexities.
Project Goals
- •Deliver immersive XR simulation platform guiding users step-by-step through procedures
- •Build structured courses → modules → tasks → steps for clear procedural learning flows
- •Provide rich learning content including 2D/3D graphics, audio, video, narration, and quizzes
- •Implement XR interactions: ray/hand inputs, voice commands, spatial anchors, object recognition
- •Implement XR UI panels that remain readable and stable in passthrough environments
Challenges Encountered
- •Front-end integration with provided backend API without direct backend control
- •Real-time XR interactions with reliable object recognition in immersive environments
- •Multi-format media management while maintaining XR-acceptable performance
- •Implementing OAuth and deep-linking with consistent behavior across platforms
- •Shifting XR SDK and system software landscape requiring continuous adaptation
Our Approach & Solution
The team implemented a data-driven XR structure and reusable media/interaction systems, translating design and backend requirements into functional XR experiences with modularity, scalability, and immersive guidance.
Design & Development
- •Implemented JSON schema support for runtime procedural content and media definitions
- •Built data-driven XR structure (courses → modules → tasks → steps) with progress tracking
- •Unified media delivery (2D/3D, audio/video, narration, quizzes) into reusable panel systems
- •Integrated XR interaction and spatial anchoring for guided, step-by-step experiences
Technologies & Tools
- •Unity Input System, Addressables, UniTask
- •Unity XR and Oculus/Meta SDKs for XR, anchors, and voice
- •AVPro Video for media playback
- •Best.HTTP and Newtonsoft.Json for API communication
- •FMOD for gamified audio cues and procedural narration
Creative & Technical Innovations
- •Single media panel rendering multiple content types (text, quizzes, 2D/3D, audio/video)
- •Voice-based media control (play/stop/seek/volume)
- •Spatial anchors for stable, persistent object placement
- •Remote media URL resolution via API for streaming content
- •Deep-link OAuth workflow for institutional login
- •Gamified audio system with dynamic feedback and cues
Visuals & Media
OAuth Institutional Login
Demonstrates secure institutional login flow using deep-link authentication, enabling seamless account setup and secure access for medical students and residents.
Spatial Anchors
Shows how spatial anchors are placed and how content/UI aligns to them, ensuring precise, stable object placement in mixed environments across sessions and devices.
Procedure/Course Flow
Walks through the immersive, step-by-step training experience, including module navigation, media playback, and interactive XR content.
Results & Impact
The XR application delivered robust, scalable, and immersive training while maintaining alignment with backend systems, ensuring usability, and adapting to evolving XR SDKs and headset software.
Application Functionality
- •Delivered fully functional, data-driven XR medical training application
- •Established stable XR foundation despite rapidly evolving SDKs
- •Integrated JSON schema directly into runtime logic for consistent structure
Performance & Media
- •Achieved stable performance with dynamic media streaming and spatial anchors
- •Unified media system supporting multiple content types in single panel
- •Gamified audio integrated via FMOD without impacting XR performance
Content Workflow & Scalability
- •Simplified content workflow through flexible JSON-based course structure
- •New procedures can be added without code changes
- •Designed for future expansion with in-app course selection
Authentication & Tracking
- •Successfully integrated secure login with OAuth flows
- •Deep links, analytics, and progression tracking in cohesive experience
Key Takeaways
The project demonstrates how a unified, data-driven XR framework can deliver immersive step-by-step training experiences while maintaining usability, scalability, and alignment with backend systems.
Why This Project Stands Out
- •Data-driven XR training delivered through unified media and interaction pipeline
- •Combines voice commands, spatial anchors, and rich multimedia in consistent UI framework
- •XR UI/UX designed for readability, stability, and comfort in passthrough environments
What We're Proud Of
- •Modular course architecture that scales across multiple procedures
- •Flexible media-panel system supporting multiple content types
- •Multi-input interaction model (voice, ray, controllers, hand) for accessible XR training
Get in Touch
Looking to build high-impact XR medical or technical training simulations? Contact FierceGuppy Game Labs to turn your procedures and educational content into interactive, step-by-step XR experiences.