Lightning Talks

The current staging system for neuroblastoma depends on IDRFs (Image Defined Risk Factors). Virtual Reality has the potential to more accurately stage these tumors. VR has proven useful for preoperative planning and anatomical understanding for surgeons as well as for patients/parents as part of the informed consent process.

Imagine a hearing device that let you hold a quiet conversation in a crowded restaurant, chat with someone who speaks a different language, or understand what the pilot just said over the public address system. Assistive and augmentative listening technologies, such as hearing aids, wireless broadcast systems, and extended-reality headsets, can change the way we hear by processing the sounds around us. Recent advances in wireless connectivity, ubiquitous acoustic sensors, and novel signal processing algorithms could dramatically improve the performance of these systems and enable new experiences that would be impossible today. Prof. Corey will provide an overview of his team's work in the newly established Listening Technology Lab at UIC and DPI.

Powerful and affordable immersive visualization anywhere along the virtual to physical continuum is now available. We identify challenges, present results, and sketch possible solutions towards making extended reality an effective and widely adopted educational tool.

Mixed Reality (MR) systems aim to create seamless, immersive experiences. Achieving this requires understanding how users perceive, navigate, and interact with their environment. Yet, current MR systems often fall short, especially as user perception shifts over time and environments grow more complex. At the core of this limitation is the inability of MR systems to reason about the human-system loop in real-time. In this talk, I will present my work that addresses this gap by introducing a system-level measurement framework to sense and interpret user experience in real-time. Specifically, I use reaction time, a continuous behavioral signal, to assess the presence, a central experiential construct in MR reflecting the feeling of “being there” in a virtual environment. This work identifies how reaction time reflects both scene fidelity and the user’s cognitive state. Grounded in cognitive science and validated through human-subject studies, it establishes a real-time link between subjective perception and system interaction. The result is a non-intrusive method to detect perceptual shifts as they occur, enabling systems that adapt alongside the user’s changing experience. I will conclude with my broader vision for adaptive, human-centered MR systems that interpret and align with perception as it unfolds, advancing immersive system design grounded in real-time behavioral signals.

Experience design of multi-sensory, co-present, social and data-intensive immersive experiences created by Dr. Ruth West and her team at the UNT xREZ Art + Science Lab.

Extended Reality (XR) will transform the way we interact with digital information, and promise a rich set of applications, ranging from productivity and architecture to interaction with smart devices. Current approaches to interactive XR systems, however, are static, and users need to manually adjust factors such as the visibility, placement and appearance of their user interface every time they change their task or environment. This is distracting and leads to information overload. To overcome these challenges, we aim to understand and predict how users perceive and interact with digital information, and use this information in context-aware systems that automatically adapt when, where and how to present virtual elements.

Modern mixed and augmented reality (MR/AR) experiences can perform below expectations, or even pose security risks, when deployed in unknown real-world environments: for instance, virtual content may appear misaligned with respect to the real world, or may block user's view of important real-world objects. Fortunately, modern vision-language models, such as GPT-4o and Claude 3, have sufficiently powerful real world understanding capabilities to enable AR/MR experience quality monitoring and evaluation, and can be coupled with fast-acting edge computing based mechanisms to allow rapid responses to cases where improperly positioned AR/MR content may significantly affect user performance or endanger the user. This talk describes our recently developed VLM-based approach to AR/MR experience quality evaluation and showcases an implementation of this approach that ensures that AR/MR virtual content does not block user's view of important real-world objects. The talk is based on research that is funded by an NSF AI Institute and by a DARPA Young Faculty Award.

Extended reality applications present a unique set of platform architecture challenges due to their real-time constraints paired with strict power, performance, and area requirements. We are developing a hardware-software co-design testbed to facilitate insight into full-stack design and characterization of XR platforms. By leveraging Chipyard (an open-source RISC-V design framework) and ILLIXR to bring up an XR runtime and map it to RISC-V hardware, we aim to enable users to explore the entire SoC design space in a way that was previously infeasible.

Augmented and virtual reality (AR/VR) combines many artificial intelligence (AI) models to implement complex applications. Unlike traditional AI workloads, those in AR/VR involve (1) multiple concurrent AI pipelines (cascaded AI models with control/data dependencies), (2) highly heterogeneous modality and corresponding model structures, and (3) heavy dynamic behavior based on the user context and inputs. In addition, AR/VR requires a (4) real-time execution of those AI workloads on (5) energy-constrained wearable form-factor devices. All together, it creates significant challenges to the AI system design targeting for AR/VR. In this talk, I will introduce case studies into hardware, software, and AI model design for AR/VR that show how the challenges form unique research problems and provide research opportunities in the AI system design for AR/VR.

As embodied AI agents become more capable, the missing piece isn’t more compute — it’s more human-like behavior to learn from.

This talk introduces Omnia’s behavioral dataset initiative, which captures real-world spatial interactions from XR platforms like Apple Vision Pro and Meta Quest. Through our immersive tutorial engine, how2TO, users perform multi-step physical tasks — from replacing a car battery to setting up electronics — while their gaze, gestures, voice, and object interactions are logged.

We explore how this data can power robotic imitation learning, enhance sim-to-real pipelines in environments like Isaac Sim, and serve as a bridge between human instruction and autonomous execution. The result is a scalable new dataset infrastructure at the intersection of XR, robotics, and physical AI.

Streaming high-fidelity 3D content requires tens of gigabits per second (post compression), far exceeding current Internet capabilities. As immersive AR/VR applications grow, efficient 3D representations are critical for real-time streaming. Among various options, meshes offer the best balance of quality and efficiency, yet there is limited work on compressing time-varying meshes (TVMs), which change dynamically in structure and topology. In this talk, I will introduce TVMC, a novel compression method that significantly reduces bandwidth requirements for real-time streaming of large-scale 3D scenes.

Computation-communication trade-offs offer exciting opportunities for performance advancement of immersive systems. This talk will highlight a recent related study (SIGMETRICS ’24) where we explored the user of neural computation at a receiving client in an immersive media streaming system to compensate for inadequate communication without disrupting the playback of the media presentation. We introduce BONES, a control algorithm that jointly manages the network and computational resources in the system to maximize the quality of experience of the user. BONES leverages Lyapunov optimization to solve the problem of interest online with near-optimal performance, making it the first of its kind. Experiments demonstrate significant gains over a broad variety of state-of-the-art methods, with minimal overhead.

We discuss a cross-layer design for networked VR applications that elevates a network data unit to a usable rendering unit for VR applications, with the aim of enhancing visual quality, especially over constrained and variable networks.

Extended Reality (XR) has emerged as a promising platform for immersive learning experiences; however, its effectiveness depends on accurately capturing and interpreting learners' cognitive, affective, and behavioral states in real time. This brief paper introduces the XR-based Learning Experience Analytics & Dashboard System (XR-LEADS), designed to predict learners' Quality of Experience (QoE) by integrating multimodal data from devices such as HoloLens 2 and NEON eye-tracking glasses. Fourteen multimodal metrics—including head and hand positions, eye gaze, and pupil size—are used by a deep learning model to compute a Learning Effectiveness Index (LEI). Leveraging the Framework of Interactive Learning Experience in XR Environments, XR-LEADS provides a structured method for analyzing and optimizing learner engagement within immersive contexts. Preliminary findings suggest that real-time QoE and LEI prediction enables more adaptive and personalized XR learning experiences, offering actionable insights beneficial to educators and learners alike.

Personal computing has evolved from desktops to mobile devices, and Extended Reality (XR) is poised to become the next major platform. However, its adoption will be gradual rather than immediate, making the integration of existing ecosystems a significant challenge. In this talk, we will explore strategies and design principles for transitioning from traditional environments to integrated XR ecosystems. We will also highlight how modern AI can automate and optimize this process, facilitating a seamless, user-centered evolution toward spatial computing.

The vision of seamless cyber-human interaction through immersive computing stands at a critical crossroads. As commercial AR glasses and VR head-mounted displays make immersive experiences increasingly accessible to everyday users, their evolution from single-user, contained experiences to ubiquitous, many-user environments creates profound security and privacy challenges. In this talk, I will map the emerging battlegrounds where attacks and defenses collide in immersive computing ecosystems. By examining the unique headwear, content, and user factors in immersive systems, I will present research on novel threat vectors and defense mechanisms. I will conclude my talk by discussing my ongoing and future research toward trustworthy immersive computing.