Hrvoje Benko
Affiliate Professor | University of Washington
Augmented Reality in the Age of Agentic AI: Why AI is the Only Way AR Succeeds and What is Still Missing?
Hrvoje Benko, University of Washington
Abstract
Augmented Reality technology has been searching for a killer app for decades. AR combined with an AI agent holds the promise to provide us with assistance at the right time, at the right place and with the right information. This can fundamentally alter our human abilities and give us all superpowers. However, before we can deliver truly useful all-day mobile AI+AR experiences, we must first solve three fundamental problems that remain unsolved. This presentation will explore these three key areas of deficiency in the current landscape of immersive technology: the limitations of existing wearable displays, the gap in current AI models' understanding of the human world that prevents true relevance, and the challenge of designing effective interaction methods for this new technology beyond simple voice commands.
Biography
Hrvoje Benko is a research scientist specializing in the intersection of Artificial Intelligence, Human-Computer Interaction (HCI), Augmented Reality (AR), and Wearable Computing. With over 20 years of experience at the vanguard of industrial research—including over a decade at Microsoft Research and executive leadership at Meta Reality Labs—he is passionate about inventing solutions that enable people to use their gestures, gaze and voice to express themselves while harnessing the contextualized understanding of their environment for better interactions.
He has invented numerous handheld and wristband input and haptic devices, developed techniques for retargeting haptics by leveraging dynamic VR illusions, created room-scale, interactive, projection-based Augmented Reality environments, as well as championed the idea and the first implementations of multitouch interactions on any surface via wearable depth cameras, thus enabling touch interactions for both Augmented and Virtual Reality scenarios. For the latter, he has received the ACM UIST Lasting Impact Award in 2022 for his co-authored work "OmniTouch: Wearable Multitouch Interaction Everywhere".
He is an Affiliate Full Professor at the University of Washington Information School and he has coauthored more than 70 issued patents and 80 scientific articles, 13 of which received best paper awards or honorable mentions at the top HCI conferences. He held the position of Director of Research Science at Meta Reality Labs Research where he led the development of novel interactions, devices and interfaces for Contextualized AI, Augmented and Virtual Reality. Prior to Meta, he was a Principal Researcher at Microsoft Research, where he worked on novel haptic handheld devices, multi-touch interactions, large-scale projection-mapping environments, and novel AR/VR interactions and technologies. He received a Ph.D. in Computer Science from Columbia University. For his research contributions, he was inducted into the SIGCHI Academy (Class of 2023), and the IEEE VTCG Virtual Reality Academy (Class of 2026).