DEMO
Rahul Singh, Devdeep Ray, Connor Smith, Sarita Adve
We demonstrate a device-cloud system for real-time, open-vocabulary, queryable semantic mapping on low-power XR devices. Semantic mapping constructs a persistent 3D representation of the environment annotated with object-level semantics, enabling applications such as spatial object search, context-aware assistants, and accessibility tools. Existing open-vocabulary semantic mapping pipelines, however, require server-class GPUs and are infeasible on power-constrained XR hardware.
Our system offloads semantic mapping to the cloud using an object-centric distributed architecture where objects are the fundamental units of storage, update, transmission, and computation. This design enables bandwidth-efficient depth streaming, efficient server-side parallel mapping, and scalable device-local maps that support low-latency querying even under network disruption.
In our demonstration, users wear a Meta Quest headset, walk through a physical space, and ask natural-language questions such as "Where are my keys?" The system builds a semantic map in real time, and users can ask natural-language queries such as "Where are my keys?" and see matched objects highlighted directly in the headset..