Introduction

Meta invited us to push the boundaries of spatial computing by demonstrating how the intersection of Mixed Reality (MR) and Generative AI can redefine human interaction within an environment.

We developed Spatial Lingo, a high-impact, open-source exemplar project that transforms any physical room into a dynamic, AI-powered language learning laboratory.

The brief

The objective was to create a focused, real-world application that serves as a "gold-standard" for the global developer community. Meta challenged us to incorporate their latest APIs and SDKs – including the Passthrough Camera API (PCA) and Llama API – into an experience that was not only technically impressive but showcased a seamless bridge between physical and digital spaces, blending function with elegance.

Key requirements included the integration of:

• Interaction SDK & Voice SDK

• MRUK (Mixed Reality Utility Kit) for high-level scene understanding

• Passthrough Camera API

• Llama API for generative intelligence

The solution

We created Spatial Lingo, a contextual language practice instructor that uses real-time AI to identify real-world objects and label them in a user’s target language. Instead of static flashcards, users practice vocabulary dynamically by exploring their actual environment, making the learning process spatial, responsive, and endlessly replayable.

Spatial intelligence & scene understanding: Using MRUK, the application intelligently scans the user’s room to identify the most optimal open space to anchor a "Language Tree." This persistent virtual asset acts as a visual progress tracker; as users identify objects and complete challenges, they accumulate berries and watch their tree grow within their physical home.

AI-powered vision: By combining the Passthrough Camera API with Sentis and YOLO, Spatial Lingo identifies real-world objects in real-time. The Llama API then processes these identifications to generate dynamic word clouds – consisting of nouns, adjectives, and verbs – tailored to the specific objects found in the user’s room.

Natural interaction: To make the environment feel reactive and alive, we integrated Voice SDK (TTS/STT) with Llama to create Golly Gosh, an interactive AI companion. The companion provides real-time guidance and feedback, allowing users to practice their pronunciation and syntax through natural, voice-based dialogue.

The roadmap

By "stress-testing" the SDKs during development, we provided critical feedback loops and feature suggestions that were integrated into Meta's long-term roadmap. This collaboration ensured that the codebase remained clean, well-documented, and ready to serve as a definitive blueprint for future developers.

The impact

Spatial Lingo successfully leveraged the hardware capabilities of the Quest 3, demonstrating that the intersection of generative AI and mixed reality is a functional tool for education and utility rather than just a concept.

By open-sourcing the project, we have provided the industry with a foundational template for building intelligent MR applications.

Anyone who takes inspiration from Spatial Lingo can find the source assets and code on GitHub.