Voice-first interaction with a real-time multi-modal model (audio + text) over WebSocket manages conversation and tool calling, infers human intent, and uses a fast mini AI model to translate intent into cinematic prompts. Workflow automates the intent-driven drift of the world model through those prompts and renders the resulting output onto the canvas element of the page through constantly streaming data using WebRTC.

Multiple experience modes (dreamlike or website) are configurable through environment variables and alter prompt engineering that impacts conversational style and the visual stream. A bespoke cognitive and visual state management system is at the heart of both agents and their outputs, making orchestration more natural and cohesive. R&D was built in close partnership with Fantasy leadership and used in thought leadership, organic accounts growth, and business development.

Stack highlights

OpenAI Realtime beta, Odyssey.ml 2-Pro early access model, Brave Search API.