Real-Time Speech-to-Speech Systems for Embodied AI

Overview

At IQAI I work on embodied AI and real-time LLM systems: an embodied humanoid that people speak to, that reasons about what it hears, and that responds in natural speech and physical action with conversational latency. This page describes the publicly shareable aspects of that work.

Engineering problem

Conversational, embodied systems are a stress test for multimodal machine learning. Streaming speech recognition, language reasoning, speech synthesis and robot control must run as one pipeline, and the experience is governed by end-to-end latency, turn-taking behaviour and graceful handling of interruptions and failures. The hardest problems are rarely a single model’s accuracy; they are the interactions between components and the reliability of the whole system over days of operation.

Areas of work

• Real-time speech-to-speech pipelines. Built pipelines integrating streaming speech recognition, LLM reasoning and text-to-speech for an embodied humanoid AI system.
• Latency and reliability engineering. Benchmarked GPT-4o, Gemini, Groq, ElevenLabs and Cartesia across latency and reliability, and optimised voice-activity detection to reduce conversational turn-detection latency.
• Robot control. Developed a Python robot control framework and action API supporting locomotion, gaze, end-effector control and real-time interpolated joint control across simulation and physical hardware.
• Autonomous behaviour. Designed and deployed an event-driven system connecting on-chain events to autonomous robot behaviours, which executed 98 action sequences over 10 days without failure.

Why it matters

Working on systems people talk to in real time has sharpened my research questions: how to make multimodal models robust to the messiness of real inputs, how to reason across modalities under tight latency and compute budgets, and how to make model behaviour predictable enough to trust in interactive and physical settings.