AI with a body
Characters you can talk to, systems that understand how you move, models that hold up in the real world – AI in immersive experiences, where it earns its place
In a sentence
We've been putting machine learning into interactive experiences since 2015 and real-time conversational AI into them since 2023. Most of that work is under NDA, so this page describes it by shape – what we built, with what, and what was hard – rather than by client.
What we build
Real-time voice characters and assistants
This is the bulk of our recent AI work. A person speaks; a character listens, thinks and replies in a voice that fits, fast enough that the conversation feels like one. We build the whole loop: speech recognition, the language model that decides what to say, the guardrails around it, speech synthesis, and the lip-sync, gaze and body language that keep the character in the world rather than floating over it.
We have extensive experience with ElevenLabs for real-time voice, usually combined with a language model from Anthropic (Claude), OpenAI or Google for the conversation itself – chosen per project, not by habit. The hard parts are never the model: they're latency you don't notice, turn-taking that doesn't talk over people, knowing what the character must never say, and a graceful hand-off to a human when that's the right thing.
The shape of it, from projects we can't name: a voice-led companion built to be talked to in confidence – which is why the personal parts run on the device rather than in the cloud. Assistants that talk visitors through an exhibit. Characters in a headset who remember what you told them ten minutes ago.
And one we can: for Riptide: Intermission (2023), an 80-minute mixed-reality meditation with XR Stories, each visitor began with an AI-led onboarding interview on Charisma's platform. Charisma built the conversation; we built the VR experience that listened to it – taking what each person had said and shaping how their journey looked and behaved, so no two were the same.
And a footnote from before language models could hold a conversation: in 2017, for Channel 4's The Robot Will See You Now, we built the emotive face for Jess, a robot that sat in a room with real couples and families – idle, listening, thinking, reacting, on an Android device. The tools of the day could spot keywords and sentiment, not follow a conversation; what made Jess work was design, timing and a human in the loop. The lesson has outlasted the tech: people will open up to a machine only if they can read it.
Machine learning for bodies and gestures
Our first shipped machine-learning system was The Boris Games in 2015: a Kinect game you play entirely by signing, with a recogniser trained on Makaton signs – BAFTA-nominated, and runner-up at NHK's Japan Prize. The same instinct runs through Rehab Trainer, where gesture recognition turns stroke physiotherapy into games patients want to repeat, and through the body tracking and spatial audio that make our accessibility work usable by blind and deaf audiences. Embodied interaction is the thread through most of what we've done; AI is what lets a system understand a body well enough to respond to it.
AI that watches video
A more recent strand, and one we can only describe in outline: pipelines that take large volumes of video, run vision and language models over them, and turn what's in the footage into structured findings a person can review, query and act on – work that a team had been doing by eye, one recording at a time. Built as batch processing on Google Cloud, with the model outputs checked against human judgement before anyone trusts them. It's the same instinct as the rest of our work: AI doing the watching, people doing the deciding.
Physical AI: when the model has to hold up on hardware
Our work with REK sits where game-engine simulation meets real robots: a multiplayer simulator that trains human pilots, the same control policies running in the sim as on the hardware, and a loop tight enough that piloting feels like driving. The AI in play is the trained balance and motion that keeps a robot upright while a person controls it – not autonomy. It's taught us what it takes for a model to survive contact with the physical world, a live audience and no second take. It isn't our first time putting software in a robot, either: the emotive face for Channel 4's Jess (2017) ran on an Android device in a robot's head – no AI to speak of, but the same rule about reacting on cue, on camera, in front of real people.
AI in the tools and the pipeline
Some of the most valuable AI work never appears in the experience. We build tools for artists and researchers, and AI has changed what those tools can do and how quickly we can build them. A renderer that had been quoted in months was ported in weeks. It's a major part of how we work – research, code, tooling – applied with judgement: a person who knows what good looks like, pointing it in the right direction.
Consultancy: should AI be in this at all?
Often the most useful thing we do is the first conversation. We'll look at a brief, a prototype or a product and tell you where AI would genuinely earn its place, what it would cost in latency, privacy and money, which parts should run on-device, and which parts don't need AI at all. We've done this as immersive advisors on a three-year Wellcome Trust research programme (LivingBodiesObjects) and as the technical second opinion inside agencies and studios – a day, a week, or across a project. You get a plan you can build with anyone, including us.
How we think about it
On-device where the data is personal. Health, children, anything said in confidence: the personal parts stay on the device or on your premises, and the cloud is used for what genuinely needs it. We'll show you the trade-off before you choose.
A person stays in the loop. REK's robots are driven by people; our companions hand off to humans when they should; our characters have hard limits on what they say. AI is at its best when it makes a person more capable, not when it replaces one quietly.
We'll tell you when it's the wrong tool. Some of the most useful conversations we have end with a simpler answer than the brief asked for. We'd rather build the right thing than the fashionable one.
Accessible by design. Voice, body tracking and spatial audio are also the technologies that open experiences to people a screen and controller shut out. We design for that from the start – it's why the accessibility team at HSBC commissioned us, not the marketing team.
Who this is for
Studios and agencies who need a partner for the AI, physical or embodied part of a project – we're used to working white-label inside other teams' pipelines. Museums, health and education organisations with an audience who'd benefit from something they can talk to. Anyone with a live event, a headset or a room, and a question about whether AI belongs in it – which is a consultancy conversation before it's a build.
Questions we get asked
- Can it run on-device, without sending data to the cloud?
- Often, yes. Where the data is personal we favour on-device or on-premises models and use cloud services only for the parts that genuinely need them. That includes native Android - Meta Quest headsets are Android underneath, and we've shipped on Android devices since the Channel 4 robot face in 2017. We'll tell you the trade-off in latency, quality and cost before you commit.
- Which models and voice platforms do you work with?
- For real-time voice, ElevenLabs, usually with a language model from Anthropic (Claude), OpenAI or Google for the conversation. We've built experiences driven by Charisma's character dialogue, and with open models where on-device is the requirement. We pick per project; we're not tied to a vendor.
- Do you build customer-service chatbots or call-centre voice agents?
- Not as a rule. Our work is characters, companions and assistants inside experiences – VR, mixed reality, installations, health and learning apps, live events. If you need a support bot for a website, there are specialists who do that better than we would.
- Can AI characters work inside a VR or mixed-reality headset?
- Yes – that's most of what we do. Speech in, speech out, and a character that responds in real time on Meta Quest, HTC Vive, PC VR and mixed reality. The hard parts are latency, turn-taking and keeping the character in the world; those are the parts we've solved before.
- Can you work under NDA or white-label?
- Most of our AI work is under NDA already – which is why this page describes it by shape rather than by client. We regularly work white-label inside other studios' and agencies' teams and pipelines.
- Can you use AI to analyse video or other large sets of content?
- Yes. We build pipelines that run vision and language models over large volumes of video and turn the results into structured findings people can review and act on – with human checks on the model output. Most of this work is under NDA.
- Do you offer consultancy without a build?
- Yes. A day, a week or a seat across a project: we assess where AI would earn its place, what it would cost in latency, privacy and money, and what should run on-device. You get a plan you can build with anyone, including us.
- Will you tell us if AI is the wrong tool?
- Yes. We'd rather build the right thing than the fashionable one.
Something in mind?
A character people could talk to, a companion inside an app, a robot that needs a pilot, a tool that needs to understand a body, hours of video that need reading – or just a question about whether AI belongs in your project at all – we consult as well as build. Tell us a little about it and we'll come back to you.
