Robots don't come with keyboards: how a robot hears you

Why does a robot need speech to text at all?

A robot has no keyboard, and its user often has busy hands, so speaking is the obvious way to give it a job. A speech to text model is the first thing that command meets, and in published robot systems everything after it takes the transcript on trust.

Our view, argued below, is that this ear is the weakest part of a talking robot, and that the best remedies so far handle the errors that get through: a robot that asks when it is unsure, and an emergency stop that never depends on speech.

Talking robots are also rarer than the coverage suggests. The International Federation of Robotics counts 542,000 industrial robots installed in 2024 (4,664,000 at work) and close to 20 million consumer service robots sold that year, mostly floor cleaners and lawn mowers. Humanoids are a sliver: in figures reported by TechNode and CGTN, Omdia estimated about 13,000 shipped in 2025 and IDC about 18,000. IDC ranked entertainment and commercial performances first by share, and factories and warehouses last.

Voice is where the leaders say they are heading. Google DeepMind says Gemini Robotics understands “commands phrased in everyday, conversational language”, and Figure’s post on Helix says new skills “could be obtained instantly just by talking to robots in natural language”. Agility Robotics’ solutions page, by contrast, has its Digit humanoid taking jobs from warehouse software, with no mention of voice. Forecasts reported by Investing.com count different things: Goldman Sachs expects humanoid unit sales of 6.48 million by 2035, Morgan Stanley 13 million humanoids in use. The roboticist Rodney Brooks, in a 2025 essay, puts the first profitable humanoid deployment “more than ten years away”.

How does a spoken command become robot motion?

It passes through a chain of stages, and the model that plans the motion usually sees only the text produced halfway along:

1. Microphones pick up the room; an array can steer towards the speaker, and echo cancellation removes the robot’s own loudspeaker.
2. Voice activity detection, often with a wake word, decides that someone is talking to the robot.
3. A streaming recogniser writes the words down as the person talks.
4. A language model turns the text into a plan tied to what the cameras see.
5. A controller turns the plan into motion, above a safety layer that uses no language.
6. The robot often replies aloud, and its microphones hear that too.

The published robot “brains” sit at step four and read text. SayCan, a 2022 Google robotics project, was evaluated on 101 instructions given as text, choosing a correct plan 84% of the time and completing 74%. Google DeepMind’s RT-2 (2023) writes the robot’s actions as text tokens, and the Helix and Gemini Robotics posts say little about how speech becomes text.

Where voice appears, it is added in front. In Physical Intelligence’s Hi Robot paper, the robot takes “text commands or spoken language (which is then transcribed into text)”, picked up by a clip-on microphone. 1X says its NEO home humanoid uses “an offboard speech-to-speech LLM” to extract the goal. Older robots matched a fixed list: NAO’s documentation says the small humanoid’s recogniser, by default, expects one of its set phrases, “nothing more, nothing less”.

Models that take audio straight in exist in labs and previews. A 2025 paper on a robot policy that hears speech directly reported 73.2% success at three tasks in a row in a simulated benchmark, against 61.0% for a comparable policy fed a separate recogniser’s transcript, in the authors’ own comparison. Google’s Gemini Robotics-ER 2 streaming preview takes raw 16 kHz audio, as a cloud service for developers. In the papers and company pages we read, we could not find a robot on sale that maps speech straight to motion, so whatever the recogniser gets wrong becomes the plan.

How did speech to text get this good, and what counts as accurate?

Mostly through neural networks after 2012 and ever more training audio, and every accuracy figure belongs to one kind of recording. A 2004 history of the field starts in 1952, with a Bell Labs system that recognised single digits from one speaker. IBM’s Shoebox of 1961 added six command words to the digits, and Carnegie Mellon’s Harpy, from the 1970s, knew 1,011 words. Statistical models called hidden Markov models were the standard until 2012, when a review by four research groups reported that deep neural networks cut errors on Switchboard, a benchmark of recorded phone calls, from 27.4% to 18.5%.

CTC (2006) and the RNN transducer (2012) let a network write words without being told where each one starts, and the transducer can do it as you speak. The Conformer (2020) added convolution to the 2017 Transformer, self-supervised pretraining (2020) learned from 53,000 hours of untranscribed audio plus ten minutes of transcribed speech.

Claims of human-level scores are disputed. In 2016 Microsoft reported 5.8% on Switchboard against 5.9% for professional transcribers and called it human parity. IBM’s researchers measured their best transcriber at 5.1%, and at 6.8% on CallHome, harder calls between friends and family, and found 36 of the 40 Switchboard evaluation speakers in its training data. Microsoft reported 5.1% the next year.

Scores on clean audio say little about a room. A 2020 paper set the best published results, as low as 2% to 4% errors on read audiobooks, beside three commercial systems on the authors’ own recordings of customer service calls: 13.7% to 23.3% errors, highest on bookings and telecoms calls full of dates, money, places and product names. Streaming costs accuracy too: holding one model to about 80 ms of delay instead of 960 ms raised its errors on clean audiobook speech from 2.50% to 3.01%.

Why is a robot a harder listener than a phone?

A phone sits close to your mouth and makes no noise. A robot hears you from a metre or more away, with motors, fans and a loudspeaker centimetres from its microphones. In the table, 0 dB means voice and noise are equally loud, and below zero the noise is louder.

The usual fix for the robot’s own voice is to switch its microphones off while it talks, which loses any reply that starts early. A 2025 editorial says robots “frequently cannot hear the human interlocutor”, typically for want of enough well-placed microphones.

In lab work, more microphones help, and so does telling the recogniser what the robot knows. In NOTSOFAR-1, a 2024 challenge on transcribing real meetings recorded in 30 rooms, the best multichannel system made 10.8% errors against 22.2% for the best single-microphone one. Telling the acoustic model whether the motors were running cut phoneme errors by 11%, biasing the recogniser towards objects the robot could see cut errors by 59%, and video of the speaker’s lips took them from 25.8% to 5.8%, averaged over noisy conditions. We could not find a speech recognition benchmark built for robots, comparable to CHiME, or a robot maker that publishes an error score for its robot’s ear.

What does a misheard word cost a robot?

A misheard word can cost an unsafe movement near people, and the first measurements of that risk are recent.

In an August 2026 preprint, researchers at Heriot-Watt University fed simulated recognition errors into two safety benchmarks for AI that plans a robot’s actions. On one, the planner accepted 55.33% of harmful instructions written cleanly; sound-alike word swaps raised that to 60.00%, and severe background noise (about 80% of words wrong) to 67.33%, turning 27 refusals into executable plans. It is one preprint testing one planner model, so it shows a mechanism and gives no rate for any real robot.

The field’s main defence, a robot that asks, has costs. KnowNo (2023) asks for help whenever more than one plan is plausible; to reach a 75% success target on real hardware it asked on 92% of trials, and its guarantee assumes its input text describes the scene correctly. In a 2023 study, people rated an interrupting robot “significantly less helpful”. The answer can be misheard too: in Amazon’s explanation of a 2018 incident, quoted by TechCrunch, an Echo took background talk for its wake word, then “send message”, then a contact’s name, and finally “right” in answer to its own confirmation question.

The stop should therefore never depend on speech. ISO 13850 describes the emergency stop device as a “manually actuated control device” and its function as “initiated by a single human action”; we could not find a clause that mentions voice in the standard’s public preview. Researchers who added a voice stop to a mobile factory robot wrote that it “should not be installed as the only emergency stopping mechanism”. On MIT’s 2010 robotic forklift, 26 of 36 shouted stop commands were caught on the first try.

Surgeons have spoken to robots since the FDA cleared voice control for AESOP, an endoscope-holding arm, on 3 May 1996. One 1998 study found the foot pedal faster and another found voice faster, and a 2006 paper still called a manual shut-off “important should the robot fail to respond to the voice command ‘Stop’”.

Researchers have also slipped commands into phone and smart-speaker assistants with ultrasound people cannot hear and with a laser from 110 metres away. A 2026 preprint shows the reverse: short “safety-plausible phrases” slipped into a robot’s audio can halt it, so an attacker can stop a robot that obeys every spoken stop.

Where should the listening run: on the robot or in the cloud?

For the ear, on the robot: the network is the slowest and least predictable part of a voice command, and speech recognition now fits on a small computer. Heavy reasoning can go elsewhere.

People leave about 208 ms between turns on average. A Berkeley robot that sent camera images to servers measured round trips of 437.63 ms to a distant data centre and 119.40 ms to a nearby GPU; “communication time dominates the computation time”, its authors wrote. On one university’s Wi-Fi, one delay in a hundred took about 250 ms, across more than 47,000 devices. Boston Dynamics’ talking Spot, a 2023 demo built on cloud services, took “sometimes 6 seconds or so” to reply, and hospital delivery robots moving between floors “are frequently disconnected with the network”.

The European Data Protection Board’s 2021 guidelines on voice assistants say “voice data is inherently biometric personal data” and describe the usual split: wake word detection “carried out exclusively locally”, most other processing on remote servers. A robot’s microphones hear everyone in the room, and smart speakers played two rounds of 134 hours of TV woke by mistake 0.95 times an hour. On the company pages we read, we could not find a robot maker that says what happens to a voice command when the Wi-Fi drops.

Recognisers are now small enough for a robot. In 2019 Google shrank a phone recogniser from 450 MB to 80 MB, and a 4-bit Conformer came out 5.8 times smaller with no loss of accuracy. Leading humanoids keep the body’s models on board: Figure says Helix “runs entirely onboard”, Google DeepMind says its on-device robot model operates “independent of a data network”, and 1X says Redwood runs on NEO while voice goes offboard.

On whether robots should connect at all, NIST’s 2023 security guide for operational technology, the computers that run industrial plants, says both that devices with no business need to communicate “should not be connected (i.e., air gapped)” and that “relying on traditional measures (e.g., air gaps) is not enough”. A March 2026 preprint by Microsoft and UC Berkeley researchers found that larger onboard GPUs drain a robot’s battery up to 160% faster, while for offloaded work “even a few tens of milliseconds of additional latency” can cut manipulation accuracy by over 10%. The position we can defend is a robot that keeps working when the network drops and connects when that helps.

What will decide whether robots really hear us?

The first thing to watch is whether makers measure the ear the robot’s way: at a distance, with the motors running, while the robot talks, and with children speaking. Beyond that score, a robot needs a calibrated “I am not sure”, so it asks when it should and only then, and makers should treat vision and context as part of hearing, since the largest lab gains came from them.

For now a person stays in the loop. 1X’s page for NEO says owners can “schedule a 1X Expert to guide it through unknown tasks”. In a 2025 study with ten users, tasks done by touch took 27.8 seconds on average against 77.2 by speech, and 81.6% of touch commands were carried out against 67.2% of spoken ones, though people preferred speech for 90% of their exploring.

Industrial noise is the hard case. A 2021 study of a warehouse in Costa Rica measured 80 to 90 dB(A) at its loading docks, around the 8-hour average of 85 dBA at which OSHA requires a hearing conservation program. With 60 dB of interference and commands spoken 10 dB louder, a 2020 study recognised 91.43% of spoken commands and called the result “definitely not acceptable in industrial environments”.