Can today’s large language models safely run real robots on their own? A new evaluation from safety-focused research firm Andon Labs says: not yet. In tests that used multiple LLMs to interpret natural-language instructions and control physical machines, the systems showed they can understand what people want—but they still make frequent, sometimes serious mistakes once the rubber meets the road. The takeaway is clear: LLM-powered robots remain promising, but they are not ready for unattended, reliable, or safety-critical deployment in real-world environments.
What worked well
– Natural-language understanding: The models generally parsed plain-English commands, followed multi-step directions, and explained their reasoning.
– High-level planning: They could outline task sequences and propose reasonable next actions, especially in clean, constrained setups.
– Adaptability in simple scenarios: When conditions closely matched their expectations, the robots often performed adequately.
Where things broke down
– Real-world perception gaps: Noisy sensor input, imperfect object detection, and occlusions led to wrong decisions about what was where.
– Physical common-sense errors: Models that excel with text still struggle with weight, friction, stability, and safe force application.
– Ambiguity and misinterpretation: Vague commands or subtle context changes produced incorrect or unsafe motions.
– Error compounding: Small missteps cascaded, with weak recovery behavior once a task drifted off plan.
– Overconfidence and hallucination: The models sometimes asserted success or “saw” objects that weren’t present, undermining trust and safety.
Why the lab-to-life gap persists
– Domain shift is hard: Performance that looks strong in simulation doesn’t fully transfer to cluttered, dynamic, and imperfect real environments.
– Timing and latency: LLMs can be slow or inconsistent, which is a poor match for the fast, closed-loop control that robots require.
– Incomplete grounding: Without robust multimodal grounding, language reasoning doesn’t reliably map to precise, safe actions.
– Safety isn’t a single feature: It requires layered protections, deterministic guardrails, and fail-safe behaviors that general-purpose LLMs don’t yet provide.
What “not ready” means in practice
– No autonomous operation in public or safety-critical spaces without close human oversight.
– Use only in constrained, well-instrumented environments with strict limits on speed, force, and workspace.
– Expect supervised assistance, not set-and-forget autonomy, for the near term.
Smarter paths forward
– Hybrid stacks over pure-LM control: Pair LLMs for high-level reasoning with classical motion planners, verified controllers, and hard safety limits.
– Structured interfaces: Constrain the model to approved skill libraries and checklists rather than free-form command generation.
– Sensing and grounding upgrades: Improve perception, calibration, tactile feedback, and multimodal training so the model’s “understanding” reflects physical reality.
– Rigorous evaluation: Red-team testing, transparent metrics, and scenario coverage that reflects the messy real world, not just sanitized demos.
– Human-in-the-loop workflows: Require confirmations for risky steps, provide clear aborts, and record decisions for auditability and continuous improvement.
Near-term use cases that still make sense
– Operator assistants that translate natural-language goals into safe, pre-validated skill sequences.
– Quality checks and step-by-step verification where the model proposes actions and a human approves execution.
– Cobots in tightly controlled cells performing repetitive tasks with limited autonomy and strong physical safeguards.
The bottom line
Andon Labs’ experiments underscore a pivotal reality for embodied AI: language models are powerful planners and communicators, but they’re not yet dependable pilots for real-world robots. They can understand what to do, but too often they fumble the how. Until perception, grounding, and safety engineering catch up, the responsible path is hybrid systems, rigorous guardrails, and human supervision.
For teams building robotic products or deploying automation on factory floors, that means optimizing for reliability over novelty. Use LLMs to make robots easier to instruct and monitor, not as the sole brain behind every move. The promise of natural-language robotics is real, but so are the risks—and today’s science says to keep both eyes open.
