Robot brain builders are pushing out of their GPT-2 era

What happened

Physical AI is one of the hottest sectors in venture investing, with companies raising billions to apply the tools that gave us Large Language Models to robotics. That excitement helped deliver a big IPO for Unitree, China’s leading robot maker, which saw the company valued at $66 billion after its arrival on China’s equivalent of the NASDAQ.

” Kendall argues that it’s too early to commit to any one hardware platform—advances in sensors and other components are coming quickly, and a truly general model should be more agnostic. ” Gervet also touched on another hot topic in the sector: How specifically to focus your physical AI business.

Robotics companies that are targeting specific tasks are getting their robots out in the field—Gritt is building solar farms, Agility is deploying robots in industrial settings, and Bedrock is operating excavators autonomously. Meanwhile, general-purpose humanoids aren’t getting out of the labs. “No customer cares about the general purpose robot that works at 80% success rate,” Gervet said of the dilemma.

“We see a lot of other players go general, but there is no value provided because there’s no vertical focus. ” The temptation to get invest in a specific vertical, however, is tempting because it provides not just revenue but also real-world deployment data. While task-specifc data might not have enough diversity to push general purpose models forward, it is an important for making a robot that adds value.

Bedrock CTO Kevin Peterson noted that his company was just starting with excavation as a way to understand the challenges of “manipulation in the wild,” but plans to develop an intelligence layer that stretches across a series of construction machines. Managing all that data is a challenge, especially because of the density of visual and lidar data.

Why it matters

This week, however, the bottom fell out, and the company lost nearly half of its value. Analysts point to one obvious issue: While the robots’ physical capabilities are improving, they still lack the know-how to actually do value-creating work. At last week’s Actuate conference, a gathering of developers building AI brains for robots, the excitement was clear.

The event has tripled in size since it kicked off in 2023, and had 1500 attendees, according to the organizer, Foxglove, a company that helps physical AI model builders manage and visualize their data. ” That crisis is the lack of high-quality training data for AI models.

Attempts to build generalized robots that can do any task are still far off, and using end-to-end learning for specific tasks still hasn’t delivered products with reliable, commercial performance. For developers, the answer is to better mimic the advances of the frontier AI labs—find or create more diverse data sets, mess with different training regimes, and figure out better reinforcement learning scenarios.

Harry Mellsop, a founder of Antioch, a startup that building simulation tools for model builders, suggests physical AI is in its “GPT 2 era,” the OpenAI model that pre-dated the arrival of ChatGPT. More data and compute will be needed to get over the hump, particularly GPUs optimized for ray tracing, which are used to create high-fidelity simulations.

The furthest ahead are autonomous vehicles, in part because of the ability to collect relevant data from cars driven by people, and in part because the main task is to avoid contact, not manipulate the physical environment. Much of the tooling for model-building comes from autonomous vehicle companies; Foxglove, for example, was founded by former employees at Cruise, General Motor’s erstwhile self-driving effort.

And now those car companies are increasingly betting that their investments in ML tooling will allow them to compete with dedicated humanoid makers. Tesla is already trying this with its Optimus robot, and now both AV-focused Wayve and ride-share giant Uber have now launched robotics labs focused on humanoid form factors as R&D efforts.

“I think you need to start in vehicles…manipulation robotics is like self-driving five years ago,” Alex Kendall, the CEO of Wayve, told TechCrunch. “The data infrastructure, the simulation, ML ops infrastructure, will probably be shared, but the specific world model for the simulator will be a different post-training.

What to watch

Foxglove announced a new product this week, built on top of an Nvidia’s Cosmos open weight world model, that allows engineers to search that data with sophisticated natural language queries to build out evaluations and simulations. The goal is faster triage and debugging so model builders can iterate faster.

So what will be the fabled ChatGPT moment for physical AI that Sam Altman recently said is just a few years away?