Self-Driving Cars Could Someday Take Requests

What happened

This article is part of our exclusive IEEE Journal Watch series in partnership with IEEE Xplore. The idea of letting a machine do the driving for you may put a lot of people off autonomous vehicles. But research could make it possible to backseat-drive an autonomous vehicle just as you might with a human driver.

The system relies on a model predictive-path integral controller previously developed by the researchers, which identifies multiple paths the vehicle could take to reach its goal and then judges them on various criteria, including speed, steering angle, and collision probability. It then finds an optimal path that is a combination of the trajectories that scored best on those judging criteria.

The team combined this with OpenAI’s GPT-4o-mini model to parse passengers’ natural-language suggestions and use them to tune how the controller chooses its path. The model is given the users’ prompt and a natural-language description of the scenario the vehicle is operating in. The description was handwritten by the researchers for the purposes of the study, but it could ultimately be provided directly by a car’s perception system, says Martinez-Baselga.

The model doesn’t directly tweak the settings of the controller; it uses the prompt to rate the relative importance of the judging criteria the controller uses to assess trajectories. This rating is then used to adjust each criteria up or down either side of a safe baseline set by the researchers.

So, if a user says they are feeling dizzy, the LLM will dial up parameters that encourage smooth steering and gentle acceleration to make the vehicle favor more sedate travel. Prior to making any changes, however, the model first presents the user with a natural-language description of the adjustments it plans to implement. The user can then sign off on the plan or make further suggestions.

Why it matters

Self-driving cars carefully balance a host of parameters to ensure a smooth ride, including things like speed, acceleration, and the smoothness of turns. But human driving preferences can often vary depending on how much of a rush they’re in, whether they’re feeling carsick, or how busy the traffic is.

These cars have a software component called the motion planner, which is responsible for choosing a safe and efficient path through traffic. The motion planner is normally tuned by engineers before the vehicles hit the road so that there’s little scope for passengers to adjust a vehicle’s driving style on the fly.

But now researchers at the Delft University of Technology (TU Delft) in the Netherlands have developed a system that uses a large language model (LLM) to translate natural-language user requests such as “I am running late, go fast” into adjustments to a self-driving control system. The researchers posted their preprint on arXiv and are presenting the work at the IEEE Intelligent Transportation Systems Conference in September.

The system doesn’t give users direct control over the vehicle’s driving decisions; it simply tunes the parameters of a safety-aware motion-planning algorithm, which helps to keep the vehicle’s behavior within safe bounds. And the system keeps the human in the loop by describing how it’s going to alter its behavior in nontechnical language, and by asking the passenger to confirm before making changes.

When the system was tested in simulation, the researchers found it adjusted the speed and smoothness of driving in line with natural-language instructions. “The motion-planning problem is not only about reaching a place while avoiding collisions, it’s also how you do it,” says lead author Diego Martinez-Baselga, a postdoctoral researcher at TU Delft.

” Previous research has investigated the potential of using LLMs and video-language models (VLMs) to direct decision-making for self-driving vehicles, but the researchers deliberately targeted driving style instead. Using LLMs and VLMs to directly control vehicles faces several challenges, says Martinez-Baselga.

These include relatively slow response times, which can make these models unsuitable for the fast-paced decision-making required in driving, and the fact that they can’t provide concrete performance guarantees in the way a deterministic motion planner can. Instead, the researchers used an LLM’s language and reasoning capabilities to translate fuzzy human preferences into something a vehicle’s motion planner can use.

What to watch

The system is also interactive, so the user can request further adjustments if the vehicle’s behavior doesn’t match expectations or the user‘s preferences change. Martinez-Baselga says this human-in-the-loop system allows the passenger to catch instances when the model misinterprets prompts. But it also helps deal with the inherent subjectivity of suggestions like “go faster” or the possibility that models don’t accurately describe changes they plan to make.

In that case the passenger can simply follow up with additional prompts “as you would do if you were in a taxi or with a friend that is driving,” says Martinez-Baselga. The researchers tested the system in the popular self-driving simulator nuPlan in scenarios that involved merging onto a busy highway.