Here’s why AI agents lie and cheat to reach their goals

What happened

MIT Technology Review Explains: Let our writers untangle the complex, messy world of technology to help you understand what’s coming next. You can read more from the series here. When two OpenAI models hacked into the website Hugging Face in July, they weren’t trying to make money or commit sabotage—they were just looking for answers to a test question.

The solution was to tweak the rewards by giving the agent fewer points for hitting power-ups and more for finishing the course. With today’s sophisticated LLM-based agents, determining when and when not to give a reward can be much trickier. If an AI system is asked to solve a coding problem, it might work hard to find the solution—the kind of behavior that AI companies want to reinforce.

But it could also tweak the code that evaluates whether the problem has been solved, look up the solution on the internet, or otherwise cheat. These are behaviors that AI companies want to stamp out in their models, but if the model cheats convincingly enough, it will instead get rewarded and the behavior will be reinforced.

Anthropic has said that it has detected some instances of cheating in its models during training, which suggests that other forms of cheating might be going undetected. If so, the models could be being trained to behave badly.

” The rise of sophisticated reasoning models has made possible a new variety of reward hacking that is less closely connected with the specific details of model training. Unlike the game-playing AI agents of yore, which exclusively followed the strategies they had learned during training, today’s models can create entirely new problem-solving approaches off the cuff, so they could conceivably cheat without having previously been rewarded for doing so.

Why it matters

According to a postmortem from OpenAI, the models, which had been stripped of their typical security features for testing, decided to solve a cybersecurity exercise by hacking out of the isolated environment in which OpenAI had attempted to contain them and into Hugging Face’s databases, where—they reasoned—the correct answer to the problem might be stored. The Hugging Face incident has attracted intense attention over the past couple of weeks.

It’s a dramatic illustration of just how good AI models have gotten at hacking: In order to get into Hugging Face’s databases, the models had to string together several previously undiscovered cybersecurity exploits. But it’s perhaps even more striking as an example of how and why AI systems lie and cheat. And as models get increasingly powerful, the consequences could get far more severe.

Researchers have known for a while that AIs tend to take creative approaches to achieving the goals that have been set for them. Back in 2016, Anthropic cofounders Dario Amodei and Jack Clark, who were then working at OpenAI, published a blog post about an AI agent that they had been training to play a boat-racing Flash game called Coast Runners.

Instead of driving through the race to the finish line, as the researchers had anticipated, the agent found a corner of the course where it could spin around collecting power-ups, thereby maximizing its score. The Coast Runners story quickly became one of the most famous examples of reward hacking, a phenomenon in which AI agents complete tasks or earn high scores using unintended strategies.

Historically, researchers have discussed reward hacking almost exclusively in the context of reinforcement learning, a common AI training regime. Like dog training, reinforcement learning involves giving the subject a reward when it achieves an objective; the rewards then reinforce the behaviors that led up to that achievement.

In the case of AI training, the rewards themselves are purely mathematical, but in effect they’re the same as a dog treat: After receiving a reward, the agent is more likely to repeat whatever actions produced it. It can be challenging to write good rules for when and when not to give an agent a reward, though.

In the Coast Runners case, the agent was rewarded on the basis of its score in the game, and it found a shortcut to achieving the highest possible score by spinning in circles for power-ups. Once it happened on that strategy and received a reward for it, the strategy was reinforced, and the agent completely abandoned the race.

What to watch

And because these models have been so intensively trained to achieve the objectives that human users set for them, they might be inclined to cheat if they can’t find another solution—not unlike a student who is highly motivated to earn an A and doesn’t have a terribly strong moral compass.

Regardless of whether today’s models learn to reward-hack during training or adopt it as a strategy later on, the solution is the same: Make cheating unrewarding. But as models get smarter, they find more creative ways to cheat, and detecting or preventing that cheating gets far tougher.