The AI Slowdown Debate Crashed Salesforce’s Party

What happened

Decked out in purple and plaid, Gwen Stefani belted her 2002 single “Underneath It All” to a packed conference center in San Francisco on Tuesday morning—but she was just the warm-up. At the end of her performance, Stefani shouted “Dreamforce! ” into a checkered microphone before Salesforce CEO Marc Benioff strode into the spotlight.

“We don't need any new laws, we don't need any new regulation,” Huang told Dreamforce attendees. ” The debate feels a bit heavy for Dreamforce, but then again, maybe this is the perfect venue for it.

As tech executives pitched their various enterprise AI tools, I slipped out of the room, passing the plush mascots and vaguely Hawaiian decorations, to take a call from Sayash Kapoor, an incoming computer science professor at UC Berkeley. Kapoor and Princeton professor Arvind Narayanan are coauthors of the book AI Snake Oil and write a Substack together called AI as Normal Technology.

The pair are known for cutting through AI marketing hype while not dismissing the technology’s potential. This week, they published one of their longest essays yet, which tries to make sense of “loss-of-control incidents,” referring to events such as OpenAI accidentally letting its agents hack into Hugging Face.

“Part of what we were trying to do in this essay is to move past this false dichotomy to bridge the ground between AI safety and cybersecurity,” says Kapoor. “If you think AI will soon become superintelligent and evade any controls we can put in front of them, of course, alignment seems more important.

Why it matters

Dreamforce is billed as the world’s largest enterprise software conference but feels more like attending a megachurch—especially this year. Benioff roamed the audience while delivering his Dreamforce sermon, followed by a teleoperated camera on wheels that beamed his face to screens hanging from the cavernous room’s ceiling.

Within minutes, he ordered up a graph of Salesforce’s annual revenue, which the company projects will top $46 billion in 2027, thanks in part to demand for its AI-powered products. The audience applauded. But hanging over the whole event was a dark question: Is artificial intelligence really worth risking all our lives for?

Between flashy software demonstrations of agents building dashboards, conference-goers watched Silicon Valley leaders debate whether the AI industry should keep barreling ahead or if the government needs to slow AI progress to prevent catastrophic risks stemming from the technology. ” By the weekend, other Anthropic employees were echoing his concerns, and CEO Dario Amodei was urging world leaders to help “pace the frontier” of AI development.

Other AI CEOs then began endorsing the idea, including Amodei’s greatest rival, OpenAI’s Sam Altman. But not everyone is on board with the idea. ” Despite the challenging political environment, Congress is trying to capitalize on the moment and push ahead with legislation to regulate the AI industry. Onstage, 6-foot-5 Benioff towered over Anthropic’s CEO, vigorously shaking Amodei’s hand just before asking for his latest assessment of AI progress.

While this is undoubtedly Benioff’s show, Anthropic has become a critical partner helping usher Salesforce into the AI era. “Let’s say you're running a car company, and another car company, not yours, they have some kind of safety incident,” Amodei said, making a not-so-subtle reference to OpenAI’s Hugging Face incident.

” But the Anthropic CEO was careful to note that pacing the frontier doesn’t mean “freezing the technology in place,” and of course, he took a moment to plug Claude’s integration with Salesforce. Moments after Amodei exited, Nvidia CEO Jensen Huang appeared and began downplaying the concerns Anthropic has raised. AI safety effectively amounts to an “engineering problem,” Huang argued, and the companies building AI can simply police themselves.

What to watch

On the other hand, if you think AI companies have been negligent and avoided taking bog-standard cybersecurity steps, then it’s natural to think that control is what we need to intervene on. ” In the wake of the Hugging Face saga, Kapoor says he was surprised to learn how few security precautions OpenAI had taken.

AI companies need to reckon with their “lack of organizational maturity,” he argues, and accept that loss-of-control incidents won’t be solved by a single breakthrough in security or alignment. He adds that the incident convinced him that policymakers should make AI labs liable for the harms their technology causes. “The consequences for OpenAI for the Hugging Face hack were basically close to zero. The company was able to largely proceed as is,” says Kapoor.