Sam Altman and AI’s decel debate

What happened

” On the latest episode of TechCrunch’s Equity podcast, Kirsten Korosec, Sean O’Kane, and I discussed how Altman’s comments were probably prompted by a recent hack in which an OpenAI agent breached Hugging Face’s systems. ” “It was more like Nixon’s people breaking into Watergate than some real stealthy cyber-op, because it didn’t need to be, and it wasn’t instructed to be,” Sean said.

But it’s worth coming back to one of the points that we also wrote about at TechCrunch, that this specific hack — yes, it was caused by an OpenAI model, but it sounds like they just didn’t secure the testing site properly. In theory, this model should not have been able to get online.

Now, of course, if you have a powerful misaligned AI, the risks of that human error go up dramatically. But it does start from just the fact that they didn’t secure things the way they should have. Sean: I think that’s right. I think your point is well taken in the sense of, we shouldn’t only think about this in some linear fashion and whether things are accelerating or decelerating.

There’s a lot that could and should be said about just how responsible these companies are being. Lorenzo, one of our colleagues, also wrote a really good piece walking through how serious security researchers who pay attention to this stuff think that the hack really was.

It really does seem like, on both sides of this hack, there were steps that probably should have been taken that would have prevented it. And one of the things that I found most interesting in that story was that some of the researchers were pointing out that what this model did was not some new advanced thing.

Why it matters

” Keep reading for a preview of our conversation, edited for length and clarity. Sean O’Kane: Maybe we’ve finally hit an inflection point here. I think a big driver of this has to be what we talked about last week, with one of OpenAI’s models breaking into Hugging Face’s data and apparently breaching a few other things around the internet, as well.

[Altman’s] not calling for a pause, like we’ve seen some people in the tech industry try to do in the past. ” And we’ll see how this holds. Any caution that we see some of these labs throw out there often gets reversed when the incentives push them forward to resume, full speed ahead. So I remain skeptical, big surprise.

Kirsten Korosec: Now I will say this — [Altman] might have been careful with his words, but OpenAI and Anthropic did [support] a petition that does reflect what he did talk about. And I do agree with you, I think that a lot of this was very much triggered by Hugging Face. It probably spooked him and certainly a lot of people in the industry.

” I don’t know if they can do that. I’ll be curious to see if they manage both. ] deceleration the right framework to be thinking about this? Because it kind of suggests that there’s only one path and we’re all stuck on this path. All we get to decide — inasmuch as we get to decide at all — is, do we speed up or do we slow down?

As opposed to — again, I’m going to really torture this metaphor — but do we build different guardrails? Do we choose different paths? I’m just very resistant to this framework. As opposed to saying, “Okay, if we’re not happy about what models are doing right now, what else can we do? ” And I don’t think it is.

One thing that I did want to emphasize again, because it’s been really interesting to see the level of alarm around this — this sense of, “What if we have these autonomous agents and models just running around hacking each other, trying to prevent hacks, it’s just all getting out of our control,” leading to all these broader debates about alignment that Rebecca Bellan did a great piece about.

What to watch

It was really very human in the way that it thought about trying to break into trying — not to anthropomorphize, but the way that it thought about breaking into Hugging Face, and that it was also very loud and messy and wasn’t really trying to hide its tracks.

It was more like Nixon’s people breaking into Watergate than some real stealthy cyber-op, because it didn’t need to be, and it wasn’t instructed to be. That should have been more easily preventable. And hopefully, this is a sign that these companies will take this forward and be more careful about that stuff.