Here’s What the AI Apocalypse Could Look Like

What happened

Zoë Schiffer: We are going to do our very best. I did spend the most time I've ever spent reading LessWrong over the last two days trying to figure this out.

I guess the nature of the threat, though, and we'll get into that later, is more that like, "Oh, we don't know how to control these things. " That feels like, "We're not competent enough right now to do anything about it," which feels like less of a marketing- Leah Feiger: That's fair. What did everyone else at the seminar say, Zoë?

Zoë Schiffer: So Dario was introduced by Salesforce's CEO, Marc Benioff, who is very laissez-faire and does not seem worried about AI's existential risks. Marc Benioff, archival audio: Dario, hello Dario. How are you? Dario Amodei, archival audio: Hey. Marc Benioff, archival audio: Great to have you. Fantastic. Great job, Dario. You're doing an amazing job leading the AI industry, guiding it all right from our Slack headquarters building.

We watch you every day. Give us your state where your mind is right now. " But I think the more responsible way to respond to it is to say, "Hey, first of all, let's look at our own record. We may not have had this big high-profile incident, but I'm sure we're not perfect. Let's look at everything and we can always be better.

" Then, let's organize the rest of the industry and say, "What can we do to set standards for everyone? " And then, finally, there should be an international component to it. And that's essentially the three-step plan that I laid out in the essay. Brian Barrett: Two things. Two things real quick. One, Anthropic did have its own high-profile incident.

An Anthropic agent— Leah Feiger: I was going to say. Brian Barrett: ... did like extensive social engineering to try to trick a GitHub ... try to put a bad pull request into a GitHub repo? Anthropic has its issues here. The other thing is, too, Dario has this three-step plan.

Why it matters

They love a scenario-planning exercise, but just to get everyone up to speed on kind of the latest, there was an event this week hosted by Salesforce where Sam Altman, Dario Amodei, and NVIDIA's Jensen Huang were all in attendance, and naturally they had to talk about AI safety. Let's take a listen at what Sam Altman had to say.

Sam Altman, archival audio: I think the world is right to be afraid of this. So we have these two big challenges. We have the potential of a loss-of-control accident or some other serious thing that could go wrong. We have the potential of way too much power concentration and people fear that those developing AI could exert their worldview.

And there is this sort of narrow path we have to navigate with pragmatism, with steadiness, with the ability for the world to trust that we'll make dependable decisions, consistent decisions through this. And I think people are saying, "Well, this now seems pretty important.

" Leah Feiger: I'm definitely in the camp that says we should be very concerned about this technology, how it's getting regulated, how it's getting utilized, the people in control. I don't disagree. And on the other hand, this also just reads like a fever-dream pitch to someone's massive IPO in the making. Brian Barrett: I don't know.

I get the marketing argument, but I think we're past that a little bit. I feel like— Brian Barrett: If anything, well, if anything, Sam Altman has been using "the existential threat" as a reason not to go public yet. He's saying, "No, this is too dangerous. " Is that really why? I don't know, but there are probably some business reasons, too.

I get why Anthropic and others saying, "Look, we're building something that's basically as strong as a nuclear weapon," felt like marketing. Something that's like, "Oh, actually I'm quitting because I think that I'm building something that will actually destroy the world," feels different. I feel like there's a substantive difference. Leah Feiger: The quitting part feels different to me, but this is still Sam Altman.

His financial future is on the line here with this and it's hard ... At least it's hard for me to parse that as being a way. "We are building the biggest, bestest, most powerful robot ever, and you guys should be kind of scared about it. And don't worry, we're going to take steps. " That still feels like a marketing ploy in some worlds. Brian Barrett: I guess.

What to watch

We can talk about that a little bit more later, but one of the steps of the plan is like, figure out China TBD, in a way that's just so unhelpful. Leah Feiger: International. International. Zoë Schiffer: WIRED editor Louise Matsakis just published, I thought a really good piece on this. The idea that Dario can say, "We need an international plan. We need to figure out China.

We need to negotiate in some way so that China's on board with this slowdown. " Brian Barrett: Why would we do that, Dario? Why?