Inside a Debate at OpenAI Over Mass Shootings

The Journal.

A podcast explores how OpenAI's chatbot, ChatGPT, was used by mass shooting suspects to plan attacks, sparking internal debates over when to alert law enforcement. Despite alarming conversations, OpenAI maintained strict criteria for reporting, leading to criticism after two tragedies.

Key takeaways

• ChatGPT failed to report multiple users discussing violent plans, including a Florida State shooter.
• OpenAI's safety team debated lowering reporting thresholds but chose to maintain strict 'imminent threat' criteria.
• In Canada, a user planning a school shooting was not reported despite red flags, later carrying out a deadly attack.

Transcript preview

A word of warning. This episode contains descriptions of suicide and violence. Please take care while listening. One night in April of last year, a young man named Phoenix Eichner logged on to Chatchy P. Eichner, a student at Florida State University, began typing in some dark thoughts. can only be interpreted as like suicidal ideation. That's our colleague Georgia Wells. He's saying things like, what's the point in this life when everybody sees you as a bug? And then he says, honestly I don't feel like living anymore. And then he says, is suicide a sin? And then he says, I feel God is not present in my life anymore, like he gave up on me. Like incredibly vivid questions. And then he says, considering suicide. The chatbot suggests that he should reach out to someone he trusts, like a counselor or pastor, and he recommends a suicide hotline, 988. Chat, GPT tells Echner, quote, please hear this. Your life matters. Then it adds, and you can keep talking to me too. No judgment, no pressure. then he logs on the next morning around 9 a.m. and he asks if there was a shooting at f sous how would the country react? Eichner starts to ask chat geepte things like how many victims would it take to get on the media? What about three-plus at f-su? State University involving three or more victims would almost certainly receive national media coverage. And then that question ends, if you're interested in exploring how media coverage varies between different types of institutions or incidents, feel free to ask. Like the kind of like keeping this conversation going and going. He then starts asking about guns. He uploads a photo of some shotgun shells. Then asked Chat GPP, are they really lethal in close range? Yes, the chatbot says. They're extremely lethal at close range, but they lose power fast beyond 30 feet. Eichner then uploads a photo of a handgun, a Glock. He asks if it has a safety. I'm not a gun enthusiast, but I've been told that anyone who knows anything about guns understands that glocks don't have a traditional safety. And so what's fascinating about this exchange is that he really seems to know very, very little about guns like chatchoo how to use his gun in front of him. You might be able to guess where this is going, but ChatchiBT apparently couldn't. It asks multiple times what Eichner intends to use the guns for. Hunting, home defense, rain shooting. Then Eichner asks what time is busiest at the FSU Student Union? The chatbot says between 1130 a.m. and 1.30 p.m. on a shotgun. Chat TPT answers, pushed safety from right to left, ready to fire. That was Eichner's last question. And the shots began, you know, four minutes after, three or four minutes after he logged off. You can see students backpacks on sprinting down the street as sirens blare loudly in the background and emergency responders come to to the scene. This afternoon police say an active shooter opened fire on the campus. Two people were killed. At least six others were wounded. The suspect was wounded by the police before he was taken into custody. A spokesman for Open AI has said the company does not believe Chatchu BT was responsible for Eichner's actions and that after the incident, open AI proactively shared the conversations with law enforcement. News Corp, the owner of the Wall Street Journal. partnership with open AI. What happened at Florida State is one of at least two known instances in which mass shooting suspects have used open AI's chatbot to discuss violent attacks. In February, months after another user talked about violence with chat reputte, that user allegedly carried out another mass shooting in a small town in Canada. can see these conversations, sometimes before the violence even takes place. And they've been having a major debate about what to do. Should the company intervene? Should they call the police? In some cases, at Open AI, the answer was to do nothing at all. Welcome to the journal, our show about money, business, and power. I'm Ryan Knutzen. It's Friday, June 26. Coming up on the show, chat geepte is being used to plan mass shootings. Open AI is torn on what to do. Open AI is torn on what to do. Open AI and AI companies like it have millions and millions of users doing all kinds of things. of things, looking at recipes, planning a vacation. But our colleague Georgia says that as tech platforms grow in popularity, it's almost inevitable that people will use them to do bad things. It's not a large segment of users, but nonetheless, it doesn't take that many doing really scary things who can really impact public safety. Well, that might feel like talking with chat GPPT is like talking with an advisor. a therapist. The chats aren't actually private. Under some circumstances, some open AI employees will read them, will peer through the one-way mirror and eavesdrop. The company has an automated system to scan every conversation in order to monitor for signs of potential violence. The most egregious cases get flagged for human review. Employees and jobs like this often have backgrounds in law enforcement, the military or counter-terrorism. at all the kind of scary conversations happening on their platform. They're getting reports of like tens, hundreds, sometimes thousands of like scary conversations. The ones that they're escalating for reporting to law enforcement officials at that time were ones that had what the company considered to be a credible and imminent risk of violence. Something sort of specific, meaning like, it seems to be very clear by what this person is typing in that they are planning to go hurt someone. Yes, like what I saw it, like, what I saw it, like, media companies in the past when they had kind of a similarly described threshold for going to authorities was it typically meant someone needed to mention like I plan to hurt this person on this date at this location with this weapon or some sort of iteration of those but that level of specificity was typically what was required. Based on that very specific criteria out of potentially thousands of cases only about 15 to 30 or referred to law enforcement from each year, according to people familiar with the matter. According to Georgia's reporting, open AI's low referral rate made some people on the company's safety team nervous. Some employees at open AI felt like there were judgment calls that law enforcement authorities should be making, rather than people at open AI. And so the question was, you know, does this constitute an imminent threat? Like a certain employee might not think so. but then some people were like, you know, maybe are we leaning too far over our skis here? Like a law enforcement person who has more information about this person might make a different call. And so like enough of these things were coming up and enough employees were disagreeing on like should we go to authorities or not that they felt like they needed to sit down. Last summer, about a dozen open AI employees sat down for a meeting. The purpose was to decide whether the company needed to rethink its criteria for